@Eben-Siyabalapitiya

DEVLOG 008 - The Website & Wrapping Everything Up

Everything is finally done after a long grind.

Building the Website

Since this is my first ever open source project I really wanted the website to be really good. Not just functional but actually good. So I locked for hours and paid attention to every single detail.

Started with a blue colour scheme but it wasn’t sitting right with LUMA’s darker aesthetic so i decided to switch everything to a deeper darker palette and it immediately i knew it was perfect for her.

I went through a lot of iterations getting the look right, smooth scroll animations, hover effects, proper transitions throughout. Kept going back and refining until it felt clean and polished.

Bugs I ran into along the way:

• Files weren’t downloading when the download button was clicked tracked down the issue and got it fixed

• Some animations were firing at the wrong time on scroll it was all so cooked so i had to adjust the trigger points for all of them so everything feels smooth and clean

• A few layout inconsistencies on smaller screens that needed sorting

• Hover states on some cards weren’t behaving correctly so i hadda clean those up as well

• Some placeholder sections needed proper content wired in before publishing so i had to collect all the files and add it to its proper spots

Each thing small on its own but there were a lot lot lot of them so they all adds up.

What’s on the Site?

Made sure everything anyone needs to build their own LUMA is actually there:

• Full build guide with step by step instructions

• Wiring diagram so its not just the chart from before, a proper schematic so anyone can see visually how it all connects before they start wiring

• 3MF file set up in Bambu Lab with the right settings infill, wall thickness, all the print settings dialed in. One click download and it’s ready to print

• CAD file included so anyone can open it up and customize the enclosure if they need to

• Recommendations section for parts and anything else that helps with the build

• Contact section so if anyone has questions or runs into issues building it they can reach out directly and the question will straight drop into my email ( i check my emails everyday)

Bought a proper domain for it as well with the money i had left. LUMA now has a real home on the internet and its not just a project link, an actual address anyone can search up and find.

Code on GitHub

Everything is published. Full firmware, wiring, build instructions all of it open source and ready for anyone who wants to build their own.

Honestly really proud of how far this went. Luma started as a random idea and turned into a complete project working firmware, a finished enclosure, a proper website, a domain, and a GitHub repo with everything documented. My first ever open source project and it came out wayyy better than I expected.

LUMA IS FINALLY DONE. Not just for me for anyone who wants their own little desk companion.

• GitHub → https://github.com/Eben-Siyabalapitiya/LUMA

• Website → https://meetluma.live/

DEVLOG 007 — Final Polish & Full Assembly

Mostly physical work this session but a lot
got done and LUMA is finally feeling complete.

─────────────────────────────────────────
New Enclosure & Sensor Placement
─────────────────────────────────────────

Reprinted the enclosure again with a new slot
for the second sensor. Moved things around so
the layout actually makes sense to use:

◆ Emotion sensor mounted on the top, easy
to reach, natural to tap or hold while
LUMA is sitting on your desk

◆ Mode switch moved to the back top center
so switching between face, clock, weather,
and calendar is just a tap on the back

Much more intuitive than before.

─────────────────────────────────────────
Legs
─────────────────────────────────────────

LUMA was looking too much like a plain box so
I designed and printed her some legs. Honestly
one of those small changes that made a big
difference, she looks way more like an actual
character now and it just suited her really well.

─────────────────────────────────────────
Reprints & Tolerance Fixes
─────────────────────────────────────────

Had to reprint most of the parts to get the
thickness right for the touch sensors to
actually work properly through the walls.
Also tightened up the sensor slot tolerances
that were coming out too loose. Took a few
iterations but everything fits clean now.

─────────────────────────────────────────
Full Rewire & Schematic
─────────────────────────────────────────

Rewired the whole thing properly from scratch.
Made a proper schematic diagram and followed
it this time so there are no guessing games
with the connections. Resoldered everything
cleanly so it’s solid for the final assembly
and not going anywhere.

─────────────────────────────────────────
Code & CAD Cleanup
─────────────────────────────────────────

Fixed up the remaining code bugs, cleaned
everything up and made sure it was all working
properly. Same with the CAD, went through it
and sorted all the small things that still
needed attention.

─────────────────────────────────────────
Final Print — Black
─────────────────────────────────────────

Printed everything in black for the final
assembly. Really suited LUMA. Looked way
better than the prototypes.

─────────────────────────────────────────
GitHub & Wrapping Up
─────────────────────────────────────────

Set up a proper GitHub repo, wrote everything
up, and finalized the project. LUMA is done.

Next goal is to build LUMA a website. Full build
guide, open source, everything anyone needs
to build their own.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

DEVLOG 007 — Final Polish & Full Assembly

Mostly physical work this session but a lot
got done and LUMA is finally feeling complete.

─────────────────────────────────────────
New Enclosure & Sensor Placement
─────────────────────────────────────────

Reprinted the enclosure again with a new slot
for the second sensor. Moved things around so
the layout actually makes sense to use:

◆ Emotion sensor mounted on the top, easy
to reach, natural to tap or hold while
LUMA is sitting on your desk

◆ Mode switch moved to the back top center
so switching between face, clock, weather,
and calendar is just a tap on the back

Much more intuitive than before.

─────────────────────────────────────────
Legs
─────────────────────────────────────────

LUMA was looking too much like a plain box so
I designed and printed her some legs. Honestly
one of those small changes that made a big
difference, she looks way more like an actual
character now and it just suited her really well.

─────────────────────────────────────────
Reprints & Tolerance Fixes
─────────────────────────────────────────

Had to reprint most of the parts to get the
thickness right for the touch sensors to
actually work properly through the walls.
Also tightened up the sensor slot tolerances
that were coming out too loose. Took a few
iterations but everything fits clean now.

─────────────────────────────────────────
Full Rewire & Schematic
─────────────────────────────────────────

Rewired the whole thing properly from scratch.
Made a proper schematic diagram and followed
it this time so there are no guessing games
with the connections. Resoldered everything
cleanly so it’s solid for the final assembly
and not going anywhere.

─────────────────────────────────────────
Code & CAD Cleanup
─────────────────────────────────────────

Fixed up the remaining code bugs, cleaned
everything up and made sure it was all working
properly. Same with the CAD, went through it
and sorted all the small things that still
needed attention.

─────────────────────────────────────────
Final Print — Black
─────────────────────────────────────────

Printed everything in black for the final
assembly. Really suited LUMA. Looked way
better than the prototypes.

─────────────────────────────────────────
GitHub & Wrapping Up
─────────────────────────────────────────

Set up a proper GitHub repo, wrote everything
up, and finalized the project. LUMA is done.

Next goal is to build LUMA a website. Full build
guide, open source, everything anyone needs
to build their own.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

DEVLOG 005 - Second Touch Sensor & Emotion System

Added a second capacitive touch sensor on a new
GPIO pin dedicated entirely to emotions.

How It Works

Built a full gesture detection system on the new
sensor:

◆ Single tap → excited
◆ Hold for 800ms → cozy
◆ Rapid tap 3+ times → mad

Each emotion has 5 or 6 distinct face variants
that cycle using weighted random probabilities
so it never shows the same reaction twice in
a row.

◈ Mad - red angry eyes in different shapes:
wide glare, tiny squint, psycho stare,
disgusted droop

◈ Excited - big round happy eyes, bouncy
small eyes, wide open stare

◈ Cozy - half closed drowsy eyes, warm round
soft eyes, tiny sleepy eyes

Emotions auto expire after 6 seconds and return
to the default idle face automatically.

Spent a long time writing the code, fixing
errors, getting everything working and then
the compiled firmware came out at 101% of the
ESP32’s flash limit and wouldn’t upload.
Couldn’t just scrap it after all that work so
I started diagnosing what was eating the space.

The two biggest culprits:

◈ HTTPClient and ArduinoJson together were
pulling in around 60–80KB of overhead just
from the libraries alone

How I Fixed It

◆ Replaced HTTPClient and ArduinoJson with a
raw WiFiClient socket connection and three
tiny manual JSON parsers using strstr and
atof no library needed at all

◆ Replaced every cos and sin math call with
precomputed fixed point lookup tables stored
in PROGMEM completely removed the float
math library dependency

◆ Wrapped all string literals in F() to keep
them in flash instead of RAM

Got the binary down under the limit without
cutting a single feature. Everything still
works exactly the same, it just actually
fits on the board now.

Still needs some tuning though the emotion
faces need a bit more work to feel right and
there are a few things here and there that
need cleaning up. The structure and foundation
is solid but it’s not fully polished yet.
More to come on that in the next log.

DEVLOG 004 - Fixes, Wiring Cleanup & Getting It Solid

Mostly a cleanup and bug fixing session today
before moving on to the next round of upgrades.

Screen Flip

Once the screen was mounted in the enclosure
everything was upside down. One line fix changed
setRotation from 1 to 3 and that was it. Done.

Weather Screen Bug

Tracked down an annoying bug where the weather
screen would show data the first time but come
back blank after switching away and returning.

The issue was a redraw flag that never reset on
screen re entry. Fixed it by resetting the flag
in both the screen router and the mode switch
handler so weather now fully redraws every time
you land on it.

Enclosure Reprint - Screen Pocket

Reprinted the base and added a small pocket to
the cutout because the screen wasn’t sitting
flush. The soldered pins on the front of the
screen were hitting the case and pushing it up
slightly. Added a slot for them to drop into
screen sits perfectly flush now.

Wiring Issues - Touch Sensor

One of the touch sensors was activating on its
own just a red light flashing like something
was constantly triggering it with nothing
touching it.

First thought was the code since I was in the
middle of cleaning everything up anyway, so I
spent a while going through it trying to figure
out what was wrong. Nothing. Code was fine.
Turned out to be a bad ground connection on the
sensor the whole time. Resoldered it properly
and it’s been stable since. Classic hardware problem disguised as a software problem.

Overall today was about getting everything solid
and clean before the next upgrade push. Wiring
is tidy, code is cleaned up, everything is
working as expected.

DEVLOG 1 — Setting Up Vision Model Training & Project Structure

━━━━━━━━━━━━━━━━━━━━━━━
TRAINING

Found a grocery dataset on Roboflow with 80k+ images across 17 categories. Set up training on Google Colab using a free T4 GPU with YOLOv8n, 50 epochs, image size 640, batch 16, output saving straight to Google Drive. Left it running overnight.

Woke up to find it stopped at epoch 16. Colab hit the free GPU usage limit and killed the session mid run. Checked the weights folder in Drive and it looked empty at first, thought all progress was gone. Turns out Colab had been auto incrementing folder names
each restart so the actual checkpoints were in kart2/weights/ the whole time. Both last.pt and best.pt were there. Opened results.csv
and confirmed 15 full epochs completed with mAP50 around 0.87, precision and recall both near 0.99. Nothing lost.

Training is paused at epoch 15/50 waiting for GPU quota to reset. Resume plan is loading last.pt with resume=True so it picks up exactly where it left off.

━━━━━━━━━━━━━━━━━━━━━━━
BUILDING THE APP

While waiting on GPU access built out the full local pipeline.
detector.py -runs model.track() each frame using ByteTrack andreturns detections with track IDs

cart.py - handles a virtual tripwire line, items crossing down get added to the cart, crossing back up removes them, grace counter stops brief tracking losses from wiping items instantly

prices.py - reads prices.json at startup and calculates subtotal plus Ontario HST 13%

display.py - draws the tripwire line and bounding boxes onto the camera feed each frame

main.py - ties everything together, Tkinter dark themed panel showing the live cart and total, OpenCV camera window showing the annotated feed, both running together

━━━━━━━━━━━━━━━━━━━━━━

PROBLEMS

Several source files were empty on disk, had to repaste everything. Stale pycache was loading a broken cached version of detector.py. Ultralytics and opencv were not installed locally. prices.json was empty and throwing a JSON error on startup. Wrong camera index was hitting the built in webcam instead of the external one. Local
best.pt was a placeholder with nothing in it, temporarily swapped in yolov8n.pt to test the pipeline then pulled the real checkpoint
down from Drive.

━━━━━━━━━━━━━━━━━━━━━━━
WHERE THINGS ARE AT

Pipeline is fully working end to end. Camera feed shows the tripwire line and bounding boxes, items crossing get added to the cart panel with their price, subtotal and HST update live, total shows in green. Detection is decent for 15 epochs, right general categories but not perfectly precise yet which is expected. Once GPU access comes back finishing the remaining 35 epochs then swapping in the final best.pt.

DEVLOG 003 — First Real Assembly

Now that the firmware was in a solid place, it was
time to actually build the thing.

Got into CAD properly and started designing the
enclosure for real not just roughing it out
anymore, actually planning where every component
sits and how it all fits together.

Originally I was just going to use the bare ESP32
board and solder everything directly onto it, but
ended up switching to a breakout board for the ESP32
instead which was the right call. That change meant
I had to scale the enclosure up a bit to fit
everything properly, so I reworked the design, got
the dimensions right, and sent it to print.

Once it came off the printer I grabbed the calipers
and started checking everything:

◆ ESP32 breakout board fit perfectly in its slot
◆ Screen seated clean in the cutout
◆ Touch sensor connected and sitting in the
right spots
◆ Overall fit was exactly where I wanted it

Pretty satisfying moment honestly.

But then I started running into the first real
problems.

◈ The walls ended up being too thick, so the
capacitive touch sensors could barely detect
anything through them touch was sluggish and
inconsistent, not responsive at all

◈ The screen had to be mounted the way it is
because flipping it would’ve sent the wires
running from the pins straight into the
breakout board no room for that. Which left
the display showing everything upside down

Both things need fixing, but overall for a first
proper assembly it came out pretty well. It’s
starting to look like an actual product, which
is a good feeling.

DEVLOG 002 — Firmware & Making It Actually Look Good

First thing I tackled was getting the core firmware
structured properly. Built out the full mode switching
system FACE, CLOCK, WEATHER, and CALENDAR all
cycling on a single small capacitive touch sensor.

Then got into the connectivity side of things:

◆ WiFi connecting on boot automatically

◆ NTP time sync so the clock always pulls real
time without me hardcoding anything

◆ OpenWeatherMap API hooked up parsing
temperature, humidity, wind speed, and condition

◆ Calendar built from the actual current date with
a real day grid and event highlights

Everything was working at this point but honestly
it looked terrible just white text on a black
screen, nothing impressive at all. It was functional
but it didn’t feel like anything yet.

So I basically tore everything out visually and
rebuilt every screen from scratch.

That’s where it started actually feeling like a
real product.

◈ Gradient header bar across every screen

◈ Weather display completely redone color coded
temperature + pixel art icon system that draws
different visuals depending on the condition
(sun, clouds, rain, thunderstorm, snow, fog
all drawn in code, no images)

◈ Calendar got a proper grid with weekday headers,
today highlighted, and event colors

◈ Clock got a big clean digital readout with a
seconds progress bar filling across the screen
in real time

◈ Scrolling agenda strip at the bottom of the
calendar showing the next upcoming event

Going from plain text to all of that in one session
was a big jump. That was the moment it started
looking like something I’d actually want on my desk.

DEVLOG 001 - Planning out & Testing
So basically I had this idea of building a little desktop robot that could show a face, clock, weather, and calendar and I just wanted something that felt alive on my desk, not another boring info display. Decided to go with the ESP32 and the ST7735 TFT display, ordered the parts and waited.
Parts came in and before doing anything permanent I threw everything onto a breadboard first just to test connections and make sure the components were actually talking to each other. Got the ESP32 seated, wired up the TFT with the SPI lines, and powered it on and immediately the display was just… white. Solid white, nothing rendering at all. I genuinely thought it was a code issue so I started digging through everything:

Rewrote the initialization logic multiple times
Swapped library settings, tried different pin configs
Spent way too long convinced it was something in the code

Eventually I just rewired the whole breadboard from scratch out of frustration and it just worked. Wiring issue the whole time, one of the connections on the breadboard just wasn’t seated properly into the rail. Such a simple thing but it ate up so much time.
After that everything started clicking. Got the display confirmed working, flashed the RoboEyes library on it and saw the eyes blink on actual hardware for the first time that was a real moment honestly. Super small but it just felt like okay this is actually going somewhere. First boot was literally just a face sitting there blinking on a breadboard with loose wires everywhere, nothing else running, but that was enough to know the foundation was solid. Ended the session jumping into CAD and just roughing out what the enclosure could look like, trying to figure out how everything fits together in a way that actually looks good sitting on a desk.

DEVLOG 3 — Training the First Gestures
Finally got everything working. I set up a data collection script that opens the webcam, lets a gesture label be selected, and then starts recording when space is pressed. From there, every frame grabs 63 normalized MediaPipe landmark values and saves them into a CSV row with the label attached. I collected about 300 samples per gesture, which sounds like a lot, but it’s actually pretty quick roughly 30–40 seconds per gesture since it just keeps recording while the sign is held, with natural movement built in. First quick test was super simple: yes/no, thumbs up vs thumbs down. Then I ran training and somehow hit 100% test accuracy by epoch 3, which honestly caught me off guard. The model itself is pretty small and just four dense layers (128 → 64 → 32 → output). But since the input is already structured landmark data instead of raw images, it picks up the patterns almost immediately.

Back with an update, so this took way longer than it should have. First thing, I run pip install tensorflow and immediately get slapped with ERROR: Could not find a version that satisfies the requirement tensorflow. Turns out TensorFlow just straight up doesn’t support Python 3.14 yet, so I had to go download Python 3.11 separately and run everything with py -3.11 from there on out. Thought I was done but then pandas starts giving ImportError saying Please upgrade numpy to >= 1.26.0, so I upgrade numpy like it asks… and now TensorFlow throws AttributeError: _ARRAY_API not found because apparently TensorFlow 2.13 needs numpy <= 1.24.3. So these two libraries are just directly fighting each other and I’m stuck in the middle. Ended up pinning numpy to 1.26.4 which somehow makes both of them happy, no idea why that exact version is the magic number but it works. And THEN on top of all that, VS Code kept ignoring the Python 3.11 interpreter I picked and running 3.14 anyway, so I gave up on the play button completely and just run everything from terminal now with py -3.11 script.py.

About a week ago I started my sign language translator and spent the first day mapping out the core architecture before writing any code. I planned out the 3 stage pipeline I’m building MediaPipe captures the webcam feed and extracts 21 points per hand, those get normalized relative to the wrist so distance from the camera doesn’t matter, then TensorFlow takes those 63 numbers and classifies them into a word. I also worked out why I’m going with this landmark based approach instead of the typical CNN on images method most tutorials use feeding a model 921,600 raw pixel values means it has to learn what a hand even looks like before it can learn gestures, whereas feeding it 63 clean numbers means it only has to learn the gesture itself. After locking in the architecture I spent the rest of that first day writing the code and split everything into three separate files so each part of the pipeline has its own job.

collect_data.py: opens up the webcam and runs MediaPipe hand detection, I hold a gesture and hit space to start recording samples for it. Every frame it grabs the 21 landmark points, normalizes them based on the wrist position and hand size, and saves those 63 numbers along with the gesture label as a row in a CSV. (have not tested it yet but it should work)

train_model.py: takes that CSV, encodes all the gesture labels, splits everything into training and test sets, and builds a small dense neural network 128 → 64 → 32 → output using TensorFlow. It saves the trained model and label encoder so the detector script can use them later.

run_detector.py: loads that trained model and runs live prediction, pulling the same normalized landmarks from the webcam and showing the predicted gesture in real time.
i didn’t get to test any of it BUT HOPING SOON