NIMBL — Devlog #2: TUI Design(Hell)

This devlog covers how I actually built the thing, and the nasty bugs I had to fix. Also that I should probably pause this to further learn frontend. Cause iam trash at frontend

Choosing OpenTUI (and Fighting It)

I needed a terminal UI framework that could run under Bun on Windows. Textual was too heavy. Ink (React) had reconciler conflicts. OpenTUI was the right call. It’s what OpenCode uses in production, it’s fast, and its SolidJS works well with the reactive state model. But it wasn’t plug and play. Which I didn’t expect.

Out of the 9 hours I spent on this phase, maybe 2 were actual feature work. The rest was bug fixing.

First wall: the native DLL kept crashing. OpenTUI tries to pass color objects to a Windows native DLL, but the way they format the data doesn’t match what the DLL expects. Every time the TUI tried to draw anything, Bun would crash. I couldn’t figure this one out on my own. Had to get help from an AI coding agent (the newly bought OpenCode Go) to trace through the crash logs and find the fix. The solution was simple once found: use color hex strings like "#06402b" everywhere instead of color objects. OpenTUI handles the conversion internally and the DLL stops crashing.

Second wall: wrong import path. OpenTUI’s SolidJS build tries to import from a file path that Bun 1.3.14 can’t load properly, causing a different kind of crash. One line change in the library file. Change solid-js/dist/solid.js to solid-js and it worked. Small fix, but took hours to find because the error message just said “Cell” with no explanation. Probally should have used Ai for that

Studying OpenCode’s Architecture

Instead of guessing the layout, I got AI to pull anomalyco/opencode from GitHub and read through their TUI package. Two key discoveries:

The prompt input works like OpenCode’s. I tried using the single-line input component first, but it wouldn’t submit on Enter. OpenCode uses a multi-line textarea with a trick: they grab the text directly from the component on every keystroke, and when you press Enter (without Shift or Ctrl), they intercept it, stop it from making a new line, and send the message instead. I do the exact same thing now and it actually works.

The green bar on the left. Every input box and every message in OpenCode has a thin vertical colored line on its left edge. It’s just a simple border, but it makes the whole interface feel structured and intentional. I borrowed that. Green accent bar for user prompts, dimmer gray for NIMBL replies. Cheap to render, huge visual upgrade.

TUI Structure

Two screens, one signal: view() toggles between "home" and "chat".

Home screen – Centered layout with the ASCII NIMBL logo, tagline, and a bordered textarea. The green left-border accent matches my brand.

Chat screen – Scrollbox of message bubbles (each has an accent bar + label), with a persistent input at the bottom. Token count and estimated cost update in the status bar on every response. /quit and /clear are wired in both views.

What’s Next

the app works. I type a prompt, Enter submits it, the API calls, and the response renders in the chat view with token usage shown. The next plan is adding all the modes, AI provider switching, plan and build mode, and letting the AI read and write files. Plus the /compact command from OpenCode. The screenshots show how it looks now and how it looked before. Heavily inspired by OpenCode and some elements are even copied, so the readme will fully credit them. Also I’m allowed to use them as they are under MIT license. Bless OpenCode.

Stack: Bun 1.3.14 · OpenTUI 0.4.5 · SolidJS 1.9.10 · TypeScript strict · Vercel AI SDK 7

NIMBL — Devlog #1: Building a Token-Efficient CLI Companion

I started NIMBL because I wanted a coding assistant that lives in the terminal and actually conserves tokens instead of burning through context windows. The goal for this first phase was to build a lightweight, responsive tool with a polished feel without unnecessary bloat.

---

Adapting OpenCode’s TUI Architecture

My initial approach was building a terminal interface from scratch using standard Textual widgets. I got a basic mockup working, but the component architecture felt rigid and didn’t scale well. Instead of continuing with a fragile layout, I pulled down the OpenCode CLI source code (anomalyco/opencode), studied its TUI architecture, and adapted its two-column layout system for NIMBL.

• Decoupled Components: Refactored the app into a persistent sidebar (housing the model selector and session history), a main chat panel, and a live status bar.
• Theme & Identity: Extracted the CSS structure into styles.tcss. I kept the dark background (#0a0a0a) but shifted the accent color from orange to a custom green (#4ade80) to give NIMBL its own identity.

---

Multi-Provider Backend & Failover

The backend is where the core engineering happened. I built a multi-provider API client that speaks OpenAI-compatible endpoints across Hack Club AI, Google AI Studio, Groq, and local Ollama.

• Automatic Failover: If an active provider drops or hits a rate limit, the client automatically rotates to the next available provider without interrupting the session.
• Streaming & Telemetry: Responses stream directly into the chat panel in real-time, with token usage tracked on every API call to keep resource consumption visible.

---

Context Budget Allocator & /compact

To keep context usage strictly governed, I built a context engine driven by a token budget allocator. Instead of letting prompts grow infinitely, the context window is portioned into fixed allocations:

• 15% System Prompts: Reserved for base instructions and active mode constraints.
• 25% Retrieved Context: Allocated for project structure, active files, and RAG retrieval.
• 35% Conversation History: Dedicated to the active chat thread.
• Scratchpad Space: The remaining allocation is preserved for scratch work and generation.

The system automatically warns you when reaching 60% capacity and provides a /compact command to summarize and prune older messages.

---

Modes, Gamification, and CLI

NIMBL features three operational modes, each with its own system prompt and context state:

• Assist Mode: A direct coding partner focused on concise, minimal code suggestions.
• Learn Mode: A Socratic tutor that guides you through concepts using progressive hints rather than giving answers immediately.
• Review Mode: A teaching code reviewer that explains why code is problematic rather than just flagging syntax.

I also integrated a learning tracker that records XP, streaks, and skill mastery as you work through concepts, alongside a prompt cache layer that estimates savings from repeated system prefixes.

Everything is wrapped in Click commands: nimbl run launches the TUI, nimbl ask handles one-shot terminal queries, and nimbl setup walks through API key configuration.

---

Next Steps

• UI Event Integration: Wire TUI message events (MessageSubmitted) directly to the backend agent logic.
• Telemetry Refinement: Connect the status bar directly to real-time token metrics.
• Local File Tooling: Implement tools to let the agent read and write files.

MIRA — Day 13: Backend Bug Cleanup, Test Suite Expansion, and AI Tooling

Today was split between fixing silent backend bugs in MIRA-AI, expanding my test suite to 120 passing tests, and configuring my external AI development tools.

---

Backend Bug Cleanup

I ran a full diagnostic check across the codebase and resolved 15 specific bugs:

• Duplicate Camera Service: Found and removed a bug in main.py that was initializing two separate camera service instances on startup.
• TFLite Tracking Guard: Disabled ByteTrack tracking when running TFLite models in camera_service.py, since TFLite inference doesn’t support ByteTrack tracking.
• Modern FastAPI Lifecycle: Migrated from deprecated on_event handlers to the modern asynccontextmanager lifespan pattern across my API services.
• Code Quality: Updated remaining deprecated .dict() calls to .model_dump() for Pydantic v2, fixed an immutable frozenset cleanup bug in my tests, and resolved 183 Ruff linter warnings.

---

Expanding the Test Suite to 120 Tests

I wrote new unit test suites for previously uncovered backend modules:

• Dashboard Models & WebSockets: Added 30+ tests covering Pydantic model serialization and boundary conditions, plus 15+ tests for WebSocket event callbacks and metric queuing.
• Test Isolation: Fixed a bug in test_logger.py where the root logger was leaking state between test runs, and fixed brittle Python version checks in test_serialization.py.
• Current Status: My test suite is now sitting at 120 out of 120 passing tests with 0 Ruff lint errors.

---

AI Developer Tooling & DigitalOcean

Outside the core MIRA repository, I spent time configuring my external OpenCode AI setup across multiple providers (OpenRouter, local Docker, Ollama, and GitHub Copilot):

• Fixed an endpoint URL misconfiguration in my OpenRouter settings and tested connectivity across 23 models.
• Added handling for “reasoning models” that return output in a reasoning field rather than standard content.
• Researched how to use my $200 DigitalOcean student credit before it expires on July 31st. Since student credits cannot be applied to GPU Droplets, I plan to deploy a 128 GB Memory-Optimized CPU Droplet to run Ollama with 70B models for offloaded testing.

---

Next Steps

• Hardware Benchmarks: Run real-world FPS and latency tests on a Raspberry Pi Zero 2W.
• Trash Class Data: Collect targeted training data for the general “Trash” class, which remains my weakest class at 7.1% mAP50.
  Also go over the website cause it looks pretty ai generated right now

MIRA — Day 12: Codebase Audit, EXP-017 Results, and Cross-Platform Fixes

Before diving into codebase fixes, I finished training EXP-017 on Kaggle using the merged mira_all dataset. It reached 59.3% mAP50—which is slightly below EXP-014’s 60.7%. This confirmed a key lesson for me: dataset quality and class balance matter far more than sheer image count. After that, I shifted my focus to running a full diagnostic audit of my codebase to clean up bugs and edge cases.

---

Fixing the Health Check

My health check command (mira doctor) was crashing immediately on Windows with a UnicodeEncodeError due to special status symbols (✓, ✗, ⚠).

• I replaced them with plain ASCII characters ([OK], [!], [FAIL]).
• Added checks for missing camera backends on Linux so the health check runs cleanly end-to-end regardless of the OS.

---

Making Torch and TensorFlow Optional

MIRA supports both PyTorch (.pt) and TFLite (.tflite) models, but requiring both libraries installed was frustrating—PyTorch alone takes up over 800 MB.

• I refactored my model adapters and dashboard to use lazy-imports.
• If PyTorch isn’t installed, PyTorch models gracefully report as unsupported without crashing the application.
• This allows a lightweight setup without forcing massive unnecessary downloads if someone only wants to run TFLite models on edge hardware.

---

Cross-Platform & Service Hardening

I cleaned up my startup scripts and hardened my camera service against common edge-case failures:

• Updated mira.bat with PYTHONUTF8=1 for Windows, and created a native mira.sh for Linux and macOS.
• Added OS guards around the Windows-specific DirectShow camera backend.
• Fixed a potential division-by-zero crash when video FPS drops to zero.
• Switched Linux CPU temperature reads to psutil instead of hardcoding 0°C.
• Ensured the model object memory is properly freed on shutdown.

---

Security, Pydantic v2, and Edge Cases

• Security: I restricted CORS in my web dashboard from wildcard access (*) to localhost (127.0.0.1:8000).
• Pydantic Upgrade: Replaced deprecated .dict() calls with .model_dump() for Pydantic v2 compatibility.
• Integrity: Explicitly forced UTF-8 encoding on all YAML file reads, and added SHA-256 hash verification to mira download.

---

Current System Status

After resolving these issues, my test suite is sitting at 99 out of 99 passing unit tests, my Ruff linter reports 0 errors, and all 17 CLI commands are fully functional. I also prepared my HackClub StarDance submission writeup and updated the main README with current CLI specs and experiment data.

---

Next Steps

• Hardware Benchmarking: Measure real-world FPS and inference latency on a Raspberry Pi Zero 2W.
• Trash Class Data: Collect targeted training images to improve the general “Trash” class, which remains my weakest performer (7.1% mAP50).

in this pic you can see a sneakpeak to a new feature

MIRA — Day 11: Dashboard Redesign, Dual Modes, and EXP-017 Datasets

Today was all about front-end overhauls and preparation for my next training run. I redesigned my web interface, split it into two dedicated operating modes, and finalized my biggest dataset merge yet to prepare for training EXP-017.

---

Clean Light Theme Dashboard

I shifted the entire MIRA web dashboard from my old dark theme to a clean light theme with a mutiple colors.

JUFO vs. Stardance Modes

I introduced two separate dashboard modes that I can select on-the-fly using URL parameters:

• JUFO Mode (?mode=jufo): A technical view tailored for competition judges, featuring deep metrics, historical training graphs, and real-time backend diagnostics.
• Stardance Mode (?mode=stardance): An interactive, simplified playground designed for quick live demonstrations with clean control toggles.

---

System Health & Live Settings

I replaced my static history table with a real-time System-Health panel displaying CPU, RAM, temperature, and inference latency using animated progress bars.

I also added an “Einstellungen” (Settings) tab to Stardance mode, allowing me to adjust model weights (via a dynamic API model picker), confidence thresholds, IoU, and camera resolution directly from the web UI without needing to restart the Flask backend.

---

Threaded Camera Pipeline

My camera stream was bottlenecking my inference engine. I decoupled the frame capture from the detection loop by moving the capture process into a dedicated Python threading.Thread. The web stream now runs at a smooth 20 FPS at 320px resolution. I also patched Camera.release() to verify the stream is open before shutting down, preventing a common camera-close crash.

---

Datasets & EXP-017 Preparation

I successfully merged all four of my data sources (TACO, TrashNet, Roboflow, and WaRP) into a unified dataset named mira_all. It contains 9,774 images across 5 classes.

While generating the ZIP file, I hit an archive path encoding issue that broke folder hierarchies. I solved this by using just the normal windows option to zip files instead of the powershell way which I wanted to test out

My current baseline model (EXP-014) is still performing well with an 85.8% F1-score on my field benchmark, though the general “Trash” class remains my weakest spot at 37.6% due to how much random variation “trash” actually has.

---

Pipline

I also worked on pipline and am working on making it plug play allowing user to get any 3rd party model plug it in and use the full research pipline to retrain it compare it benchmark it and do a lot of stuff. I also added the option to add custom datasets and expanded on the CLI with mutiple custom commands this is probally, the second last post because I am nearly finished.
The ui for the dashboard was made with ne gpt 5.6 sol because I just couldnt come up with a good ui.

Next Steps

• Train EXP-017: Upload the clean mira_all dataset and run training on Kaggle.
• Logger Analysis: Review the data from my new last_per_class.json file to see which items are being misidentified during continuous live runs.

MIRA — Day 10: Removing Streamlit, Safety, and Purging my LLM Tools

I spent the last week cleaning up a massive amount of technical debt. After about 40 commits, touching 168 files, and adding 6,186 lines of code while removing 1,200 lines of old code, the repo is finally much cleaner.

The biggest changes were replacing Streamlit with a Flask-SocketIO server, auditing my project for the Jugend Forscht competition, and removing my custom AI development tools (which I built earlier to help me write code).

---

Replacing Streamlit with Flask + SocketIO

Streamlit was a bottleneck for real-time video streaming and thread-safe operations because it re-runs the entire script on user interaction.

• The Architecture: I built a custom Flask and SocketIO dashboard under src/dashboard_flask/. Live camera frames are captured in a background thread, encoded to JPEG, and pushed over WebSockets.
• Thread Safety: I implemented four threading locks to prevent race conditions: camera_lock, model_lock, inventory_lock, and config_lock.
• Bugs Fixed: I resolved an unnecessary BGR-to-RGB conversion that was washing out my live camera colors, and wrapped my configuration updater in config_lock to prevent multi-thread memory corruption.

---

Modularizing my Inference Pipeline

To clean up the backend, I consolidated all duplication:

• I deleted the separate live and debug scripts, merging their logic into a single src/inference_engine.py module.
• This engine handles the threaded camera feed, YOLO11n prediction loop, and my BYTETrack fallback.
• All paths, labels, and parameters are now centralized in src/config.py.

---

Purging my Custom AI Tools

I built a terminal-based AI coding assistant to help me write and modify code. However, after an architectural review, I decided it does not belong in this repository.

• The Reason: MIRA stands for Machine Intelligence for Recycling Automation. The codebase should focus strictly on computer vision and robotics, not LLM wrappers.
• The Separation: I separated all AI coding utilities into a new MIRA-DevTools repository, which I will start as I near the end of MIRA.

---

JUFO Competition Audit & Paper

Since this project is for Jugend Forscht, I have been heavily working on my scientific paper (which I realized I haven’t actually mentioned in previous updates).

• AI Audit: I ran an AI-assisted audit against JUFO evaluation criteria. It originally rated my project at a 6/10 because my paper lacked a deep discussion section and YOLO11n graphs.
• The Fixes: I expanded my analysis on how the WaRP dataset’s class imbalance skewed my trash predictions, and compiled a 22-page LaTeX report with proper bibliographies and training curves.
• Quality Assurance: I added 17 unit tests and automated GitHub Actions with Pytest and Ruff. I will translate the paper to English and upload it soon.

---

Next Steps

• Folder Cleanup: Run git-filter-repo to shrink my 17 GB folder by removing cached datasets before pushing to GitHub.
• Final Training: Train one last model on my optimized dataset combination once my Kaggle GPU hours reset.

MIRA Day 9: YOLO11 Migration, INT8 Issues, and Dataset Analysis

Over the last few days, I focused on improving the project structure, simplifying the codebase, and testing different approaches for the detection pipeline.

I made around 15 commits, added roughly 2,200 lines of code, and removed around 1,000 lines that were duplicated or no longer needed.

The main changes were migrating from YOLOv8n to YOLO11n, creating an automated benchmark script, and investigating why some dataset combinations performed worse than expected.

YOLO11 and Dataset Experiments

I moved from YOLOv8n to YOLO11n because it is better suited for edge deployment while keeping the model lightweight. Since MIRA is intended to run on limited hardware, inference speed and efficiency are important.

To test different training setups, I trained four model variants using Kaggle T4 GPUs. I compared several datasets:

• TACO: Outdoor litter images.
• TrashNet: Clean recycling images from controlled environments.
• Roboflow Trash Detection: Waste images with bounding box annotations.
• WaRP: Industrial recycling images from conveyor belt environments.

The best results came from combining TACO, TrashNet, and Roboflow. This combination performed better during live camera tests because it contained both clean object images and more realistic trash scenarios.

Why WaRP Reduced Performance

Adding WaRP unexpectedly reduced the performance of my general “Trash” class.

The reason was a dataset imbalance. WaRP contains many detailed recycling categories like plastic, glass, metal, and cardboard, but it has very few examples matching my general trash category.

After mapping WaRP’s classes into my five target classes, most additional training data was added to recyclable categories. This caused the model to become more focused on plastic, paper, and glass while becoming less reliable at detecting mixed waste.

This showed that more training data does not always mean better results. The distribution and relevance of the data are equally important.

INT8 Quantization Problem

While testing my INT8 TFLite model, I noticed that the detection rate was almost zero. The model itself was working, but my confidence filtering removed valid detections.

After INT8 conversion, the output confidence values changed compared to the original model. Some detections that previously passed a 0.50 confidence threshold were now below it.

Because my pipeline still used the old threshold, valid detections were discarded.

I updated the benchmark and live detector to automatically use a lower confidence threshold for INT8 models. After this change, detections worked correctly again.

Code Cleanup and Fixes

I found several duplicated parts in the project, especially multiple versions of the bounding box drawing logic. I centralized these functions and cleaned up configuration and path handling.

I also fixed two problems in the live detection system:

1. Tracker dependency issue:
   The detector previously used BoT-SORT by default, which caused problems if the required package was missing. I changed the fallback system to use BYTETrack first.

2. Double confidence filtering:
   Confidence filtering was happening both during prediction and again during visualization. I moved this into one centralized post-processing step to avoid removing valid detections.

Repository Cleanup

The project folder has grown to around 17 GB because of duplicated datasets, training results, logs, and model checkpoints.

Before pushing everything to GitHub, I need to clean the Git history, improve the .gitignore, and remove unnecessary files. The goal is to reduce the repository size to below 500 MB.

Next Steps

• Clean the repository and remove large unused files.
• Train the final model with the improved dataset combination.
• Generate evaluation graphs and confusion matrices for the final documentation.

CLI Student Database — First Steps Into Python

Work Completed:

• Initialized a student database CLI and implemented basic data persistence using JSON.
• Built robust input validation (try/except loops) to handle invalid menu choices and prevent the program from crashing on typos (like typing abc for an integer).
• Restricted range entries (e.g., age 0–20, grades 1–13) to prevent garbage data insertion.
• Expanded the menu to a full CRUD flow: Add (with duplicate detection), View, Search (case-insensitive partial match), Update, Delete (with confirmation), and Statistics.
• Implemented an auto-save routine that writes to the JSON file after every database mutation (Add, Update, Delete) to prevent data loss.
• Refactored the validation parameters into centralized constants at the top of the file to make maintenance easier.

---

The Struggle: Smart Edits & Empty Inputs

The biggest logic challenge was the Update feature. I didn’t want to force users to re-type a student’s age and grade if they only wanted to change one of them.

My solution was checking if the input is empty using .strip() and evaluating it dynamically:

new_age = input("New age (press Enter to keep current): ").strip()
if new_age:
    database[name]["age"] = int(new_age)

If they just hit Enter, Python evaluates the empty string as falsy and safely skips the update, leaving the old value intact. It was a simple fix, but it made the CLI feel better

---

Next Steps:

• Sorting data: Study how to sort dictionary outputs by name, age, or grade.
• Grade labels: Map numerical grades to letter categories (A/B/C/D/F). So 90 would be A
• Nested structures: Expand the schema to support multiple subjects per student.
• Data Export: Learn the csv module to output student lists to a spreadsheet format.
• System Refactoring: Move the program toward Object-Oriented Programming (OOP) classes, and eventually migrate JSON file storage to a lightweight SQLite database.

---

Conclusion

A few hours ago this was a script that crashed on bad input and forgot your data. Now it’s a small application with validation, persistence, and a proper menu. The biggest win wasn’t any single feature — it was realizing that I can take a broken thing, isolate the problem, and fix it incrementally. That’s the actual skill. It was a great excersice to pracitce coding while I wait for my MIRA training to complete.

MIRA — Day 8: The Invisible Boxes (and Why Two Models Were Lying to Me)

BUGS

After hours of digging through Ultralytics internals with AI assistance, I found not one but three separate bugs all stacking on top of each other.

Bug #1: Tracker

Ultralytics 8.4.84 silently switched the default tracker from ByteTrack to something called TRACKTRACK. I never asked for this. Nobody told me.

TRACKTRACK has internal confidence thresholds that are absurdly high:
track_high_thresh: 0.6 # Only detections >= 0.6 are “real” new_track_thresh: 0.7 # New objects need >= 0.7 to be tracked
Here’s the problem: INT8 quantization drops confidence scores by 20-40%. So a detection that would score 0.55 in the original PyTorch model now scores ~0.35 after quantization. TRACKTRACK looks at 0.35, says “too low,” and silently throws it away. No error. No warning. Just… gone.

Bug #2: The Double Filter

Even if TRACKTRACK somehow kept a box, my drawing function had its own confidence filter set to 0.5. So a box at conf=0.4 that survived the tracker got killed again at the drawing stage.

Bug #3: Nothing Stopped the User from Loading the Wrong Model

Nothing in the code prevented me from loading mira_classifier_int8.tflite — a classification model — into the detection pipeline. Which i failed to realise. The code just checked if the file ended in .tflite and said “sure, load it as a detector.” It would’ve failed silently or produced garbage.

live_detection.py — The Big One

I rewrote the core inference loop. Three major changes:

Forced ByteTrack everywhere. PyTorch models now explicitly use tracker="bytetrack.yaml" instead of whatever Ultralytics defaults to. No more silent tracker swaps.

Bypassed the tracker entirely for INT8 models. TFLite INT8 now uses model.predict() instead of model.track(). No tracker = no threshold filtering = boxes actually appear. Conf is auto-overridden to 0.25 at startup because INT8 models need lower thresholds.

Added model validation. If you try to load a classifier model into the detection pipeline, you now get a clear error:
ERROR: ‘mira_classifier_int8.tflite’ is a CLASSIFIER model, not a detector. Live detection requires a detection model (.pt or detection .tflite). Use ’mira eval-class –model mira_classifier_int8.tflite –exp ’ instead.
No more silent failures.

dashboard.py — Same Story, Streamlit Edition

The Streamlit dashboard had the exact same tracker and threshold bugs. Fixed identically: ByteTrack forced, INT8 models use predict(), classifier models filtered from the sidebar before you can even select them.

cli.py — First Line of Defence

Updated cli.py and added mutiple new commands to make it easier to use also made it possible to use in the termnail through a bat file so u can run the scripts in powershell if you want if you just add .\mira and then commands. Also Added validation in the live subcommand so classifier models get rejected before the subprocess even spawns. Faster error, cleaner output.

Folder Cleanup Reality

Checked the project size today: 17.3 GB. That’s absurd for a recycling classifier. Most of it is:

• Failed experiment outputs
• Raw dataset images that should’ve been gitignored
• Multiple copies of the same model in different formats
• Private test data that never should’ve been committed

Need to do a serious cleanup before pushing anything.

What’s Next

Downloading three new datasets to retrain on:

• SortWaste (WACV 2026) — real conveyor belt images, 87k bounding boxes
• TACO — waste in wild environments (forests, streets, beaches)
• NAVER Recycle Trash — 21k images, indoor + outdoor

Target: Combine everything into ~20,000 images, retrain with YOLO11n, hit 80%+ mAP50 with actual real-world robustness.

MIRA — Day 7: The “Stage B” Breakthrough (Multi-Object Detection)

The Engineering Change: Scrapping the Foundation

Stage A (Classification) was a controlled laboratory success, but it failed the “Real World Test”, and wasnt able to accuractly depict objects with a live cam feed, I also hit a major roadblock: my auto-labeling script used Canny Edge detection which, due to table glare and shadows, mistook my entire desk for a single object. This taught me the most important lesson in AI: Garbage In, Garbage Out (GIGO).

To save MIRA, I had to delete my original dataset and move to a professional Object Detection architecture (YOLOv8-Nano).

---

Data Engineering: Mapping 64 Classes to 5

I forked a high-variance “Wild Trash” dataset containing thousands of images of waste in complex environments (grass, dirt, water).

• The Problem: The dataset had 64 highly specific classes (e.g., “Aerosol,” “Egg carton,” “Plastic lid”).
• The Solution: I wrote a custom Class Remapping Engine in Python. This script programmatically rewrote thousands of .txt label files, collapsing 64 sub-categories into MIRA’s 5 core functional classes: Glass, Metal, Paper, Plastic, and Trash.
• Result: A robust, “Super-Dataset” of ~3,300 images ready for spatial training.

Cloud-Accelerated Training

Training an object detector on a local CPU would have taken days. I migrated the pipeline to Google Colab, utilizing an NVIDIA Tesla T4 GPU.

• Strategy: I implemented a patience=0 strategy and tuned the learning rate to 1e-3 to force the model to converge over 100 epochs despite the high variance of the “Wild” data.
• Outcome: Achieved an mAP50 of 82.3% on the combined dataset. The “Nano” architecture was chosen specifically to ensure the final weights (approx 6MB) can run standalone on a Raspberry Pi.

The Live Multi-Object Tracking Engine

The most exciting part is the new live_detection.py. MIRA no longer just “names” an image; it locates and tracks items in 3D space.

• ByteTrack Integration: I enabled the ByteTrack algorithm (persist=True). Now, the AI assigns a unique ID to every object. If a bottle is briefly covered by a hand, the Kalman Filter “remembers” its position, preventing the future servos for the robot from jittering.
• Performance:
  • Latency: ~40ms (Inference)
  • Throughput: ~15.6 FPS
  • Simultaneous Detection: Successfully tested with 3+ materials in the frame at once.

Diagnostic: Identifying Edge Cases

Testing today revealed a specific weakness: Metal cans in vertical orientation (opening facing the camera). Because the feature profile changes from a cylinder to a dark circle, detection confidence drops.

• Fix Strategy: I will supplement the dataset with ~50 manual “end-on” photos of cans to patch this “Canonical View Bias” before the hardware build.

What’s Next

Creating a dashboard to see everything regarding the AI and trying to quantize it further

this was highly aided by AI this time as I just couldnt fix and needed AI help to fix it. Just wanted to check how big my folder is and its 17.3 GB i really need to clean it up I still have alot failed experiements and private data inside

MIRA — Day 6: Post-Training Quantization and Signal Smoothing

What I Built Today

Today I achieved major optimization milestones, Finally creating the INT8 Quantizatio

Post-Training INT8 Quantization

I implemented src/quantize.py using TensorFlow’s TFLiteConverter. The script center-crops and resizes 100 representative images from the dataset to dynamically calibrate activation ranges, converting the uncompressed Keras model to full 8-bit integer precision.

Compression Summary:

• Keras Full Model: 23.48 MB
• Standard TFLite: 8.49 MB
• Quantized INT8 TFLite: 2.61 MB (9.0x smaller than Keras binary)

Zero-Dependency Live TFLite Inference

I completely created the live webcam inference stream (src/live_inference_tflite.py). By implementing a standard mathematical Softmax and image scaling pipeline in pure NumPy, the script now starts instantly and runs with minimal CPU overhead.

Jitter Reduction via Exponential Moving Average (EMA)

To prevent erratic jumps in presicion so it jumping from 60 to 70 in a second caused by camera focus and exposure changes, I implemented a temporal smoothing filter on the output probabilities:

smoothed_probs = alpha * raw_probs + (1 - alpha) * smoothed_probs

Furture plans

So after thinking about it I found 2 logical problems in my model. FIrst of all the dataset it trained on is still to limited and there is not enough variaton in objects.
Secondly the whole Infracstructer is wrong and at the moment its classifier and not a detecter so Its nor able to detect mutipile objects which is important for the Robot arm its also not able to discern mutipile objects on the same screen. So the next Part will porbally be a long post in which I fix all these bugs and add New featurs

MIRA — Day 5: Transfer Learning, and Two-Stage Fine-Tuning

1. Summary

Since the last update, which was quite a while ago because I needed to learn some more basics, MIRA has evolved. By implementing transfer learning with MobileNetV2, and performing partial layer fine-tuning, overall validation accuracy was raised from 61% to 87.42%, while validation loss dropped from 1.06 to 0.31. Which is very important.

---

3. Key Technical Breakthroughs & Refactors

The “Paper” Class Feature Extraction Bottleneck

In the initial baseline model, paper recall was extremely poor (0.07). The model regularly confused paper with metal or plastic.

• The Fix: Using the pre-trained ImageNet weights of MobileNetV2 resolved this. The model now extracts high-level semantic texture and edge properties. In EXP-003, paper reached a 96% precision rate. but instead metal got a bit worse and has now reached 77% from its orignal 86%

Refactoring src/evaluate.py

To prevent difficulties in strcuturing and avoid having multiple different evaluation scripts for each model , I refactored evaluate.py using Python’s argparse library. The script now accepts command-line arguments to evaluate any model dynamically so i can just use a comand and where it should save its finding and it will run the script acording to that:

python src/evaluate.py --model mira_fine_tuned_model.keras --exp EXP-003_FineTuning

It also dynamically scales the loaded image resolution (224x224 for MobileNetV2 vs. 180x180 for baseline) to avoid input shape mismatches

Implementation of Two-Stage Fine-Tuning

After training the custom dense classification head on frozen MobileNetV2 features, I wrote train_fine_tune.py to unfreeze the base model from layer 100 onwards (leaving the low-level edge-detection layers 0–99 intact). Recompiling with an ultra-low learning rate (1e-5) allowed the model to subtly adjust its deep weights to the recycling dataset:

This successfully drove validation loss down to 0.3148, showing that the network is making predictions with significantly higher statistical confidence.

Automated Latency Benchmarking

I integrated CPU latency tracking directly into the evaluation loop using time.perf_counter() to record inference times in milliseconds per image, This will be used later to see how it performs on small hardware and different

I would advise everybody who wants to do something similar to use the resources i linked.

---

4. Next Milestones

1. Real-time Live Webcam Inference (src/live_inference.py): Deploying the mira_fine_tuned_model.keras on a live video stream to evaluate performance and classification stability under varying lighting and structural angles and add these kinds of borders which state what the computer thinks the object i show is
2. Embedded Planning: Utilising INT8 to quantize the model to a fracture of its size to run on a standalone, onboard single-board computer (Raspberry Pi), moving closer to an autonomous physical robot. Which will then be able to automatically sort them.

5. Resources Used

• Framework Foundation: TensorFlow Image Classification Tutorial
• Data Pipelines: TensorFlow image_dataset_from_directory API
• Model Composition: Keras Sequential Model Documentation
• Conceptual Architecture: 3Blue1Brown Neural Networks Playlist —

MIRA — Day 4: Evaluate and important results

What I Built

Today I built evaluate.py and redid all of my images so my data and captured 800 increasing the count from 120 to 800. This allows me to load my 800-image dataset, evaluate it against a validation set, and output a visual Confusion Matrix.

Vital Changes & Difficulties Faced

1. The Alphabet-Slice Bug (Deterministic Split vs. Batch Shuffling)

When I first ran evaluate.py, my Confusion Matrix looked bizarre: I had 159 test files, but every single one of them had a true label of “plastic”!

To keep my evaluation predictions aligned with the correct labels, I had initially set shuffle=False when loading the directory. Because Keras loads folders alphabetically (glass -> metal -> paper -> plastic), disabling shuffling caused it to read the sorted list and simply slice the last 20% for testing. The last 20% of the alphabet was entirely plastic.

To fix this, I set shuffle=True so Keras would pull the exact same random validation split as the training script. To prevent the predictions from getting mismatched during shuffling, I refactored the evaluation script to pull images and labels simultaneously, batch-by-batch:

y_true = []
y_pred = []

for images, labels in val_ds:
    preds = model.predict(images, verbose=0)
    batch_preds = np.argmax(preds, axis=1)
    
    y_pred.extend(batch_preds)
    y_true.extend(labels.numpy())

No matter how randomly Keras shuffles the images, our answers and guesses are extracted at the exact same instant, making mismatching impossible.

The Paper vs. Metal Crisis 📊

With the pipeline corrected, I ran the evaluation and got a real, stable baseline accuracy of 61% worse than yestarday which was expected as the images were alot more than before and before them it was just learning the images and not guessing. Across my 159 validation images. The Confusion Matrix revealed a major failure point in my baseline CNN:

• True Paper Images: 42
• Correctly classified as Paper: 3
• Misclassified as Metal: 25 (59.5%)

My simple 3-layer CNN is nearly blind to paper textures. It looks at white paper or cardboard with bright reflections and thinks: “Ah, a bright shiny reflection! This must be a metal can.”

What’s Next

Now that I have documented my true baseline and identified its major blind spot, we are ready to implement Transfer Learning (MobileNetV2). This pre-trained model has seen millions of images and knows how to distinguish between the flat texture of paper/cardboard and the metallic shine of aluminum. Also i noticed that I struggled quite alot with this part so iam going to pause this project a bit to do another project ot solidify my python basics and continue from there onwards.

Resources & Documentation

• Keras Sequential Model API — Documentation used to structure layers, dropout, and data augmentation.
• Scikit-Learn Classification Report — Reference for evaluating precision and recall.
• Scikit-Learn Confusion Matrix Display — Documentation used to plot the final matrix.

MIRA — Day 3 Creating Model and training on Limited data

What I Built Today

Today I built train_baseline.py, the first complete CNN training pipeline for MIRA. The script loads the dataset, applies data augmentation, trains a 3-layer convolutional neural network, plots the accuracy and loss curves, and saves the trained model to disk. I followed the official TensorFlow image classification tutorial and adjusted it to fit the MIRA dataset structure and 4-class recycling problem.

model = Sequential([
  data_augmentation,
  layers.Rescaling(1./255),
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Dropout(0.2),
  layers.Flatten(),
  layers.Dense(128, activation='relu'),
  layers.Dense(num_classes, name="outputs")
])

Difficulty: Image Stretching

The first training run loaded images incorrectly, the model was receiving stretched and distorted frames because image_dataset_from_directory resizes images to the target dimensions by default, squashing non-square images. Fixed by adding crop_to_aspect_ratio=True, which crops from the center instead of stretching:

train_ds = tf.keras.utils.image_dataset_from_directory(
  data_dir,
  validation_split=0.2,
  subset="training",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size,
  crop_to_aspect_ratio=True)

Difficulty: Overfitting

Early runs showed training accuracy climbing while validation accuracy stayed flat or dropped ,the model was memorizing the training images rather than learning generalizable features. This happens when the dataset is too small relative to the model’s capacity. Fixed with two countermeasures: data augmentation (RandomFlip, RandomRotation, RandomZoom) to artificially vary the training images, and Dropout(0.2) to prevent individual neurons from becoming too specialized.

Results

val_acc = 0.8235 @ epoch 20
val_loss = 0.4030 @ epoch 20
best val_acc = 0.8824 @ epoch 19
best val_loss = 0.3646 @ epoch 19
dataset = 126 images / 4 classes

The high epoch-to-epoch variance in validation accuracy (swings of ±18%) is happening because of the low validation set containing only 17 images, one wrong prediction equals a 5.9% accuracy shift. Results will stabilize further after expanding the dataset.

Coming from C/C++

Well nothing to say because I never coded anything like this in C/C++ so this one will be the last Coming from C/C++

What’s Next

I will capture 200 images per class (800 total). So my results are actually meaningful and verfiable also in the images I will adjust lighning and more

After the dataset is large enough:
I will Write evaluate.py, load the saved model, run it on the test set, and compute a full classification report: precision, recall, and F1-score per class. So I get a per class breakdown of the real acuraccy of the certain folder each and not just the total median.

Resources Used

• TensorFlow image classification tutorial
• TensorFlow image_dataset_from_directory API
• Keras Sequential model API
• 3Blue1Brown — Neural Networks series

Credits

Also wanted to shoutout RUMI where i got insperation for the naming scheme from.

MIRA — Day 3 visualise data set

What I Built Today

Today I wrote visualize_dataset.py from scratch using the Python os, random, and Matplotlib documentation. The script counts images per class folder, samples up to 5 random images per class, and displays them in a labeled 4×5 grid. This is a sanity check before the model training you cannot train reliably on a dataset you have never inspected.

import os
import random
import cv2
import matplotlib.pyplot as plt

DATA_DIR = "../data/"
CLASSES = ['glass', 'metal', 'paper', 'plastic']
SAMPLES = 5

total_files = 0
for class_name in CLASSES:
    folder = os.path.join(DATA_DIR, class_name)
    num_files = len([f for f in os.listdir(folder) if f.lower().endswith(('.jpg', '.jpeg', '.png'))])
    print(f"{class_name:<10}: {num_files} images")
    total_files += num_files
print(f"{'Total':<10}: {total_files} images")

fig, axes = plt.subplots(len(CLASSES), SAMPLES, figsize=(15, 10))

for row, class_name in enumerate(CLASSES):
    folder = os.path.join(DATA_DIR, class_name)
    files = [f for f in os.listdir(folder) if f.lower().endswith(('.jpg', '.jpeg', '.png'))]
    sampled = random.sample(files, min(SAMPLES, len(files)))

    for col, filename in enumerate(sampled):
        img = cv2.imread(os.path.join(folder, filename))
        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
        axes[row, col].imshow(img)
        axes[row, col].axis('off')
        if col == 0:
            axes[row, col].set_ylabel(class_name, fontsize=12)

plt.tight_layout()
plt.show()

How It Works

The script has two distinct parts. First, the counter loop iterates over each class folder using os.listdir(), filters for image file extensions, and accumulates a total. Second, the grid section uses plt.subplots() to create a fixed axes array indexed by axes[row, col], each cell addressed independently by class row and sample column. random.sample(files, min(SAMPLES, len(files))) guards against crashes when a folder has fewer than 5 images by capping k to whatever is available.
So if I only got three it just going to use three and not crash like it would normaly do for this part I ran my Code through ki afterwards and he told me to add it

The one conversion that matters: cv2.cvtColor(img, cv2.COLOR_BGR2RGB) before passing to Matplotlib. OpenCV loads images in BGR channel order; Matplotlib expects RGB. Without this, all reds appear blue. I forgot this but when the pictures came out weird i remeberd the tutorial by Tech with Tim where he mentions it mutipile times I would advise everybody to watch his openCV turorial

Difficulty

Straightforward session. Coming from C/C++, the structure of iterating over a list of strings and using them as both display labels and filesystem paths felt natural immediately. plt.subplots() returning a 2D array of axes objects is the same concept as a 2D array in C — index by row and column. No significant blockers.

Resources used:

• Matplotlib docs: pyplot.subplots, pyplot.imshow, Axes.set_ylabel
• Python docs: os.path.join, os.listdir
• OpenCV docs: cvtColor color conversion codes

Dataset Status

glass : 32 images
metal : 31 images
paper : 30 images
plastic : 33 images
Total : 126 images

Also all this images are not in the Github at the moment because these aren’t high quality and good images on which i can train my model

What’s Next

Dataset is captured and verified. Next is the first CNN training pipeline in TensorFlow/Keras, train_baseline.py. This means learning ml and taking my first steps into it . The target for the first run is a accuracy above 75%.

MIRA — Day 2: Dataset Capture Pipeline

What I Built

Wrote capture_frame.py from scratch using only the OpenCV docs, no tutorials for the code, no AI-generated code. The script opens a live webcam feed and lets me build my own labeled image dataset by pressing keystroke shortcuts to save frames directly into categorized folders.

Evolution of the Capture Logic

In my first draft, I managed file naming with a manual integer counter. It worked, but had a critical flaw: every time the script restarted, the counter reset to 1 and began overwriting existing files. Also through out this project I have always specified the Data type because i got advice from a friend to do that.

Initial counter approach:

i = int(1)
if key == ord('1'):
    filepath = os.path.join(glass_folder, f'frame_{i}.jpg')
    cv2.imwrite(filepath, frame)
    i += 1

I replaced this with timestamp-based naming using datetime.now().strftime("%Y%m%d_%H%M%S"), which ensures every file has a unique name regardless of how many times the script restarts:

glass_20260625_143022.jpg
glass_20260625_143022.jpg

Make it more efficent to scale

My first working draft had four separate folder variables and a long if/elif chain, one block per material class. Functional, but not scalable because this would mean: adding a fifth class would mean copy-pasting another entire block.

First draft, repetitive structure:

glass_folder  = "../data/glass/"
metal_folder  = "../data/metal/"


if key == ord('1'):
    filepath = os.path.join(glass_folder, f'glass_{timestamp}.jpg')
    cv2.imwrite(filepath, frame)
elif key == ord('2'):
    filepath = os.path.join(metal_folder, f'metal_{timestamp}.jpg')
    cv2.imwrite(filepath, frame)

Refactored:

CLASSES = {
    '1': ('glass',   '../data/glass/'),
    '2': ('metal',   '../data/metal/'),
}

key_char = chr(key)
if key_char in CLASSES:
    label, folder = CLASSES[key_char]
    filepath = os.path.join(folder, f'{label}_{timestamp}.jpg')
    cv2.imwrite(filepath, frame)

Adding a fifth class now takes one line in the dictionary. The save logic doesn’t change at all which is pretty neat this was quite difficult because I never worked with classes like this but I managed to do it after googling and asking AI how to set up Classes but not providing it any context so I can write it myself.

Coming from C/C++

One thing I found difficult to utilise where f strings. I dont know if a equalivent exists in C/C++ but I never used it so it was quite weird utilising them but they’re very convient and this is another positive aspect of the high abstraction.

What’s Next

The dataset capture pipeline is working. Next session I will write visualize_dataset.py, a script that loads random samples from each class folder, displays them in a grid, and prints the class distribution as a count per folder. This will then be the basis for my CNN pipelane

---

Resources & Documentation

• OpenCV 4.x Documentation
• Python Datetime Guide
• OS Path Management
• Tech With Tim - OpenCV Python Tutorial Video 3

MIRA Project - Architecture & Infrastructure

Work Completed:

• Initialized project architecture and created the virtual environment (.venv).

• Structerd the directory (data/glass, data/metal, data/paper, data/plastic) in preparation for supervised machine learning pipelines.

• Wrote and executed a system boot script to test Python and terminal output. Which just utilises time.sleep() and print().

• linked the local workspace to GitHub and pushed the first commit.

• Studied Python syntax, focusing on loops, variables, and type hinting, easy cause I alredy got basics from other languages.

Next Step:
Studying OpenCV documentation and computer vision fundamentals to create without using AI capture_frame.py. The goal is to build an automated dataset generation script where a keystroke triggers webcam capture and pushes the image directly to the designated categorization folder So i can just make my data set generation easier or maybe I am going to use a public one iam still undecided.

Reflection: Python vs. C/C++
The logic of Python is simple, but adjusting to the syntax is quite weird. Coming from lower-level languages that require strict structuring (like C/C++), the “freedom” and abstraction of Python takes some getting used to. First of all, the absence of semicolons (;) to terminate lines and the reliance on indentation for loops and functions feels unusual.

However, the freedom makes certain things simpler. For example, Python’s for loops:

names: list[str] = ['Alice', 'Bob', 'Charlie', 'Anni-Frid']

for name in names:
    print(f'Hello, {name}')

In C, you have to manually manage the loop counter and the array index, which overcomplicates a simple process:

for (int i = 0; i < 4; i++) {
    printf("Hello, %s\n", names[i]);
}

Conclusion
For a moment I wasn’t sure if this would really be possible but I gained a lot more confidince through learning a bit and I think this will probally be a long term Project because of the Scale and future stuff I have in mind.

Project Update: Bug Fixes

Overview

This phase was all about polishing the user experience, squashing gameplay bugs, and adding new repository documentation. After getting feedback I decided to change The whole UI

Dynamic Theming & UI Improvements

One of the major frontend improvements in this sprint was fixing the player icon chooser button functionality in the web interface. Fully redesignen the Front-End and more.

I first of all started with replacing my .root with some new colors which are greenish. I also added some simple fade in/out animations to all Popups.

Another bug was that it couldn’t quite fit in a screen of the size 1980 x 1080. I fixed that by making it possible to scroll down so everything you need for the game is shown on your screen. The way i tested it is by local hosting it and then pressin F12 in the browser and then you could adjust the size of the Window which was pretty neat.

Also fixed a lot more of UI bugs such as weird buttons and more but this was the important stuff.

Squashing Gameplay Bugs

I knocked out two major bugs in this update:

• Trading System Fix:
  There was a bug where it wasnt possible for people to only request something without giving anything or make only money transactions so I fixed this.

• Special Property Popups: I resolved a rendering issue with the popups for special properties, specifically the Airports and Companies (Utilities). Now, when a player interacts with these tiles, the modal correctly pulls the specific data object and displays the unique rent calculations and purchase prices.

Documentation Revamp

• I updated the README.md to ensure the decumentation was on point for anyone cloning the repo.
• I refactored and renamed SHOWCASE.md to Instructions.md. This provides a much clearer, step-by-step guide on how the game operates and how to get the local server running instead of just showcasing the Repo more.

Conclusion

The feedback was essential and really showed me how sloppy I worked it should now work perfectly even though i couldnt test it for 4 people

Update: UI/UX Fixes, Auction System Fixes, and Property Manager Integration FInal

This update focused on providing a better user experience and just making it less buggy

Key Changes

1. UI/UX Improvements

• Refactored layout components to improve visual consistency.
• Updated navigation paths so it makes more sense for the User and is just more logical
• Adjusted CSS rules to improve the game across different screen sizes.

2. Auction System Fixes

• Addressed synchronization issues with the auction state.
• Fixed logical errors in bid processing and countdown timers.

3. Property Manager Integration

• Fixed the Property Manager feature, allowing users to track and organize their properties.
• Integrated the property database schema with the existing auction mechanisms.
• Created user views for listing and editing properties. So just new UI

4. General Bug Fixes & Stability

• Cleaned up left over parts of old system