KAIA - Local AI Agent

I remade the terminal UI! I was researching different UI’s i could make KAIA (yes its named jarvis on my end), and instead of going for a generic, jarvis like UI, I actually took inspiration from an open source coding agent called Opencode, whose UI would look good on jarvis. i am in the middle of re-working it, so this is the first rough draft.

Im currently working on a big issue where my groq model (through API key) is doing some weird stuff, like mis-routing, not understanding the prompt, and picking the wrong tool at times. Currently have to fix that, or my fallback chain wont work as intended. (groq always has issuses). I also asked ChatGPT what hardware stuff i can build for/around KAIA, and it lwk gave me a lot. Here’s the full breakdown of what happened though: Right now, I’m working through Phase 2B, which is basically focused on making KAIA’s routing system much more reliable.The current plan is:2B.0 — Telemetry:I’m adding tracking to the classifier so KAIA records which classifier actually made the decision, what intent it selected, and its confidence when available. This lets me see why KAIA routed something incorrectly instead of just knowing that it was wrong.I’m also modifying the benchmark system so every test case can save this information individually. That way I can look at a failed case and see something like:Request → “Write a script…”
Expected → coding
Actual → tool_use
Classifier → local_nn
Confidence → 0.91
Provider → …
Latency → …2B.1 — Classifier V2:Once the current benchmark finishes, I’m going to replace the current cheap classifier prompt with a much more structured one. It’ll have explicit definitions for the five main intents:codingtool_usereasoningself_modchatI’m also adding few-shot examples, fine-grained intents, complexity, tool_required, and a confidence score.One of the main things I’m trying to fix is the classifier confusing “write code that does X” with “KAIA should do X.“For example:“Write a script that watches my Downloads folder”
→ codingversus:“Watch my Downloads folder”
→ tool_useThey’re very similar linguistically, but they should go down completely different paths.2B.2 — Cheap-classifier gate:After V2 is working, I’m adding a gate that decides whether KAIA should trust the cheap classifier.There will be different modes, including:OFF
CONFIDENCE
AGREEMENTWith confidence mode, KAIA can basically say:cheap classifier
↓
confidence high?
↙ ↘
YES NO
↓ ↓
accept escalate
↓
stronger modelThe agreement mode goes further by comparing the cheap classifier’s decision against a stronger classifier. If they disagree, KAIA can escalate instead of blindly choosing one.2B.3 — Acceptance + checkpoint:Finally, I’m going to establish actual acceptance criteria for the routing system and generate a Phase 2B report.I’ll measure things like:intent accuracyfine-intent accuracyrouting accuracyclassifier confidencefallback behaviorprovider reliabilitylatencycostThen I’ll create a Phase 2B checkpoint so I have a known-good baseline before moving on.

Still working on the intelligence overhaul! Along the way, I’ve also slipped in some CAD/Onshape improvements and email capabilities, which somehow took brain.py from 3,000+ lines to around 4,500 lines 😭.The overhaul is going really well so far. One of the biggest discoveries was that some of JARVIS’s “intelligence” problems weren’t actually intelligence problems at all — I found bugs in my evaluator that were incorrectly marking things as failures even when JARVIS had completed them successfully. I’ve also found some real issues in the routing and memory systems, so I’m fixing those while making JARVIS faster and more reliable. At this point, I’m focusing less on just adding more features and more on making the features I already have actually work together intelligently. The goal is for JARVIS to feel less like a collection of tools and more like an actual agent that can understand what I’m asking, decide what needs to happen, use the right tools, recover when something goes wrong, and verify its own work.

Finished the taxonomy today, took 2 days for that. I re-trained KAIA and now the results are around 93% accuracy for tool calling and 66% for intent classify (which sounds bad but ill be tweaking it throughout the day)

Currently working on the intent classifer for the neural network, and asking claude what a taxonomy in this context means. There are a lot of stuff i need to learn before i continue with this. however, the good thing is i did train the neural network on existing back and forth logs with me an KAIA, and for tool calling, we got around a 98% accuracy, which ill take!

Getting a rough version of the UI set up so I can finally ship this project! For this UI, this is only version one, so don’t mind the bad quality for the web app. Also worked on the svelte design a bit, and to be honest its coming along. but its no where near publishing on the web.

Just finished setting up a simple neural network, and fixed some frontend issues. KAIA now has a webapp version (yet to be published) and a local neural network for small tasks, such as intent routing or when KAIA is offline.

Working on testing the AI Agent’s features, such as having it write code for its very own version where KAIA is a webapp! Currently named jarvis for personal use, however will be changed once project is shipped.

Updating the Agent workflow. I also started 2 different types of web apps, one with React, and the other with Svelte. The svelte one is currently going better, however im not sure if ill actually post the svelte one. Plus, i might leave that off since i need to test KAIA more, there are some bugs that i dont really want in the first version.