Im currently working on a big issue where my groq model (through API key) is doing some weird stuff, like mis-routing, not understanding the prompt, and picking the wrong tool at times. Currently have to fix that, or my fallback chain wont work as intended. (groq always has issuses). I also asked ChatGPT what hardware stuff i can build for/around KAIA, and it lwk gave me a lot. Here’s the full breakdown of what happened though: Right now, I’m working through Phase 2B, which is basically focused on making KAIA’s routing system much more reliable.The current plan is:2B.0 — Telemetry:I’m adding tracking to the classifier so KAIA records which classifier actually made the decision, what intent it selected, and its confidence when available. This lets me see why KAIA routed something incorrectly instead of just knowing that it was wrong.I’m also modifying the benchmark system so every test case can save this information individually. That way I can look at a failed case and see something like:Request → “Write a script…”
Expected → coding
Actual → tool_use
Classifier → local_nn
Confidence → 0.91
Provider → …
Latency → …2B.1 — Classifier V2:Once the current benchmark finishes, I’m going to replace the current cheap classifier prompt with a much more structured one. It’ll have explicit definitions for the five main intents:codingtool_usereasoningself_modchatI’m also adding few-shot examples, fine-grained intents, complexity, tool_required, and a confidence score.One of the main things I’m trying to fix is the classifier confusing “write code that does X” with “KAIA should do X.“For example:“Write a script that watches my Downloads folder”
→ codingversus:“Watch my Downloads folder”
→ tool_useThey’re very similar linguistically, but they should go down completely different paths.2B.2 — Cheap-classifier gate:After V2 is working, I’m adding a gate that decides whether KAIA should trust the cheap classifier.There will be different modes, including:OFF
CONFIDENCE
AGREEMENTWith confidence mode, KAIA can basically say:cheap classifier
↓
confidence high?
↙ ↘
YES NO
↓ ↓
accept escalate
↓
stronger modelThe agreement mode goes further by comparing the cheap classifier’s decision against a stronger classifier. If they disagree, KAIA can escalate instead of blindly choosing one.2B.3 — Acceptance + checkpoint:Finally, I’m going to establish actual acceptance criteria for the routing system and generate a Phase 2B report.I’ll measure things like:intent accuracyfine-intent accuracyrouting accuracyclassifier confidencefallback behaviorprovider reliabilitylatencycostThen I’ll create a Phase 2B checkpoint so I have a known-good baseline before moving on.