NIMBL - Devlog #4:

This devlog is about the one thing NIMBL was built for: proving it saves tokens without making answers worse. I built a real benchmark, ran it against opencode, and got cooked.

The Tier B benchmark

I generated a ca 150-file TypeScript “storefront” app with planted bugs, decoy files, and hidden golden test suites. 25 tasks spread across retrieval, bug-fixing, multi-file work, test-writing, shell loops, delegation, and long-horizon fixes.

The key part: every bug is only caught by hidden tests, so an agent can’t cheat by reading the grader. Ran it live on the same model for both harnesses: deepseek-v4-flash, 3 samples per task, 75 runs per side.

And critically — every single result is saved raw, not summarized. Full event streams, tool calls, token splits, everything, so nothing can be hidden.

The results

solved mean tokens NIMBL (hybrid) 68/75 28,004 opencode 75/75 23,352

opencode solved more and used fewer tokens. That stings, but it’s the truth, and the raw data is committed.

Where NIMBL genuinely won: retrieval tasks (+63% fewer tokens) and latency (26s vs 49s per task). Pure lookup-and-answer is where the context index shines.

What went wrong

Shell mismatch (the big one). NIMBL runs commands through PowerShell but never told the model that. So the model wrote POSIX (find | sort, 2>&1, head) and 60 of 113 shell calls failed. opencode tells the model it’s PowerShell, and the same model wrote working commands. The fix is a one-line hint in the tool description.
Analysis paralysis. One long-horizon task spent 78 tool steps reading files, never ran a single test, never edited anything. All 3 samples failed while opencode fixed it every time.
Delegation overhead. Subagent “research” cost 7k-27k tokens per task with no payoff. opencode solved the same tasks inline for a fraction of that.
Hallucinated answers. One retrieval task answered “42” with zero tool calls, never reading the file that had the answer.

Where it stands

NIMBL has permissions, sessions, providers, budgets, context selection, structural extraction, tokenizers, and a real benchmark harness that saves raw output for every run. It’s genuinely usable for simple coding.

But the benchmark says what the benchmark says: the token win is retrieval-only for now. The next devlog should be about the shell fix, the delegate budget, and seeing if the numbers move.