NIMBL — Devlog #5:

This Devlog ran another benchmark: 25 tasks × 3 samples, hidden golden tests only, and the same model on both harnesses (deepseek-v4-flash). I also used these results to fix the mistakes I found and run the benchmark again, which is running rn.
NImbl is ca 40% more token efficent half the price and caches better.

The Results

NIMBL: solved 68/75 tasks, with total tokens of 2,386,450, wall time of 37.7 min (2,264 s) and throughput 1,054 tk/s.

opencode: solved 75/75 tasks, with total tokens of 2,436,642, wall time of 71.2 min (4,270 s), throughput 571 tk/s.

NIMBL finished 1.9× faster and actually used fewer total tokens than opencode. The tradeoff is that it didn’t finish 7 tasks, and those failed runs still spent real tokens, so this wasn’t a cheap excuse to give up early.

On the 4-mode run NIMBL also used around 40% fewer tokens per solved task (26.2–28.0k vs opencode’s 46.8k, −40% to −44% in every mode). That shows the savings were never only coming from retrieval tasks.

By Category results

bug-fix tasks: −7.2%
retrieval tasks: −51.1%
multi-file tasks: −16.3%
running agent delegation: +130.2%
long-hunt: −35.3%
shell-loop: +6.3%
test-write: +116.9%

What It Says

The results are pretty decent, and they are getting better. Retrieval now uses half the tokens at a third of the latency: 7 seconds compared to opencode’s 19 seconds.

NIMBL is faster in every category, even delegation, where opencode takes 116 seconds per task. The architecture is doing great, and reference cost lands at about half of opencode’s (~$0.26–0.28 per solved task vs ~$0.51).

The losses are concentrated in two specific places. Delegation costs 130% more tokens — 373,493 compared to 162,256 — and test-write costs 117% more and it did all this and still failed so it practically wasted tokens

The reason for that is simple: we had a hardcoded limit of 8 tool calls, which wasn’t enough for these long tasks, opencode ran ~100. So NIMBL kept reading files and crashed when trying to run an edit, because it ran out of tool calls. The fix was basic: instead of hardcoding every tool call limit universally, adjust them according to the task.

Running Now

A new benchmark is running rn trying to use these fixes to finally close that gap of tool calls and get fully functional harness which is cheaper and better after local benchmarks work I will run it in a real benchmark like swe lite or sth