after ~18 hours over two days of benchmarking, 17 models were evaluated on the waifmark 2 benchmark.

(💢💢💢 benchmarking time got recorded to root ~/ and so I can’t include in this project time)

Final findings:
1: higher param models like Qwen3.5 9B and Gemma 4 12B are able to score high despite their low Q3 quants, with only their speed dragging them down
2: Nanbeige 4.2 3B and Granite 4.2 8B at Q4 are not considered to be mainstream open models (i.e. competing with Qwen or Gemma), but scored incredibly well given their small size. The only downside came with unoptimised runs and thinking traces inflating the time/response metric.
3: every AI lab except for Liquid (LFM) saw a direct proportionality trend, that for a higher score a higher time/response is required.
4: As a larger portion of the benchmark is now tied to Agentic tasks in a sandbox, there is a much larger variance in the distribution of scores with more recent models scoring comparatively higher, with less models scoring in-between compared to Waifmark 1.

The Pareto Frontier for Waifmark is:
Qwen3.5 9B (89%) -> LFM2.5 2.6B (55%) -> Ling 3.0 Tiny (35%) ->Qwen3 4B Instruct 2507 (21%).

Now, the issues faced:

• UI issues from the refactor to FastAPI + frontend; fixed with page indexing.
• During benchmarking, many models timed out due to unstable mlx weights, a corrupted test bank during Day 4 with 6 times the number of actual roleplay scenarios caused a benchmark time of over 12 hours; once I realised the mistake at the end (due to ~600+ tasks that had to be reviewed) I reran the benchmark with the correct test bank (which had also been recently changed to 1/3 agentic + 2/3 roleplay, and consists of 4 times more questions than Waifmark 1.
• over 260+ out of ~600 total tasks had to be re-reviewed by a human, as the LLM-as-a-Judge flagged the answer. Most were agentic tasks that the benchmarked LLM did not provide an answer to but had demonstrated steps in its reasoning, with toolcall accuracies above 0%. This took a significant portion of my time to audit all scores.

See images for results, or navigate to:
https://qubixal.github.io/waifmark/chart.html