waifmark-2

As requested, README has been updated and rewritten by a human.
There are also now three ways to install - directly through PyPI by doing pip install waifmark, or downloading waifmark-macos.zip and unzipping.

there’s not a lot otherwise to say about this devlog

it was found that I accidentally committed using personal details and a local email.

The gh repo has been wiped and recreated; here’s proof of the original repo and work.
Thank you reviewers for your understanding.

Waifmark 2 has been updated with a new UI!

After criticisms of the previous UI being too AI, I’ve refactored it with a cleaner, pastel aesthetic.

I kinda like it!
tell me what i should change next -»>

(readme has been humanified too…)

after ~18 hours over two days of benchmarking, 17 models were evaluated on the waifmark 2 benchmark.

(💢💢💢 benchmarking time got recorded to root ~/ and so I can’t include in this project time)

Final findings:
1: higher param models like Qwen3.5 9B and Gemma 4 12B are able to score high despite their low Q3 quants, with only their speed dragging them down
2: Nanbeige 4.2 3B and Granite 4.2 8B at Q4 are not considered to be mainstream open models (i.e. competing with Qwen or Gemma), but scored incredibly well given their small size. The only downside came with unoptimised runs and thinking traces inflating the time/response metric.
3: every AI lab except for Liquid (LFM) saw a direct proportionality trend, that for a higher score a higher time/response is required.
4: As a larger portion of the benchmark is now tied to Agentic tasks in a sandbox, there is a much larger variance in the distribution of scores with more recent models scoring comparatively higher, with less models scoring in-between compared to Waifmark 1.

The Pareto Frontier for Waifmark is:
Qwen3.5 9B (89%) -> LFM2.5 2.6B (55%) -> Ling 3.0 Tiny (35%) ->Qwen3 4B Instruct 2507 (21%).

Now, the issues faced:

• UI issues from the refactor to FastAPI + frontend; fixed with page indexing.
• During benchmarking, many models timed out due to unstable mlx weights, a corrupted test bank during Day 4 with 6 times the number of actual roleplay scenarios caused a benchmark time of over 12 hours; once I realised the mistake at the end (due to ~600+ tasks that had to be reviewed) I reran the benchmark with the correct test bank (which had also been recently changed to 1/3 agentic + 2/3 roleplay, and consists of 4 times more questions than Waifmark 1.
• over 260+ out of ~600 total tasks had to be re-reviewed by a human, as the LLM-as-a-Judge flagged the answer. Most were agentic tasks that the benchmarked LLM did not provide an answer to but had demonstrated steps in its reasoning, with toolcall accuracies above 0%. This took a significant portion of my time to audit all scores.

See images for results, or navigate to:
https://qubixal.github.io/waifmark/chart.html

finally, benchmarking has begun.
The benchmark was re-evaluated meticulously based on Qwen3.5-4B as a ~50% guideline for benchmarking.
Then, I set up my now dead macbook pro (dead display, dead battery) to a portable monitor to use as basically a cloud server and started benchmarking. This is the fourth night of trying (after benchmarking failed over the night due to the computer not plugging in and losing all charge (day 1), automation logic not working (day 2), LFM weights getting stuck (day 3) so hopefully it works today with the 17 consecutive models to be benchmarked.

using 17 models, including:
MiniCPM5-1B (0.57GB)
LFM2.5-2.6B GGUF (1.59GB)
Granite3B (2.09G8)
G9V3 (1.77GB)
Nanbeige (2.50GB)
Granite8B (4.98GB)
Ling(4.58GB)
Falcon (4.28GB)
LFM8B (4.51GB)
Gemma E2B(4.04GB)
Gemma E4B (4.79GB)
Gemma12B (5.30GB)
Qwen3.5-9B (4.35GB)
Qwen3-4B-2507 (2.1GB)
Qwen3.5-2B (1.60GB)
Qwen3.5-4B (2.83GB)
DeepSeek-R1-Qwen3-8B (4.29GB)
Note, all models were selected to fit within 6-8GB vram total so it could be evaluated on my local hardware.

devlog 4 - UI refactor
streamlit was replaced with FastAPI + a frontend and i think it looks much better than previously now.

issues + features:

• hugging face’s api (i.e. HfApi.list_models() was not filtered correctly during search)
• direct model import based on local dir is now possible
• openrouter judge was outdated and replaced, now user-configurable
• install now possible in a oneliner pip install -r requirements.txt && waifmark
• if a benchmark was cancelled, previously a bg server process vLLM workers would leak; instead process groups (start_new_session=True) and os.killpg gets rid of all processes when a session is ended.
• streamlit ran CORS and XSRF disabled; rebound to localhost

devlog 3:
it’s been even longer but I’ve finally done enough to call this a “devlog”.

basically everything has been fully rewritten in python. A few (quite detrimental) bug fixes are:

• nonexistent error recovery system (i.e. pausing benchmark + invalid responses)
• scoring rubric weights ignored in prompt sent to judge
• save files mislabelling
• A/B scoring only having one slider for better model
• several UI patches (floating navbar -> broken margins)
• added leaderboard from the original index.html currently visible on the github under “Leaderboard”, displaying top scores.

To work on:

• largely inconsistent and seemingly inflated scoring
• check if benchmark can actually be completed, finalise benchmarking criteria + process
• benchmark the rest of the models
  i hope this can be done within 2 more devlogs.

Image 1:
much more simplified run system (compared to before).
Image 2:
the og graph, now in v2!

Dev2

It’s been so long but the UI looks a little better now (?) The process has been basically fully streamlined:

1. HF model download
   -> pick quant/download type
2. Benchmarking
   -> start server -> run benchmark, score/100 evaluation, ETA and live console monitoring
3. Result auditing (Will continue to work on this)
   -> human review responses flagged by LLM judge
4. Results showcase/leaderboard
   -> WIP

Progress related to benchmarking:

• 3x question count compared to Waifmark 1
• ~2x faster parallel automated scoring
• memory system (short/long ctx), agentic toolcalling (basic file reading/shell/research) functions fully implemented as part of local agentic benchmarking.

Devlog 0.1

(For a future project.)
I’m trying out LLM benchmarking on my M1 Macbook Pro that probably needs a break!

• Prior to Stardance -
  Waifmark 1 was a benchmark testing local agentic capabilities and speech persona of small locally hosted (V)LLMs.
  However, my benchmarking process for Waifmark 1 was unstandardised and troublesome, and I kept all the data in excel out of all places.

• Current stage -
  Waifmark 2 is in the works. The benchmark is now evaluated by an LLM-as-a-Judge that can flag and pass low-confidence outputs for human review (as is industry standard).

Using the wonderful streamlit library I built a basic app that can download you a model from hf, serve the model and benchmark it in 3 steps. Unfortunately I cannot show any more behind the process as of now, but I’m very excited to join Stardance and to see what changes can be observed moving from Waifmark 1 -> 2!