Devlog #4 — wait, why am I training on Wikipedia?

Last week I got le gros chaton (my from-scratch transformer) to actually train. Ran it on wikitext-103 in a Colab notebook on a free T4, 12k steps, val loss hit 3.73. Felt insane for something I wrote myself from PyTorch primitives.

But then I looked at what it generated. It could do English-ish. It knew about Singapore and stuff. The moment I asked it to write code? Total garbage. Fair enough. It has never seen a single line of code.

the pivot

I thought about what I actually want this thing to do. Answer: a coding agent. Something that mogs on terminal-bench, that you give a task and it runs commands, fixes its own bugs, gets it done. Qwen-coder territory.

Wikitext won’t get me there. You can’t learn Python from Wikipedia articles about the snake.

So the plan changed. The big project is now le fat chaton, the bigger sibling. And it’s a coder.

the architecture stuff

I went down a rabbit hole on what frontier coding models do. They’re basically all mixture-of-experts (MoE) now. Qwen3-coder is 480B params / 35B active. Deepseek-coder-v2-lite is 16B / 2.4B active. The trick:

• big total params (knows a lot)
• small active params per token (runs fast and cheap)

Runs like a small model, knows like a big one. Exactly what I want for a snappy terminal agent.

So I rewrote the model. Added an MoE class. A gate routes each token to its top-2 experts out of 8, with a load-balance loss so the router doesn’t pick the same expert every time. Also threw in:

1. SwiGLU — gated MLP, basically free quality, every Llama uses it
2. GQA — smaller KV cache, faster long-context decoding (the agent harness needs this)
3. Shared expert — DeepSeek-style, one always-on expert for common knowledge

Fat config math: ~10.25B total / ~3.65B active. Not building that locally, it would crash my PC again.

the data question

Wikitext was a fine proof. For the coder I need code. Options:

• smollm-corpus python-edu — 7.6M educational Python files, high quality, but gated behind S3 (slow pull)
• starcoderdata python — ~50GB Python, streams fast from HF, the Codellama workhorse
• the-stack-v2 — 900B tokens, overkill for my budget

Going with starcoderdata Python, blended with ~15-20% cosmopedia (prose) so it has general knowledge. That blend is the Qwen-coder recipe.

am I warm-starting from wikitext?

No. Different architecture entirely (17M dense vs 10B MoE, weights don’t fit). Even if they did, warm-starting a coder from prose wastes compute unlearning prose. Fresh random init on the mixed corpus. Clean slate.

This went from “lets make a little model that talks” to “lets build a coding agent that mogs the big labs” real fast. Ambitious for a solo student with a 2070 and $30 of Modal credits? Absolutely. Doing it anyway? Yeah.

Next: the eval harness so I can measure if this is making the model better at code (pass@1, pass@5 on HumanEval). Val loss going down means nothing if it still can’t write a