@unknown_lizzard

Devlog #4 — wait, why am I training on Wikipedia?

Last week I got le gros chaton (my from-scratch transformer) to actually train. Ran it on wikitext-103 in a Colab notebook on a free T4, 12k steps, val loss hit 3.73. Felt insane for something I wrote myself from PyTorch primitives.

But then I looked at what it generated. It could do English-ish. It knew about Singapore and stuff. The moment I asked it to write code? Total garbage. Fair enough. It has never seen a single line of code.

the pivot

I thought about what I actually want this thing to do. Answer: a coding agent. Something that mogs on terminal-bench, that you give a task and it runs commands, fixes its own bugs, gets it done. Qwen-coder territory.

Wikitext won’t get me there. You can’t learn Python from Wikipedia articles about the snake.

So the plan changed. The big project is now le fat chaton, the bigger sibling. And it’s a coder.

the architecture stuff

I went down a rabbit hole on what frontier coding models do. They’re basically all mixture-of-experts (MoE) now. Qwen3-coder is 480B params / 35B active. Deepseek-coder-v2-lite is 16B / 2.4B active. The trick:

• big total params (knows a lot)
• small active params per token (runs fast and cheap)

Runs like a small model, knows like a big one. Exactly what I want for a snappy terminal agent.

So I rewrote the model. Added an MoE class. A gate routes each token to its top-2 experts out of 8, with a load-balance loss so the router doesn’t pick the same expert every time. Also threw in:

1. SwiGLU — gated MLP, basically free quality, every Llama uses it
2. GQA — smaller KV cache, faster long-context decoding (the agent harness needs this)
3. Shared expert — DeepSeek-style, one always-on expert for common knowledge

Fat config math: ~10.25B total / ~3.65B active. Not building that locally, it would crash my PC again.

the data question

Wikitext was a fine proof. For the coder I need code. Options:

• smollm-corpus python-edu — 7.6M educational Python files, high quality, but gated behind S3 (slow pull)
• starcoderdata python — ~50GB Python, streams fast from HF, the Codellama workhorse
• the-stack-v2 — 900B tokens, overkill for my budget

Going with starcoderdata Python, blended with ~15-20% cosmopedia (prose) so it has general knowledge. That blend is the Qwen-coder recipe.

am I warm-starting from wikitext?

No. Different architecture entirely (17M dense vs 10B MoE, weights don’t fit). Even if they did, warm-starting a coder from prose wastes compute unlearning prose. Fresh random init on the mixed corpus. Clean slate.

This went from “lets make a little model that talks” to “lets build a coding agent that mogs the big labs” real fast. Ambitious for a solo student with a 2070 and $30 of Modal credits? Absolutely. Doing it anyway? Yeah.

Next: the eval harness so I can measure if this is making the model better at code (pass@1, pass@5 on HumanEval). Val loss going down means nothing if it still can’t write a

DEVLOG #3: teaching it to talk (fine-tuning)

the base model was a completer. ask it “what is 2+2?” and it just continues the sentence. fine-tuning is what makes it actually answer: same model, more training, but on prompt/response pairs.

i wrote ~48 Q&A pairs by hand. trivia, math, jokes, hellos. all in the same rigid Q: …\nA: format. new trick: loss masking. only the response counts toward the loss, not the prompt, so i set prompt targets to -100 (pytorch’s cross_entropy ignores that automatically, zero model changes). tested the mask boundary in isolation first. wrong mask = 400 steps of training the wrong thing.

the run: load base weights, 400 steps, lower lr. loss went 12.3 to 0.02.then the reality check. it nails everything in the set. capitals, math, jokes, “who are you,” verbatim. but something NOT in the set? slop. 17M params + 48 examples is a lookup table, not a reasoner. real chat models are billions of params. not a bug, just the scale.but it’s finally coherent. the first base model was word salad. now you type a question and get a real answer back that makes sense.

it’s tiny and dumb and can’t generalize, but it actually talks. that’s the win.

full pipeline done: tokenizer, model from scratch, pretrain on wikipedia, fine-tune, chat. mine, it talks, i understand every piece.

DEVLOG #2: real data, real loss, and a lesson about memorizing

split my data into 90% train / 10% val. before i only had one number, training loss, and it lies to you. you can’t tell learning from memorizing with just that. now i check both every 200 steps. rule: both go down = learning. train down but val up = memorizing = useless.also made the model bigger, 7M -> 17.7M params. immediately ran out of gpu memory lol. dropped batch size to 32 to fit my 8GB card.training: 3000 steps. loss started at 158 (should be ~10 for random , my first pass blew up and made garbage). recovered fast though, didn’t ruin the run.

right now the llm is spitting out gibberish and random words but after finetuning it should sound more coherent

DEVLOG #1: wiki text training

so i finally got the model to train on actual data instead of that one paragraph about singapore nature reserves lol. before this the whole thing was basically a joke, i trained a 7M parameter transformer on like 134 tokens and it just spat out jumbled fragments of the same sentence over and over. cool proof it worked but definitely not a “language model” in any real sense.anyway. the big move this week was swapping out the tiny json file for wikitext-2. that’s a huggingface dataset of wikipedia text, about 2.4 million tokens once you filter out the empty lines. that’s… a lot more than 134 lol. encoding it took maybe half a minute with tiktoken and the whole tensor fits on my gpu no problem (it’s only like 8MB of ints).

DEVLOG #1: the birth of cortex-cli
this is a similar type of agent harness tui as pi.dev or opencode, I started making this because I was very interested in agentic workflows and interfaces. For the most part it was easy to make but I hit a while designing the backend, I found it really hard calling the agent and proccessing tool calls correctly, so I inspired myself from pi.dev. I think the main feature of the TUI is that the elements are clickable and actually work in the terminal, at first I found it very hard to do and it felt impossible but after a bit of research I figured it out!

furthermore, I decided to chose a very intuitive and minimal UI because other clis felt very hard to get used to.

Devlog 1: Beginning of OrbitScope

Started working on OrbitScope, this is my Slack bot for the Stardance Slack Bot mission. First, I made a command to ping the bot: /orbitscope-ping(as a sanity check).

Since I love space and everything about it, I decided to make a command thats shows you a random pic from space using the APOD nasa API, the api was pretty tricky to get working at first, but after some research I managed to make it work pretty well.
command: /orbitscope-space_pic

While I was coding this I noticed that sending the url of the nasa image wasn’t showing the image in the slack workspace but I found out that I could use: “type: image” which tells the slack bot to send the url as an image.

Furthermore, I started expanding my commands, I made a similar one to the space_pic command, instead of sending a APOD pick from the nasa api I used the mars rover pictures api, so it ended up being pretty easy to code.
command: /orbitscope-mars

DEVLOG 3: minigames!

added a small incremental game called “stardust” this is a very simple game where you mine stardust and upgrade your drill to mine more stardust

have fun!

DEVLOG 2: rocket builder

added a new rocket builder app, now you can build the artemis V rocket!

I was having an issue with each element stacking on top of eachother but I ended up figuring it out quite quickly!

DEVLOG 1: OrbitOS is officialy released!

packaged with top of the line, military grade software, OrbitOS gives you access to Artemis Launch control.

In the beginning, I had a bit of trouble figuring out the css, but it ended up going very well, all the buttons work and you can even check out the source code. I chose a very blocky and minimal UI for this because it matches the mission control feeling very well therefor this was heavily inspired from actual NASA mission control software(complete lie).

DEVLOG 1: telokine is kinda working?? 🤖

So far I have been developing Telokine for Stardance and it allows you to train a robot without writing any code.

I built this because I wanted RL without having to write complex code or/and make complex jupyter notebooks, this is very simple and kid-friendly way to understand how it works visually.

anyways,
This works with your gpu and/or cpu, but expect some bugs as it’s not completely finished yet(expect new features soon!).

(forgot to add it to hackatime so it just shows up as 15 minutes of work)(it took me more like 5-8 hours), I used ai for the frontend and cuda.
Links

GitHub repo

Live demo (UI only)

DEVLOG 1: telokine is kinda working?? 🤖

So far I have been developing Telokine for Stardance and it allows you to train a robot without writing any code.

I built this because I wanted RL without having to write complex code or/and make complex jupyter notebooks, this is very simple and kid-friendly way to understand how it works visually.

anyways,
This works with your gpu and/or cpu, but expect some bugs as it’s not completely finished yet(expect new features soon!).

(forgot to add it to hackatime so it just shows up as 15 minutes of work)(it took me more like 5-8 hours), I used ai for the frontend and cuda.
Links

GitHub repo

Live demo (UI only)