DEVLOG #2: real data, real loss, and a lesson about memorizing

split my data into 90% train / 10% val. before i only had one number, training loss, and it lies to you. you can’t tell learning from memorizing with just that. now i check both every 200 steps. rule: both go down = learning. train down but val up = memorizing = useless.also made the model bigger, 7M -> 17.7M params. immediately ran out of gpu memory lol. dropped batch size to 32 to fit my 8GB card.training: 3000 steps. loss started at 158 (should be ~10 for random , my first pass blew up and made garbage). recovered fast though, didn’t ruin the run.

right now the llm is spitting out gibberish and random words but after finetuning it should sound more coherent