DEVLOG #1: wiki text training

so i finally got the model to train on actual data instead of that one paragraph about singapore nature reserves lol. before this the whole thing was basically a joke, i trained a 7M parameter transformer on like 134 tokens and it just spat out jumbled fragments of the same sentence over and over. cool proof it worked but definitely not a “language model” in any real sense.anyway. the big move this week was swapping out the tiny json file for wikitext-2. that’s a huggingface dataset of wikipedia text, about 2.4 million tokens once you filter out the empty lines. that’s… a lot more than 134 lol. encoding it took maybe half a minute with tiktoken and the whole tensor fits on my gpu no problem (it’s only like 8MB of ints).