Switched to using the fineweb edu dataset, instead of shakespear. This is the dataset I will use to train my actual model on. It’s a highly currated database of the internet, and is similar to what GPT2 was trained on (although the actual db they trained it on is closed source). I also reformated the sampling system so you can put in a query and get a response, as long as the query takes up less tokens than the context. Other than swapping out the training file, it’s the exact same. Next, I will probably implement a special ending token to my tokenizer, and also increase the scale of my model a little bit. I can’t increase it all the way to what Karpathy does in his tutorial (he uses cloud computing, so he paid to rent out like 8 64GB GPUs, or something like that), since I don’t want to spend money training it. However I think I can squeeze a little more out of my laptop.