The first thing I did was fix an issue where the weights didn’t load for the shakespeare tab. A lot of people commented on my ship saying that the shakespeare tab didn’t work. They were getting an error saying that the weights didn’t load, but I couldn’t reproduce the error. The code for it and the other models is identical, so I couldn’t figure out why the weights weren’t loading for the shakespeare tab but worked for everything else. Defeated, I asked Claude if it could find the error. I am glad I did, because I would have never figured it out by myself. The reason why it wasn’t working was because I called the weights file “gpt.json” which commonly stands for Google Publisher Tag, and is often blocked automatically by ad blockers. So some people have ad blockers that were preventing the weights file from being loaded, resulting in the error! I changed it and hopefully it should work now.
I also tried increasing the model size to just over 1M parameters, around 10x the size of the previous model I made. Unfortunately, my computer kept running out of memory and I had to keep restarting the training. I tried to optimize the memory usage a bit, and reduced the batch size by half, but it still crashed a bunch. I got it down to a loss of around 4.1, but then it crashed my entire computer and I lost the weights :( I don’t think scaling up is a feasible solution, so I will have to try and find a different way to improve.@Definitly-Not-Me suggested that I try overfitting a model on a small dataset, to see if it’s a bug in my code, so I will try that next. I might also look into training on Kaggle or Google Colab, if the overfitting on a small dataset test goes well.