Added an EOS ID, which is a special token that the model can output to indicate that it’s done. I built a script to insert this in between passages in the training data, so it can learn when it should shut up (otherwise it will yap forever, until it hits the max number of output tokens). This meant I had to retrain the entire model, but got it down to a loss of around 3.46. Obviously, it’s not very good. But I’m pretty sure the only way to improve is to scale up the model size and training time, something I don’t really have the capacity to do (this model trained for 15 hours on my laptop). I will probably continue by adding this model to my demo website, and build a chatGPT like UI, just for fun. After that, I think it might be time to ship!