I made a bunch of tiny improvements. I didn’t change the architecture at all, but I did change the optimizer to AdamW. The optimizer is what controls how much each parameter is changed each epoch. Before, I was using a set learning rate, of about .1 for the first 80% of training then decreasing to .01 for the last 20%. Decreasing it at the end is important because as the loss settles into a local minimum, it might not be able to go to the lowest point because it keeps overshooting it since the lr is too high. Conversely, we can’t keep the lr too small the whole time because it might settle into a very tiny local minima that it might have been able to go over with a higher learning rate. It would also take forever. AdamW helps improve the optimizing step by changing the learning rate per parameter, helping the model train as fast as possible without overfitting. Overfitting isn’t really an issue I’m worrying about since my model is pretty small (around 56k params), so it’s not really large enough to learn the ins and outs of my dataset. Anyway I also changed the logging step, so it now logs validation dataset loss in addition to the training dataset loss. I split the dataset into train and validation so I can make sure it’s not overfitting. Currently the val loss is at 1.76, which is a lot better than where we were at before, which makes me hopeful. Training has around another 7,000 epochs to go, but Stardance is yelling at me I should write another devlog now so here you go, Stardance. I’m looking forward to sampling the model and seeing how much it’s improved!