Alright! So today was very productive:

1. I trained a lot more models for benchmarking based off MSE

2. I increased model size, because with seeding support, I could find the limits of my model size and the most optimal seeds for training and…

3. Added seeding support for benchmarking. Seeding lets me actually make sure that my models are consistent during training, and I can repeat a trial if something goes wrong

4. Added logging; Each training cycle now also creates a csv file with the epoch number, the train score, and the value score (how well it generalized vs how close it is to the training data)

5. I doubled my dataset, just by inverting the current data, and fixed the training script. The training script would cause spikes in the model’s val score, making it worse, and it couldnt recover from those spikes

But… with the now ~6x bigger model and double the data, training on just my cpu is taking forever, but theres not much I can do about that for now :P