Small update on building a Language Model

So after getting the architecture down and adding collecting some data, I started the training phase. That was one week ago ….
Training is surprisingly (or not ?) actually more than just shoving in more data and expect it to work. Right now I’m struggling with finding sensible hyperparameters and getting the model to output something coherent.

What’s new

I mainly worked on the training loop and minor optimizations to hidden layers to distract myself while Pytorch was frying my APU. Of course there were also bugs in the code that had to fix on the way.

For monitoring the training I decided to use plain TUI built with Rich but I’m currently adding a csv logging for better history tracking.

At first I was planning to run the training on my laptop but I understood why billions dollar companies needed so many datacenters. Anyway, I deployed it on Kaggle which also was harder than it should be.

Next

• Take a break,
• Improve training loop,
• Deploy on Kaggle
• Forget about it