Two steps before the throne

Finally deployed the training script to Kaggle. Now all there’s to be is wait and hope.

A quick list of what’s new:

• Divided the training loop in many functions to handle dataloading, parsing, logging and more;
• Managed to find a copy of gutenberg books to serve as dataset;
• Implemented checkpoints in the training so that I will not lose all progress if I run out of compute;
• Dataloading is now done using np.memmap ensuring that only a portion of tokens are loaded in memory;
• Scaled the value of hyprparameters based on configuration of gpt-2 small (my main source is Chatgpt/GLM and a rough guess of number of epochs);
• Fixed a “storage leak” where the best model checkpoints where saved per epoch instead of overwriting previous one (totally my fault)

Next:

• Fix my nonexistant git history while waiting for the training to finish