Now we’re talking folks

Reached a milestone. After nearly 10 hours of training on OpenwebText, the model can now formulate somewhat grammatically correct sentences.

It’s not over though.

Training was interrupted because apparently kaggle has a time limit for how long a notebook can run (on top on the weekly time limit). Sadly I have no interest in buying cloud compute for that, so I will just look for a richer dataset, reduce the number of epoch and crank up the learning rate.