YACK

Now we’re talking folks

Reached a milestone. After nearly 10 hours of training on OpenwebText, the model can now formulate somewhat grammatically correct sentences.

It’s not over though.

Training was interrupted because apparently kaggle has a time limit for how long a notebook can run (on top on the weekly time limit). Sadly I have no interest in buying cloud compute for that, so I will just look for a richer dataset, reduce the number of epoch and crank up the learning rate.

I’m tired boss

Sorry, stardance but for my own good and sanity, I think I will have to give up or half bake this project. I really tried 😞

Two steps before the throne

Finally deployed the training script to Kaggle. Now all there’s to be is wait and hope.

A quick list of what’s new:

• Divided the training loop in many functions to handle dataloading, parsing, logging and more;
• Managed to find a copy of gutenberg books to serve as dataset;
• Implemented checkpoints in the training so that I will not lose all progress if I run out of compute;
• Dataloading is now done using np.memmap ensuring that only a portion of tokens are loaded in memory;
• Scaled the value of hyprparameters based on configuration of gpt-2 small (my main source is Chatgpt/GLM and a rough guess of number of epochs);
• Fixed a “storage leak” where the best model checkpoints where saved per epoch instead of overwriting previous one (totally my fault)

Next:

• Fix my nonexistant git history while waiting for the training to finish

Small update on building a Language Model

So after getting the architecture down and adding collecting some data, I started the training phase. That was one week ago ….
Training is surprisingly (or not ?) actually more than just shoving in more data and expect it to work. Right now I’m struggling with finding sensible hyperparameters and getting the model to output something coherent.

What’s new

I mainly worked on the training loop and minor optimizations to hidden layers to distract myself while Pytorch was frying my APU. Of course there were also bugs in the code that had to fix on the way.

For monitoring the training I decided to use plain TUI built with Rich but I’m currently adding a csv logging for better history tracking.

At first I was planning to run the training on my laptop but I understood why billions dollar companies needed so many datacenters. Anyway, I deployed it on Kaggle which also was harder than it should be.

Next

• Take a break,
• Improve training loop,
• Deploy on Kaggle
• Forget about it

Devlog 0: Build a Large/Small/Whatever LM

Overview

Hello Stardance! Hello world appetIntern disdainwartsnine.
Those were the first words of my Language Model, am feeling so fogging proud right now. I’m trying to get into ML for the next year (not just because of the hype). For my first projects, I tried building a simple perceptron as well as understanding the algorithm behind Tic-Tac-Toe. Nothing grandiose. Why aim so high for my next project since I’m not that knowledgeable ? Because I’m short of time and there are full tutorial available, that’s all.

Goal & MVP

The MVP goal is to be able to generate coherent text (will probably be gibberish since I’m not a billion dollars corpo) and implement simple RAG to compensate for the lack of intelligence.

I plan to upgrade it as I learn more about LLMs and Transformers.

The Current State

I just got the basic unoptimized architecture down.
It tooks so much time to understand the Transformer architecture and all the maths that goes behind it (thanks to 3Blue1Brown it was manageable). Now I need to train & evaluate the “Advanced Autocomplete Machine” so that It can start acting more like GPT2.

Picture:

Liking my rice ? 😏