tiny-llm: getting the data pipeline and model architecture in place

Before writing any model code, I needed a way to actually get training data into a usable format. prepare.py handles that: it streams a dataset from HuggingFace (starting with TinyStories, ~470M tokens), tokenizes it with the GPT-2 BPE tokenizer via tiktoken, and writes it out as flat uint16 arrays split into train.bin and val.bin.

A few details mattered here even though I didn’t write this part myself:

• Tokens are stored as uint16 instead of the more obvious int32/int64 — GPT-2’s vocab is 50,257 tokens, which fits comfortably under 65,536, so uint16 cuts file size in half for free.
• The split between train/val happens per document, not per token. Each story gets an end-of-text token appended, and documents are randomly assigned to train or val as a whole — so a story never gets cut in half across the train/val boundary and leaks information.
• Files are written so they can be read later with np.memmap, meaning the corpus never has to fit in RAM — training just maps the file and reads slices from disk on demand.

With the data pipeline sorted, I moved on to the part I actually wanted to understand deeply: the model itself. That’s model.py, CausalSelfAttention, MLP, Block, and GPT, all written and shape-tested from scratch.

Having both pieces done means the two “boring but necessary” and “the actual point of the project” halves are finally connected: real tokenized text on one end, a transformer architecture ready to consume it on the other.

Next: train.py — wiring prepare.py’s output into model.py via memmap batching, cross-entropy loss, AdamW, and a training loop.