GPT-2 124M Reproduction

Reproduced GPT-2 124M (the old 2019 model) from scratch, trained on ~10B tokens across 2x RTX 5090 in ~7 hours. Final val loss ~3.10.

Dataset

• FineWeb-Edu sample-10BT, ~10B tokens
• GPT-2 BPE (tiktoken), vocab 50257 padded to 50304
• 100 uint16 shards (99 train, 1 val), ~19GB
• Streaming download + multiprocessing tokenizer — switching to imap_unordered with a bigger chunksize took tokenizing from ~2 hours to ~6 minutes

Architecture

GPT-2 124M: embd=768, block_size=1024, weight-tied embeddings (~124M params)

• 12 blocks, each with causal self-attention (12 heads, flash attention) + MLP (Linear → GELU → Linear, 4x)
• Pre-LayerNorm + residuals on both sublayers

Training

• bf16 + torch.compile, AdamW, OneCycleLR cosine with warmup, grad clip 1.0
• 0.5M-token effective batch (524,288), held fixed via gradient accumulation
• DDP across 2x RTX 5090 on RunPod, 19,073 steps (~1 epoch), ~7 hours
• Checkpoint every 200 steps + auto-resume babysit script

Results

• Val loss ~3.10
• Coherent, topical English continuations. It’s a base model, so it continues a prompt rather than answering it .
• Shipped an inference script + Gradio demo, uploaded to Hugging Face

What was actually hard

The architecture was basically the char transformer from last devlog, just wider (124M vs ~10M) with BPE instead of 65 chars. The model was the easy part.

The real work was scale engineering: sharding 10B tokens to stream off disk, gradient accumulation to hit a 0.5M-token batch on cards that OOM at batch 8, multi-GPU gradient sync with DDP, mixed precision, and checkpoint/resume so a 7-hour cloud run survives interruptions. That’s what turns a transformer into a trained LM.