Devlog

2026-07-31 — Ported training to actual hardware, kicked off full run

The training pipeline (README, configs/qlora.yaml) was written for a 16 GB
AMD RX 6800 XT (ROCm, gfx1030). This machine has no such card — its only GPUs
are an NVIDIA RTX 5060 Laptop (8 GB, CUDA) and a weak AMD iGPU. The config’s
RDNA2-only choices were actively hurting: fp16-only, 4-bit disabled, and
flash-linear-attention never installed.

Changes:

• Installed CUDA torch (2.13.0+cu130) instead of the ROCm wheel.
• Installed flash-linear-attention — enables the Gated-DeltaNet fast path
  transformers otherwise silently falls back from (up to ~18x slower per the
  README’s own measurement). Added to pyproject.toml.
• configs/qlora.yaml: re-enabled quantization.load_in_4bit and switched
  compute_dtype to bfloat16. Both were disabled specifically for RDNA2’s
  missing bf16-dot instruction, which doesn’t apply on NVIDIA. Drops base
  weights ~8.1 GB → ~2.6 GB.
• 00_check_env.py: was ROCm-only and hard-failed a working CUDA setup
  (/dev/kfd checks, “torch is a ROCm build”). Branched it so AMD-only gates
  no longer block a valid NVIDIA run; VRAM threshold now reads the actual
  load_in_4bit setting instead of a hardcoded 15 GB.
• max_seq_length: 2048 → 1024. Smoke run measured 6.64/7.5 GB peak VRAM
  at 1024 — 2048 would very likely OOM mid-run on this 8 GB card.

Smoke run (20 steps): eval_loss 0.966 → 0.741, eval accuracy 0.756 → 0.793.
Real learning signal, despite the script’s own crude first/last train-loss
check flagging “NOT DECREASING” (too few steps, too noisy to read that way).

Full run started ~16:00, effective batch 16, ~1288 steps, ~27s/step →
~9.5h ETA. Checkpoints every 50 steps to training/out/renma-qlora.

Known non-issue: TRL logs “Mismatch between tokenized prompt and the start of
tokenized prompt+completion” on essentially every example. Expected —
the dataset intentionally ends prompts mid-template (dangling <think>, see
commit e04b564), so the prompt/completion split doesn’t land on a token
boundary. Cosmetic, not a masking bug (eval loss/accuracy both improve
cleanly through it).