Devlog

2026-07-31 — Ported training to actual hardware, kicked off full run

The training pipeline (README, configs/qlora.yaml) was written for a 16 GB
AMD RX 6800 XT (ROCm, gfx1030). This machine has no such card — its only GPUs
are an NVIDIA RTX 5060 Laptop (8 GB, CUDA) and a weak AMD iGPU. The config’s
RDNA2-only choices were actively hurting: fp16-only, 4-bit disabled, and
flash-linear-attention never installed.

Changes:

• Installed CUDA torch (2.13.0+cu130) instead of the ROCm wheel.
• Installed flash-linear-attention — enables the Gated-DeltaNet fast path
  transformers otherwise silently falls back from (up to ~18x slower per the
  README’s own measurement). Added to pyproject.toml.
• configs/qlora.yaml: re-enabled quantization.load_in_4bit and switched
  compute_dtype to bfloat16. Both were disabled specifically for RDNA2’s
  missing bf16-dot instruction, which doesn’t apply on NVIDIA. Drops base
  weights ~8.1 GB → ~2.6 GB.
• 00_check_env.py: was ROCm-only and hard-failed a working CUDA setup
  (/dev/kfd checks, “torch is a ROCm build”). Branched it so AMD-only gates
  no longer block a valid NVIDIA run; VRAM threshold now reads the actual
  load_in_4bit setting instead of a hardcoded 15 GB.
• max_seq_length: 2048 → 1024. Smoke run measured 6.64/7.5 GB peak VRAM
  at 1024 — 2048 would very likely OOM mid-run on this 8 GB card.

Smoke run (20 steps): eval_loss 0.966 → 0.741, eval accuracy 0.756 → 0.793.
Real learning signal, despite the script’s own crude first/last train-loss
check flagging “NOT DECREASING” (too few steps, too noisy to read that way).

Full run started ~16:00, effective batch 16, ~1288 steps, ~27s/step →
~9.5h ETA. Checkpoints every 50 steps to training/out/renma-qlora.

Known non-issue: TRL logs “Mismatch between tokenized prompt and the start of
tokenized prompt+completion” on essentially every example. Expected —
the dataset intentionally ends prompts mid-template (dangling <think>, see
commit e04b564), so the prompt/completion split doesn’t land on a token
boundary. Cosmetic, not a masking bug (eval loss/accuracy both improve
cleanly through it).

Renma Devlog #1 

Dataset built, smoke test passed, full QLoRA run not started. 

Dataset:
12,000 examples (11,760 train / 240 val) from MathNet (18,823 kept of 27,817) + Hendrycks MATH (2,354 kept of 7,500, tiers 1–2 only), 149 dupes removed.
Tier mix: 1—2.5%, 2—8.7%, 3—43.9%, 4—36.1%, 5—8.8%.
Open question: tier 1 is thin (2.5%) — decide before full run whether to upsample or accept the skew. 

LoRA targeting:
Qwen3.5-4B is hybrid — 24/32 layers are Gated-DeltaNet (in_proj_qkv/in_proj_z/out_proj), only 8 are standard attention (q/k/v/o_proj).
Targeting just the usual four names would silently train ¼ of the network with no error.
qlora.yaml now targets both.
Still need to verify empirically via trainable param count after get_peft_model(). 

ROCm/bitsandbytes:
gfx1030 + rocm70 build segfaults with no traceback (no gfx1030 code objects despite support matrix).
Bisected to rocm714 build working — now forced via BNB_ROCM_VERSION=714 in rocm.py.
Also: flash-linear-attention/causal_conv1d are CUDA-only, so Gated-DeltaNet falls back to torch_chunk_gated_delta_rule (correct, 2–3x slower). 

Next:
Run full QLoRA training.