feat: add gradient accumulation and offload checkpoint writes to a background worker
- Accumulate gradients across micro-steps: zero grad only at window start, scale loss by acc_steps, and step optimizer/scheduler at window end.
- Move snapshot serialization to a background thread so disk I/O and the “latest” copy no longer stall training; recovery checkpoints stay synchronous for crash-durable resume.
- Clone state to CPU (_to_cpu) before enqueueing to avoid GPU OOM and in-place corruption by later training updates.
- Treat rolling recovery checkpoints as a minimal resume set (LoRA+RNG only); full snapshots still store optimizer/scheduler/scaler state. Resume from a rolling checkpoint rebuilds optimizer momentum and LR schedule from scratch.
- Make CaptionProcessor RNG thread-safe with a lock for concurrent prefetch.
- Add flush()/shutdown() for deterministic test state and clean process exit.
Comments 0
No comments yet. Be the first!
Sign in to join the conversation.