You are browsing as a guest. Sign up (or log in) to start making projects!

5h 0m 5s logged

feat: add gradient accumulation and offload checkpoint writes to a background worker

  • Accumulate gradients across micro-steps: zero grad only at window start, scale loss by acc_steps, and step optimizer/scheduler at window end.
  • Move snapshot serialization to a background thread so disk I/O and the “latest” copy no longer stall training; recovery checkpoints stay synchronous for crash-durable resume.
  • Clone state to CPU (_to_cpu) before enqueueing to avoid GPU OOM and in-place corruption by later training updates.
  • Treat rolling recovery checkpoints as a minimal resume set (LoRA+RNG only); full snapshots still store optimizer/scheduler/scaler state. Resume from a rolling checkpoint rebuilds optimizer momentum and LR schedule from scratch.
  • Make CaptionProcessor RNG thread-safe with a lock for concurrent prefetch.
  • Add flush()/shutdown() for deterministic test state and clean process exit.

Commit

0
22

Comments 0

No comments yet. Be the first!