A pipeline materializes every component before the first stage runs, so peak
memory is the sum of all components even though no two are needed at the same
moment. The CPU offload flags cannot help with this: they act after loading,
and on a unified-memory device moving weights to the host frees nothing because
it is the same pool.
Add lazy_module_load, off by default. Heavy components become a LazyModule
proxy that loads on first use, and the pipeline installs a release hook on the
last stage that holds each one. Peak becomes the largest overlapping set
instead of the sum.
Measured on MiniMax-H3 r16, 121 GiB GB10, 1 GPU, 192x320, 4 steps. Peak CUDA
allocated is 57.7 GiB, reached during conditioning where the text encoder and
video VAE overlap. The four deferred components account for 96.3 GiB together,
which is what stays resident without the flag: text encoder 48.0, DiT 37.8,
video VAE 9.7, audio VAE 0.8. Generation completed and wrote a video.
Details worth flagging:
The release hook lives on PipelineStage.__call__, not in the pipeline stage
loop, so pipelines that override forward still free.
The proxy forwards __class__, so isinstance stays honest. A proxy answering
False to isinstance(module, FSDPModule) would take the wrong branch silently.
Self-returning methods hand back the proxy rather than the component. Stages
write self.vae = self.vae.to(device) in a dozen places; returning the component
there would replace the proxy with a reference the pipeline cannot release, and
the run would look normal while freeing nothing.
Releasing is a latency cost, never a correctness one: a released component
reloads on next access. Training keeps everything resident and warns if the
flag is set. If no stage holds a deferred component the pipeline warns rather
than silently doing nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
