KyleandClaude Opus 5
43644de268
A pipeline materializes every component before the first stage runs, so peak memory is the sum of all components even though no two are needed at the same moment. The CPU offload flags cannot help with this: they act after loading, and on a unified-memory device moving weights to the host frees nothing because it is the same pool. Add lazy_module_load, off by default. Heavy components become a LazyModule proxy that loads on first use, and the pipeline installs a release hook on the last stage that holds each one. Peak becomes the largest overlapping set instead of the sum. Measured on MiniMax-H3 r16, 121 GiB GB10, 1 GPU, 192x320, 4 steps. Peak CUDA allocated is 57.7 GiB, reached during conditioning where the text encoder and video VAE overlap. The four deferred components account for 96.3 GiB together, which is what stays resident without the flag: text encoder 48.0, DiT 37.8, video VAE 9.7, audio VAE 0.8. Generation completed and wrote a video. Details worth flagging: The release hook lives on PipelineStage.__call__, not in the pipeline stage loop, so pipelines that override forward still free. The proxy forwards __class__, so isinstance stays honest. A proxy answering False to isinstance(module, FSDPModule) would take the wrong branch silently. Self-returning methods hand back the proxy rather than the component. Stages write self.vae = self.vae.to(device) in a dozen places; returning the component there would replace the proxy with a reference the pipeline cannot release, and the run would look normal while freeing nothing. Releasing is a latency cost, never a correctness one: a released component reloads on next access. Training keeps everything resident and warns if the flag is set. If no stage holds a deferred component the pipeline warns rather than silently doing nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.6 KiB
Offloading
This page describes how to use offloading techniques for inference to reduce GPU memory usage while maintaining acceptable performance.
Default Behavior
dit_cpu_offload: bool = True
use_fsdp_inference: bool = False
dit_layerwise_offload: bool = True
text_encoder_cpu_offload: bool = True
image_encoder_cpu_offload: bool = True
vae_cpu_offload: bool = True
pin_cpu_memory: bool = True
lazy_module_load: bool = False
On unified-memory accelerators such as NVIDIA GB10 and Apple silicon, FastVideo
detects the selected device inside each worker and disables all five host-offload
modes before loading modules. Host and accelerator allocations share one physical
pool there, so offload adds transfers and duplicate residency instead of freeing
memory. CUDA FSDP sharding remains enabled when requested; MPS continues to
disable FSDP. pin_cpu_memory is not an offload mode and is left unchanged.
Behavior Explanation
!!! note
For CLI usage, replace underscores (_) with hyphens (-).
use_fsdp_inference
Enables FSDP for inference. The model weights are sharded across multiple GPUs to reduce memory usage per GPU, and weights are broadcast to all GPUs layer by layer during inference.
Performance Impact
FSDP inference introduces negligible performance overhead due to weight prefetching. Performance overhead may be visible when GPU interconnect is slow (e.g., multiple consumer-level GPUs connected by slow PCIe without GPU P2P support).
Usage Recommendation
We recommend enabling this option when multiple GPUs are available.
dit_cpu_offload
Enables CPU offloading for FSDP inference. When enabled, the model weights are offloaded to CPU memory, and the weight of each layer is moved to GPU memory only when that layer is being computed.
Performance Impact
The PyTorch FSDP implementation does not overlap computation and data transfer perfectly for inference, so enabling this option will harm performance.
Usage Recommendation
This option only takes effect when FSDP is enabled. For single GPU usage, we recommend using dit_layerwise_offload instead.
dit_layerwise_offload
This option is similar to dit_cpu_offload, but with two key differences:
- It overlaps computation and PCIe data transfer
- It only works for single GPU inference
Performance Impact
This option introduces negligible performance overhead.
Usage Recommendation
We recommend enabling this option for single GPU usage. This option is not compatible with FSDP.
text_encoder_cpu_offload
When enabled, the text encoder model weights are offloaded to CPU memory, and text encoding is computed on CPU.
Performance Impact
This option significantly slows down text encoding computation, but text encoding is usually not the bottleneck.
Usage Recommendation
We recommend enabling this option only when OOM happens.
image_encoder_cpu_offload and vae_cpu_offload
When enabled, the weights are stored in CPU memory and moved to GPU memory when the corresponding module is being computed. After computation, the weights are moved back to CPU memory.
Performance Impact
These options introduce performance overhead due to PCIe data transfer.
Usage Recommendation
We recommend enabling these options when OOM happens.
lazy_module_load
Every option above moves weights between host and device. This one changes whether they are in memory at all.
By default a pipeline loads every component before the first stage runs, so
peak memory is the sum of all of them even though no two are needed at the same
moment. With lazy_module_load enabled, each heavy component loads on first use
and is freed once the last stage that needs it has returned, so peak memory
becomes the largest overlapping set instead of the sum. For a text-to-video
pipeline that is roughly max(text encoder, DiT + VAE) rather than
text encoder + DiT + VAE.
Performance Impact
A freed component is read from disk again on the next generation, so a multi-prompt run pays one reload per component per request. For a large text encoder that is tens of seconds.
Usage Recommendation
Enable this when a model does not fit at load time, which the CPU offload options above cannot help with because they act after loading. It is particularly relevant on unified-memory devices, where host and device draw on the same pool and moving weights to the host frees nothing. Leave it off when the model already fits.
This option applies to inference only. Training keeps every component resident and logs a warning if the flag is set.
General Recommendations
Single GPU Inference
We recommend enabling dit_layerwise_offload. If OOM happens, also enable image_encoder_cpu_offload and vae_cpu_offload. If OOM still happens, consider enabling text_encoder_cpu_offload.
Multi-GPU Inference
We recommend enabling use_fsdp_inference and disabling both dit_layerwise_offload and dit_cpu_offload. If OOM happens, consider enabling text_encoder_cpu_offload, image_encoder_cpu_offload, and vae_cpu_offload. If OOM still happens, consider enabling dit_cpu_offload.
When the Model Does Not Fit at Load Time
The offload options only help once loading has finished. If the run dies while
components are still being placed, or if the machine has unified memory so
there is no separate host pool to offload into, enable lazy_module_load.
Examples
Single GPU with Layerwise Offloading
from fastvideo import VideoGenerator
generator = VideoGenerator.from_pretrained(
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
num_gpus=1,
# Recommended for single GPU
dit_layerwise_offload=True,
# Enable if OOM happens
vae_cpu_offload=True,
image_encoder_cpu_offload=True,
text_encoder_cpu_offload=True,
# Speeds up CPU-GPU transfer
pin_cpu_memory=True,
)
prompt = "A curious raccoon peers through a vibrant field of yellow sunflowers."
video = generator.generate_video(prompt, output_path="output/", save_video=True)
Multi-GPU with FSDP
from fastvideo import VideoGenerator
generator = VideoGenerator.from_pretrained(
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
num_gpus=2,
# Recommended for multi-GPU
use_fsdp_inference=True,
dit_layerwise_offload=False,
dit_cpu_offload=False,
# Enable if OOM happens
vae_cpu_offload=True,
image_encoder_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=True,
)
prompt = "A majestic lion strides across the golden savanna."
video = generator.generate_video(prompt, output_path="output/", save_video=True)