alexzms
57a2fa9ec1
Make compile + FA4-FP4 attention work and add a parametrized speed harness.
- flash_attn.py: wrap `_nvfp4_quantize_for_fa4` as a torch.library.custom_op
+ register_fake so torch.compile/dynamo treats the flashinfer FP4 Q/K quant
as an opaque leaf (can't trace its functools.cache lock / JIT subprocess);
pre-build the FP4 quant module in FlashAttentionImpl.__init__ to warm the
cache before compile. Required for compile + nvfp4_fa4 to run at all.
- bench_ltx2_3_distilled_t2v.py: parametrized t2v harness (compile / nvfp4 /
fa4 / cudagraph mode / resolution / frames / steps via env) with per-stage
timing.
- basic_ltx2_3_distilled_i2v_{uncompiled,compiled}{,_nvfp4}.py: i2v variants.
- INSTRUCTION_ltx23_nvfp4.md: full environment + command-line reproduction.
Result (t2v 832x1280, 8+3, GB300): nvfp4+FA4+cudagraphs = 3.00s e2e, the
fastest config (vs bf16+FA4+cudagraphs 3.81s, nvfp4+FA4 default 6.35s). The
NVFP4 denoise penalty is per-step launch-bound; CUDA graphs removes it.
Basic Video Generation Tutorial
The VideoGenerator class provides the primary Python interface for doing offline video generation, which is interacting with a diffusion pipeline without using a separate inference api server.
Requirements
- At least a single NVIDIA GPU with CUDA 12.4.
- Python 3.10-3.12
Installation
If you have not installed FastVideo, please following these instructions first.
Usage
The first script in this example shows the most basic usage of FastVideo. If you are new to Python and FastVideo, you should start here.
# if you have not cloned the directory:
git clone https://github.com/hao-ai-lab/FastVideo.git && cd FastVideo
python examples/inference/basic/basic.py
For an example on Apple silicon:
python examples/inference/basic/basic_mps.py
For an example running DMD+VSA inference:
python examples/inference/basic/basic_dmd.py
For the typed config/request path added during the inference API refactor:
python examples/inference/basic/basic_dmd_new_api.py
Basic Walkthrough
All you need to generate videos using multi-gpus from state-of-the-art diffusion pipelines is the following few lines!
from fastvideo import VideoGenerator
def main():
generator = VideoGenerator.from_pretrained(
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
num_gpus=1,
)
prompt = ("A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes "
"wide with interest. The playful yet serene atmosphere is complemented by soft "
"natural light filtering through the petals. Mid-shot, warm and cheerful tones.")
video = generator.generate_video(prompt)
if __name__ == "__main__":
main()