hao-ai-lab-FastVideo/docs/inference/optimizations.md

Optimizations

This page describes the various options for speeding up generation times in FastVideo.

!!! note "On a DGX Spark (GB10)?" Several options on this page behave differently on the GB10's unified-memory hardware — some give little or nothing there. See DGX Spark: Performance & Tuning for what actually helps on that platform and why. Two Sparks, one clip: Pair two NVIDIA DGX Sparks.

Table of Contents

• Optimized Attention Backends

  • Flash Attention
  • Sliding Tile Attention (Archived)
  • Sage Attention
  • Sage Attention 3

• FP8 Weight Quantization

• Adaptive Guidance (CFG gating)

• torch.compile

Attention Backends

Available Backends

• Torch SDPA: FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA
• Flash Attention 2 and 3: FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
• Video Sparse Attention: FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN
• Sage Attention: FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN
• Sage Attention 3: FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN_THREE
• Attn-QAT inference (modified SageAttention3 FP4, sm_120/RTX 5090): FASTVIDEO_ATTENTION_BACKEND=ATTN_QAT_INFER
• Video MoBA Attention: FASTVIDEO_ATTENTION_BACKEND=VMOBA_ATTN
• Sparse Linear Attention: FASTVIDEO_ATTENTION_BACKEND=SLA_ATTN
• SageSLA Attention: FASTVIDEO_ATTENTION_BACKEND=SAGE_SLA_ATTN
• Sliding Tile Attention (archived branch only): FASTVIDEO_ATTENTION_BACKEND=SLIDING_TILE_ATTN

Configuring Backends

There are two ways to configure the attention backend in FastVideo.

1. In Python

In python, set the FASTVIDEO_ATTENTION_BACKEND environment variable before instantiating VideoGenerator like this:

os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "VIDEO_SPARSE_ATTN"

2. In CLI

You can also set the environment variable on the command line:

FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN python example.py

Flash Attention

FLASH_ATTN

We recommend always installing Flash Attention 2:

uv pip install flash-attn==2.7.4.post1 --no-build-isolation

And if using a Hopper+ GPU (ie H100), installing Flash Attention 3 by compiling it from source (takes about 10 minutes for me):

git clone https://github.com/Dao-AILab/flash-attention.git && cd flash-attention

cd hopper
uv pip install ninja
python setup.py install

Flash Attention 4 (opt-in)

FastVideo never auto-selects FlashAttention-4 (flash_attn.cute) just because it is installed: its CuTeDSL kernels JIT-compile per shape family and can fail at runtime on some GPU/shape combinations. To use FA4, install the pinned flash-attn-4 build (see the flash-attn-4 source in pyproject.toml) and set:

UV_TORCH_BACKEND=cu130 uv pip install -e ".[fasth3]"
export FASTVIDEO_FA4=1

On GPUs below sm90 a capability gate routes to FlashAttention-2 the calls FA4 cannot serve there: grad-enabled (training) attention (FA4's backward requires sm90+) and GQA attention (FA4's pack_gqa fails to JIT-compile below sm90). On sm90+ both run on FA4. If FA4 is unusable while FASTVIDEO_FA4=1 is set, FastVideo fails loudly instead of silently falling back.

MiniMax-H3 can additionally use FA4's packed-varlen entry point for its long, single-sequence dense DiT self-attention:

export FASTVIDEO_FA4=1
export FASTVIDEO_MINIMAX_H3_FA4_PACKED_VARLEN=1

This route is inference-only and remains disabled by default. Runtime guards keep masked attention, batch sizes above one, unequal query/key lengths, grad-enabled calls, and NVFP4 on their established paths. It does not apply to the Preview checkpoint's sparse VSA blocks. Packed-varlen changes floating-point reduction order relative to fixed-length FA4, so treat it as a speed/quality evaluation option rather than an exact-parity mode.

FP4 Flash Attention 4 (Blackwell only)

FLASH_ATTN with --nvfp4_fa4

On Blackwell GPUs (B200/B300), you can enable FP4 quantized Q/K attention for up to 1.31x kernel speedup over BF16 FA4, peaking at 2018 TFLOPS. This quantizes Q and K to NVFP4 E2M1 with per-block E4M3 scale factors while keeping V in BF16 or FP8.

See the Attn-QAT paper and flash-attention-fp4 benchmark results for details.

Requirements

• GPU: NVIDIA Blackwell (sm100a or sm103a) — B200, B300, GB200, GB300
• CUDA: 13.0+
• Python: 3.10 or 3.11

Installation

Install the FP4 flash attention kernel (without upgrading your existing torch):

# branch fix/cutlass-dsl-4.5 carries the cutlass-dsl 4.5 fix (cute.core.ThrMma
# -> cute.ThrMma); switch back to @fp4 once hao-ai-lab/flash-attention-fp4#2 merges.
pip install --no-deps "git+ssh://git@github.com/hao-ai-lab/flash-attention-fp4.git@fix/cutlass-dsl-4.5#subdirectory=flash_attn/cute"
pip install "nvidia-cutlass-dsl>=4.5.2" apache-tvm-ffi flashinfer-python

The --no-deps flag prevents upgrading torch/torchvision. Use the supported PyTorch 2.12.0 and CUDA 13 environment for this kernel.

Branch-to-nvidia-cutlass-dsl compatibility (the fork tracks the CuTe DSL API surface closely):

fork branch	cutlass-dsl	notes
fp4	==4.4.2 (+ nvidia-cutlass-dsl-libs-base==4.4.2)	validated set on GB200: quack-kernels==0.4.1, flashinfer-python==0.6.8, CUTE_DSL_ENABLE_TVM_FFI=1, FASTVIDEO_FA4=1
fix/cutlass-dsl-4.5	>=4.5.2	carries the cute.core.ThrMma -> cute.ThrMma fix
any	4.6-era	unsupported: cute.make_fragment was removed at module level; fails at CuTe JIT trace

FASTVIDEO_FA4=1 is required alongside the fork: it ships no compiled FlashAttention-2, so dense attention paths raise ImportError without the FA4 opt-in. The same kernel also serves ATTN_QAT_INFER on sm_100a/sm_103a (datacenter Blackwell) — the selection log's receipt line (ATTN_QAT_INFER resolved: ...) records the arch, kernel, and quantization mode that actually bound.

Usage

Enable FP4 attention via the --nvfp4_fa4 flag:

python examples/inference/optimizations/fp4_attn_wan2_1_1_3b.py --nvfp4_fa4

Or in Python via the engine.attention.nvfp4_fa4 field (resolution sets the env vars):

from fastvideo import VideoGenerator
gen = VideoGenerator.from_pretrained(
    "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    {
        "engine": {
            "attention": {"nvfp4_fa4": True},
            "num_gpus": 1,
            "use_fsdp_inference": False,  # FSDP is incompatible with FP4 pointer path
        },
    },
)
gen.generate(request={"prompt": "A raccoon in sunflowers", "output": {"save_video": True}})

Known Limitations

• engine.use_fsdp_inference: true is incompatible with the FP4 path (FSDP shards invalidate tensor pointers)
• Per-call cosine similarity vs BF16: ~0.99 (slight quantization error accumulates over denoising steps)
• Only supports headdim >= 128

NVFP4 + Attn-QAT (modified SageAttention3, Blackwell sm_120)

ATTN_QAT_INFER with transformer_quant=nvfp4_qat

Runs the DiT fully in 4-bit: NVFP4 linear layers (activations quantized on the fly) plus the modified SageAttention3 FP4 attention backend. This is the inference half of the Quantization-Aware Distillation (QAD) recipe and the path used for the RTX 5090 release.

The attn_qat_infer kernel hard-gates on sm_120 (consumer Blackwell / RTX 5090); on other GPUs the backend logs a notice and falls back to Flash Attention. See the Attn-QAT paper.

Enable both halves — attention via the env var, linear via transformer_quant:

import os
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "ATTN_QAT_INFER"

from fastvideo import VideoGenerator
gen = VideoGenerator.from_config({
    "model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    "engine": {
        "num_gpus": 1,
        "use_fsdp_inference": False,     # FSDP shards invalidate the FP4 tensor pointers
        # Wan-2.1 uses the nvfp4_qat config (NVFP4 is LTX2-specific).
        "quantization": {"transformer_quant": "nvfp4_qat"},
    },
})
gen.generate(request={"prompt": "A raccoon in sunflowers", "output": {"save_video": True}})

Or run the example script:

python examples/inference/optimizations/nvfp4_qat_wan2_1_1_3b.py
python examples/inference/optimizations/nvfp4_qat_wan2_1_1_3b.py --bf16  # baseline

Sliding Tile Attention (Archived)

SLIDING_TILE_ATTN

The full STA integration in fastvideo/ is archived from main and preserved at:

• https://github.com/hao-ai-lab/FastVideo/tree/sta_do_not_delete

We keep STA off main because we believe VSA is strictly better than STA for the actively maintained FastVideo path.

Kernel code in fastvideo-kernel is still retained. For mask search and STA inference workflow, see STA docs.

Video Sparse Attention

VIDEO_SPARSE_ATTN

Video Sparse Attention is provided by fastvideo-kernel. See VSA docs for installation details.

Sage Attention

SAGE_ATTN

To use SageAttention 2.1.1, please compile from source:

git clone https://github.com/thu-ml/SageAttention.git
cd sageattention
python setup.py install  # or uv pip install -e .

Sage Attention 3

SAGE_ATTN_THREE

SageAttention 3 is an advanced attention mechanism that leverages FP4 quantization and Blackwell GPU Tensor Cores for significant performance improvements.

Hardware Requirements

• RTX5090

Installation

Note that Sage Attention 3 requires python>=3.13, torch>=2.8.0, and CUDA 13. If you are using uv and torch==2.8.0, make sure that sentencepiece==0.2.1 in the pyproject.toml file.

To use Sage Attention 3 in FastVideo, follow the README.md in the linked repository to install the package from source.

V-MoBA / SLA / SageSLA

These backends are model-specific and require the corresponding kernels and dependencies. Use the support matrix and model examples to confirm compatibility before enabling them.

FP8 Weight Quantization

transformer_quant="FP8"

Quantizes DiT linear layers (attention projections and FFN) to FP8 e4m3.

On GPUs older than sm89, the FP8 matmul falls back to a bf16 dequant path automatically.

Requirements

• GPU: sm89+ (H100, L40S, RTX 4090, or newer) for hardware FP8 compute
• ROCm: CDNA4 (MI350X / MI355X, gfx950) runs the FP8 _scaled_mm path through hipBLASLt (OCP e4m3fn). MI300X (gfx942) only exposes the fnuz FP8 formats and takes the bf16 dequant fallback like a pre-sm89 GPU.
• No additional packages required beyond the base FastVideo install

Usage

from fastvideo import VideoGenerator

gen = VideoGenerator.from_config({
    "model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    "engine": {"quantization": {"transformer_quant": "FP8"}},  # per-tensor (default)
})
gen.generate(request={"prompt": "A raccoon in sunflowers", "output": {"save_video": True}})

engine.quantization.transformer_quant takes a quantization registry name and builds that config with its default arguments. To pass constructor arguments, such as per-channel granularity, set the config instance on the DiT config through pipeline.model.generic.dit instead:

from fastvideo import VideoGenerator
from fastvideo.layers.quantization import get_quantization_config

gen = VideoGenerator.from_config({
    "model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    "pipeline": {
        "model": {
            "generic": {
                "dit": {"quant_config": get_quantization_config("FP8")(granularity="channel")},  # slower, higher accuracy
            },
        },
    },
})

Or run the example script:

python examples/inference/optimizations/fp8_wan2_1_1_3b.py
python examples/inference/optimizations/fp8_wan2_1_1_3b.py --granularity channel
python examples/inference/optimizations/fp8_wan2_1_1_3b.py --bf16  # baseline

Granularity

Mode	Weight scales	Activation scales	Speed	Accuracy
tensor (default)	per-tensor	per-tensor	faster	lower
channel	per-output-channel	per-token (rowwise)	slower	higher

torch.compile

FastVideo can torch.compile the DiT (transformer) for a substantial end-to-end speedup. It is off by default and enabled per-run.

Enabling

generator = VideoGenerator.from_pretrained(
    "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    {"engine": {"compile": {"enabled": True}}},
)

A complete A/B example (eager vs compiled, warmup excluded) is in examples/inference/optimizations/torch_compile_example.py.

fastvideo generate is config-file driven; to enable torch.compile from the CLI, set the relevant field in your run config and pass it via fastvideo generate --config run.yaml. There is no top-level --enable-torch-compile flag on the subcommand.

Only DiT submodules that declare _compile_conditions are compiled (most shipped models). The text encoder and VAE are not compiled by this flag.

Regional fullgraph compile (experimental)

inference_torch_compile is a stricter, kwargs-free variant that ports the training-side regional compile of #1718 to inference: the loader wraps each _compile_conditions block in torch.compile(fullgraph=True) with inductor options={"emulate_precision_casts": True} right after the transformer loads. The ordinary compile path keeps its historical compiler-disabled attention boundary by default; regional compile opts in only the compatible attention instances owned by this transformer. MiniMax-H3 VSA is supported only by the inference-only sm_100a tile-64 route (FASTVIDEO_VSA_SM100A=1 and VSA_tile_size=64); the loader probes that route before capture and keeps the transformer eager when the kernel or device is unsupported. Legacy VSA, MiniMax-H3 tile-256 VSA, and the explicit FASTVIDEO_DISABLE_ATTENTION_COMPILE=1 escape hatch keep the transformer eager with one warning instead of failing mid-denoise.

generator = VideoGenerator.from_config({
    "model_path": "MiniMaxAI/MiniMax-H3",
    "engine": {"compile": {"regional": True}},  # or FASTVIDEO_INFERENCE_TORCH_COMPILE=1
})

In a YAML config, set generator.engine.compile.regional: true.

Do not combine it with torch_compile_kwargs['mode'] (the loader injects inductor options, and torch.compile forbids mode+options); it is independent of enable_torch_compile, and when both are set the regional compile wins for the DiT.

What to expect from generic compile

The Wan result below measures the existing generic engine.compile.enabled: true path. It is useful evidence that compile can help, but it is not a benchmark or numerical gate for the stricter regional fullgraph path above.

Config	Effect
Wan2.1-T2V-1.3B, A100-80GB, 480×832×81f, 50 steps	end-to-end 259.7s → 198.1s (−23.7%); per-step 4.91 → 3.78 s/it

The speedup is configuration-dependent — it varies with model, resolution, step count, and GPU. Treat the number above as one measured data point, not a guarantee; benchmark your own config (recipe below).

There is a one-time graph-build cost on the first generation (tens of seconds to minutes, model-dependent). It amortizes over subsequent generations with the same input shapes. Always exclude the first (warmup) generation when measuring steady-state latency — measuring the warmup is the most common way to wrongly conclude "compile is slower".

Regional MiniMax-H3 accuracy caveat (job 2660). On one GB200 at 768×1344×124, the native 50-point schedule ran exactly 49 transformer forwards. After one warmup, three fixed-prompt/fixed-seed repeats averaged 185.08s → 157.01s end to end and 174.90s → 147.12s denoising. Each leg was independently pixel-deterministic, but compiled output did not match eager: mean absolute pixel error 20.247/255, PSNR 16.67 dB, mean SSIM 0.7108, and mean MS-SSIM 0.6370 across 124 frames. Treat regional MiniMax-H3 compile as an opt-in performance experiment, not an eager-parity-safe mode.

The same caveat applies to sparse MiniMax-H3 regional compile. Its mask compaction, sm_100a launch, trained compression gates, and inference-only H3 fusions are fullgraph-compatible, but compilation can still change model numerics. The FastH3 all profile enables regional compile by default because it is the fastest measured route at 124, 243, and 345 frames. Use --no-inference-torch-compile when comparing against the eager sparse-DiT route.

Numerics. Inductor's lowering is designed to preserve eager semantics within floating-point tolerance, but per-model equivalence is not asserted by any standing SSIM regression here — the SSIM tests in fastvideo/tests/ssim/ run with enable_torch_compile disabled. If you depend on compile output staying close to eager (or your previous compiled run), run an MS-SSIM gate on your config, especially when combining engine.compile.enabled: true with other numerics-affecting flags (quantized attention backends, FP4, layerwise offload edge cases).

Known interactions

• Layerwise CPU offload (engine.offload.dit_layerwise: true, the default): the offload hook previously caused an implicit graph break once per transformer layer, fragmenting the compiled region. Addressed in hao-ai-lab/FastVideo#1365 — keep that fix to get a clean compiled region under the default offload path.
• mode="reduce-overhead" / CUDA graphs: not yet supported by regional compile because that path injects inductor options, and PyTorch rejects mode together with options. The generic compile path can accept mode, but CUDA-graph compatibility remains backend- and shape-dependent. FA2/FA3 inference and FA4 expose traceable custom-op boundaries. MiniMax-H3's sm_100a tile-64 inference route is the only VSA path in the regional support envelope; other VSA paths and the FA3 grad-enabled path remain outside it. Use the default inductor mode shown above unless your exact configuration has its own gate.

Extra torch.compile options live at engine.compile.backend, fullgraph, mode, and dynamic; any other torch.compile kwargs go in engine.compile.extras. Set them in the nested config, in a config file, or as a CLI dotted override (for example --generator.engine.compile.mode reduce-overhead). Example (currently not recommended — see the CUDA-graphs caveat above):

VideoGenerator.from_pretrained(
    "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    {"engine": {"compile": {"enabled": True, "mode": "reduce-overhead"}}},  # may error today
)

Benchmarking torch.compile

Same discipline as attention backends — same prompt, same seed, same config; discard the first generation (graph build):

import time
from fastvideo import VideoGenerator

gen = VideoGenerator.from_pretrained("your-model-id", {"engine": {"compile": {"enabled": True}}})
req = {"prompt": "Your prompt", "sampling": {"seed": 1024},
       "output": {"save_video": False}}
gen.generate(req)                                            # warmup: graph build, discard
t0 = time.perf_counter()
gen.generate(req)                                            # measured: shapes reused
print(f"compiled steady-state: {time.perf_counter() - t0:.2f}s")

See examples/inference/optimizations/torch_compile_example.py for a baseline-vs-compile A/B with the warmup correctly excluded.

Benchmarking different optimizations

To benchmark backend performance, generate the same prompt with the same seed and compare end-to-end generation times:

import os
import time

for backend in ["TORCH_SDPA", "FLASH_ATTN", "SAGE_ATTN"]:
    os.environ["FASTVIDEO_ATTENTION_BACKEND"] = backend
    generator = VideoGenerator.from_pretrained("your-model-id")
    start_time = time.perf_counter()
    generator.generate({
        "prompt": "Your prompt",
        "sampling": {"seed": 1024},
    })
    elapsed = time.perf_counter() - start_time
    print(f"{backend}: {elapsed:.2f}s")

Note: reinstantiate VideoGenerator after changing FASTVIDEO_ATTENTION_BACKEND.

Adaptive Guidance (CFG gating)

CFG gating accelerates classifier-free guidance by reusing the cached noise_pred_cond - noise_pred_uncond delta after a configurable fraction of the denoising schedule, skipping the unconditional model forward for the remaining steps. The technique is the LinearAG variant of Adaptive Guidance (Castillo et al. 2023, arXiv:2312.12487).

Enabling

Set the FASTVIDEO_CFG_GATE_STEP environment variable to a float in [0, 1]:

Value	Behavior
1.0 (default)	Disabled — legacy two-pass CFG every step.
0.5	Cache the delta after len(timesteps) * 0.5 steps; reuse for the rest.
0.0	Cache from the very first step (most aggressive).

export FASTVIDEO_CFG_GATE_STEP=0.5

Trade-offs

• Memory: one extra model-output-sized tensor per rank held during the gating window.
• Quality: VBench-measured quality is preserved within noise on 4 of 5 dimensions at FASTVIDEO_CFG_GATE_STEP=0.5 for Wan T2V 1.3B per the PR's reported numbers (see #1372).
• Speed: ~22% e2e on 4xL40S and ~24% on 1xH100 at the same settings.

Default behavior is byte-for-byte equivalent to the legacy two-pass CFG path; the feature is fully opt-in.