hao-ai-lab-FastVideo/docs/inference/optimizations.md

Optimizations

This page describes the various options for speeding up generation times in FastVideo.

Table of Contents

• Optimized Attention Backends

  • Flash Attention
  • Sliding Tile Attention (Archived)
  • Sage Attention
  • Sage Attention 3

Attention Backends

Available Backends

• Torch SDPA: FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA
• Flash Attention 2 and 3: FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
• Video Sparse Attention: FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN
• Sage Attention: FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN
• Sage Attention 3: FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN_THREE
• Attn QAT Infer: FASTVIDEO_ATTENTION_BACKEND=ATTN_QAT_INFER
• Attn QAT Train: FASTVIDEO_ATTENTION_BACKEND=ATTN_QAT_TRAIN
• Video MoBA Attention: FASTVIDEO_ATTENTION_BACKEND=VMOBA_ATTN
• Sparse Linear Attention: FASTVIDEO_ATTENTION_BACKEND=SLA_ATTN
• SageSLA Attention: FASTVIDEO_ATTENTION_BACKEND=SAGE_SLA_ATTN
• Sliding Tile Attention (archived branch only): FASTVIDEO_ATTENTION_BACKEND=SLIDING_TILE_ATTN

Configuring Backends

There are two ways to configure the attention backend in FastVideo.

1. In Python

In python, set the FASTVIDEO_ATTENTION_BACKEND environment variable before instantiating VideoGenerator like this:

os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "VIDEO_SPARSE_ATTN"

2. In CLI

You can also set the environment variable on the command line:

FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN python example.py

Flash Attention

FLASH_ATTN

We recommend always installing Flash Attention 2:

pip install flash-attn==2.7.4.post1 --no-build-isolation

And if using a Hopper+ GPU (ie H100), installing Flash Attention 3 by compiling it from source (takes about 10 minutes for me):

git clone https://github.com/Dao-AILab/flash-attention.git && cd flash-attention

cd hopper
pip install ninja
python setup.py install

Sliding Tile Attention (Archived)

SLIDING_TILE_ATTN

The full STA integration in fastvideo/ is archived from main and preserved at:

• https://github.com/hao-ai-lab/FastVideo/tree/sta_do_not_delete

We keep STA off main because we believe VSA is strictly better than STA for the actively maintained FastVideo path.

Kernel code in fastvideo-kernel is still retained. For mask search and STA inference workflow, see STA docs.

Video Sparse Attention

VIDEO_SPARSE_ATTN

Video Sparse Attention is provided by fastvideo-kernel. See VSA docs for installation details.

Sage Attention

SAGE_ATTN

To use SageAttention 2.1.1, please compile from source:

git clone https://github.com/thu-ml/SageAttention.git
cd sageattention
python setup.py install  # or pip install -e .

Sage Attention 3

FastVideo now exposes two SageAttention3-compatible backends with distinct environment variable values:

• SAGE_ATTN_THREE: the regular upstream SageAttention3 backend imported from the sageattn3 package.
• ATTN_QAT_INFER: the inference CUDA-kernel backend imported from the in-repo attn_qat_infer package.

SAGE_ATTN_THREE

SageAttention 3 is an advanced attention mechanism that leverages FP4 quantization and Blackwell GPU Tensor Cores for significant performance improvements.

Hardware Requirements

• RTX5090

Installation

Note that Sage Attention 3 requires python>=3.13, torch>=2.8.0, CUDA >=12.8. If you are using uv and using torch==2.8.0 make sure that sentencepiece==0.2.1 in the pyproject.toml file.

To use Sage Attention 3 in FastVideo, follow the README.md in the linked repository to install the package from source.

Attn QAT Infer

ATTN_QAT_INFER

This backend uses the attn_qat_infer implementation that lives in the fastvideo-kernel repository alongside the fastvideo_kernel Triton kernels. Use this backend when you want to run the dedicated FP4 inference CUDA kernel directly during inference.

For the full Attention QAT guide, including Wan 2.1 14B checkpoint download, example editing steps, training launchers, and troubleshooting, see Attention QAT.

This backend currently assumes access to the in-repo fastvideo-kernel checkout or an equivalent editable/source install that exposes:

• attn_qat_infer

Example:

os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "ATTN_QAT_INFER"

QAT Attention

ATTN_QAT_TRAIN

This backend uses the FastVideoKernel Triton attention implementation from fastvideo_kernel.triton_kernels.attn_qat_train. Use it when you specifically want the training-oriented Triton attention path rather than the attn_qat_infer CUDA kernel path.

The dedicated Attention QAT page covers when to use ATTN_QAT_TRAIN versus ATTN_QAT_INFER, the ready-made training launchers, and the end-to-end Wan 2.1 14B inference workflow.

This backend currently assumes access to an install that exposes:

• fastvideo_kernel

Example:

os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "ATTN_QAT_TRAIN"

V-MoBA / SLA / SageSLA

These backends are model-specific and require the corresponding kernels and dependencies. Use the support matrix and model examples to confirm compatibility before enabling them.

Benchmarking different optimizations

To benchmark backend performance, generate the same prompt with the same seed and compare end-to-end generation times:

import os
import time

for backend in ["TORCH_SDPA", "FLASH_ATTN", "SAGE_ATTN"]:
    os.environ["FASTVIDEO_ATTENTION_BACKEND"] = backend
    generator = VideoGenerator.from_pretrained("your-model-id")
    start_time = time.perf_counter()
    generator.generate_video(
        prompt="Your prompt",
        seed=1024,
    )
    elapsed = time.perf_counter() - start_time
    print(f"{backend}: {elapsed:.2f}s")

Note: reinstantiate VideoGenerator after changing FASTVIDEO_ATTENTION_BACKEND.