hao-ai-lab-FastVideo/examples/inference/basic/README.md

Basic Video Generation Tutorial

The VideoGenerator class provides the primary Python interface for doing offline video generation, which is interacting with a diffusion pipeline without using a separate inference api server.

Requirements

• At least a single NVIDIA GPU with CUDA 12.4.
• Python 3.10-3.12

Installation

If you have not installed FastVideo, please following these instructions first.

Usage

The first script in this example shows the most basic usage of FastVideo. If you are new to Python and FastVideo, you should start here.

# if you have not cloned the directory:
git clone https://github.com/hao-ai-lab/FastVideo.git && cd FastVideo

python examples/inference/basic/basic.py

Apple Silicon (FastMetal-QAD)

Use the MLX runtime with FastMetal-QAD. See the MLX install guide.

hf download FastVideo/FastMetal-1.3B-QAD --local-dir ./FastMetal-1.3B-QAD

python examples/inference/basic/mlx_wan_prompt_to_video.py \
  --model-root ./FastMetal-1.3B-QAD \
  --mlx-checkpoint ./FastMetal-1.3B-QAD \
  --prompt "A bird's-eye view of a misty forest valley at dawn."

5B uses mlx_wan22_generate.py with FastVideo/FastMetal-5B-QAD.

examples/inference/basic/basic_mps.py is the older PyTorch MPS demo.

FastH3 V1 T2VA also runs through the native MLX runtime. Convert the DiT to INT8, INT6, or INT4 first, then run:

python examples/inference/basic/mlx_fasth3.py \
  --model-root ./FastH3-Preview-v0.2 \
  --mlx-checkpoint ./FastH3-MLX/int6 \
  --prompt "(S1) A presenter says <d>[English] Fast H3 is amazing.</d>" \
  --height 480 --width 832 --num-frames 124 \
  --output-path ./outputs/fasth3_int6.mp4

FastH3 V2 is a separate checkpoint and script. Convert with --include-vsa and run mlx_fasth3_8step.py. Do not reuse a V1 DiT.

Pass --fast for temporal RIFE fast mode and --fast-spatial for spatial fast mode (reduced-canvas denoise + pixel-space upsample); the two compose. V1 VSA is opt-in: convert with --include-vsa and pass --vsa. V2 turns VSA on by default. See the MLX install guide. This MLX entrypoint supports T2VA only; FL2VA, Ref2VA, and two-pass refinement remain follow-up work. INT6/INT8/INT4 are weight-only; VSA attention activations stay BF16. Dense-only V1 checkpoints keep working for dense inference.

The complete setup and conversion commands are in the MLX install guide.

For an example running DMD+VSA inference:

python examples/inference/basic/basic_dmd.py

For the typed config/request path added during the inference API refactor:

python examples/inference/basic/basic_dmd_new_api.py

FastH3 Preview

The verified basic FastH3 example runs the few-step (4-forward, DMD2-distilled) MiniMax-H3 preview, generating synchronized video and audio with its trained block-sparse VSA attention:

UV_TORCH_BACKEND=cu130 uv pip install -e ".[fasth3]"

This installs the pinned FA4 CuTe package and FastVideo kernel release used by the measured GB200 profile. Then run:

python examples/inference/basic/basic_fasth3.py --prompt "your prompt"

The default checkpoint, FastH3 Preview v0.2, is public on the Hub under the MiniMax H3 Community License. Review its model card and license before use or redistribution.

The default all profile is the fastest measured four-GPU Preview recipe on GB200. It selects VSA sparsity 0.9 with 64-token tiles and the ptx_blackwell sparse kernel, enables FA4 for eligible non-VSA paths, regionally compiles and replicates the sparse DiT, compiles and temporally parallelizes the video VAE with the gather strategy, and pins CPU-offloaded component memory. It also pins the benchmark protocol: five sigma-grid points (exactly four DiT forwards), one excluded seed-999 warmup, then three timed seed-1000 requests with distinct output paths.

The equivalent explicit command is:

python examples/inference/basic/basic_fasth3.py \
  --prompt "your prompt" \
  --profile all \
  --num-gpus 4 \
  --steps 5 \
  --vsa-sparsity 0.9 \
  --vsa-tile-size 64 \
  --vsa-kernel ptx_blackwell \
  --compile-vae \
  --parallel-vae \
  --replicated-dit \
  --pin-cpu-memory \
  --fa4 \
  --no-torch-compile \
  --inference-torch-compile \
  --ulysses-a2a off \
  --warmup \
  --repeats 3 \
  --seed 1000 \
  --warmup-seed 999

all enables the inference-only H3 fusions and regional compile. Both can change floating-point operation order, so this is a report-only performance profile rather than an exact-parity route. Use --profile strict to disable the H3 fusions while preserving regional compile, or --profile strict --no-inference-torch-compile for the eager strict route. Individual --no-* switches are available for portability and attribution; in particular, use --vsa-kernel triton --no-fa4 if the Blackwell kernels are unavailable. --h3-sequential-load / --no-h3-sequential-load override the auto split that releases Qwen3-VL before DiT/VAE load (on by default on GB10, off on discrete GPUs). The script preserves the warmup and each measured video under distinct paths, then prints per-request wall time, the worker's peak memory, and a warmup-excluded median. Peak memory is the worker's allocator high-water mark: it is never reset between generations, so it includes model load and every prior run, and only rank 0's value is reported. It is populated by the mp backend only: the Ray backend leaves peak_memory_mb unset, so --execution-backend ray prints no Peak memory line.

One script covers each validated duration; regional compile is the fastest measured DiT route for all three:

# 5 s
python examples/inference/basic/basic_fasth3.py \
  --prompt "your prompt" --output outputs/fasth3_5s
# 10 s
python examples/inference/basic/basic_fasth3.py \
  --prompt "your prompt" --num-frames 243 --output outputs/fasth3_10s
# 15 s
python examples/inference/basic/basic_fasth3.py \
  --prompt "your prompt" --num-frames 345 --output outputs/fasth3_15s

Pass --no-inference-torch-compile to recover the eager sparse-DiT route.

Hopper (H100 80 GB)

Hopper has no FA4 and no ptx_blackwell VSA kernel, so the GB200 profile above does not apply there. The measured Hopper route keeps the same checkpoint, four forwards and 90% VSA sparsity and changes three things: the 64-token VSA path runs on the ThunderKittens sm_90a kernel, which fastvideo-kernel picks on sm_90 whenever it is built with cd fastvideo-kernel && CMAKE_ARGS='-DFASTVIDEO_KERNEL_BUILD_TK=ON' ./build.sh (--vsa-kernel tk keeps the ptx_blackwell route off and fails early when that extension is missing), the DiT is FSDP-sharded across the GPUs instead of replicated (--no-replicated-dit; a replicated 66 GB BF16 DiT does not fit 80 GB), and the Qwen3-VL text encoder stays resident as block-scaled FP8 (35.5 GB instead of 66.7 GB) rather than being offloaded on every request:

python scripts/checkpoint_conversion/quantize_minimax_h3_text_encoder_fp8.py \
  --source ./FastH3-Preview-v1/text_encoder --output ./FastH3-TextEncoder-FP8

python examples/inference/basic/basic_fasth3.py \
  --prompt "your prompt" \
  --num-gpus 4 --no-replicated-dit \
  --vsa-kernel tk --no-fa4 \
  --text-encoder-weights ./FastH3-TextEncoder-FP8 \
  --no-offload-text-encoder --no-offload-vae \
  --height 544 --width 960 --num-frames 345

examples/inference/basic/basic_fasth3_h100.yaml is the same profile for fastvideo generate --config. Measured points on this route (345 frames, 24 FPS, stereo audio, warm, loading and compilation excluded):

GPUs	Resolution	Playback	Generation	Source
8x H100 80 GB	1344x768	14.375 s	13.506 s (13.503 / 13.506 / 13.554)	hlander-ai/minimax-h3 on FastVideo b2db0c0 with these patches, Triton VSA
4x H100 80 GB, NVLink	960x544	14.375 s	12.98-14.02 s over 8 consecutive clips	Windflow streaming worker on FastVideo b2db0c0 with these patches, TK VSA

The 4x H100 row was measured through a downstream streaming worker rather than this script; basic_fasth3.py numbers for that shape will follow in a later PR. The FP8 text encoder is a precision change relative to the stock BF16 encoder; the other two changes are lossless.

FastH3 Preview LoRAs

The LoRA release runs on top of MiniMaxAI/MiniMax-H3 with the same default compile, fusion, FA4, VSA, and parallel-VAE profile as the full FastH3 example:

bash examples/inference/basic/run_fasth3_lora_preview_vsa_datafree.sh \
  --prompt "your prompt"

The four release launchers are:

• run_fasth3_lora_preview_vsa_datafree.sh
• run_fasth3_lora_preview_vsa_synthetic_step1300.sh
• run_fasth3_lora_preview_vsa_synthetic_step1900.sh
• run_fasth3_lora_preview_dense_datafree.sh

Each downloads its exact private adapter file from FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA; authenticate with hf auth login first. Pass --lora-strength 0.5 to interpolate every adapter payload at half strength. Strength 1 applies the published rank-64 adapter at its trained scale and approximates the full student; 0 removes its weight deltas. VSA launchers still use sparse attention at strength 0 and require FastVideo's tile-64 VSA kernel; the dense launcher selects FA4. Each launcher writes to its own variant directory by default so comparison outputs do not collide.

FastH3 OmniRef PDD (Ref2VA)

basic_fasth3_omniref_pdd.py runs a Parallel Decoding Distillation Ref2VA student in eight transformer forwards. The export carries only its transformer_ref, scheduler configs, and fastvideo_inference.json; the script links them with the base MiniMax-H3 components into one local model directory. Its 128-token VSA tiles need the ptx_blackwell kernel:

python examples/inference/basic/basic_fasth3_omniref_pdd.py \
  --model-path <local export directory or Hugging Face repo id> \
  --video reference.mp4 --image character.png --prompt "your prompt"

See FastH3 distilled checkpoint schedules for the contract and the reference-video sparsity policy. For the consumer NVFP4 stack (RTX 5090, RTX PRO 6000, DGX Spark; FP8 on the RTX 4090) add --profile <name>, and see FastH3 OmniRef inference.

Basic Walkthrough

All you need to generate videos using multi-gpus from state-of-the-art diffusion pipelines is the following few lines!

from fastvideo import VideoGenerator

def main():
    generator = VideoGenerator.from_pretrained(
        "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
        num_gpus=1,
    )

    prompt = ("A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes "
             "wide with interest. The playful yet serene atmosphere is complemented by soft "
             "natural light filtering through the petals. Mid-shot, warm and cheerful tones.")
    video = generator.generate_video(prompt)

if __name__ == "__main__":
    main()