diff --git a/docs/contributing/env_vars.md b/docs/contributing/env_vars.md index 62b6c56d6..746ddab17 100644 --- a/docs/contributing/env_vars.md +++ b/docs/contributing/env_vars.md @@ -250,7 +250,7 @@ longer exists also fails the test, so the fixing pull request deletes its entry. | `FASTVIDEO_VBENCH_FULL_INFO_JSON` | str | unset | eval | Path to VBench_full_info.json, used instead of the vendored copy. Deprecated names: `VBENCH_FULL_INFO_JSON`. | | `FASTVIDEO_FVD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the FVD metric. | | `FASTVIDEO_FAD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the audio Frechet distance metric. | -| `FASTVIDEO_H3_VSA_FP4` | bool | `0` | attention | Run MiniMax-H3 VSA attention on the block-sparse SageAttention3 FP4 kernel (sm_120, no-grad, single sequence-parallel rank). | +| `FASTVIDEO_H3_VSA_FP4` | bool | `0` | attention | Run MiniMax-H3 VSA attention on the block-sparse SageAttention3 FP4 kernel (sm_120 or GB10 sm_121, no-grad, single sequence-parallel rank). | | `FASTVIDEO_H3_VSA_HEADS_FIRST_TILE` | bool | `0` | performance | VSA-H3 64/128-token tiles: scatter rows straight into the heads-first layout the sparse kernels read, removing their per-call transpose copy of query, key and value. Bit-identical; no-grad, uncompiled CUDA only. | | `FASTVIDEO_H3_VSA_TILE_FIRST` | bool | `0` | attention | Single-rank MiniMax-H3 VSA with one tile gather of the block input instead of separate Q/K/V/gate scatters. | | `FASTVIDEO_H3_VSA_SM89_KERNEL` | one of original, bf16, int8 | `original` | attention | Fine-attention kernel for MiniMax-H3 VSA on sm_89: original, bf16, or int8 (INT8 QK, BF16 PV). | @@ -271,6 +271,7 @@ longer exists also fails the test, so the fixing pull request deletes its entry. | `FASTVIDEO_H3_VAE_TILE_BATCH` | int | `1` | performance | Spatial tiles per MiniMax-H3 video VAE decoder call; 1 decodes per tile. | | `FASTVIDEO_H3_REF2VA_MEMO_ENTRIES` | int | `0` | performance | Entries in each content-keyed MiniMax-H3 memo of reference encodes (Qwen3-VL Ref2VA presentation, VAE keyframe latents); repeat references are reused exactly. 0 disables. | | `FASTVIDEO_H3_VAE_TILE_PARALLEL` | bool | `0` | performance | Split MiniMax-H3 VAE spatial tiles (not only temporal chunks) across the sequence-parallel ranks in the parallel decode and keyframe encode; bitwise equal to the serial path. | +| `FASTVIDEO_H3_VAE_INT8_OVERLAY` | bool | `1` | performance | Apply the INT8 ConvRot decoder overlay when the MiniMax-H3 VAE folder ships one; 0 decodes with the dense weights. | | `FASTVIDEO_H3_VAE_INT8_SHARED_QKV` | bool | `0` | performance | Share the INT8 activation rotation and quantization across the MiniMax-H3 VAE Q/K/V projections. | | `FASTVIDEO_H3_VAE_INT8_TRANSPOSE_VIEW` | bool | `0` | performance | Use transposed weight views in the MiniMax-H3 VAE INT8 projections. | | `FASTVIDEO_H3_VAE_INT8_FUSED_DEQUANT` | bool | `0` | performance | Fused dequantization epilogue for the MiniMax-H3 VAE INT8 projections. | diff --git a/docs/getting_started/installation/spark_performance.md b/docs/getting_started/installation/spark_performance.md index 3c69e61a8..7f7e2a71e 100644 --- a/docs/getting_started/installation/spark_performance.md +++ b/docs/getting_started/installation/spark_performance.md @@ -111,7 +111,7 @@ this hardware, for reasons specific to it: | bf16 VAE decode | ~1.14×, lossless; ~5–7% e2e on few-step | ✅ default for Wan | | VSA (video sparse attention) | works out of the box (Triton kernel auto-selects on `sm_121`) | ✅ automatic | | Building FlashAttention | **no speedup** — Torch SDPA already hits an efficient flash kernel on `sm_121`, and FA2 ties it | ❌ not worth building | -| `torch.compile` of the VAE decode | recompile storm (per-frame varying shapes) → ~1.1× | ❌ dead end | +| `torch.compile` of the Wan VAE decode | recompile storm (per-frame varying shapes) → ~1.1× | ❌ dead end (the H3 ViT decoder compiles well, see [FastH3](#fasth3-decode-gemms-and-attention-on-gb10)) | | Linear (fp8 / nvfp4) quantization on long-sequence models (e.g. Cosmos) | ~nothing — see below | ❌ wrong lever here | | FP4 attention (`ATTN_QAT_INFER`) | works on `sm_121` (runtime allowlist landed in #1647; kernel build is #1598); helps, but needs a QAT-trained checkpoint | ⚠️ opt-in — see below | | FP4 linear on short-sequence models (LTX2) | up to −24% denoise at 1080p (#1594) | ⚠️ model/resolution-dependent | @@ -260,8 +260,8 @@ It does not use frame dropping or spatial upscaling. ```bash FASTVIDEO_MINIMAX_H3_FUSIONS=all \ -FASTVIDEO_NVFP4_MM_BACKEND=cutlass \ -FASTVIDEO_H3_VAE_TILE_BATCH=1 \ +FASTVIDEO_NVFP4_MM_BACKEND=cudnn \ +FASTVIDEO_H3_VAE_TILE_BATCH=12 FASTVIDEO_H3_VAE_INT8_OVERLAY=0 \ FASTVIDEO_VSA_TRITON=1 FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 \ FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 \ FASTVIDEO_STAGE_LOGGING=1 \ @@ -276,7 +276,7 @@ H3 permits frame counts of `17n+5`; 124 is the nearest legal count above five seconds and 243 is the nearest above ten seconds. For Trim, use `basic_fasth3_spark_pruned_nvfp4.yaml` with the same environment. -Both recipes keep the encoder, DiT and VAEs resident, disable compilation, +Both recipes keep the encoder, DiT and VAEs resident, compile only the VAE decoder, and use `h3_dit_vsa`. On two Sparks, use the corresponding `basic_fasth3_spark_pair_{pruned,v2}_nvfp4.yaml` after following the [pair setup guide](spark_pair.md). The pair uses SP2/TP1 and parallel VAE @@ -309,6 +309,63 @@ The prompt JSON must contain `latency-ceramics-005` and `latency-harbor-005`. Pass `--width 1344 --height 768` for the native 768p protocol. Use the Trim repository and config for its corresponding run. +### FastH3 decode, GEMMs and attention on GB10 + +The recipes decode with the light VAE's dense weights +(`FASTVIDEO_H3_VAE_INT8_OVERLAY=0`), 12 spatial tiles per decoder call and a +compiled decoder (`compile.vae_enabled: true`). The INT8 ConvRot overlay that +ships next to the VAE is slower on GB10 and stops tile batching from helping. +Decoder-only timings on identical 832x480, 124-frame latents (one GB10): + +| Light VAE decoder | 1 tile per call | 12 tiles per call | PSNR vs FP32 decode | +|---|---:|---:|---:| +| INT8 ConvRot overlay, fp16 autocast | 42.2 s | 43.5 s | 63.4 dB | +| Dense, fp16 autocast | 24.4 s | 14.4 s | 77.5 dB | +| Dense, fp16 autocast, compiled | 21.6 s | 9.8 s | 77.7 dB | + +The first one or two calls in a process include decoder compiles; later calls +in the same process reuse them. + +NVFP4 linears use FlashInfer's cuDNN backend (`FASTVIDEO_NVFP4_MM_BACKEND=cudnn`). +The CUTLASS backend used on RTX Blackwell is much slower on GB10 for the +FFN projections, and both backends return identical outputs. FlashInfer +`mm_fp4` with 19,392 activation rows (FastH3 at 832x480, 124 frames): + +| Projection (K→N) | `cutlass` | `cudnn` | `auto` | +|---|---:|---:|---:| +| attention q/k/v, gate (5376→7168) | 5.5 ms | 10.0 ms | 10.0 ms | +| attention out (7168→5376) | 5.5 ms | 4.6 ms | 8.3 ms | +| FFN in (5376→28672) | 52.9 ms | 19.6 ms | 20.0 ms | +| FFN out (14336→5376) | 26.6 ms | 8.9 ms | 9.0 ms | + +`FASTVIDEO_H3_VSA_FP4=1` runs VSA-H3's block-sparse attention on the FP4 +SageAttention3 kernel, as on RTX Blackwell. It is off by default and needs +`fastvideo-kernel` built for `12.1a` (the default when building on a Spark). +On a captured FastH3 Trim attention input (19,392 tokens, 56 heads, 64-token +tiles) it runs in 29 ms against 59 ms for Triton VSA, with a relative L2 error +of 0.13 versus an FP32 reference (0.002 for Triton BF16). + +It changes the output for now. The released FastH3 checkpoints were not trained with +FP4 attention, so the same seed gives a different, though coherent, clip. On V2 +at 832x480 with four prompts, denoising was 24-33% faster (60-68 s to 43-51 s) +and the clips scored LPIPS 0.39-0.55 against the BF16-attention clips. The +recipes leave it off; review outputs before enabling it. + +Measured on one Spark (832x480, 124 frames, seed 1234, the ceramics release +prompt), timing warm `generate()` calls in one process through finished MP4 +output: + +| Model and configuration | End to end | Denoise | Video decode | +|---|---:|---:|---:| +| Trim, previous recipe (CUTLASS, INT8 overlay, 1 tile per call) | 121-122 s | 77-78 s | 43 s | +| Trim, current recipe | 68 s | 58 s | 9.5 s | +| Trim, current recipe + `FASTVIDEO_H3_VSA_FP4=1` | 54 s | 43 s | 9.6 s | +| V2, previous recipe | 139-141 s | 95-96 s | 42-43 s | +| V2, current recipe | 79 s | 68 s | 9.6 s | + +Within each configuration, repeated calls produced bit-identical frames +except the previous V2 recipe, whose repeated calls diverged. + ### Released model measurements The released Trim stack at revision `cae9ceb6feefe77d34a56640782cda3909363f19` diff --git a/examples/inference/basic/basic_fasth3_spark_pruned_nvfp4.yaml b/examples/inference/basic/basic_fasth3_spark_pruned_nvfp4.yaml index 5ad449512..14f7e9642 100644 --- a/examples/inference/basic/basic_fasth3_spark_pruned_nvfp4.yaml +++ b/examples/inference/basic/basic_fasth3_spark_pruned_nvfp4.yaml @@ -4,9 +4,16 @@ # # GB10 uses Triton VSA. The release packs attention, FFN and VSA gate weights. # The NVFP4 encoder and lightweight VAE ship in the same repository. +# The VAE decodes with its dense weights, 12 tiles per call and a compiled +# decoder: on GB10 the INT8 ConvRot overlay is slower and less accurate. +# NVFP4 GEMMs use FlashInfer's cuDNN backend, 2-3x faster than CUTLASS on the +# H3 FFN shapes on GB10 with identical outputs. +# Optional, off by default: FASTVIDEO_H3_VSA_FP4=1 runs VSA on the FP4 attention kernel +# (fastvideo-kernel built for 12.1a). It is faster but changes the output for now; +# see spark_performance.md. # -# FASTVIDEO_MINIMAX_H3_FUSIONS=all FASTVIDEO_NVFP4_MM_BACKEND=cutlass \ -# FASTVIDEO_H3_VAE_TILE_BATCH=1 \ +# FASTVIDEO_MINIMAX_H3_FUSIONS=all FASTVIDEO_NVFP4_MM_BACKEND=cudnn \ +# FASTVIDEO_H3_VAE_TILE_BATCH=12 FASTVIDEO_H3_VAE_INT8_OVERLAY=0 \ # FASTVIDEO_VSA_TRITON=1 FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 \ # FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 FASTVIDEO_STAGE_LOGGING=1 \ # fastvideo generate --config examples/inference/basic/basic_fasth3_spark_pruned_nvfp4.yaml @@ -31,7 +38,7 @@ generator: lazy_module_load: false compile: enabled: false - vae_enabled: false + vae_enabled: true pipeline: workload_type: t2v vae_tiling: true diff --git a/examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml b/examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml index 27531db0e..5bab303fc 100644 --- a/examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml +++ b/examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml @@ -4,9 +4,16 @@ # # GB10 uses Triton VSA. The release packs attention, FFN and VSA gate weights. # The NVFP4 encoder and lightweight VAE ship in the same repository. +# The VAE decodes with its dense weights, 12 tiles per call and a compiled +# decoder: on GB10 the INT8 ConvRot overlay is slower and less accurate. +# NVFP4 GEMMs use FlashInfer's cuDNN backend, 2-3x faster than CUTLASS on the +# H3 FFN shapes on GB10 with identical outputs. +# Optional, off by default: FASTVIDEO_H3_VSA_FP4=1 runs VSA on the FP4 attention kernel +# (fastvideo-kernel built for 12.1a). It is faster but changes the output for now; +# see spark_performance.md. # -# FASTVIDEO_MINIMAX_H3_FUSIONS=all FASTVIDEO_NVFP4_MM_BACKEND=cutlass \ -# FASTVIDEO_H3_VAE_TILE_BATCH=1 \ +# FASTVIDEO_MINIMAX_H3_FUSIONS=all FASTVIDEO_NVFP4_MM_BACKEND=cudnn \ +# FASTVIDEO_H3_VAE_TILE_BATCH=12 FASTVIDEO_H3_VAE_INT8_OVERLAY=0 \ # FASTVIDEO_VSA_TRITON=1 FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 \ # FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 FASTVIDEO_STAGE_LOGGING=1 \ # fastvideo generate --config examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml @@ -31,7 +38,7 @@ generator: lazy_module_load: false compile: enabled: false - vae_enabled: false + vae_enabled: true pipeline: workload_type: t2v vae_tiling: true diff --git a/fastvideo/envs.py b/fastvideo/envs.py index 696a32b6d..24380b894 100644 --- a/fastvideo/envs.py +++ b/fastvideo/envs.py @@ -581,7 +581,7 @@ FASTVIDEO_FAD_REF_FEATURES = EnvStr(None, FASTVIDEO_H3_VSA_FP4 = EnvBool(False, category="attention", doc="Run MiniMax-H3 VSA attention on the block-sparse SageAttention3 FP4 kernel " - "(sm_120, no-grad, single sequence-parallel rank).") + "(sm_120 or GB10 sm_121, no-grad, single sequence-parallel rank).") FASTVIDEO_H3_VSA_HEADS_FIRST_TILE = EnvBool(False, category="performance", doc="VSA-H3 64/128-token tiles: scatter rows straight into the heads-first " @@ -663,6 +663,10 @@ FASTVIDEO_H3_VAE_TILE_PARALLEL = EnvBool(False, doc="Split MiniMax-H3 VAE spatial tiles (not only temporal chunks) across " "the sequence-parallel ranks in the parallel decode and keyframe encode; " "bitwise equal to the serial path.") +FASTVIDEO_H3_VAE_INT8_OVERLAY = EnvBool(True, + category="performance", + doc="Apply the INT8 ConvRot decoder overlay when the MiniMax-H3 VAE folder " + "ships one; 0 decodes with the dense weights.") FASTVIDEO_H3_VAE_INT8_SHARED_QKV = EnvBool(False, category="performance", doc="Share the INT8 activation rotation and quantization across the " diff --git a/fastvideo/models/loader/component_loader.py b/fastvideo/models/loader/component_loader.py index 521f49fe1..3925a5c3d 100644 --- a/fastvideo/models/loader/component_loader.py +++ b/fastvideo/models/loader/component_loader.py @@ -974,9 +974,9 @@ class VAELoader(ComponentLoader): if class_name == "AutoencoderKLMiniMaxH3": from fastvideo.models.vaes.minimax_h3_int8_convrot import ( dense_vae_safetensors, - find_int8_convrot_vae_path, + int8_convrot_overlay_to_apply, ) - int8_convrot_path = find_int8_convrot_vae_path(model_path) + int8_convrot_path = int8_convrot_overlay_to_apply(model_path) safetensors_list = dense_vae_safetensors(safetensors_list) if not safetensors_list: raise ValueError(f"No safetensors files found in {model_path}") diff --git a/fastvideo/models/vaes/minimax_h3_int8_convrot.py b/fastvideo/models/vaes/minimax_h3_int8_convrot.py index d9a12a01f..0681a6450 100644 --- a/fastvideo/models/vaes/minimax_h3_int8_convrot.py +++ b/fastvideo/models/vaes/minimax_h3_int8_convrot.py @@ -351,3 +351,13 @@ def dense_vae_safetensors(paths: list[str]) -> list[str]: def find_int8_convrot_vae_path(model_path: str | Path) -> Path | None: candidate = Path(model_path) / INT8_CONVROT_FILENAME return candidate if candidate.is_file() else None + + +def int8_convrot_overlay_to_apply(model_path: str | Path) -> Path | None: + """The ConvRot overlay shipped in ``model_path``, unless ``FASTVIDEO_H3_VAE_INT8_OVERLAY=0`` skips it.""" + path = find_int8_convrot_vae_path(model_path) + if path is not None and not envs.FASTVIDEO_H3_VAE_INT8_OVERLAY.get(): + logger.info("Skipping MiniMax-H3 INT8 ConvRot VAE overlay %s (FASTVIDEO_H3_VAE_INT8_OVERLAY=0); " + "decoding with the dense weights", path) + return None + return path diff --git a/fastvideo/tests/vaes/test_minimax_h3_int8_convrot.py b/fastvideo/tests/vaes/test_minimax_h3_int8_convrot.py index 165ccc41e..99ecc2935 100644 --- a/fastvideo/tests/vaes/test_minimax_h3_int8_convrot.py +++ b/fastvideo/tests/vaes/test_minimax_h3_int8_convrot.py @@ -5,13 +5,17 @@ from __future__ import annotations import json import torch +import pytest import torch.nn as nn from safetensors.torch import save_file +import fastvideo.envs as envs from fastvideo.models.vaes.minimax_h3_int8_convrot import ( + INT8_CONVROT_FILENAME, Int8ConvRotLinear, _int8_linear_from_tensors, dense_vae_safetensors, + int8_convrot_overlay_to_apply, overlay_minimax_h3_int8_convrot_decoder, parse_comfy_quant_marker, regular_hadamard, @@ -82,6 +86,20 @@ def test_dense_vae_safetensors_drops_convrot_overlay() -> None: ] +@pytest.mark.parametrize(("enabled", "ships_overlay", "applied"), [ + (True, True, True), + (False, True, False), + (True, False, False), +]) +def test_int8_convrot_overlay_follows_the_overlay_switch(tmp_path, env_overrides, enabled: bool, ships_overlay: bool, + applied: bool) -> None: + if ships_overlay: + (tmp_path / INT8_CONVROT_FILENAME).write_bytes(b"") + env_overrides.enter_context(envs.FASTVIDEO_H3_VAE_INT8_OVERLAY.override(enabled)) + expected = tmp_path / INT8_CONVROT_FILENAME if applied else None + assert int8_convrot_overlay_to_apply(tmp_path) == expected + + def test_parse_comfy_quant_marker_reads_padded_uint8() -> None: payload = json.dumps({"format": "int8_tensorwise", "convrot": True, "convrot_groupsize": 256}).encode() blob = torch.zeros(72, dtype=torch.uint8)