nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[perf]: faster FastH3 decode and NVFP4 GEMMs in the DGX Spark recipes

Browse Source 
On GB10 the light H3 VAE's INT8 ConvRot overlay decodes 832x480x124 in
42 s and gains nothing from tile batching. The dense weights take 24 s
with one tile per decoder call, 14 s with 12 tiles per call and 9.8 s
compiled, at higher fidelity (77.7 vs 63.4 dB PSNR against an FP32
decode). FASTVIDEO_H3_VAE_INT8_OVERLAY=0 now skips the overlay (default
on, so other recipes are unchanged), and the single-Spark recipes use it
with FASTVIDEO_H3_VAE_TILE_BATCH=12 and a compiled VAE decoder.

FlashInfer's CUTLASS mm_fp4 is 2.7-3x slower than its cuDNN backend for
the H3 FFN projections on GB10 with identical outputs, so the recipes
select FASTVIDEO_NVFP4_MM_BACKEND=cudnn. FlashInfer autotuning was not
adopted: it made repeated calls non-deterministic.

One Spark, 832x480x124, seed 1234, warm calls: FastH3 Trim 121-122 s to
68 s (denoise 78 -> 58 s, decode 43 -> 9.5 s); FastH3 V2 139-141 s to
79 s. Repeated calls are bit-identical, which the previous V2 recipe was
not.

FP4 attention (FASTVIDEO_H3_VSA_FP4=1, built for sm_121a since #1598) is
documented for GB10 as opt-in and stays off: it is 24-33% faster in
denoise on V2 but changes the output (LPIPS 0.39-0.55 against BF16
attention), and the checkpoints were not trained for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Aryan Kumar
2026-10-09 11:43:20 -07:00
co-authored by Claude Opus 5.5
parent d0b91810b2
commit 1c1aa75f33
8 changed files with 118 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/contributing/env_vars.md
+2 -1
View File
@@ -250,7 +250,7 @@ longer exists also fails the test, so the fixing pull request deletes its entry.
| `FASTVIDEO_VBENCH_FULL_INFO_JSON` | str | unset | eval | Path to VBench_full_info.json, used instead of the vendored copy. Deprecated names: `VBENCH_FULL_INFO_JSON`. |
| `FASTVIDEO_FVD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the FVD metric. |
| `FASTVIDEO_FAD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the audio Frechet distance metric. |
| `FASTVIDEO_H3_VSA_FP4` | bool | `0` | attention | Run MiniMax-H3 VSA attention on the block-sparse SageAttention3 FP4 kernel (sm_120, no-grad, single sequence-parallel rank). |
| `FASTVIDEO_H3_VSA_FP4` | bool | `0` | attention | Run MiniMax-H3 VSA attention on the block-sparse SageAttention3 FP4 kernel (sm_120 or GB10 sm_121, no-grad, single sequence-parallel rank). |
| `FASTVIDEO_H3_VSA_HEADS_FIRST_TILE` | bool | `0` | performance | VSA-H3 64/128-token tiles: scatter rows straight into the heads-first layout the sparse kernels read, removing their per-call transpose copy of query, key and value. Bit-identical; no-grad, uncompiled CUDA only. |
| `FASTVIDEO_H3_VSA_TILE_FIRST` | bool | `0` | attention | Single-rank MiniMax-H3 VSA with one tile gather of the block input instead of separate Q/K/V/gate scatters. |
| `FASTVIDEO_H3_VSA_SM89_KERNEL` | one of original, bf16, int8 | `original` | attention | Fine-attention kernel for MiniMax-H3 VSA on sm_89: original, bf16, or int8 (INT8 QK, BF16 PV). |
@@ -271,6 +271,7 @@ longer exists also fails the test, so the fixing pull request deletes its entry.
| `FASTVIDEO_H3_VAE_TILE_BATCH` | int | `1` | performance | Spatial tiles per MiniMax-H3 video VAE decoder call; 1 decodes per tile. |
| `FASTVIDEO_H3_REF2VA_MEMO_ENTRIES` | int | `0` | performance | Entries in each content-keyed MiniMax-H3 memo of reference encodes (Qwen3-VL Ref2VA presentation, VAE keyframe latents); repeat references are reused exactly. 0 disables. |
| `FASTVIDEO_H3_VAE_TILE_PARALLEL` | bool | `0` | performance | Split MiniMax-H3 VAE spatial tiles (not only temporal chunks) across the sequence-parallel ranks in the parallel decode and keyframe encode; bitwise equal to the serial path. |
| `FASTVIDEO_H3_VAE_INT8_OVERLAY` | bool | `1` | performance | Apply the INT8 ConvRot decoder overlay when the MiniMax-H3 VAE folder ships one; 0 decodes with the dense weights. |
| `FASTVIDEO_H3_VAE_INT8_SHARED_QKV` | bool | `0` | performance | Share the INT8 activation rotation and quantization across the MiniMax-H3 VAE Q/K/V projections. |
| `FASTVIDEO_H3_VAE_INT8_TRANSPOSE_VIEW` | bool | `0` | performance | Use transposed weight views in the MiniMax-H3 VAE INT8 projections. |
| `FASTVIDEO_H3_VAE_INT8_FUSED_DEQUANT` | bool | `0` | performance | Fused dequantization epilogue for the MiniMax-H3 VAE INT8 projections. |
docs/getting_started/installation/spark_performance.md
+61 -4
View File
@@ -111,7 +111,7 @@ this hardware, for reasons specific to it:
| bf16 VAE decode | ~1.14×, lossless; ~5–7% e2e on few-step | ✅ default for Wan |
| VSA (video sparse attention) | works out of the box (Triton kernel auto-selects on `sm_121`) | ✅ automatic |
| Building FlashAttention | **no speedup** — Torch SDPA already hits an efficient flash kernel on `sm_121`, and FA2 ties it | ❌ not worth building |
| `torch.compile` of the VAE decode | recompile storm (per-frame varying shapes) → ~1.1× | ❌ dead end |
| `torch.compile` of the Wan VAE decode | recompile storm (per-frame varying shapes) → ~1.1× | ❌ dead end (the H3 ViT decoder compiles well, see [FastH3](#fasth3-decode-gemms-and-attention-on-gb10)) |
| Linear (fp8 / nvfp4) quantization on long-sequence models (e.g. Cosmos) | ~nothing — see below | ❌ wrong lever here |
| FP4 attention (`ATTN_QAT_INFER`) | works on `sm_121` (runtime allowlist landed in #1647; kernel build is #1598); helps, but needs a QAT-trained checkpoint | ⚠️ opt-in — see below |
| FP4 linear on short-sequence models (LTX2) | up to −24% denoise at 1080p (#1594) | ⚠️ model/resolution-dependent |
@@ -260,8 +260,8 @@ It does not use frame dropping or spatial upscaling.
```bash
FASTVIDEO_MINIMAX_H3_FUSIONS=all \
FASTVIDEO_NVFP4_MM_BACKEND=cutlass \
FASTVIDEO_H3_VAE_TILE_BATCH=1 \
FASTVIDEO_NVFP4_MM_BACKEND=cudnn \
FASTVIDEO_H3_VAE_TILE_BATCH=12 FASTVIDEO_H3_VAE_INT8_OVERLAY=0 \
FASTVIDEO_VSA_TRITON=1 FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 \
FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 \
FASTVIDEO_STAGE_LOGGING=1 \
@@ -276,7 +276,7 @@ H3 permits frame counts of `17n+5`; 124 is the nearest legal count above
five seconds and 243 is the nearest above ten seconds.
For Trim, use `basic_fasth3_spark_pruned_nvfp4.yaml` with the same environment.
Both recipes keep the encoder, DiT and VAEs resident, disable compilation,
Both recipes keep the encoder, DiT and VAEs resident, compile only the VAE decoder,
and use `h3_dit_vsa`. On two Sparks, use the corresponding
`basic_fasth3_spark_pair_{pruned,v2}_nvfp4.yaml` after following the
[pair setup guide](spark_pair.md). The pair uses SP2/TP1 and parallel VAE
@@ -309,6 +309,63 @@ The prompt JSON must contain `latency-ceramics-005` and
`latency-harbor-005`. Pass `--width 1344 --height 768` for the native 768p
protocol. Use the Trim repository and config for its corresponding run.
### FastH3 decode, GEMMs and attention on GB10
The recipes decode with the light VAE's dense weights
(`FASTVIDEO_H3_VAE_INT8_OVERLAY=0`), 12 spatial tiles per decoder call and a
compiled decoder (`compile.vae_enabled: true`). The INT8 ConvRot overlay that
ships next to the VAE is slower on GB10 and stops tile batching from helping.
Decoder-only timings on identical 832x480, 124-frame latents (one GB10):
| Light VAE decoder | 1 tile per call | 12 tiles per call | PSNR vs FP32 decode |
|---|---:|---:|---:|
| INT8 ConvRot overlay, fp16 autocast | 42.2 s | 43.5 s | 63.4 dB |
| Dense, fp16 autocast | 24.4 s | 14.4 s | 77.5 dB |
| Dense, fp16 autocast, compiled | 21.6 s | 9.8 s | 77.7 dB |
The first one or two calls in a process include decoder compiles; later calls
in the same process reuse them.
NVFP4 linears use FlashInfer's cuDNN backend (`FASTVIDEO_NVFP4_MM_BACKEND=cudnn`).
The CUTLASS backend used on RTX Blackwell is much slower on GB10 for the
FFN projections, and both backends return identical outputs. FlashInfer
`mm_fp4` with 19,392 activation rows (FastH3 at 832x480, 124 frames):
| Projection (K→N) | `cutlass` | `cudnn` | `auto` |
|---|---:|---:|---:|
| attention q/k/v, gate (5376→7168) | 5.5 ms | 10.0 ms | 10.0 ms |
| attention out (7168→5376) | 5.5 ms | 4.6 ms | 8.3 ms |
| FFN in (5376→28672) | 52.9 ms | 19.6 ms | 20.0 ms |
| FFN out (14336→5376) | 26.6 ms | 8.9 ms | 9.0 ms |
`FASTVIDEO_H3_VSA_FP4=1` runs VSA-H3's block-sparse attention on the FP4
SageAttention3 kernel, as on RTX Blackwell. It is off by default and needs
`fastvideo-kernel` built for `12.1a` (the default when building on a Spark).
On a captured FastH3 Trim attention input (19,392 tokens, 56 heads, 64-token
tiles) it runs in 29 ms against 59 ms for Triton VSA, with a relative L2 error
of 0.13 versus an FP32 reference (0.002 for Triton BF16).
It changes the output for now. The released FastH3 checkpoints were not trained with
FP4 attention, so the same seed gives a different, though coherent, clip. On V2
at 832x480 with four prompts, denoising was 24-33% faster (60-68 s to 43-51 s)
and the clips scored LPIPS 0.39-0.55 against the BF16-attention clips. The
recipes leave it off; review outputs before enabling it.
Measured on one Spark (832x480, 124 frames, seed 1234, the ceramics release
prompt), timing warm `generate()` calls in one process through finished MP4
output:
| Model and configuration | End to end | Denoise | Video decode |
|---|---:|---:|---:|
| Trim, previous recipe (CUTLASS, INT8 overlay, 1 tile per call) | 121-122 s | 77-78 s | 43 s |
| Trim, current recipe | 68 s | 58 s | 9.5 s |
| Trim, current recipe + `FASTVIDEO_H3_VSA_FP4=1` | 54 s | 43 s | 9.6 s |
| V2, previous recipe | 139-141 s | 95-96 s | 42-43 s |
| V2, current recipe | 79 s | 68 s | 9.6 s |
Within each configuration, repeated calls produced bit-identical frames
except the previous V2 recipe, whose repeated calls diverged.
### Released model measurements
The released Trim stack at revision `cae9ceb6feefe77d34a56640782cda3909363f19`
examples/inference/basic/basic_fasth3_spark_pruned_nvfp4.yaml
+10 -3
View File
@@ -4,9 +4,16 @@
#
# GB10 uses Triton VSA. The release packs attention, FFN and VSA gate weights.
# The NVFP4 encoder and lightweight VAE ship in the same repository.
# The VAE decodes with its dense weights, 12 tiles per call and a compiled
# decoder: on GB10 the INT8 ConvRot overlay is slower and less accurate.
# NVFP4 GEMMs use FlashInfer's cuDNN backend, 2-3x faster than CUTLASS on the
# H3 FFN shapes on GB10 with identical outputs.
# Optional, off by default: FASTVIDEO_H3_VSA_FP4=1 runs VSA on the FP4 attention kernel
# (fastvideo-kernel built for 12.1a). It is faster but changes the output for now;
# see spark_performance.md.
#
# FASTVIDEO_MINIMAX_H3_FUSIONS=all FASTVIDEO_NVFP4_MM_BACKEND=cutlass \
# FASTVIDEO_H3_VAE_TILE_BATCH=1 \
# FASTVIDEO_MINIMAX_H3_FUSIONS=all FASTVIDEO_NVFP4_MM_BACKEND=cudnn \
# FASTVIDEO_H3_VAE_TILE_BATCH=12 FASTVIDEO_H3_VAE_INT8_OVERLAY=0 \
# FASTVIDEO_VSA_TRITON=1 FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 \
# FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 FASTVIDEO_STAGE_LOGGING=1 \
# fastvideo generate --config examples/inference/basic/basic_fasth3_spark_pruned_nvfp4.yaml
@@ -31,7 +38,7 @@ generator:
lazy_module_load: false
compile:
enabled: false
vae_enabled: false
vae_enabled: true
pipeline:
workload_type: t2v
vae_tiling: true
examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml
+10 -3
View File
@@ -4,9 +4,16 @@
#
# GB10 uses Triton VSA. The release packs attention, FFN and VSA gate weights.
# The NVFP4 encoder and lightweight VAE ship in the same repository.
# The VAE decodes with its dense weights, 12 tiles per call and a compiled
# decoder: on GB10 the INT8 ConvRot overlay is slower and less accurate.
# NVFP4 GEMMs use FlashInfer's cuDNN backend, 2-3x faster than CUTLASS on the
# H3 FFN shapes on GB10 with identical outputs.
# Optional, off by default: FASTVIDEO_H3_VSA_FP4=1 runs VSA on the FP4 attention kernel
# (fastvideo-kernel built for 12.1a). It is faster but changes the output for now;
# see spark_performance.md.
#
# FASTVIDEO_MINIMAX_H3_FUSIONS=all FASTVIDEO_NVFP4_MM_BACKEND=cutlass \
# FASTVIDEO_H3_VAE_TILE_BATCH=1 \
# FASTVIDEO_MINIMAX_H3_FUSIONS=all FASTVIDEO_NVFP4_MM_BACKEND=cudnn \
# FASTVIDEO_H3_VAE_TILE_BATCH=12 FASTVIDEO_H3_VAE_INT8_OVERLAY=0 \
# FASTVIDEO_VSA_TRITON=1 FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 \
# FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 FASTVIDEO_STAGE_LOGGING=1 \
# fastvideo generate --config examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml
@@ -31,7 +38,7 @@ generator:
lazy_module_load: false
compile:
enabled: false
vae_enabled: false
vae_enabled: true
pipeline:
workload_type: t2v
vae_tiling: true
fastvideo/envs.py
+5 -1
View File
@@ -581,7 +581,7 @@ FASTVIDEO_FAD_REF_FEATURES = EnvStr(None,
FASTVIDEO_H3_VSA_FP4 = EnvBool(False,
category="attention",
doc="Run MiniMax-H3 VSA attention on the block-sparse SageAttention3 FP4 kernel "
"(sm_120, no-grad, single sequence-parallel rank).")
"(sm_120 or GB10 sm_121, no-grad, single sequence-parallel rank).")
FASTVIDEO_H3_VSA_HEADS_FIRST_TILE = EnvBool(False,
category="performance",
doc="VSA-H3 64/128-token tiles: scatter rows straight into the heads-first "
@@ -663,6 +663,10 @@ FASTVIDEO_H3_VAE_TILE_PARALLEL = EnvBool(False,
doc="Split MiniMax-H3 VAE spatial tiles (not only temporal chunks) across "
"the sequence-parallel ranks in the parallel decode and keyframe encode; "
"bitwise equal to the serial path.")
FASTVIDEO_H3_VAE_INT8_OVERLAY = EnvBool(True,
category="performance",
doc="Apply the INT8 ConvRot decoder overlay when the MiniMax-H3 VAE folder "
"ships one; 0 decodes with the dense weights.")
FASTVIDEO_H3_VAE_INT8_SHARED_QKV = EnvBool(False,
category="performance",
doc="Share the INT8 activation rotation and quantization across the "
fastvideo/models/loader/component_loader.py
+2 -2
View File
@@ -974,9 +974,9 @@ class VAELoader(ComponentLoader):
if class_name == "AutoencoderKLMiniMaxH3":
from fastvideo.models.vaes.minimax_h3_int8_convrot import (
dense_vae_safetensors,
find_int8_convrot_vae_path,
int8_convrot_overlay_to_apply,
)
int8_convrot_path = find_int8_convrot_vae_path(model_path)
int8_convrot_path = int8_convrot_overlay_to_apply(model_path)
safetensors_list = dense_vae_safetensors(safetensors_list)
if not safetensors_list:
raise ValueError(f"No safetensors files found in {model_path}")
fastvideo/models/vaes/minimax_h3_int8_convrot.py
+10
View File
@@ -351,3 +351,13 @@ def dense_vae_safetensors(paths: list[str]) -> list[str]:
def find_int8_convrot_vae_path(model_path: str | Path) -> Path | None:
candidate = Path(model_path) / INT8_CONVROT_FILENAME
return candidate if candidate.is_file() else None
def int8_convrot_overlay_to_apply(model_path: str | Path) -> Path | None:
"""The ConvRot overlay shipped in ``model_path``, unless ``FASTVIDEO_H3_VAE_INT8_OVERLAY=0`` skips it."""
path = find_int8_convrot_vae_path(model_path)
if path is not None and not envs.FASTVIDEO_H3_VAE_INT8_OVERLAY.get():
logger.info("Skipping MiniMax-H3 INT8 ConvRot VAE overlay %s (FASTVIDEO_H3_VAE_INT8_OVERLAY=0); "
"decoding with the dense weights", path)
return None
return path
fastvideo/tests/vaes/test_minimax_h3_int8_convrot.py
+18
View File
@@ -5,13 +5,17 @@ from __future__ import annotations
import json
import torch
import pytest
import torch.nn as nn
from safetensors.torch import save_file
import fastvideo.envs as envs
from fastvideo.models.vaes.minimax_h3_int8_convrot import (
INT8_CONVROT_FILENAME,
Int8ConvRotLinear,
_int8_linear_from_tensors,
dense_vae_safetensors,
int8_convrot_overlay_to_apply,
overlay_minimax_h3_int8_convrot_decoder,
parse_comfy_quant_marker,
regular_hadamard,
@@ -82,6 +86,20 @@ def test_dense_vae_safetensors_drops_convrot_overlay() -> None:
]
@pytest.mark.parametrize(("enabled", "ships_overlay", "applied"), [
(True, True, True),
(False, True, False),
(True, False, False),
])
def test_int8_convrot_overlay_follows_the_overlay_switch(tmp_path, env_overrides, enabled: bool, ships_overlay: bool,
applied: bool) -> None:
if ships_overlay:
(tmp_path / INT8_CONVROT_FILENAME).write_bytes(b"")
env_overrides.enter_context(envs.FASTVIDEO_H3_VAE_INT8_OVERLAY.override(enabled))
expected = tmp_path / INT8_CONVROT_FILENAME if applied else None
assert int8_convrot_overlay_to_apply(tmp_path) == expected
def test_parse_comfy_quant_marker_reads_padded_uint8() -> None:
payload = json.dumps({"format": "int8_tensorwise", "convrot": True, "convrot_groupsize": 256}).encode()
blob = torch.zeros(72, dtype=torch.uint8)