hao-ai-lab-FastVideo/docs/getting_started/installation/spark_performance.md

DGX Spark (GB10): Performance & Tuning

You have FastVideo installed on a DGX Spark — this page is what to run next. It covers which models are practical on the GB10, what actually makes them faster, and what won't help (and why), so you don't burn a night tuning knobs that can't move on this hardware.

:::tip[TL;DR]

• Use distilled few-step models (e.g. FastVideo/FastWan2.1-T2V-1.3B-Diffusers). They run in ~40 s/video. Full-step models are 12–47 min on the GB10.
• On few-step models, VAE decode is the bottleneck, not attention — it's bandwidth-bound on the Spark's unified memory.
• bf16 VAE decode is the real, lossless lever (FastVideo already turns it on for Wan). FlashAttention, linear quantization, and torch.compile of the VAE give little or nothing here — see the table below.
• Heavy runs can make the box unreachable — run generations with VAE tiling on and nice -n 19. See Running safely. :::

The hardware reality (this explains everything below)

The GB10 pairs a Blackwell GPU (sm_121) with 128 GB of unified LPDDR5X memory (~270 GB/s) shared between CPU and GPU. That bandwidth is roughly 10× below a datacenter GPU's HBM. Two consequences drive every tuning decision:

1. Memory-bandwidth-bound stages hurt disproportionately. VAE decode moves a lot of data and becomes the dominant cost on short (few-step) generations.
2. Compute-bound stages scale with step count. Full-step diffusion (50+ steps) is denoise-bound and simply takes a long time here.

Use distilled few-step models

The single biggest lever on the GB10 is model choice. A 3-step distilled model is ~18× faster than the full-step version of the same architecture:

Model	Steps	Time / video	Bottleneck
FastWan2.1-T2V-1.3B (distilled)	3	~40 s	VAE decode
Wan2.1-T2V-1.3B (full-step)	50	~12 min	denoise
Cosmos-Predict2.5-2B (full-step)	51	~47 min	denoise
LTX2.3-distilled (+audio)	8	~6 min	mixed

The bottleneck flips from decode to denoise at around 4 steps. Below that, you're paying mostly for VAE decode; above it, mostly for the denoising loop.

:::note[Few-step timings are noisy — measure in-process] On a 3-step run, one-time per-process startup (Triton autotune, allocator warmup) dominates and never amortizes, so single-run totals wobble ~±30%. Compare levers back-to-back in one process or as medians, never as two separate single runs. The reproduction script does this for you. :::

bf16 VAE decode — the real lever (already on for Wan)

Because few-step generation is decode-bound, VAE decode precision is where the time is. Decoding in bf16 instead of fp32 is essentially lossless (MS-SSIM ~0.9999 vs fp32 on the identical latent) and ~1.14× faster — worth roughly 5–7% end-to-end on a decode-bound few-step model.

FastVideo already defaults Wan's decode to bf16 (vae_decode_precision="bf16", with encode kept at fp32), so for the recommended Wan/FastWan models there's nothing to set. If you run a model that still defaults to an fp32 decode, set the decode-only override yourself:

from fastvideo.configs.pipelines.base import PipelineConfig

pipeline_config = PipelineConfig.from_pretrained(model_id)
pipeline_config.vae_decode_precision = "bf16"   # decode-only; leaves encode precision alone

Decode is output-only, so lowering its precision is safe. (Encode seeds the denoising trajectory for I2V/causal models, so that stays at the pipeline's default — don't lower vae_precision blindly for those.)

Memory: one unified 128 GB pool

The GB10 has no separate VRAM — CPU and GPU share one 128 GB LPDDR5X pool (~118 GB usable). Two practical consequences:

• nvidia-smi reports memory as [N/A] on the GB10, and the system "used" figure conflates CPU + GPU + cache, so it's only a soft upper bound — treat the whole 128 GB as one shared budget. For a worker-measured figure, use FastVideo's own peak_memory_mb (reported on the generation result and by the performance benchmark), which is measured inside the worker that runs the model. It is the worker's allocator high-water mark, never reset between generations, so it includes model load and every prior run rather than a single run's delta. The mp backend populates it; the Ray backend leaves peak_memory_mb unset, so --execution-backend ray prints no peak-memory figure.
• The 128 GB is a working-set ceiling, not storage — the model cache lives on the NVMe (3.7 TB, ample). What has to fit in 128 GB is the weights, activations, and KV cache — and, critically, the VAE decode buffers, which is why tiling matters (an untiled high-res decode can spike the pool into swap and lock the box).

The recommended few-step models are comfortable here: their weights are small (1.3–2 B) and few-step generation keeps activations modest — a Wan2.1-1.3B few-step generation peaks at ~8.4 GB (measured), a small fraction of the pool. The pressure comes from decode resolution/frames, not the model — a 1080p×121-frame untiled decode is what pushes the pool toward its ceiling, which is why VAE tiling stays on by default.

What helps vs. what doesn't on the GB10

The honest summary — most "obvious" GPU optimizations don't move the needle on this hardware, for reasons specific to it:

Lever	Effect on the GB10	Use it?
Distilled few-step model	~18× vs full-step	✅ the primary lever
bf16 VAE decode	~1.14×, lossless; ~5–7% e2e on few-step	✅ default for Wan
VSA (video sparse attention)	works out of the box (Triton kernel auto-selects on sm_121)	✅ automatic
Building FlashAttention	no speedup — Torch SDPA already hits an efficient flash kernel on sm_121, and FA2 ties it	❌ not worth building
torch.compile of the Wan VAE decode	recompile storm (per-frame varying shapes) → ~1.1×	❌ dead end (the H3 ViT decoder compiles well, see FastH3)
Linear (fp8 / nvfp4) quantization on long-sequence models (e.g. Cosmos)	~nothing — see below	❌ wrong lever here
FP4 attention (ATTN_QAT_INFER)	works on sm_121 (runtime allowlist landed in #1647; kernel build is #1598); helps, but needs a QAT-trained checkpoint	⚠️ opt-in — see below
FP4 linear on short-sequence models (LTX2)	up to −24% denoise at 1080p (#1594)	⚠️ model/resolution-dependent

Why linear quantization is the wrong lever on long-sequence models

Quantizing the linear (GEMM) layers is a natural first instinct, but on a long-sequence video model it buys almost nothing on the GB10. A video-DiT denoise step is dominated by O(N²) attention at these sequence lengths (tens of thousands of tokens); the linear layers are a small single-digit fraction of the work. Quantizing them faster leaves the attention-bound total essentially unchanged — measured at ~1% on Cosmos-2.5, i.e. noise, and full-step CFG models also lose quality to per-step quantization error.

The same mechanism does help on short-sequence models: LTX2's aggressive VAE compression gives it short attention sequences, so FP4 linear reaches −24% there (#1594). The rule: on the GB10, the lever that matters is attention (sparse or FP4), not the linear layers — unless the model has short sequences.

FP4 on the GB10 (opt-in)

Block-scaled FP4 works on sm_121 under CUDA 13:

• FP4 attention (FASTVIDEO_ATTENTION_BACKEND=ATTN_QAT_INFER, #1598) is numerically correct on the GB10 and ~6% faster end-to-end generation, but it only preserves quality on a quantization-aware-distilled checkpoint (e.g. FastVideo/FastWan-QAD-1.3B) — stock weights aren't trained to tolerate it.
• FP4 linear helps only where sequences are short (LTX2, above).

The qad_fp4_ab.py harness reproduces the FP4 attention A/B on the QAD checkpoint.

Running safely (don't lock the box)

The GB10 is easy to make unreachable — a heavy build or an untiled high-res decode starves the ~20 ARM cores and unified memory, sshd can't get cycles, and you're locked out at "Connection timed out during banner exchange" until the box is power-cycled. To avoid it:

• Inference: keep VAE tiling on (the default), use sane resolution/frames, and run under nice -n 19:

  nice -n 19 nohup python your_script.py > run.log 2>&1 &
  

• Builds (flash-attn, kernel): nice -n 19, MAX_JOBS=2, nohup. Never a bare foreground high-parallelism build.

• FastVideo automatically disables DiT layerwise/CPU offload and encoder/VAE CPU offload after each worker binds its GB10 device. Do not force those modes back on: "CPU" offload uses the same unified RAM. Multi-GPU FSDP sharding remains available because it partitions weights without parking them in a separate host pool.

• Older MiniMax H3 / FastH3 bf16 weights need deferred loading on one GB10. The full Qwen3-VL conditioner is tens of gigabytes of BF16. If the DiT and VAEs load while that encoder is still resident, the process can be killed by earlyoom. On unified memory, lazy_module_load auto-enables and owns that split (encoder, then DiT, then VAE; DiT can drop before decode). Sequential load is the H3-only fallback when lazy is off. Keep deferred loading for those older checkpoints. The trimmed NVFP4 encoder and light VAE in the V2 resident recipe are a different memory profile. Geometry scalars come from checkpoint config.json, not live weights. See Offloading.

• FastH3 TAEH3 (--video-decode-backend taeh3) is an opt-in preview decoder. T2VA never materializes the 9.7 GiB video VAE (DiT still loads after Qwen via sequential start). On this box, alpine 768×1344×124 decoded in 2.4 s versus 68 s for the full VAE, and one T2VA generation finished in 224 s end-to-end. Reconstruction is approximate, not lossless. FL2VA/Ref2VA still need the full VAE to encode references.

• Two Sparks, one clip. Sequence parallel (sp_size=2) over the QSFP RoCE link ran one 768×1344×124 FastH3 recipe in 292 s vs 374–393 s on one GB10, and a 345-frame (~14.4 s) clip in 587 s. Other heights, widths, and frame counts are valid. Weights stay replicated, so lazy module load (auto on GB10) is still required on each box. Bring-up and knobs: Pair two NVIDIA DGX Sparks.

• A worker's SIGTERM log and traceback show where it was interrupted, not why it was selected; confirm the cause in the earlyoom service or system logs. A later SIGKILL or kernel OOM kill cannot be caught and reported by Python.

Gotchas specific to the GB10

A few things that surprise people on this box (beyond the memory notes above):

• Don't force TORCH_SDPA on a VSA checkpoint (FastWan, LTX2.3-distilled). The SDPA path builds a model without the gate weights the checkpoint carries and fails to load. Run the model natively — VSA auto-routes to its Triton kernel on sm_121.
• Few-step timings are noisy run-to-run (~±30%) — one-time startup dominates a 3-step run. Compare in-process / as medians, never two separate single runs (the benchmark script does this).
• nvidia-smi shows [N/A] for memory — see Memory.
• Cosmos-2.5 uses a Qwen2.5-VL text encoder; make sure you're on a FastVideo build recent enough to include its transformers-compatibility handling before running it.
• MiniMax H3 worker init can look healthy and still die on the first generate if deferred loading is off (--no-lazy-module-load and sequential also off) and encoder, VAE, and DiT load together. On GB10 the log should show lazy_module_load owns deferral (or, if lazy is off, sequential Released MiniMax-H3 text encoder after conditioning before Loading MiniMax-H3 denoise modules).

FastH3 V2 NVFP4 on one Spark

This recipe uses the full V2 eight-forward transformer, the 50-layer NVFP4 Qwen3-VL encoder, and the light H3 video VAE. Its configuration keeps all three resident on one GB10. This stack fits in the Spark's unified memory; benchmark your installed runtime and review the clips before publishing a speed claim. The earlier bf16 H3 memory guidance above concerns a larger checkpoint.

Install FastVideo following the Spark install guide. The released repositories are complete inference stacks:

Model	Repository	Packed DiT profile
FastH3 V2	FastVideo/FastVideo-FastH3-8-Step-V2-NVFP4-Consumer	h3_dit_vsa
FastH3 Trim	FastVideo/FastVideo-FastH3-Trim-8-Step-NVFP4	h3_dit_vsa

Each ships its own trained schedule, NVFP4 transformer and 50-layer NVFP4 text encoder, lightweight 26-layer video VAE with the INT8-weight overlay, and audio VAE. Download the complete repository; the runtime selects these components from its model index. The released Trim transformer also packs attention and VSA gates, unlike the earlier FFN-only pruned export. Neither release requires local checkpoint conversion or a separate encoder download.

hf download FastVideo/FastVideo-FastH3-8-Step-V2-NVFP4-Consumer
hf download FastVideo/FastVideo-FastH3-Trim-8-Step-NVFP4

Keep fastvideo_inference.json with the transformer if you stage the stack in a local directory. It declares the trained eight-forward ladder and video/audio shifts of 10/3. The recipes use num_inference_steps: 9 for nine sigma points and eight DiT forwards.

On GB10 (FlashInfer 0.6.18 and 0.7.0), FastVideo completes each NVFP4 activation quantization on the host before its GEMM. Without this completion fence, identical H3 requests produced different DiT latents and occasionally corrupt video. The fence costs no measurable denoise time, because the GPU stays saturated. It applies to sm_121; other architectures retain asynchronous execution.

Run examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml from the repository root. It uses 832x480, 124 frames and seed 1234, VSA sparsity 0.8 with 64-token tiles, and the light H3 VAE through the h3-vae decode backend. It does not use frame dropping or spatial upscaling.

FASTVIDEO_H3_VSA_FP4=1 \
FASTVIDEO_MINIMAX_H3_FUSIONS=all \
FASTVIDEO_NVFP4_MM_BACKEND=cudnn FASTVIDEO_NVFP4_MM_ROW_SPLIT=table \
FASTVIDEO_H3_VAE_TILE_BATCH=12 FASTVIDEO_H3_VAE_INT8_OVERLAY=0 \
FASTVIDEO_VSA_TRITON=1 FASTVIDEO_VSA_PTX_BLACKWELL=0 FASTVIDEO_FA4=0 \
FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 \
FASTVIDEO_STAGE_LOGGING=1 \
nice -n 19 fastvideo generate \
  --config examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml

The defaults produce a roughly five-second clip at 24 fps. For native 768p, set --request.sampling.width 1344 and --request.sampling.height 768, keeping 124 frames. For the separate ten-second setting, use 243 frames. H3 permits frame counts of 17n+5; 124 is the nearest legal count above five seconds and 243 is the nearest above ten seconds.

For Trim, use basic_fasth3_spark_pruned_nvfp4.yaml with the same environment. Both recipes keep the encoder, DiT and VAEs resident, compile only the VAE decoder, and use h3_dit_vsa. On two Sparks, use the corresponding basic_fasth3_spark_pair_{pruned,v2}_nvfp4.yaml after following the pair setup guide. The pair uses SP2/TP1 and parallel VAE gathering, with tile batch 1 on each worker.

For release timing, create one generator per model/resolution. Run one untimed ceramics warmup, then two timed ceramics calls and two timed harbor calls in that same process, using the exact release prompt strings, seed 1234 and the settings above. Measure each generate() call through finished MP4 output and report the median of the two timed calls per prompt. Keep the warmup excluded. Record the model revision, code commit, command, environment and peak-memory scope with the results. Review every clip's video and audio before publishing a quality or speed claim.

The benchmark helper requires a local stack so it can validate the schedule before loading. For example:

hf download FastVideo/FastVideo-FastH3-8-Step-V2-NVFP4-Consumer \
  --local-dir ./FastH3-V2-Consumer
python examples/inference/basic/benchmark_fasth3_spark_nvfp4.py \
  --config examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml \
  --model-path ./FastH3-V2-Consumer --frames 124 \
  --prompts /path/to/benchmark_prompts.json \
  --output-dir outputs/fasth3_spark_v2_nvfp4/benchmark-124

Set the environment from the generation command above before benchmarking. The prompt JSON must contain latency-ceramics-005 and latency-harbor-005. Pass --width 1344 --height 768 for the native 768p protocol. Use the Trim repository and config for its corresponding run.

FastH3 decode, GEMMs and attention on GB10

The recipes decode with the light VAE's dense weights (FASTVIDEO_H3_VAE_INT8_OVERLAY=0), 12 spatial tiles per decoder call and a compiled decoder (compile.vae_enabled: true). The INT8 ConvRot overlay that ships next to the VAE is slower on GB10 and stops tile batching from helping. Decoder-only timings on identical 832x480, 124-frame latents (one GB10):

Light VAE decoder	1 tile per call	12 tiles per call	PSNR vs FP32 decode
INT8 ConvRot overlay, fp16 autocast	42.2 s	43.5 s	63.4 dB
Dense, fp16 autocast	24.4 s	14.4 s	77.5 dB
Dense, fp16 autocast, compiled	21.6 s	9.8 s	77.7 dB

The first one or two calls in a process include decoder compiles; later calls in the same process reuse them.

NVFP4 linears use FlashInfer's cuDNN backend (FASTVIDEO_NVFP4_MM_BACKEND=cudnn). The CUTLASS backend used on RTX Blackwell is much slower on GB10 for the FFN projections, and both backends return identical outputs. FlashInfer mm_fp4 with 19,392 activation rows (FastH3 at 832x480, 124 frames):

Projection (K→N)	cutlass	cudnn	auto
attention q/k/v, gate (5376→7168)	5.5 ms	10.0 ms	10.0 ms
attention out (7168→5376)	5.5 ms	4.6 ms	8.3 ms
FFN in (5376→28672)	52.9 ms	19.6 ms	20.0 ms
FFN out (14336→5376)	26.6 ms	8.9 ms	9.0 ms

FASTVIDEO_H3_VSA_FP4=1 runs VSA-H3's block-sparse attention on the FP4 SageAttention3 kernel, as on RTX Blackwell. It needs fastvideo-kernel built for 12.1a on GB10 or 12.0a on sm_120. A missing extension raises an error; select BF16 VSA explicitly by setting h3_vsa_fp4.enabled: false in the YAML. On a captured FastH3 Trim attention input (19,392 tokens, 56 heads, 64-token tiles) it runs in 29 ms against 59 ms for Triton VSA, with a relative L2 error of 0.13 versus an FP32 reference (0.002 for Triton BF16).

The FastH3-QAD-v2 recipes, basic_fasth3_spark_qad_v2_nvfp4.yaml and basic_fasth3_spark_pair_qad_v2_nvfp4.yaml, select FastVideo/FastH3-QAD-v2 explicitly. The older v2 files continue to select V2. Both recipes enable FP4 in YAML through pipeline.experimental.h3_vsa_fp4.enabled; shell flags are not required for that selection. FastH3-QAD-v2 keeps its 64-token tiles and sparsity 0.8, and protects text and audio query rows with FP8 Q/K and BF16 dense attention. These are inference settings. The published FastH3-QAD-v2 checkpoint used linears-only QAD with BF16 attention, so this does not reproduce its training attention numerics.

The earlier unprotected FP4 path is faster (on a 5090, 480p5 8L measured about 12.6 s versus 19 s on bf16 VSA) but changes the output. On V2 at 832x480 with four prompts, denoising was 24-33% faster (60-68 s to 43-51 s) and the clips scored LPIPS 0.39-0.55 against the BF16-attention clips. Validate clips before a quality comparison. On a 32 GB RTX 5090 the measured recipe also sets FASTVIDEO_H3_ENCODER_LAYERWISE=1 and FASTVIDEO_H3_PARK_MODULES=vae,audio_vae.

Measured on one Spark (832x480, 124 frames, seed 1234, the ceramics release prompt), timing warm generate() calls in one process through finished MP4 output:

Model and configuration	End to end	Denoise	Video decode
Trim, previous recipe (CUTLASS, INT8 overlay, 1 tile per call)	121-122 s	77-78 s	43 s
Trim, current recipe	68 s	58 s	9.5 s
Trim, current recipe + FASTVIDEO_H3_VSA_FP4=1	54 s	43 s	9.6 s
V2, previous recipe	139-141 s	95-96 s	42-43 s
V2, current recipe	79 s	68 s	9.6 s

Within each configuration, repeated calls produced bit-identical frames except the previous V2 recipe, whose repeated calls diverged.

Row splitting is off globally. The single-Spark cuDNN recipe opts into FASTVIDEO_NVFP4_MM_ROW_SPLIT=table for the shapes measured on GB10. Leave it off on other GPUs unless a matched benchmark validates the split there.

Optional NVFP4 video decoder

FASTVIDEO_H3_VAE_NVFP4_DECODER=/path/to/decoder.pt swaps in an NVFP4 decoder distilled with quantization-aware training and exported by scripts/distill/minimax_h3_nvfp4_decoder/export_deploy.py. At load the INT8 overlay is skipped, the VAE is depth-cut to the decoder's block count (the 26-layer light VAE hosts an 8-block decoder; a 36-block decoder needs the full MiniMaxAI/MiniMax-H3 vae/ in the stack's vae/ folder), and only the packed FP4 weights stay resident (0.31 GiB for 8 blocks instead of 2.3 GiB). The decode stage runs bf16 autocast for these decoders, the precision they were trained in. The decoder GEMMs use the cuDNN backend on GB10; outputs match CUTLASS bit for bit.

The variable also accepts .safetensors files written by fastvideo.models.vaes.minimax_h3_nvfp4_checkpoint: either the same bf16 master weights, or pre-packed NVFP4 weights from scripts/distill/minimax_h3_nvfp4_decoder/export_packed.py. A packed file is read straight onto the GPU and never materializes the full-precision weights of the quantized linears; it decodes bit-identically to packing the bf16 file at load.

V2 at 832x480, 124 frames, seed 1234, one Spark, warm calls:

Video decoder	End to end	Denoise	Video decode	Peak allocated
Light VAE, INT8 overlay, 1 tile per call, eager	112.4 s	68.4 s	42.7 s	59.4 GiB
Light VAE, dense, 12 tiles per call, compiled (recipe)	78.9 s	68.1 s	9.6 s	64.3 GiB
8-block NVFP4 decoder	70.9 s	68.2 s	1.6 s	58.0 GiB

The 8-block decoder decodes the same with or without compile.vae_enabled and does not need tile batching. Its output is lossy against the light VAE (31 dB PSNR on this clip); review clips before using it.

Released model measurements

The released Trim stack at revision cae9ceb6feefe77d34a56640782cda3909363f19 completed the native 832x480, 124-frame protocol on one Spark, seed 1234. The tested code is 6ccdbc761e6b854003f472e39b826c06cb54de60, using the resident recipe above. Medians exclude warmups and cover finished MP4 output.

Released model	Resolution	One Spark, ceramics / harbor	Two Sparks
FastH3 Trim NVFP4	832x480	124.090 / 127.519 s	Pending
FastH3 Trim NVFP4	1344x768	Review pending	Pending
FastH3 V2 NVFP4-Consumer	832x480	Repeatability review pending	Pending
FastH3 V2 NVFP4-Consumer	1344x768	Review pending	Pending

The completed Trim 480p batch used one extra untimed harbor warmup in the same process, six calls total. Both warmups are excluded from the medians. Each prompt's three decoded videos and audio streams match exactly, and sampled frames are coherent. These checks do not establish BF16 parity, speech accuracy or lip sync. Pending cells are not measured substitutes from older checkpoints.

Reproduce these numbers

Two scripts under examples/inference/optimizations/ reproduce the claims on your own GB10:

# Headline: few-step generation timing (median) + the bf16-vs-fp32 decode A/B.
# FASTVIDEO_STAGE_LOGGING=1 also prints the denoise / decode / text split.
FASTVIDEO_STAGE_LOGGING=1 nice -n 19 \
    python examples/inference/optimizations/spark_benchmark.py

# FP4 attention quality/speed A/B on the QAD checkpoint (one arm per run).
QAD_LINEAR=0 FASTVIDEO_ATTENTION_BACKEND=ATTN_QAT_INFER nice -n 19 \
    python examples/inference/optimizations/qad_fp4_ab.py

See also the Optimizations reference for the full list of attention backends and quantization options.