[docs] DGX Spark (GB10) performance & tuning guide + reproduction examples (#1631)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
This commit is contained in:
Raghav K
co-authored by
SolitaryThinker
SolitaryThinker
parent
b963a24612
commit
a2bfc7cdb2
@@ -5,6 +5,7 @@ FastVideo supports the following hardware platforms:
|
||||
|
||||
- [NVIDIA CUDA](installation/gpu.md)
|
||||
- [NVIDIA DGX Spark / GB10 (ARM64 + CUDA 13)](installation/spark.md)
|
||||
([performance & tuning](installation/spark_performance.md))
|
||||
- [Apple silicon](installation/mps.md)
|
||||
|
||||
## Quick Installation
|
||||
|
||||
@@ -137,6 +137,13 @@ If you hit other issues, please open an issue on our
|
||||
our [Slack community](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ)
|
||||
for additional support.
|
||||
|
||||
## Next: performance & tuning
|
||||
|
||||
Installed and verified? See [DGX Spark: Performance & Tuning](spark_performance.md)
|
||||
for which models are practical on the GB10, what makes them faster, and what
|
||||
won't help on this hardware (and why) — so you don't spend a night tuning knobs
|
||||
that can't move here.
|
||||
|
||||
## Development Environment Setup
|
||||
|
||||
If you're planning to contribute to FastVideo please see the
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
# DGX Spark (GB10): Performance & Tuning
|
||||
|
||||
You have FastVideo [installed on a DGX Spark](spark.md) — this page is what to
|
||||
run next. It covers **which models are practical on the GB10, what actually
|
||||
makes them faster, and what won't help (and why)**, so you don't burn a night
|
||||
tuning knobs that can't move on this hardware.
|
||||
|
||||
!!! tip "TL;DR"
|
||||
- **Use distilled few-step models** (e.g. `FastVideo/FastWan2.1-T2V-1.3B-Diffusers`).
|
||||
They run in ~40 s/video. Full-step models are 12–47 min on the GB10.
|
||||
- On few-step models, **VAE decode is the bottleneck**, not attention — it's
|
||||
bandwidth-bound on the Spark's unified memory.
|
||||
- **bf16 VAE decode** is the real, lossless lever (FastVideo already turns it
|
||||
on for Wan). **FlashAttention, linear quantization, and `torch.compile` of
|
||||
the VAE give little or nothing here** — see the table below.
|
||||
- Heavy runs can make the box unreachable — run generations with VAE tiling on
|
||||
and `nice -n 19`. See [Running safely](#running-safely-dont-lock-the-box).
|
||||
|
||||
## The hardware reality (this explains everything below)
|
||||
|
||||
The GB10 pairs a Blackwell GPU (`sm_121`) with **128 GB of unified LPDDR5X memory
|
||||
(~270 GB/s) shared between CPU and GPU**. That bandwidth is roughly **10× below a
|
||||
datacenter GPU's HBM**. Two consequences drive every tuning decision:
|
||||
|
||||
1. **Memory-bandwidth-bound stages hurt disproportionately.** VAE decode moves a
|
||||
lot of data and becomes the dominant cost on short (few-step) generations.
|
||||
2. **Compute-bound stages scale with step count.** Full-step diffusion (50+
|
||||
steps) is denoise-bound and simply takes a long time here.
|
||||
|
||||
## Use distilled few-step models
|
||||
|
||||
The single biggest lever on the GB10 is **model choice**. A 3-step distilled
|
||||
model is ~18× faster than the full-step version of the same architecture:
|
||||
|
||||
| Model | Steps | Time / video | Bottleneck |
|
||||
|---|---|---|---|
|
||||
| FastWan2.1-T2V-1.3B (distilled) | 3 | **~40 s** | VAE decode |
|
||||
| Wan2.1-T2V-1.3B (full-step) | 50 | ~12 min | denoise |
|
||||
| Cosmos-Predict2.5-2B (full-step) | 51 | ~47 min | denoise |
|
||||
| LTX2.3-distilled (+audio) | 8 | ~6 min | mixed |
|
||||
|
||||
The bottleneck flips from decode to denoise at around **4 steps**. Below that,
|
||||
you're paying mostly for VAE decode; above it, mostly for the denoising loop.
|
||||
|
||||
!!! note "Few-step timings are noisy — measure in-process"
|
||||
On a 3-step run, one-time per-process startup (Triton autotune, allocator
|
||||
warmup) dominates and never amortizes, so single-run totals wobble ~±30%.
|
||||
Compare levers **back-to-back in one process or as medians**, never as two
|
||||
separate single runs. The [reproduction script](#reproduce-these-numbers)
|
||||
does this for you.
|
||||
|
||||
## bf16 VAE decode — the real lever (already on for Wan)
|
||||
|
||||
Because few-step generation is decode-bound, VAE decode precision is where the
|
||||
time is. Decoding in **bf16 instead of fp32 is essentially lossless** (MS-SSIM
|
||||
~0.9999 vs fp32 on the identical latent) and ~1.14× faster — worth roughly
|
||||
5–7% end-to-end on a decode-bound few-step model.
|
||||
|
||||
**FastVideo already defaults Wan's decode to bf16** (`vae_decode_precision="bf16"`,
|
||||
with encode kept at fp32), so for the recommended Wan/FastWan models there's
|
||||
nothing to set. If you run a model that still defaults to an fp32 decode, set the
|
||||
decode-only override yourself:
|
||||
|
||||
```python
|
||||
from fastvideo.configs.pipelines.base import PipelineConfig
|
||||
|
||||
pipeline_config = PipelineConfig.from_pretrained(model_id)
|
||||
pipeline_config.vae_decode_precision = "bf16" # decode-only; leaves encode precision alone
|
||||
```
|
||||
|
||||
Decode is output-only, so lowering its precision is safe. (Encode seeds the
|
||||
denoising trajectory for I2V/causal models, so that stays at the pipeline's
|
||||
default — don't lower `vae_precision` blindly for those.)
|
||||
|
||||
## Memory: one unified 128 GB pool
|
||||
|
||||
The GB10 has **no separate VRAM** — CPU and GPU share one 128 GB LPDDR5X pool
|
||||
(~118 GB usable). Two practical consequences:
|
||||
|
||||
- **`nvidia-smi` reports memory as `[N/A]`** on the GB10, and the system "used"
|
||||
figure conflates CPU + GPU + cache, so it's only a soft upper bound — treat the
|
||||
whole 128 GB as one shared budget. For a per-run figure, use FastVideo's own
|
||||
`peak_memory_mb` (reported on the generation result and by the performance
|
||||
benchmark), which is measured inside the worker that runs the model.
|
||||
- **The 128 GB is a *working-set* ceiling, not storage** — the model cache lives
|
||||
on the NVMe (3.7 TB, ample). What has to fit in 128 GB is the weights,
|
||||
activations, and KV cache — and, critically, the **VAE decode buffers**, which
|
||||
is why tiling matters
|
||||
(an untiled high-res decode can spike the pool into swap and lock the box).
|
||||
|
||||
The recommended few-step models are comfortable here: their weights are small
|
||||
(1.3–2 B) and few-step generation keeps activations modest — a Wan2.1-1.3B
|
||||
few-step generation peaks at **~8.4 GB** (measured), a small fraction of the pool.
|
||||
The pressure comes from **decode resolution/frames**, not the model — a
|
||||
1080p×121-frame untiled decode is what pushes the pool toward its ceiling, which
|
||||
is why VAE tiling stays
|
||||
on by default.
|
||||
|
||||
## What helps vs. what doesn't on the GB10
|
||||
|
||||
The honest summary — most "obvious" GPU optimizations don't move the needle on
|
||||
this hardware, for reasons specific to it:
|
||||
|
||||
| Lever | Effect on the GB10 | Use it? |
|
||||
|---|---|---|
|
||||
| Distilled few-step model | ~18× vs full-step | ✅ **the primary lever** |
|
||||
| bf16 VAE decode | ~1.14×, lossless; ~5–7% e2e on few-step | ✅ default for Wan |
|
||||
| VSA (video sparse attention) | works out of the box (Triton kernel auto-selects on `sm_121`) | ✅ automatic |
|
||||
| Building FlashAttention | **no speedup** — Torch SDPA already hits an efficient flash kernel on `sm_121`, and FA2 ties it | ❌ not worth building |
|
||||
| `torch.compile` of the VAE decode | recompile storm (per-frame varying shapes) → ~1.1× | ❌ dead end |
|
||||
| Linear (fp8 / nvfp4) quantization on long-sequence models (e.g. Cosmos) | ~nothing — see below | ❌ wrong lever here |
|
||||
| FP4 attention (`ATTN_QAT_INFER`) | works on `sm_121` (runtime allowlist landed in #1647; kernel build is #1598); helps, but needs a QAT-trained checkpoint | ⚠️ opt-in — see below |
|
||||
| FP4 linear on short-sequence models (LTX2) | up to −24% denoise at 1080p (#1594) | ⚠️ model/resolution-dependent |
|
||||
|
||||
### Why linear quantization is the wrong lever on long-sequence models
|
||||
|
||||
Quantizing the linear (GEMM) layers is a natural first instinct, but on a
|
||||
long-sequence video model it buys almost nothing on the GB10. A video-DiT denoise
|
||||
step is dominated by **O(N²) attention** at these sequence lengths (tens of
|
||||
thousands of tokens); the linear layers are a small single-digit fraction of the
|
||||
work. Quantizing them faster leaves the attention-bound total essentially
|
||||
unchanged — measured at ~1% on Cosmos-2.5, i.e. noise, and full-step CFG models
|
||||
also lose quality to per-step quantization error.
|
||||
|
||||
The same mechanism **does** help on **short-sequence** models: LTX2's aggressive
|
||||
VAE compression gives it short attention sequences, so FP4 linear reaches −24%
|
||||
there (#1594). The rule: **on the GB10, the lever that matters is attention
|
||||
(sparse or FP4), not the linear layers** — unless the model has short sequences.
|
||||
|
||||
### FP4 on the GB10 (opt-in)
|
||||
|
||||
Block-scaled FP4 works on `sm_121` under CUDA 13:
|
||||
|
||||
- **FP4 attention** (`FASTVIDEO_ATTENTION_BACKEND=ATTN_QAT_INFER`, #1598) is
|
||||
numerically correct on the GB10 and ~6% faster end-to-end generation, but it only preserves
|
||||
quality on a **quantization-aware-distilled checkpoint** (e.g.
|
||||
`FastVideo/FastWan-QAD-1.3B`) — stock weights aren't trained to tolerate it.
|
||||
- **FP4 linear** helps only where sequences are short (LTX2, above).
|
||||
|
||||
The [`qad_fp4_ab.py`](#reproduce-these-numbers) harness reproduces the FP4
|
||||
attention A/B on the QAD checkpoint.
|
||||
|
||||
## Running safely (don't lock the box)
|
||||
|
||||
The GB10 is easy to make **unreachable** — a heavy build or an untiled high-res
|
||||
decode starves the ~20 ARM cores and unified memory, `sshd` can't get cycles, and
|
||||
you're locked out at *"Connection timed out during banner exchange"* until the box
|
||||
is power-cycled. To avoid it:
|
||||
|
||||
- **Inference:** keep **VAE tiling on** (the default), use sane resolution/frames,
|
||||
and run under `nice -n 19`:
|
||||
|
||||
```bash
|
||||
nice -n 19 nohup python your_script.py > run.log 2>&1 &
|
||||
```
|
||||
|
||||
- **Builds** (flash-attn, kernel): `nice -n 19`, `MAX_JOBS=2`, `nohup`. Never a
|
||||
bare foreground high-parallelism build.
|
||||
- Leave `*_cpu_offload` at the example defaults — "CPU" offload is the *same*
|
||||
unified RAM on the GB10, so the win is tiling + sane resolution, not offloading.
|
||||
|
||||
## Gotchas specific to the GB10
|
||||
|
||||
A few things that surprise people on this box (beyond the memory notes above):
|
||||
|
||||
- **Don't force `TORCH_SDPA` on a VSA checkpoint** (FastWan, LTX2.3-distilled).
|
||||
The SDPA path builds a model without the gate weights the checkpoint carries and
|
||||
fails to load. Run the model natively — VSA auto-routes to its Triton kernel on
|
||||
`sm_121`.
|
||||
- **Few-step timings are noisy run-to-run** (~±30%) — one-time startup dominates a
|
||||
3-step run. Compare in-process / as medians, never two separate single runs (the
|
||||
benchmark script does this).
|
||||
- **`nvidia-smi` shows `[N/A]` for memory** — see [Memory](#memory-one-unified-128-gb-pool).
|
||||
- **Cosmos-2.5** uses a Qwen2.5-VL text encoder; make sure you're on a FastVideo
|
||||
build recent enough to include its `transformers`-compatibility handling before
|
||||
running it.
|
||||
|
||||
## Reproduce these numbers
|
||||
|
||||
Two scripts under `examples/inference/optimizations/` reproduce the claims on
|
||||
your own GB10:
|
||||
|
||||
```bash
|
||||
# Headline: few-step generation timing (median) + the bf16-vs-fp32 decode A/B.
|
||||
# FASTVIDEO_STAGE_LOGGING=1 also prints the denoise / decode / text split.
|
||||
FASTVIDEO_STAGE_LOGGING=1 nice -n 19 \
|
||||
python examples/inference/optimizations/spark_benchmark.py
|
||||
|
||||
# FP4 attention quality/speed A/B on the QAD checkpoint (one arm per run).
|
||||
QAD_LINEAR=0 FASTVIDEO_ATTENTION_BACKEND=ATTN_QAT_INFER nice -n 19 \
|
||||
python examples/inference/optimizations/qad_fp4_ab.py
|
||||
```
|
||||
|
||||
See also the [Optimizations](../../inference/optimizations.md) reference for the
|
||||
full list of attention backends and quantization options.
|
||||
Reference in New Issue
Block a user