The audio VAE decode (~95 ms) only depends on the audio latents, which are
final when denoising returns, but it ran serially after the ~2 s video VAE
decode. FASTVIDEO_H3_OVERLAP_AUDIO_DECODE starts it on a second CUDA stream
when the video decoding stage begins and merges the waveform back afterwards.
The audio stage mutates the batch (it clears its runtime tensors), so the
thread works on a shallow copy with its own extra dict; errors are re-raised
on the main thread and the stage's deferred modules are released as
PipelineStage.__call__ would.
Measured on the stage bodies with the real video and audio VAEs (768p, 107
frames, GB200, interleaved, 3 rounds): 8.294 s -> 8.224 s (-70 ms of a 95 ms
audio decode, 74% hidden); the audio waveform is bitwise identical. The
remaining cost is power/clock contention between the two streams.
FASTVIDEO_H3_SKIP_FP32_SAMPLES_MIRROR additionally drops the fp32 [0, 1]
samples mirror that generate() builds for return_frames=True requests even
though the frames are produced from the uint8 output. It costs 116 ms and
4.18 GiB of host traffic for a 362-frame clip; with the flag result["samples"]
is None for those requests.
Found with MotionKernel (agent-driven kernel search)
The H3 dense self-attention call always ran on FlashAttention. On GB200,
cuDNN's fused SDPA engine is a few percent faster in-model at 37k-109k
tokens, but the ATen op has no shape metadata: it always reads inputs as
[B, H, S, D], so handing it contiguous [B, S, H, D] is silently misread.
Add FASTVIDEO_MINIMAX_H3_DENSE_ATTN=flash|cudnn (default flash) and route
the dense, unmasked, inference-only call through a helper that presents
BHSD strided views. Training calls, compiled regions, head sizes >128,
non-contiguous inputs, VSA blocks and NVFP4 stay on their existing paths.
FA4_* tuning and the default behaviour are unchanged.
The 36 H3 VAE decoder transformer blocks pay ~1.1 s per 768p video in separate
elementwise passes: an fp32 RMSNorm kernel plus an activation-dtype cast,
attn * scale1 plus the fp32 residual add, silu plus mul for the SwiGLU, and
ff * scale2 plus the second residual add. FASTVIDEO_H3_VAE_COMPILE_BLOCKS
routes the block loop through one regional torch.compile graph (default
Inductor mode, no CUDA graphs - the decode is device-busy), where Inductor
fuses each pair into a single pass.
The compiled region takes the block as an argument instead of wrapping
block.forward per block, so Dynamo traces one block-sized graph that every
block and temporal chunk replays.
Decode-only measurements on GB200 (768x1344, tile batch 8, interleaved, warm,
median of 3):
107 frames: 3.65 s -> 1.97 s (-46.1%), peak 16.1 -> 11.2 GiB
362 frames: 12.75 s -> 6.91 s (-45.8%), peak 16.2 -> 11.3 GiB
one-time compile: 8.9 s for the first shape (cold Inductor cache, ~5.8 MB)
Parity: Inductor only fuses pointwise passes, so the fp32 arithmetic is
unchanged; decoded 8-bit frames differ by at most two code values
(max |d| = 2/255, PSNR 72.1 dB, 99.6% identical).
Profiler interaction: with the profiler's default settings the flag does not
hang (collect 4.6 s, export 0.35 s). The hang reproduces with stack capture
(FASTVIDEO_TORCH_PROFILER_WITH_STACK=1): prof.key_averages() then does not
return in 10 minutes on the compiled run (eager: 43 s) and raises
RecursionError at the default limit, because the Inductor CPU frame tree is
much deeper. That is profiler post-processing, not the decode.
Found with MotionKernel (agent-driven kernel search)
BaseLayerWithLoRA.forward was decorated with torch.compile, so wrapping a linear also compiled its base layer.
Compiling a packed NVFP4/FP8 linear on its own changes its numerics, so a wrapped layer at strength 0 could differ
from the unwrapped layer. forward now adds no compiled region: the base layer runs exactly as it would unwrapped
(eagerly, or traced with a caller's compiled block), and only the adapter delta is compiled.
The unit lane fails on main in two contract tests. All three failures are
bookkeeping that drifted from the code, not runtime bugs:
- `test_env_policy.py::test_env_doc_table_matches_registry`: the generated
table in docs/contributing/env_vars.md is stale for
FASTVIDEO_MLX_QUANT_GEMM. Regenerated with the documented command
(`python fastvideo/tests/contract/test_env_policy.py`).
- `test_env_policy.py::test_env_access_follows_policy`: two new violations.
- fastvideo/tests/mlx/test_mlx_affine_dq_gemm.py wrote
FASTVIDEO_MLX_QUANT_GEMM through monkeypatch.setenv; now uses the
registered field's override() through the env_overrides fixture.
- fastvideo/tests/train/models/test_load_kandinsky5.py read
FASTVIDEO_ATTENTION_BACKEND with `not in os.environ`; now uses
envs.FASTVIDEO_ATTENTION_BACKEND.is_set() and override(). Its entry in
KNOWN_VIOLATIONS is deleted, as the shrinking-list rule requires.
- `test_schema_parity_inventory.py::..._extension_fields_are_classified`:
MiniMaxH3PipelineConfig declares video_scheduler_shift and
audio_scheduler_shift (training-time noise clocks rejected by
check_pipeline_config at inference); both are added to
pipeline_config_extensions.internal_only.
Test plan (CPU, no GPU required):
python -m pytest fastvideo/tests/contract/test_env_policy.py \
fastvideo/tests/api/test_schema_parity_inventory.py \
fastvideo/tests/mlx/test_mlx_affine_dq_gemm.py -q
# 45 passed
python -m pytest fastvideo/tests/contract/ fastvideo/tests/api/ -q -m "not gpu"
# 501 passed
Aryan Kumar