Add the OmniRef (Ref2VA, VSA tile 128) and V2/V2.1 Consumer (T2VA,
VSA tile 64) profiles of one quantization-aware distillation recipe:
NVFP4 W4A4 q/k/v/out and FFN linears in all 50 blocks, the FP4
sparse-attention emulator, frozen VSA gates and norms, output plus
layer-reconstruction regression against a frozen bf16 teacher, and
best-by-score checkpoints with audio eligibility guards.
Training code: the PDD-QAD method, the two H3 model plugins, the
NVFP4 emulated linear, and the tile-64/128 Triton emulator with its
PyTorch numeric reference. The VSA attention impl gains a fine
attention override hook and lets tile 128 run grad-capable forwards
through it. The trainer gains opt-in tagged checkpoints and early
stop hooks for the QAD guards.
Also add two runnable configs, the training guide, and CPU/GPU tests.
For Wan VSA inference with 64-token tiles, each attention layer scattered Q, K,
V and the gate into a tile buffer and then made four transposed copies. On
SM100, one Triton kernel (tile_to_bhsd) now does the scatter and the transposes
in one pass. Training, 256-token tiles, other GPUs and builds without the kernel
use the previous path. FASTVIDEO_DISABLE_VSA64_FUSED_LAYOUT=1 turns the fused
path off.
On GB200 the layout and attention step goes from 15.7 ms to 8.6 ms. FastWan 1.3B
frames are bit-exact with the previous path at sequence parallel 1 and 2.
On SM100, the Attn-QAT training kernel now uses a faster route for non-causal
BF16 attention with equal query and key lengths of 16384 tokens or more. The
forward pass processes full KV tiles without bounds masks, and a separate masked
step handles the partial tail tile. The backward pass uses 64x128 dQ and dK/dV
tiles. Outputs, softmax statistics and all gradients are bitwise identical to the
previous route. FASTVIDEO_ATTN_QAT_SM100_OPTIMIZED=0 turns off the SM100 route,
and FASTVIDEO_ATTN_QAT_SM100_WIDE_BWD=0 keeps the 64x64 backward tiles.
On GB200 with 6 heads and 31200 tokens, the forward kernel time goes from 9.14 ms
to 6.96 ms and the backward kernel time from 17.41 ms to 15.85 ms.
Aryan Kumar