Resolve the FP4 attention kernel and backend conflicts with main's #1980 versions
(re-adding install_fp4_vsa_attention), keep main's FP32 FSDP grouping, and keep both
save_inference and save_tagged in the checkpointer.
- Allow the frozen FP32 boundary modules through the FSDP mixed-dtype
guard when a model declares replicate_fp32_modules.
- Make the tile-64 FP4 kernel and its reference handle a listed block
with an empty query half instead of propagating NaN; add a CPU
regression test.
- Compare generated row ids in the row-plan test.
- Consume method checkpoint/stop requests after the pre-loop eval, so
best/ always refers to the evaluated weights.
- Remove the unused audio-stats module and its claim.
- Refresh the V2 model description and drop the dead compression script
reference.
- V2 config now matches the canonical 16-GPU stage 1: warmup 10,
min LR ratio 0.3, HSDP 16.
- State that the published V2.1 checkpoint is linears-only (BF16 VSA
attention); the FP4 adaptation is a separate, non-public stage.
- Name the released checkpoints and starting models.
- Vendor and pin both calibrated amax tables.
- Add the V2 prompt/embedding and OmniRef eval data-prep scripts.
- Include V2 width/height/num_frames and the train manifest in the step-0
fingerprint so a dimension change cannot reuse a stale baseline.
- Honor CompressionSpec.first_steps in the V2 rung scope.
- Mark tagged checkpoints weights-only in metadata and teach the exporter
to resolve and load them, so best/ can be exported.
- Add a tile-64 backward comparison against the STE reference.
Add the OmniRef (Ref2VA, VSA tile 128) and V2/V2.1 Consumer (T2VA,
VSA tile 64) profiles of one quantization-aware distillation recipe:
NVFP4 W4A4 q/k/v/out and FFN linears in all 50 blocks, the FP4
sparse-attention emulator, frozen VSA gates and norms, output plus
layer-reconstruction regression against a frozen bf16 teacher, and
best-by-score checkpoints with audio eligibility guards.
Training code: the PDD-QAD method, the two H3 model plugins, the
NVFP4 emulated linear, and the tile-64/128 Triton emulator with its
PyTorch numeric reference. The VSA attention impl gains a fine
attention override hook and lets tile 128 run grad-capable forwards
through it. The trainer gains opt-in tagged checkpoints and early
stop hooks for the QAD guards.
Also add two runnable configs, the training guide, and CPU/GPU tests.
On SM100, the Attn-QAT training kernel now uses a faster route for non-causal
BF16 attention with equal query and key lengths of 16384 tokens or more. The
forward pass processes full KV tiles without bounds masks, and a separate masked
step handles the partial tail tile. The backward pass uses 64x128 dQ and dK/dV
tiles. Outputs, softmax statistics and all gradients are bitwise identical to the
previous route. FASTVIDEO_ATTN_QAT_SM100_OPTIMIZED=0 turns off the SM100 route,
and FASTVIDEO_ATTN_QAT_SM100_WIDE_BWD=0 keeps the 64x64 backward tiles.
On GB200 with 6 heads and 31200 tokens, the forward kernel time goes from 9.14 ms
to 6.96 ms and the backward kernel time from 17.41 ms to 15.85 ms.
Adds opt-in regional torch.compile to the YAML-driven modular Wan trainer (training.model.enable_torch_compile / torch_compile_kwargs) with FA4 and Wan modulation forward/backward as opaque custom ops, compiled repeated DiT blocks after FSDP setup, and activation checkpointing applied before FSDP for trainable Wan transformers.
Merged with main's #1716 (WanModel resolves the checkpointing type in __init__; the PR's pre-FSDP wrap uses it and the post-load wrap is gone) and #1757 (pre_fsdp_model_transform runs before this PR's pre_fsdp_transform). Maintainer review (Swarm order #776): AdaLN skip names are canonicalized through checkpoint-wrapper prefixes, test_inference_regional_compile.py joins the unit lane, the Wan modulation custom-op test moved to the transformer lane, regional_compile is classified in the schema inventory, and a 1/2-GPU training smoke covers fully_shard(CheckpointWrapper) in eager and regional-compile modes. Validated on GB200: the FA4 CuTe forward/backward parity and fullgraph-backward tests and all four smoke cases pass.
Also exempts MMAudio recipes, whose student builds its pipeline config at init, from the fine-tuning recipe construction check that #1648 left failing on main.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adds preprocess_hunyuan15_overfit.py, which writes VAE latents plus Qwen and ByT5 text embeddings in the dual-text parquet schema (pyarrow_schema_t2v_dual_text) that main's HunyuanVideo 1.5 training (#1662) reads; main's guide and overfit config pointed at this script, but nothing in-tree produced text_embedding_2 before.
Reconciled with #1662 during maintainer review (Swarm orders #723 and #777): main's schema and collate are the base. On top of them, both text streams now share one CFG-dropout decision per row also when _sample_index is absent, and a row missing the primary text_embedding raises instead of training unconditioned. The PR's widening of the shared t2v schema and its writer changes were dropped. docs/training/hunyuan15_overfit.md now describes the in-tree script.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Aryan Kumar