- V2 config now matches the canonical 16-GPU stage 1: warmup 10,
min LR ratio 0.3, HSDP 16.
- State that the published V2.1 checkpoint is linears-only (BF16 VSA
attention); the FP4 adaptation is a separate, non-public stage.
- Name the released checkpoints and starting models.
- Vendor and pin both calibrated amax tables.
- Add the V2 prompt/embedding and OmniRef eval data-prep scripts.
Add the OmniRef (Ref2VA, VSA tile 128) and V2/V2.1 Consumer (T2VA,
VSA tile 64) profiles of one quantization-aware distillation recipe:
NVFP4 W4A4 q/k/v/out and FFN linears in all 50 blocks, the FP4
sparse-attention emulator, frozen VSA gates and norms, output plus
layer-reconstruction regression against a frozen bf16 teacher, and
best-by-score checkpoints with audio eligibility guards.
Training code: the PDD-QAD method, the two H3 model plugins, the
NVFP4 emulated linear, and the tile-64/128 Triton emulator with its
PyTorch numeric reference. The VSA attention impl gains a fine
attention override hook and lets tile 128 run grad-capable forwards
through it. The trainer gains opt-in tagged checkpoints and early
stop hooks for the QAD guards.
Also add two runnable configs, the training guide, and CPU/GPU tests.
Adds opt-in regional torch.compile to the YAML-driven modular Wan trainer (training.model.enable_torch_compile / torch_compile_kwargs) with FA4 and Wan modulation forward/backward as opaque custom ops, compiled repeated DiT blocks after FSDP setup, and activation checkpointing applied before FSDP for trainable Wan transformers.
Merged with main's #1716 (WanModel resolves the checkpointing type in __init__; the PR's pre-FSDP wrap uses it and the post-load wrap is gone) and #1757 (pre_fsdp_model_transform runs before this PR's pre_fsdp_transform). Maintainer review (Swarm order #776): AdaLN skip names are canonicalized through checkpoint-wrapper prefixes, test_inference_regional_compile.py joins the unit lane, the Wan modulation custom-op test moved to the transformer lane, regional_compile is classified in the schema inventory, and a 1/2-GPU training smoke covers fully_shard(CheckpointWrapper) in eager and regional-compile modes. Validated on GB200: the FA4 CuTe forward/backward parity and fullgraph-backward tests and all four smoke cases pass.
Also exempts MMAudio recipes, whose student builds its pipeline config at init, from the fine-tuning recipe construction check that #1648 left failing on main.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Aryan Kumar