Files

T

Zhang PeiyuanandClaude Opus 5.5 c520878fd2 [perf]: QAT/QAD add safe regional compile for modular training (#1718 )

Adds opt-in regional torch.compile to the YAML-driven modular Wan trainer (training.model.enable_torch_compile / torch_compile_kwargs) with FA4 and Wan modulation forward/backward as opaque custom ops, compiled repeated DiT blocks after FSDP setup, and activation checkpointing applied before FSDP for trainable Wan transformers.

Merged with main's #1716 (WanModel resolves the checkpointing type in __init__; the PR's pre-FSDP wrap uses it and the post-load wrap is gone) and #1757 (pre_fsdp_model_transform runs before this PR's pre_fsdp_transform). Maintainer review (Swarm order #776): AdaLN skip names are canonicalized through checkpoint-wrapper prefixes, test_inference_regional_compile.py joins the unit lane, the Wan modulation custom-op test moved to the transformer lane, regional_compile is classified in the schema inventory, and a 1/2-GPU training smoke covers fully_shard(CheckpointWrapper) in eager and regional-compile modes. Validated on GB200: the FA4 CuTe forward/backward parity and fullgraph-backward tests and all four smoke cases pass.

Also exempts MMAudio recipes, whose student builds its pipeline config at init, from the fine-tuning recipe construction check that #1648 left failing on main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-07 05:24:24 -07:00

configs

[perf]: QAT/QAD add safe regional compile for modular training (#1718 )

2026-10-07 05:24:24 -07:00

scenario

[misc] yapf pass under CI's interpreter (3.12) + pin hook language_version (#1702 )

2026-08-08 22:24:17 -07:00

launch_minimax_h3_t2va_crush_smol_validation.sh

[feat] add Minimax H3 sft pipeline (#1688 )

2026-08-07 16:12:04 -07:00

README.md

[feat] add qat docs (#1621 )

2026-07-19 18:37:47 -07:00

run_slurm.sh

[feat] add Minimax H3 sft pipeline (#1688 )

2026-08-07 16:12:04 -07:00

run.sh

[feat] Add minimal LoRA finetuning support to the YAML training stack (#1242 )

2026-05-17 14:49:13 -07:00

setup_minimax_h3_ref2va_synthetic_single_sample.sh

[feat]: add MiniMax H3 Ref2VA and LoRA training support (#1757 )

2026-10-07 03:21:45 -07:00

setup_minimax_h3_t2va_crush_smol_single_sample.sh

[feat] add Minimax H3 sft pipeline (#1688 )

2026-08-07 16:12:04 -07:00

README.md

Training

Quick Start

Single-node

bash examples/train/run.sh <config.yaml> [--dotted.key value ...]

The script auto-detects available GPUs. Override with NUM_GPUS:

NUM_GPUS=4 bash examples/train/run.sh \
    examples/train/configs/fine_tuning/wan/t2v.yaml

Multi-node (Slurm)

bash examples/train/run_slurm.sh <config.yaml> <num_nodes> [--dotted.key value ...]

bash examples/train/run_slurm.sh \
    examples/train/configs/fine_tuning/wan/t2v.yaml 4 \
    --training.distributed.hsdp_shard_dim 8 \
    --training.distributed.hsdp_replicate_dim 4

Slurm environment variables:

Variable	Default	Description
`PARTITION`	`main`	Slurm partition
`NUM_GPUS`	`8`	GPUs per node
`CPUS_PER_TASK`	`128`	CPUs per task
`MEM`	`1440G`	Memory per node
`EXCLUDE`	`""`	Nodes to exclude

Overriding config values

Any config field can be overridden from the command line using dotted keys:

bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
    --training.optimizer.learning_rate 1e-5 \
    --training.loop.max_train_steps 1000 \
    --training.checkpoint.output_dir outputs/my_experiment

Resuming from a checkpoint

bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
    --training.checkpoint.resume_from_checkpoint outputs/my_experiment/checkpoint-500

W&B Logging

Training metrics and validation videos are logged to Weights & Biases. Set your API key before launching:

export WANDB_API_KEY=your_key_here

To disable W&B (e.g. for local debugging):

export WANDB_MODE=offline

The project name and run name are set in the config under training.tracker:

training:
  tracker:
    project_name: my_project
    run_name: my_run

Directory Layout

examples/train/
├── configs/      # Single-step training configs (by method and model)
├── scenario/     # Multi-step end-to-end training pipelines
├── run.sh        # Single-node launcher
└── run_slurm.sh  # Multi-node Slurm launcher

See configs/README.md and scenario/README.md for details.

The featured QAD Wan2.1 MixKit scenario runs Attn-QAT supervised fine-tuning, checkpoint export, and three-step DMD2. See the Attn-QAT training guide.