Will LinandClaude Fable 5
8775bff28a
1. mode/options conflict: _compile_model_regions now rejects torch_compile_kwargs['mode'] with an actionable error instead of letting torch raise about an 'options' key the user never wrote (the CLI help's own example uses mode). CLI help updated. 2. Legacy stack scoping: new regional_compile flag (FastVideoArgs, default False). The modular fastvideo/train moduleloader opts in; legacy --enable-torch-compile recipes keep their pre-PR whole-model torch.compile semantics (partial graphs, kwargs passed through). 3. AC-prefix normalization in the weight loader: checkpoint_wrapper strips its prefix from state_dict() keys but not named_parameters()/ named_buffers(); with pre_fsdp_transform applying AC before load, a loaded buffer inside a wrapped block was silently converted into a TRAINABLE parameter by load_state_dict(assign=True) (verified on torch 2.12). All name-keyed lookups (buffer membership, fused weight_loader, _get_parameter_dtype at load and shard time) now compare canonical names via _strip_checkpoint_wrapper_prefix. 4. FA3: flash-attn 3's grad-enabled attention path graph-breaks, which fullgraph regional compile turns into a hard error at step 1 on Hopper hosts. The loader now detects FLASH_ATTN+fa3 and warns-and-disables compile instead of crashing, and dmd2_t2v_fa3.yaml ships the recipe with compile explicitly off (main config annotated). Tests: mode-conflict rejection; AC-prefix canonicalization incl. the buffer-stays-buffer regression under assign=True. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Training
Quick Start
Single-node
bash examples/train/run.sh <config.yaml> [--dotted.key value ...]
The script auto-detects available GPUs. Override with NUM_GPUS:
NUM_GPUS=4 bash examples/train/run.sh \
examples/train/configs/fine_tuning/wan/t2v.yaml
Multi-node (Slurm)
bash examples/train/run_slurm.sh <config.yaml> <num_nodes> [--dotted.key value ...]
bash examples/train/run_slurm.sh \
examples/train/configs/fine_tuning/wan/t2v.yaml 4 \
--training.distributed.hsdp_shard_dim 8 \
--training.distributed.hsdp_replicate_dim 4
Slurm environment variables:
| Variable | Default | Description |
|---|---|---|
PARTITION |
main |
Slurm partition |
NUM_GPUS |
8 |
GPUs per node |
CPUS_PER_TASK |
128 |
CPUs per task |
MEM |
1440G |
Memory per node |
EXCLUDE |
"" |
Nodes to exclude |
Overriding config values
Any config field can be overridden from the command line using dotted keys:
bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
--training.optimizer.learning_rate 1e-5 \
--training.loop.max_train_steps 1000 \
--training.checkpoint.output_dir outputs/my_experiment
Resuming from a checkpoint
bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
--training.checkpoint.resume_from_checkpoint outputs/my_experiment/checkpoint-500
W&B Logging
Training metrics and validation videos are logged to Weights & Biases. Set your API key before launching:
export WANDB_API_KEY=your_key_here
To disable W&B (e.g. for local debugging):
export WANDB_MODE=offline
The project name and run name are set in the config under training.tracker:
training:
tracker:
project_name: my_project
run_name: my_run
Directory Layout
examples/train/
├── configs/ # Single-step training configs (by method and model)
├── scenario/ # Multi-step end-to-end training pipelines
├── run.sh # Single-node launcher
└── run_slurm.sh # Multi-node Slurm launcher
See configs/README.md and scenario/README.md for details.
The featured QAD Wan2.1 MixKit scenario runs Attn-QAT supervised fine-tuning, checkpoint export, and three-step DMD2. See the Attn-QAT training guide.