leffffandClaude Fable 5
5aae1c093a
- Never touch the user's documented dataset: preprocess roots are now env-overridable (KANDINSKY5_OVERFIT_DATA_DIR/_OUTPUT_DIR) and the nightly keeps every artifact under a single test-owned data/kandinsky5_e2e/ root, cleaning only that - Override method.generator_update_interval to 1 in the E2E stage-2 run: with the recipe's 5, a 3-step run performed zero student updates and the exported "student" was stage-1 weights - Harden the media oracle: exact frame count/geometry (closes the SSIM shorter-clip truncation hole), per-frame luma spatial std (solid clip at reference mean scored 15.4 on the old global RGB std), consecutive-frame temporal diff (frozen clips), and MS-SSIM floor recalibrated to 0.95 against measured known-bads (solid 0.9189, frozen 0.8597, good 1.0000); known-bad clips are permanent self-calibrating regressions in fastvideo/tests/workflow/test_kandinsky5_e2e_media_oracle.py - Capability-gate ATTN_QAT_INFER: is_attn_qat_infer_available() now requires an sm_120/sm_121 device in addition to the extension import (CUDA 13 wheels can bundle the extension on H100/GB200, where the first kernel call failed instead of falling back); the real platform resolver is covered in fastvideo/tests/api/test_attn_qat_infer_capability_gate.py - Remove the now-stale golden reference: the generator-update fix changes what stage 2 trains, so the reference must be re-recorded from the corrected run (KANDINSKY5_E2E_WRITE_REFERENCE=1 bootstrap + review) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Training
Quick Start
Single-node
bash examples/train/run.sh <config.yaml> [--dotted.key value ...]
The script auto-detects available GPUs. Override with NUM_GPUS:
NUM_GPUS=4 bash examples/train/run.sh \
examples/train/configs/fine_tuning/wan/t2v.yaml
Multi-node (Slurm)
bash examples/train/run_slurm.sh <config.yaml> <num_nodes> [--dotted.key value ...]
bash examples/train/run_slurm.sh \
examples/train/configs/fine_tuning/wan/t2v.yaml 4 \
--training.distributed.hsdp_shard_dim 8 \
--training.distributed.hsdp_replicate_dim 4
Slurm environment variables:
| Variable | Default | Description |
|---|---|---|
PARTITION |
main |
Slurm partition |
NUM_GPUS |
8 |
GPUs per node |
CPUS_PER_TASK |
128 |
CPUs per task |
MEM |
1440G |
Memory per node |
EXCLUDE |
"" |
Nodes to exclude |
Overriding config values
Any config field can be overridden from the command line using dotted keys:
bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
--training.optimizer.learning_rate 1e-5 \
--training.loop.max_train_steps 1000 \
--training.checkpoint.output_dir outputs/my_experiment
Resuming from a checkpoint
bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
--training.checkpoint.resume_from_checkpoint outputs/my_experiment/checkpoint-500
W&B Logging
Training metrics and validation videos are logged to Weights & Biases. Set your API key before launching:
export WANDB_API_KEY=your_key_here
To disable W&B (e.g. for local debugging):
export WANDB_MODE=offline
The project name and run name are set in the config under training.tracker:
training:
tracker:
project_name: my_project
run_name: my_run
Directory Layout
examples/train/
├── configs/ # Single-step training configs (by method and model)
├── scenario/ # Multi-step end-to-end training pipelines
├── run.sh # Single-node launcher
└── run_slurm.sh # Multi-node Slurm launcher
See configs/README.md and scenario/README.md for details.
The featured QAD Wan2.1 MixKit scenario runs Attn-QAT supervised fine-tuning, checkpoint export, and three-step DMD2. See the Attn-QAT training guide.