shaoxiongduan
36d6f8785e
run_wan14b_held.sh: correct stage-1/D to WANTRACK_FREEZE_HEAD=0 (train the
track head through stage-1, matching the 1.3B run D which never froze it and
whose head plateaus only by ~step 4700/4800). Was 1.
Overfit + stage-1 configs (14B):
- synth_sparse_random_14b_{lr2e5,bs8,bs16}: random-warmstart overfit variants;
bs16 (global bs 16, lr 1e-4) is the stable one whose ckpt-1200 merge scored
best on gradient-SNR (track_encoder.proj 0.530 / snr 0.969).
- openvid_sparse_14b_merged_fixed2000 + stage1_16node: stage-1 from the merged
init (global bs 128, lr 1e-5, 4800 steps); 16-node variant = 64 GPU, accum 2.
gradient_flow_analysis.py: register the 14B fun-control and bs16-1200 merged
inits for SNR comparison.
720p pipeline: preprocessing_datasets.py drops the persistent decoded-pixel
reference (1.34GB/clip at 720p) after each item so preprocess RSS stays bounded;
run_preprocess_720p_held.sh + verify_{720p_parquet,row_integrity}.py +
check_720p_latents.py to re-encode OpenVid at 1280x720 and verify index/track
alignment against the 480p set; smoke_720p configs; acquire_best.sh node grabber.
Training
Quick Start
Single-node
bash examples/train/run.sh <config.yaml> [--dotted.key value ...]
The script auto-detects available GPUs. Override with NUM_GPUS:
NUM_GPUS=4 bash examples/train/run.sh \
examples/train/configs/fine_tuning/wan/t2v.yaml
Multi-node (Slurm)
bash examples/train/run_slurm.sh <config.yaml> <num_nodes> [--dotted.key value ...]
bash examples/train/run_slurm.sh \
examples/train/configs/fine_tuning/wan/t2v.yaml 4 \
--training.distributed.hsdp_shard_dim 8 \
--training.distributed.hsdp_replicate_dim 4
Slurm environment variables:
| Variable | Default | Description |
|---|---|---|
PARTITION |
main |
Slurm partition |
NUM_GPUS |
8 |
GPUs per node |
CPUS_PER_TASK |
128 |
CPUs per task |
MEM |
1440G |
Memory per node |
EXCLUDE |
"" |
Nodes to exclude |
Overriding config values
Any config field can be overridden from the command line using dotted keys:
bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
--training.optimizer.learning_rate 1e-5 \
--training.loop.max_train_steps 1000 \
--training.checkpoint.output_dir outputs/my_experiment
Resuming from a checkpoint
bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
--training.checkpoint.resume_from_checkpoint outputs/my_experiment/checkpoint-500
W&B Logging
Training metrics and validation videos are logged to Weights & Biases. Set your API key before launching:
export WANDB_API_KEY=your_key_here
To disable W&B (e.g. for local debugging):
export WANDB_MODE=offline
The project name and run name are set in the config under training.tracker:
training:
tracker:
project_name: my_project
run_name: my_run
Directory Layout
examples/train/
├── configs/ # Single-step training configs (by method and model)
├── scenario/ # Multi-step end-to-end training pipelines
├── run.sh # Single-node launcher
└── run_slurm.sh # Multi-node Slurm launcher
See configs/README.md and scenario/README.md for details.