shaoxiongduan
92cc893cfc
Training:
- finetune.py: optional track/base param groups (WANTRACK_TRACK_GROUP) so the
track pathway can take a different LR (WANTRACK_TRACK_LR_MULT); patch_embedding
gradient masked to the track slot [:, 36:]. Raises if nothing matches.
- track_warmup.py callback: hold the pretrained DiT at lr 0 while the track
pathway warms up, then ramp it back in.
14B configs + held-alloc launchers:
- finetune_wantrack_{synth_sparse_fixed,synth_sparse_random,openvid_sparse_merged_bias,
synth_stage2}_14b + openvid_14b_{warmup,lwlr} + 1.3B stage2 configs.
- run_wan14b_held.sh (held alloc + auto-restart + rank cleanup, per-stage WANTRACK
env), acquire_nodes.sh, chain_stepAB.sh, run_stepB_seed.sh, run_stepC_merge.sh,
watch_gate.sh, run_synth_stage2_slurm.sh.
Init / diagnostics:
- convert_trackwan_init_v2.py: --pe-src to lift patch_embedding[:, 36:] and
--track-src to lift track_encoder together (co-adapted merge); sharded-source support.
- check_gate_growth.py, gradient_flow_analysis.py, gradient_flow_summarize.py.
Data pipeline: segment_tracks.py sharding; decode/segment/tracks slurm scripts;
precompute_null_text, two_track_val, cfg_ablation, lwlr_experiment.
Inference: app_multi_trace.py updates.
Training
Quick Start
Single-node
bash examples/train/run.sh <config.yaml> [--dotted.key value ...]
The script auto-detects available GPUs. Override with NUM_GPUS:
NUM_GPUS=4 bash examples/train/run.sh \
examples/train/configs/fine_tuning/wan/t2v.yaml
Multi-node (Slurm)
bash examples/train/run_slurm.sh <config.yaml> <num_nodes> [--dotted.key value ...]
bash examples/train/run_slurm.sh \
examples/train/configs/fine_tuning/wan/t2v.yaml 4 \
--training.distributed.hsdp_shard_dim 8 \
--training.distributed.hsdp_replicate_dim 4
Slurm environment variables:
| Variable | Default | Description |
|---|---|---|
PARTITION |
main |
Slurm partition |
NUM_GPUS |
8 |
GPUs per node |
CPUS_PER_TASK |
128 |
CPUs per task |
MEM |
1440G |
Memory per node |
EXCLUDE |
"" |
Nodes to exclude |
Overriding config values
Any config field can be overridden from the command line using dotted keys:
bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
--training.optimizer.learning_rate 1e-5 \
--training.loop.max_train_steps 1000 \
--training.checkpoint.output_dir outputs/my_experiment
Resuming from a checkpoint
bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
--training.checkpoint.resume_from_checkpoint outputs/my_experiment/checkpoint-500
W&B Logging
Training metrics and validation videos are logged to Weights & Biases. Set your API key before launching:
export WANDB_API_KEY=your_key_here
To disable W&B (e.g. for local debugging):
export WANDB_MODE=offline
The project name and run name are set in the config under training.tracker:
training:
tracker:
project_name: my_project
run_name: my_run
Directory Layout
examples/train/
├── configs/ # Single-step training configs (by method and model)
├── scenario/ # Multi-step end-to-end training pipelines
├── run.sh # Single-node launcher
└── run_slurm.sh # Multi-node Slurm launcher
See configs/README.md and scenario/README.md for details.