Files

T

shaoxiongduan 8af0c43a65 [fix] WanTrack: enforce I2V via Fun-InP base + CLIP cross-attention

The overfit teacher ignored the first frame (validation looked T2V): the
init came from FastWan2.1-T2V-1.3B, whose image-conditioning channels are
zero/untrained and which has no CLIP pathway, so first-frame and track
conditioning received ~no gradient.

Re-base on Wan2.1-Fun-1.3B-InP (pretrained concat-I2V + CLIP):
- convert_trackwan_init: widen patch-embed base_in(36)->52, keep the 36
  pretrained image-cond channels, zero-init only the 16 track channels;
  inherit image_dim from the base config; merge sharded safetensors.
- trackwan config: image_dim=1280 builds the DiT image_embedder.
- preprocess: compute + store clip_feature (CLIP frame-0 embedding) and
  add image_encoder/image_processor to required modules.
- schema: clip_feature_{bytes,shape,dtype} in pyarrow_schema_i2v_track.
- wantrack / track_validation / trackwan_infer: feed
  encoder_hidden_states_image through train, validation, and inference.
- yaml: point at the Fun-InP init + re-preprocessed data.

Verified: step-0 validation reproduces the GT first frame (PSNR ~24-26 dB
vs ~9-11 before), i.e. I2V is enforced before any training.

2026-07-01 06:36:41 +00:00

configs

[feat] Add Wan RL DiffusionNFT training (#1450 )

2026-06-11 21:18:59 -07:00

scenario

[fix] WanTrack: enforce I2V via Fun-InP base + CLIP cross-attention

2026-07-01 06:36:41 +00:00

README.md

[misc]: reorganize training configs and add documentation (#1177 )

2026-03-26 16:38:31 -07:00

run_slurm.sh

[misc]: reorganize training configs and add documentation (#1177 )

2026-03-26 16:38:31 -07:00

run.sh

[feat] Add minimal LoRA finetuning support to the YAML training stack (#1242 )

2026-05-17 14:49:13 -07:00

README.md

Training

Quick Start

Single-node

bash examples/train/run.sh <config.yaml> [--dotted.key value ...]

The script auto-detects available GPUs. Override with NUM_GPUS:

NUM_GPUS=4 bash examples/train/run.sh \
    examples/train/configs/fine_tuning/wan/t2v.yaml

Multi-node (Slurm)

bash examples/train/run_slurm.sh <config.yaml> <num_nodes> [--dotted.key value ...]

bash examples/train/run_slurm.sh \
    examples/train/configs/fine_tuning/wan/t2v.yaml 4 \
    --training.distributed.hsdp_shard_dim 8 \
    --training.distributed.hsdp_replicate_dim 4

Slurm environment variables:

Variable	Default	Description
`PARTITION`	`main`	Slurm partition
`NUM_GPUS`	`8`	GPUs per node
`CPUS_PER_TASK`	`128`	CPUs per task
`MEM`	`1440G`	Memory per node
`EXCLUDE`	`""`	Nodes to exclude

Overriding config values

Any config field can be overridden from the command line using dotted keys:

bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
    --training.optimizer.learning_rate 1e-5 \
    --training.loop.max_train_steps 1000 \
    --training.checkpoint.output_dir outputs/my_experiment

Resuming from a checkpoint

bash examples/train/run.sh examples/train/configs/fine_tuning/wan/t2v.yaml \
    --training.checkpoint.resume_from_checkpoint outputs/my_experiment/checkpoint-500

W&B Logging

Training metrics and validation videos are logged to Weights & Biases. Set your API key before launching:

export WANDB_API_KEY=your_key_here

To disable W&B (e.g. for local debugging):

export WANDB_MODE=offline

The project name and run name are set in the config under training.tracker:

training:
  tracker:
    project_name: my_project
    run_name: my_run

Directory Layout

examples/train/
├── configs/      # Single-step training configs (by method and model)
├── scenario/     # Multi-step end-to-end training pipelines
├── run.sh        # Single-node launcher
└── run_slurm.sh  # Multi-node Slurm launcher

See configs/README.md and scenario/README.md for details.