Training:
- finetune.py: optional track/base param groups (WANTRACK_TRACK_GROUP) so the
track pathway can take a different LR (WANTRACK_TRACK_LR_MULT); patch_embedding
gradient masked to the track slot [:, 36:]. Raises if nothing matches.
- track_warmup.py callback: hold the pretrained DiT at lr 0 while the track
pathway warms up, then ramp it back in.
14B configs + held-alloc launchers:
- finetune_wantrack_{synth_sparse_fixed,synth_sparse_random,openvid_sparse_merged_bias,
synth_stage2}_14b + openvid_14b_{warmup,lwlr} + 1.3B stage2 configs.
- run_wan14b_held.sh (held alloc + auto-restart + rank cleanup, per-stage WANTRACK
env), acquire_nodes.sh, chain_stepAB.sh, run_stepB_seed.sh, run_stepC_merge.sh,
watch_gate.sh, run_synth_stage2_slurm.sh.
Init / diagnostics:
- convert_trackwan_init_v2.py: --pe-src to lift patch_embedding[:, 36:] and
--track-src to lift track_encoder together (co-adapted merge); sharded-source support.
- check_gate_growth.py, gradient_flow_analysis.py, gradient_flow_summarize.py.
Data pipeline: segment_tracks.py sharding; decode/segment/tracks slurm scripts;
precompute_null_text, two_track_val, cfg_ablation, lwlr_experiment.
Inference: app_multi_trace.py updates.
data_pipeline/generate_videos.py:
Add --image arg -> when set, forwards to VideoGenerator.generate_video
via image_path=... so the same script can drive either T2V or I2V. Used
to generate the shared-first-frame synthetic-toy overfit dataset from a
single seed image.
examples/inference/gradio/trackwan/app_action.py:
Bring the interactive drawing app in line with the training-time val
recipe: joint text+motion CFG (MotionStream Eq. 2) with w_text and
w_motion sliders + a mode radio (joint / text_only / no_track).
`on_generate` now branches on mode instead of the old bare single
forward, and text_only / no_track let the user compare against the
same counterfactuals the val callback logs.
data_pipeline/sparse_sampling_dashboard.py:
New interactive dashboard for eyeballing the sparse recipe
(1 track per SAM object + num_sampled background extras). Overlays
the FastSAM segmentation on frame 0 in the same tab20 palette used
for the tracked points, so mask/point correspondence is visible at a
glance. Sliders for num_sampled, random-vs-weighted extras mode, and
seed; a button to render the full kept-tracks overlay video.
- track_informativeness.py: compare abs-motion vs affine/low-rank residual vs
motion/subspace clustering vs DINOv2 semantic embeddings for scoring which
point tracks are informative; per-clip camera-motion metric; gradio viz.
- sampling_viz.py: render the exact _augment_tracks sampler (coverage + low-rank
weighted + uniform) on a preprocessed dataset to inspect which traces it keeps.
- find_static_windows.py: phase-correlation per-frame camera-motion signal (recovers
after turns) + greedy non-overlapping static-window extraction to build a
relatively-still clip dataset from long untrimmed videos.
- segment_viz.py: FastSAM config sweep + fixes (config/clip filters, allowed_paths).
- _augment_tracks: default random 1000-2500 track sampling (was 1-200), contiguous
chunk masking (was per-frame), diversity weighting (favor moving/unique tracks,
keep some static), and an object-coverage hook (>=1 track per SAM segment via
optional object_ids). Sampling ON by default; CFG dropout now default off.
- trackwan_infer.generate: motion classifier-free guidance (guidance_scale) using
the base no-tracks I2V branch as the unconditional.
- test_motion_cfg.py: quick guidance-scale EPE sweep.
- finetune_wantrack_wan200.yaml: 50->200 data-scale run (same params).
- droid_dataset_dashboard.py: render per-clip CoTracker overlays + motion-coverage
stats (frac_moving, mean/max motion) + near-duplicate grouping, and a gradio
gallery sorted least-motion-first to surface weak/duplicate clips.
- app_action.py: DTensor-safe checkpoint hot-swap (copy into each param's local
shard instead of load_state_dict, which errors on FSDP DTensors); a Preview-traces
button that overlays the synthetic control on the frozen first frame before
generating; and a streaming per-step denoise log.
convert_droid.py builds a set where the first frame is similar but motion is
diverse, the design that forces trace reliance: it scans DROID (LeRobot mirror),
clusters frame-0 embeddings, and writes the tightest cluster as 121-frame libx264
mp4s + manifest with ONE generic caption, dropping straight into extract_tracks ->
i2v_track preprocess. Adds finetune_wantrack_droid_overfit.yaml (Fun-Control init,
WANTRACK_AUG=0 clean overfit, fresh output_dir) and gitignores .gradio/.
Add the instruments used to diagnose whether the model follows the tracks:
- track_sensitivity.py: per-checkpoint swap-sensitivity (relative velocity shift
when only the tracks / first-frame / text change) -- intrinsic input attribution.
- control_sensitivity metric: intervention-diff (generate twice, same conditioning,
different control) scored in the moving ROI, so a static background can't dilute it.
- gen_eval_artifacts.py + app_viewer.py: pre-generate counterfactual videos
(input vs re-extracted tracks, EPE heatmap) and browse them with no GPU.
- app_action.py: mouse-draw a motion and hot-swap checkpoints.
- test_controllability.py / cotracker_epe / app.py: clip_feature + minor updates.
Initialize the 52ch TrackWan from Wan2.1-Fun-1.3B-Control so the track-slot
patch-embed channels start from Fun-Control's PRETRAINED control channels
(non-zero). Combined with the zero-init track head this gives step-0 == teacher
while still letting gradient reach the track encoder -- avoiding the double
zero-init deadlock of the from-scratch Fun-InP path (where both the track
patch-embed channels and the track-head proj are zero, so the track pathway
never trains). Adds the converter, the matching track_config channel handling,
and the InP init updates.
The overfit teacher ignored the first frame (validation looked T2V): the
init came from FastWan2.1-T2V-1.3B, whose image-conditioning channels are
zero/untrained and which has no CLIP pathway, so first-frame and track
conditioning received ~no gradient.
Re-base on Wan2.1-Fun-1.3B-InP (pretrained concat-I2V + CLIP):
- convert_trackwan_init: widen patch-embed base_in(36)->52, keep the 36
pretrained image-cond channels, zero-init only the 16 track channels;
inherit image_dim from the base config; merge sharded safetensors.
- trackwan config: image_dim=1280 builds the DiT image_embedder.
- preprocess: compute + store clip_feature (CLIP frame-0 embedding) and
add image_encoder/image_processor to required modules.
- schema: clip_feature_{bytes,shape,dtype} in pyarrow_schema_i2v_track.
- wantrack / track_validation / trackwan_infer: feed
encoder_hidden_states_image through train, validation, and inference.
- yaml: point at the Fun-InP init + re-preprocessed data.
Verified: step-0 validation reproduces the GT first frame (PSNR ~24-26 dB
vs ~9-11 before), i.e. I2V is enforced before any training.
shaoxiongduan