Training:
- finetune.py: optional track/base param groups (WANTRACK_TRACK_GROUP) so the
track pathway can take a different LR (WANTRACK_TRACK_LR_MULT); patch_embedding
gradient masked to the track slot [:, 36:]. Raises if nothing matches.
- track_warmup.py callback: hold the pretrained DiT at lr 0 while the track
pathway warms up, then ramp it back in.
14B configs + held-alloc launchers:
- finetune_wantrack_{synth_sparse_fixed,synth_sparse_random,openvid_sparse_merged_bias,
synth_stage2}_14b + openvid_14b_{warmup,lwlr} + 1.3B stage2 configs.
- run_wan14b_held.sh (held alloc + auto-restart + rank cleanup, per-stage WANTRACK
env), acquire_nodes.sh, chain_stepAB.sh, run_stepB_seed.sh, run_stepC_merge.sh,
watch_gate.sh, run_synth_stage2_slurm.sh.
Init / diagnostics:
- convert_trackwan_init_v2.py: --pe-src to lift patch_embedding[:, 36:] and
--track-src to lift track_encoder together (co-adapted merge); sharded-source support.
- check_gate_growth.py, gradient_flow_analysis.py, gradient_flow_summarize.py.
Data pipeline: segment_tracks.py sharding; decode/segment/tracks slurm scripts;
precompute_null_text, two_track_val, cfg_ablation, lwlr_experiment.
Inference: app_multi_trace.py updates.
examples/inference/gradio/trackwan/app_action.py:
Bring the interactive drawing app in line with the sparse training
regime our overfit runs used (WANTRACK_SPARSE=1). New 'Track budget'
radio: 'dense' keeps the old full 50x50 grid; 'sparse' (new default)
builds a much smaller [Tpx, num_moving + num_bg, 2] tensor -- the
moving handle points from the drawn path + num_bg random background
points from the training grid (seeded by the seed slider). New
'num_bg' slider (0-60, default 20) matches WANTRACK_EXTRA_RANDOM=20.
Overlay logic updated to draw every point in sparse mode instead of
grid-strided subsampling that assumed N==2500.
examples/train/scenario/worldmodel/finetune_wantrack_synth_sparse_random_warmstart.yaml:
Test config for the curriculum idea: start from sparse-fixed@1000
and let random-sampling sparse training take over. Compared cold-
start sparse-random (~0.8%) vs warmstart (~3.2%) vs fixed (~8.8%)
divergence.
data_pipeline/generate_videos.py:
Add --image arg -> when set, forwards to VideoGenerator.generate_video
via image_path=... so the same script can drive either T2V or I2V. Used
to generate the shared-first-frame synthetic-toy overfit dataset from a
single seed image.
examples/inference/gradio/trackwan/app_action.py:
Bring the interactive drawing app in line with the training-time val
recipe: joint text+motion CFG (MotionStream Eq. 2) with w_text and
w_motion sliders + a mode radio (joint / text_only / no_track).
`on_generate` now branches on mode instead of the old bare single
forward, and text_only / no_track let the user compare against the
same counterfactuals the val callback logs.
data_pipeline/sparse_sampling_dashboard.py:
New interactive dashboard for eyeballing the sparse recipe
(1 track per SAM object + num_sampled background extras). Overlays
the FastSAM segmentation on frame 0 in the same tab20 palette used
for the tracked points, so mask/point correspondence is visible at a
glance. Sliders for num_sampled, random-vs-weighted extras mode, and
seed; a button to render the full kept-tracks overlay video.
- _augment_tracks: default random 1000-2500 track sampling (was 1-200), contiguous
chunk masking (was per-frame), diversity weighting (favor moving/unique tracks,
keep some static), and an object-coverage hook (>=1 track per SAM segment via
optional object_ids). Sampling ON by default; CFG dropout now default off.
- trackwan_infer.generate: motion classifier-free guidance (guidance_scale) using
the base no-tracks I2V branch as the unconditional.
- test_motion_cfg.py: quick guidance-scale EPE sweep.
- finetune_wantrack_wan200.yaml: 50->200 data-scale run (same params).
- droid_dataset_dashboard.py: render per-clip CoTracker overlays + motion-coverage
stats (frac_moving, mean/max motion) + near-duplicate grouping, and a gradio
gallery sorted least-motion-first to surface weak/duplicate clips.
- app_action.py: DTensor-safe checkpoint hot-swap (copy into each param's local
shard instead of load_state_dict, which errors on FSDP DTensors); a Preview-traces
button that overlays the synthetic control on the frozen first frame before
generating; and a streaming per-step denoise log.
Add the instruments used to diagnose whether the model follows the tracks:
- track_sensitivity.py: per-checkpoint swap-sensitivity (relative velocity shift
when only the tracks / first-frame / text change) -- intrinsic input attribution.
- control_sensitivity metric: intervention-diff (generate twice, same conditioning,
different control) scored in the moving ROI, so a static background can't dilute it.
- gen_eval_artifacts.py + app_viewer.py: pre-generate counterfactual videos
(input vs re-extracted tracks, EPE heatmap) and browse them with no GPU.
- app_action.py: mouse-draw a motion and hot-swap checkpoints.
- test_controllability.py / cotracker_epe / app.py: clip_feature + minor updates.
shaoxiongduan