Training:
- finetune.py: optional track/base param groups (WANTRACK_TRACK_GROUP) so the
track pathway can take a different LR (WANTRACK_TRACK_LR_MULT); patch_embedding
gradient masked to the track slot [:, 36:]. Raises if nothing matches.
- track_warmup.py callback: hold the pretrained DiT at lr 0 while the track
pathway warms up, then ramp it back in.
14B configs + held-alloc launchers:
- finetune_wantrack_{synth_sparse_fixed,synth_sparse_random,openvid_sparse_merged_bias,
synth_stage2}_14b + openvid_14b_{warmup,lwlr} + 1.3B stage2 configs.
- run_wan14b_held.sh (held alloc + auto-restart + rank cleanup, per-stage WANTRACK
env), acquire_nodes.sh, chain_stepAB.sh, run_stepB_seed.sh, run_stepC_merge.sh,
watch_gate.sh, run_synth_stage2_slurm.sh.
Init / diagnostics:
- convert_trackwan_init_v2.py: --pe-src to lift patch_embedding[:, 36:] and
--track-src to lift track_encoder together (co-adapted merge); sharded-source support.
- check_gate_growth.py, gradient_flow_analysis.py, gradient_flow_summarize.py.
Data pipeline: segment_tracks.py sharding; decode/segment/tracks slurm scripts;
precompute_null_text, two_track_val, cfg_ablation, lwlr_experiment.
Inference: app_multi_trace.py updates.
examples/inference/gradio/trackwan/app_action.py:
Bring the interactive drawing app in line with the sparse training
regime our overfit runs used (WANTRACK_SPARSE=1). New 'Track budget'
radio: 'dense' keeps the old full 50x50 grid; 'sparse' (new default)
builds a much smaller [Tpx, num_moving + num_bg, 2] tensor -- the
moving handle points from the drawn path + num_bg random background
points from the training grid (seeded by the seed slider). New
'num_bg' slider (0-60, default 20) matches WANTRACK_EXTRA_RANDOM=20.
Overlay logic updated to draw every point in sparse mode instead of
grid-strided subsampling that assumed N==2500.
examples/train/scenario/worldmodel/finetune_wantrack_synth_sparse_random_warmstart.yaml:
Test config for the curriculum idea: start from sparse-fixed@1000
and let random-sampling sparse training take over. Compared cold-
start sparse-random (~0.8%) vs warmstart (~3.2%) vs fixed (~8.8%)
divergence.
Add the training configs used for the golf-overfit and 20-clip synthetic
overfit series exercising every combination of the recent changes:
golf_nonzero / golf_overfit_nonzero -- non-zero-init proj ablation
synth_generic / synth_varied -- prompt-conditioning ablation
(generic vs per-clip caption)
synth_varied_d64 -- id_dim=64 (paper spec)
synth_varied_d64_nobias -- + bias=False on track head
synth_varied_d64_nobias_fixed -- + WANTRACK_FIXED_SAMPLE=1
synth_curriculum -- random-sample warmstart from
a fixed-sample checkpoint
synth_varied_full_val -- max_train_steps=1 harness that
regenerates all 20 val clips
in 3 modes from a checkpoint
synth_sparse_fixed / synth_sparse_random -- 1-per-object + 20 background
extras (Yongqi's recipe)
All resume from the trackwan_1.3b_i2v_control_init family (or the
d64_nobias variant) and enable the new joint text+motion CFG (w_t=3,
w_m=1.5) in track_validation.
data_pipeline/generate_videos.py:
Add --image arg -> when set, forwards to VideoGenerator.generate_video
via image_path=... so the same script can drive either T2V or I2V. Used
to generate the shared-first-frame synthetic-toy overfit dataset from a
single seed image.
examples/inference/gradio/trackwan/app_action.py:
Bring the interactive drawing app in line with the training-time val
recipe: joint text+motion CFG (MotionStream Eq. 2) with w_text and
w_motion sliders + a mode radio (joint / text_only / no_track).
`on_generate` now branches on mode instead of the old bare single
forward, and text_only / no_track let the user compare against the
same counterfactuals the val callback logs.
data_pipeline/sparse_sampling_dashboard.py:
New interactive dashboard for eyeballing the sparse recipe
(1 track per SAM object + num_sampled background extras). Overlays
the FastSAM segmentation on frame 0 in the same tab20 palette used
for the tracked points, so mask/point correspondence is visible at a
glance. Sliders for num_sampled, random-vs-weighted extras mode, and
seed; a button to render the full kept-tracks overlay video.
Changes to the track pathway inspired by (and now matching or extending)
the MotionStream paper:
track_encoder.py:
* bias=False on temporal_conv and proj (paper Eq. 1 requires zero at
empty cells; nn.Conv3d's default bias=True was leaving a per-channel
constant term across every spatial cell, drowning the sparse signal).
* zero_init default now False (paper explicitly rejects ControlNet
zero-conv; matches Yongqi's recipe).
* WANTRACK_FIXED_SAMPLE=1 -> deterministic sinusoidal IDs (arange(N))
instead of a fresh random draw every forward. Same video always
scatters the same phi_n at the same cells across steps.
wantrack.py:
* WANTRACK_FIXED_SAMPLE plumbing: per-sample seeds derived from
info_list[i]['id'] so the same clip -> same subset every step.
* WANTRACK_SPARSE=1 sampler: 1 track per SAM object + WANTRACK_EXTRA_RANDOM
background extras (uniform or lowrank-weighted via WANTRACK_EXTRA_MODE).
Bypasses the dense 1000-2500 code path.
* on_train_start hook: WANTRACK_NONZERO_INIT=1 re-inits track_encoder.proj
with Kaiming after checkpoint load, avoiding the load/save round-trip
that corrupted our earlier attempts to force non-zero.
track_validation.py:
* paired 'no_motion_cfg' third variant per sample (with-tracks + text CFG
only, drops the motion arm) alongside the existing 'generated' (joint
text+motion CFG) and 'no_track' branches -- isolates how much of gen's
quality comes from motion CFG amplification vs the base pathway.
TrackValidation now also generates a NO-TRACK video per sample (same first-frame +
prompt, track_points=None -> zero track map), logged as track_val/no_track alongside
track_val/generated. If the two match, the model is ignoring the tracks; their
divergence quantifies track influence. Sanity-checked: identical at step 0 (proj=0),
should separate as the track pathway trains. Toggle via paired_no_track (default on).
_augment_tracks gated the object-coverage + low-rank-weighted + uniform draw on
`0 < max_points < N`, so with the default WANTRACK_MAX_POINTS=2500 and the full
2500-track grid (max_points == N) the entire subsample block was skipped -- the
model saw all 2500 tracks every step and only chunk-masking ran. Gate on
`min_points < N` instead so K~U[1000,2500] actually samples (verified: kept
1179/1437/1573 across steps, was a constant 2500).
Also: TrackValidation now applies the same sampler (loads object_ids/track_weights,
runs _augment_tracks deterministically) so validation is conditioned on -- and
overlays -- the sampled subset, not the full grid. Env-gated WANTRACK_DEBUG=1 logs
per-step sampling stats (kept count, object coverage, coord ranges). Adds a
single-video (golf) overfit config for debugging.
Sampler-on finetune configs (coverage + low-rank-weighted + uniform track
sampling, p_mask chunk masking) across three data regimes: unfiltered
egocentric EPIC-200, camera-static-filtered EPIC-200 (single kitchen), and the
Wan-200 synthetic set re-run with the sampler on (vs the AUG=0 overfit baseline).
- track_informativeness.py: compare abs-motion vs affine/low-rank residual vs
motion/subspace clustering vs DINOv2 semantic embeddings for scoring which
point tracks are informative; per-clip camera-motion metric; gradio viz.
- sampling_viz.py: render the exact _augment_tracks sampler (coverage + low-rank
weighted + uniform) on a preprocessed dataset to inspect which traces it keeps.
- find_static_windows.py: phase-correlation per-frame camera-motion signal (recovers
after turns) + greedy non-overlapping static-window extraction to build a
relatively-still clip dataset from long untrimmed videos.
- segment_viz.py: FastSAM config sweep + fixes (config/clip filters, allowed_paths).
- _augment_tracks: default random 1000-2500 track sampling (was 1-200), contiguous
chunk masking (was per-frame), diversity weighting (favor moving/unique tracks,
keep some static), and an object-coverage hook (>=1 track per SAM segment via
optional object_ids). Sampling ON by default; CFG dropout now default off.
- trackwan_infer.generate: motion classifier-free guidance (guidance_scale) using
the base no-tracks I2V branch as the unconditional.
- test_motion_cfg.py: quick guidance-scale EPE sweep.
- finetune_wantrack_wan200.yaml: 50->200 data-scale run (same params).
- droid_dataset_dashboard.py: render per-clip CoTracker overlays + motion-coverage
stats (frac_moving, mean/max motion) + near-duplicate grouping, and a gradio
gallery sorted least-motion-first to surface weak/duplicate clips.
- app_action.py: DTensor-safe checkpoint hot-swap (copy into each param's local
shard instead of load_state_dict, which errors on FSDP DTensors); a Preview-traces
button that overlays the synthetic control on the frozen first frame before
generating; and a streaming per-step denoise log.
convert_droid.py builds a set where the first frame is similar but motion is
diverse, the design that forces trace reliance: it scans DROID (LeRobot mirror),
clusters frame-0 embeddings, and writes the tightest cluster as 121-frame libx264
mp4s + manifest with ONE generic caption, dropping straight into extract_tracks ->
i2v_track preprocess. Adds finetune_wantrack_droid_overfit.yaml (Fun-Control init,
WANTRACK_AUG=0 clean overfit, fresh output_dir) and gitignores .gradio/.
Add the instruments used to diagnose whether the model follows the tracks:
- track_sensitivity.py: per-checkpoint swap-sensitivity (relative velocity shift
when only the tracks / first-frame / text change) -- intrinsic input attribution.
- control_sensitivity metric: intervention-diff (generate twice, same conditioning,
different control) scored in the moving ROI, so a static background can't dilute it.
- gen_eval_artifacts.py + app_viewer.py: pre-generate counterfactual videos
(input vs re-extracted tracks, EPE heatmap) and browse them with no GPU.
- app_action.py: mouse-draw a motion and hot-swap checkpoints.
- test_controllability.py / cotracker_epe / app.py: clip_feature + minor updates.
Initialize the 52ch TrackWan from Wan2.1-Fun-1.3B-Control so the track-slot
patch-embed channels start from Fun-Control's PRETRAINED control channels
(non-zero). Combined with the zero-init track head this gives step-0 == teacher
while still letting gradient reach the track encoder -- avoiding the double
zero-init deadlock of the from-scratch Fun-InP path (where both the track
patch-embed channels and the track-head proj are zero, so the track pathway
never trains). Adds the converter, the matching track_config channel handling,
and the InP init updates.
The overfit teacher ignored the first frame (validation looked T2V): the
init came from FastWan2.1-T2V-1.3B, whose image-conditioning channels are
zero/untrained and which has no CLIP pathway, so first-frame and track
conditioning received ~no gradient.
Re-base on Wan2.1-Fun-1.3B-InP (pretrained concat-I2V + CLIP):
- convert_trackwan_init: widen patch-embed base_in(36)->52, keep the 36
pretrained image-cond channels, zero-init only the 16 track channels;
inherit image_dim from the base config; merge sharded safetensors.
- trackwan config: image_dim=1280 builds the DiT image_embedder.
- preprocess: compute + store clip_feature (CLIP frame-0 embedding) and
add image_encoder/image_processor to required modules.
- schema: clip_feature_{bytes,shape,dtype} in pyarrow_schema_i2v_track.
- wantrack / track_validation / trackwan_infer: feed
encoder_hidden_states_image through train, validation, and inference.
- yaml: point at the Fun-InP init + re-preprocessed data.
Verified: step-0 validation reproduces the GT first frame (PSNR ~24-26 dB
vs ~9-11 before), i.e. I2V is enforced before any training.
shaoxiongduan