_augment_tracks gated the object-coverage + low-rank-weighted + uniform draw on
`0 < max_points < N`, so with the default WANTRACK_MAX_POINTS=2500 and the full
2500-track grid (max_points == N) the entire subsample block was skipped -- the
model saw all 2500 tracks every step and only chunk-masking ran. Gate on
`min_points < N` instead so K~U[1000,2500] actually samples (verified: kept
1179/1437/1573 across steps, was a constant 2500).
Also: TrackValidation now applies the same sampler (loads object_ids/track_weights,
runs _augment_tracks deterministically) so validation is conditioned on -- and
overlays -- the sampled subset, not the full grid. Env-gated WANTRACK_DEBUG=1 logs
per-step sampling stats (kept count, object coverage, coord ranges). Adds a
single-video (golf) overfit config for debugging.
Sampler-on finetune configs (coverage + low-rank-weighted + uniform track
sampling, p_mask chunk masking) across three data regimes: unfiltered
egocentric EPIC-200, camera-static-filtered EPIC-200 (single kitchen), and the
Wan-200 synthetic set re-run with the sampler on (vs the AUG=0 overfit baseline).
- track_informativeness.py: compare abs-motion vs affine/low-rank residual vs
motion/subspace clustering vs DINOv2 semantic embeddings for scoring which
point tracks are informative; per-clip camera-motion metric; gradio viz.
- sampling_viz.py: render the exact _augment_tracks sampler (coverage + low-rank
weighted + uniform) on a preprocessed dataset to inspect which traces it keeps.
- find_static_windows.py: phase-correlation per-frame camera-motion signal (recovers
after turns) + greedy non-overlapping static-window extraction to build a
relatively-still clip dataset from long untrimmed videos.
- segment_viz.py: FastSAM config sweep + fixes (config/clip filters, allowed_paths).
- _augment_tracks: default random 1000-2500 track sampling (was 1-200), contiguous
chunk masking (was per-frame), diversity weighting (favor moving/unique tracks,
keep some static), and an object-coverage hook (>=1 track per SAM segment via
optional object_ids). Sampling ON by default; CFG dropout now default off.
- trackwan_infer.generate: motion classifier-free guidance (guidance_scale) using
the base no-tracks I2V branch as the unconditional.
- test_motion_cfg.py: quick guidance-scale EPE sweep.
- finetune_wantrack_wan200.yaml: 50->200 data-scale run (same params).
- droid_dataset_dashboard.py: render per-clip CoTracker overlays + motion-coverage
stats (frac_moving, mean/max motion) + near-duplicate grouping, and a gradio
gallery sorted least-motion-first to surface weak/duplicate clips.
- app_action.py: DTensor-safe checkpoint hot-swap (copy into each param's local
shard instead of load_state_dict, which errors on FSDP DTensors); a Preview-traces
button that overlays the synthetic control on the frozen first frame before
generating; and a streaming per-step denoise log.
convert_droid.py builds a set where the first frame is similar but motion is
diverse, the design that forces trace reliance: it scans DROID (LeRobot mirror),
clusters frame-0 embeddings, and writes the tightest cluster as 121-frame libx264
mp4s + manifest with ONE generic caption, dropping straight into extract_tracks ->
i2v_track preprocess. Adds finetune_wantrack_droid_overfit.yaml (Fun-Control init,
WANTRACK_AUG=0 clean overfit, fresh output_dir) and gitignores .gradio/.
Add the instruments used to diagnose whether the model follows the tracks:
- track_sensitivity.py: per-checkpoint swap-sensitivity (relative velocity shift
when only the tracks / first-frame / text change) -- intrinsic input attribution.
- control_sensitivity metric: intervention-diff (generate twice, same conditioning,
different control) scored in the moving ROI, so a static background can't dilute it.
- gen_eval_artifacts.py + app_viewer.py: pre-generate counterfactual videos
(input vs re-extracted tracks, EPE heatmap) and browse them with no GPU.
- app_action.py: mouse-draw a motion and hot-swap checkpoints.
- test_controllability.py / cotracker_epe / app.py: clip_feature + minor updates.
Initialize the 52ch TrackWan from Wan2.1-Fun-1.3B-Control so the track-slot
patch-embed channels start from Fun-Control's PRETRAINED control channels
(non-zero). Combined with the zero-init track head this gives step-0 == teacher
while still letting gradient reach the track encoder -- avoiding the double
zero-init deadlock of the from-scratch Fun-InP path (where both the track
patch-embed channels and the track-head proj are zero, so the track pathway
never trains). Adds the converter, the matching track_config channel handling,
and the InP init updates.
The overfit teacher ignored the first frame (validation looked T2V): the
init came from FastWan2.1-T2V-1.3B, whose image-conditioning channels are
zero/untrained and which has no CLIP pathway, so first-frame and track
conditioning received ~no gradient.
Re-base on Wan2.1-Fun-1.3B-InP (pretrained concat-I2V + CLIP):
- convert_trackwan_init: widen patch-embed base_in(36)->52, keep the 36
pretrained image-cond channels, zero-init only the 16 track channels;
inherit image_dim from the base config; merge sharded safetensors.
- trackwan config: image_dim=1280 builds the DiT image_embedder.
- preprocess: compute + store clip_feature (CLIP frame-0 embedding) and
add image_encoder/image_processor to required modules.
- schema: clip_feature_{bytes,shape,dtype} in pyarrow_schema_i2v_track.
- wantrack / track_validation / trackwan_infer: feed
encoder_hidden_states_image through train, validation, and inference.
- yaml: point at the Fun-InP init + re-preprocessed data.
Verified: step-0 validation reproduces the GT first frame (PSNR ~24-26 dB
vs ~9-11 before), i.e. I2V is enforced before any training.
shaoxiongduan