Files

T

shaoxiongduan 34a0a81be1 [feat] WanTrack: bidirectional I2V + point-track (MotionStream) finetune + eval

TrackWan = Wan2.2 + MotionStream point-track motion control. Stage-1 bidirectional
I2V finetune (text kept), built on the new modular trainer.

- DiT: TrackWanTransformer3DModel (52ch in = 16 noisy + 20 I2V + 16 track), TrackEncoder
  (sinusoidal track-ID embed, scatter, 4x temporal conv, zero-init head); config + registry.
- Trainer: WanTrackModel wrapper, i2v_track preprocess pipeline + pyarrow schema,
  TrackValidationCallback, finetune_wantrack_i2v.yaml.
- data_pipeline/: video gen (Wan2.2-A14B), CoTracker3 track extraction, 52ch init
  conversion, synthetic track authoring, controllability (CoTracker EPE) eval, standalone
  inference, Gradio demo.
- eval: motion.cotracker_epe metric. research_log/ lab notebook.
- fixes: FA3 flash_attn_func tuple unwrap; decord fallback for removed torchvision.io.read_video;
  preprocess flush write_remainder.

2026-07-01 06:36:40 +00:00

2.2 KiB

Raw Blame History

Data Pipeline — Progress Log

Project: build a (video, prompt, point-tracks) dataset to train a motion-controlled real-time interactive video model (bidirectional finetune stage first).

Branch: shao/realtime-bidir. This data_pipeline/ dir holds the Stage-0 data generation scripts (everything upstream of FastVideo's preprocess pipeline).

Pipeline shape (where we are)

vidprom prompts ──> generate_videos.py ──> videos/*.mp4 + videos2caption.json + merge.txt
                                              │
                    extract_tracks.py  ──> tracks/*.npz  (+ patch points_path into the json)
                                              │
                    [TODO] v1_preprocess.py --preprocess_task i2v_points ──> parquet
                                              │
                    [TODO] bidir I2V+points trainer

Status

Located existing FastVideo preprocess stack + MatrixGame2 control-signal pattern (the template).
Confirmed env = repo .venv (/mnt/weka/home/hao.zhang/shao/FastVideo/.venv/bin/python).
Confirmed Wan2.2-T2V-A14B weights cached (~/.cache/huggingface/hub/models--Wan-AI--Wan2.2-T2V-A14B-Diffusers).
Wrote generate_videos.py (Wan2.2-14B T2V → mp4 + manifest).
Wrote extract_tracks.py (CoTracker v3 50×50 grid → npz, patch manifest).
[~] Smoke-test launched on fs-mbz-gpu-538 (run.sh --smoke, background). Node validated: internet=yes (downloads OK on the node), 8 GPUs idle, .venv imports fastvideo + torch 2.11.0/CUDA, cotracker3_offline loads (25.4M params). Prompts downloaded: 248,221 (140 MB).
Scale to 50, then 5k.
Extend preprocess with an i2v_points task (mirror MatrixGame2).

Open questions / decisions pending

Training model for the bidir stage: 1.3B/480p for plumbing bring-up vs straight to 14B (data gen is fixed at 14B/720p regardless — user confirmed).
Confirm conditioning shape: I2V + points (input image + per-frame points → video). Assumed yes.
Point sampling at train time: store full 2500-pt grid + visibility; sample 1–200 in the trainer.

See DECISIONS.md for rationale, RUNBOOK.md for exact commands.

2.2 KiB Raw Blame History Unescape Escape

Data Pipeline — Progress Log

Pipeline shape (where we are)

Status

Open questions / decisions pending

2.2 KiB

Raw Blame History