shaoxiongduan
34a0a81be1
TrackWan = Wan2.2 + MotionStream point-track motion control. Stage-1 bidirectional I2V finetune (text kept), built on the new modular trainer. - DiT: TrackWanTransformer3DModel (52ch in = 16 noisy + 20 I2V + 16 track), TrackEncoder (sinusoidal track-ID embed, scatter, 4x temporal conv, zero-init head); config + registry. - Trainer: WanTrackModel wrapper, i2v_track preprocess pipeline + pyarrow schema, TrackValidationCallback, finetune_wantrack_i2v.yaml. - data_pipeline/: video gen (Wan2.2-A14B), CoTracker3 track extraction, 52ch init conversion, synthetic track authoring, controllability (CoTracker EPE) eval, standalone inference, Gradio demo. - eval: motion.cotracker_epe metric. research_log/ lab notebook. - fixes: FA3 flash_attn_func tuple unwrap; decord fallback for removed torchvision.io.read_video; preprocess flush write_remainder.
41 lines
2.2 KiB
Markdown
41 lines
2.2 KiB
Markdown
# Data Pipeline — Progress Log
|
||
|
||
Project: build a `(video, prompt, point-tracks)` dataset to train a motion-controlled
|
||
real-time interactive video model (bidirectional finetune stage first).
|
||
|
||
Branch: `shao/realtime-bidir`. This `data_pipeline/` dir holds the **Stage-0** data
|
||
generation scripts (everything *upstream* of FastVideo's `preprocess` pipeline).
|
||
|
||
## Pipeline shape (where we are)
|
||
|
||
```
|
||
vidprom prompts ──> generate_videos.py ──> videos/*.mp4 + videos2caption.json + merge.txt
|
||
│
|
||
extract_tracks.py ──> tracks/*.npz (+ patch points_path into the json)
|
||
│
|
||
[TODO] v1_preprocess.py --preprocess_task i2v_points ──> parquet
|
||
│
|
||
[TODO] bidir I2V+points trainer
|
||
```
|
||
|
||
## Status
|
||
|
||
- [x] Located existing FastVideo preprocess stack + MatrixGame2 control-signal pattern (the template).
|
||
- [x] Confirmed env = repo `.venv` (`/mnt/weka/home/hao.zhang/shao/FastVideo/.venv/bin/python`).
|
||
- [x] Confirmed Wan2.2-T2V-A14B weights cached (`~/.cache/huggingface/hub/models--Wan-AI--Wan2.2-T2V-A14B-Diffusers`).
|
||
- [x] Wrote `generate_videos.py` (Wan2.2-14B T2V → mp4 + manifest).
|
||
- [x] Wrote `extract_tracks.py` (CoTracker v3 50×50 grid → npz, patch manifest).
|
||
- [~] Smoke-test launched on fs-mbz-gpu-538 (`run.sh --smoke`, background). Node validated:
|
||
internet=yes (downloads OK on the node), 8 GPUs idle, `.venv` imports fastvideo + torch
|
||
2.11.0/CUDA, cotracker3_offline loads (25.4M params). Prompts downloaded: 248,221 (140 MB).
|
||
- [ ] Scale to 50, then 5k.
|
||
- [ ] Extend preprocess with an `i2v_points` task (mirror MatrixGame2).
|
||
|
||
## Open questions / decisions pending
|
||
|
||
- Training model for the bidir stage: 1.3B/480p for plumbing bring-up vs straight to 14B (data gen is fixed at 14B/720p regardless — user confirmed).
|
||
- Confirm conditioning shape: I2V + points (input image + per-frame points → video). Assumed yes.
|
||
- Point sampling at train time: store full 2500-pt grid + visibility; sample 1–200 in the trainer.
|
||
|
||
See `DECISIONS.md` for rationale, `RUNBOOK.md` for exact commands.
|