shaoxiongduan
92cc893cfc
Training:
- finetune.py: optional track/base param groups (WANTRACK_TRACK_GROUP) so the
track pathway can take a different LR (WANTRACK_TRACK_LR_MULT); patch_embedding
gradient masked to the track slot [:, 36:]. Raises if nothing matches.
- track_warmup.py callback: hold the pretrained DiT at lr 0 while the track
pathway warms up, then ramp it back in.
14B configs + held-alloc launchers:
- finetune_wantrack_{synth_sparse_fixed,synth_sparse_random,openvid_sparse_merged_bias,
synth_stage2}_14b + openvid_14b_{warmup,lwlr} + 1.3B stage2 configs.
- run_wan14b_held.sh (held alloc + auto-restart + rank cleanup, per-stage WANTRACK
env), acquire_nodes.sh, chain_stepAB.sh, run_stepB_seed.sh, run_stepC_merge.sh,
watch_gate.sh, run_synth_stage2_slurm.sh.
Init / diagnostics:
- convert_trackwan_init_v2.py: --pe-src to lift patch_embedding[:, 36:] and
--track-src to lift track_encoder together (co-adapted merge); sharded-source support.
- check_gate_growth.py, gradient_flow_analysis.py, gradient_flow_summarize.py.
Data pipeline: segment_tracks.py sharding; decode/segment/tracks slurm scripts;
precompute_null_text, two_track_val, cfg_ablation, lwlr_experiment.
Inference: app_multi_trace.py updates.
24 lines
1.2 KiB
Bash
24 lines
1.2 KiB
Bash
#!/usr/bin/env bash
|
|
# Sample the WanTrack gate at each new checkpoint and append to a TSV, so we can see whether it
|
|
# is on track for the ~0.011 the successful 1.3B merged init reached. Checkpoints rotate
|
|
# (total_limit=3), so poll faster than they are produced or the early trajectory is lost.
|
|
set -uo pipefail
|
|
WORK=/mnt/lustre/vlm-s4duan; REPO=$WORK/FastVideo
|
|
OUT="${OUT:-$WORK/wantrack_14b_synth_sparse_fixed_out}"
|
|
LOG="${LOG:-$WORK/logs/gate_growth.tsv}"
|
|
cd "$REPO"; source .venv/bin/activate
|
|
export HOME=$WORK PYTHONPATH=$REPO NCCL_CUMEM_ENABLE=0 RANK=0 LOCAL_RANK=0 WORLD_SIZE=1 \
|
|
MASTER_ADDR=127.0.0.1 MASTER_PORT=29600
|
|
touch "$LOG"
|
|
while :; do
|
|
for d in "$OUT"/checkpoint-*; do
|
|
[ -f "$d/dcp/.metadata" ] || continue
|
|
step=$(basename "$d" | sed 's/checkpoint-//')
|
|
grep -q "^${step} " "$LOG" 2>/dev/null && continue
|
|
line=$(python data_pipeline/check_gate_growth.py "$d" 2>/dev/null | grep -oE "gate\[:, 36:\] std=[0-9.]+" | grep -oE "[0-9.]+$")
|
|
[ -n "$line" ] && { printf "%s\t%s\t%s\n" "$step" "$line" "$(date +%H:%M:%S)" >> "$LOG"; echo "step $step gate=$line"; }
|
|
done
|
|
pgrep -f "run_wan14b_held.sh" >/dev/null || { echo "[watch] no launcher; exiting"; break; }
|
|
sleep 240
|
|
done
|