shaoxiongduan
92cc893cfc
Training:
- finetune.py: optional track/base param groups (WANTRACK_TRACK_GROUP) so the
track pathway can take a different LR (WANTRACK_TRACK_LR_MULT); patch_embedding
gradient masked to the track slot [:, 36:]. Raises if nothing matches.
- track_warmup.py callback: hold the pretrained DiT at lr 0 while the track
pathway warms up, then ramp it back in.
14B configs + held-alloc launchers:
- finetune_wantrack_{synth_sparse_fixed,synth_sparse_random,openvid_sparse_merged_bias,
synth_stage2}_14b + openvid_14b_{warmup,lwlr} + 1.3B stage2 configs.
- run_wan14b_held.sh (held alloc + auto-restart + rank cleanup, per-stage WANTRACK
env), acquire_nodes.sh, chain_stepAB.sh, run_stepB_seed.sh, run_stepC_merge.sh,
watch_gate.sh, run_synth_stage2_slurm.sh.
Init / diagnostics:
- convert_trackwan_init_v2.py: --pe-src to lift patch_embedding[:, 36:] and
--track-src to lift track_encoder together (co-adapted merge); sharded-source support.
- check_gate_growth.py, gradient_flow_analysis.py, gradient_flow_summarize.py.
Data pipeline: segment_tracks.py sharding; decode/segment/tracks slurm scripts;
precompute_null_text, two_track_val, cfg_ablation, lwlr_experiment.
Inference: app_multi_trace.py updates.
27 lines
1.1 KiB
Bash
27 lines
1.1 KiB
Bash
#!/usr/bin/env bash
|
|
# Seed step B's output dir with step A's final checkpoint, so B starts from A while its config
|
|
# can still use resume_from_checkpoint: latest (which is what makes crash-restarts safe).
|
|
#
|
|
# Hardlinked (cp -al), not copied: a 14B training-state checkpoint is hundreds of GB (bf16
|
|
# weights + fp32 master + Adam m/v), and both dirs live on the same Lustre filesystem.
|
|
# Checkpoints are write-once, so sharing inodes is safe here.
|
|
set -euo pipefail
|
|
WORK=/mnt/lustre/vlm-s4duan
|
|
STEP="${STEP:-1000}"
|
|
SRC="${SRC:-$WORK/wantrack_14b_synth_sparse_fixed_out/checkpoint-$STEP}"
|
|
DST_DIR="${DST_DIR:-$WORK/wantrack_14b_synth_sparse_random_out}"
|
|
DST="$DST_DIR/checkpoint-$STEP"
|
|
|
|
[ -d "$SRC" ] || { echo "[B-seed] missing $SRC"; exit 1; }
|
|
[ -f "$SRC/dcp/.metadata" ] || { echo "[B-seed] $SRC incomplete (no dcp/.metadata) — refusing"; exit 1; }
|
|
|
|
if [ -d "$DST" ]; then
|
|
echo "[B-seed] $DST already exists — leaving it alone"
|
|
else
|
|
mkdir -p "$DST_DIR"
|
|
echo "[B-seed] hardlinking $SRC -> $DST"
|
|
cp -al "$SRC" "$DST"
|
|
fi
|
|
echo "[B-seed] checkpoints now in $DST_DIR:"
|
|
ls -d "$DST_DIR"/checkpoint-* 2>/dev/null | sed 's/^/ /'
|