nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Use 21 latent frames for OpenVid causal training

Browse Source
This commit is contained in:
H1yori233
2026-07-22 21:29:47 -07:00
parent e172215880
commit 2fa0f70ee8
16 changed files with 124 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/train/configs/ablation/openvid_causal_a12_a15/A12/cd.yaml
+4 -4
View File
@@ -37,10 +37,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -60,7 +60,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A12_sink1_local6_relativistic_chunk3_cd2k_from_tf3k_openvid_gbs64
run_name: A12_sink1_local6_relativistic_chunk3_cd2k_from_tf3k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -73,7 +73,7 @@ callbacks:
sampling_steps:
- 4
guidance_scale: 3.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A12/cd/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A12/sf.yaml
+4 -4
View File
@@ -53,10 +53,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -76,7 +76,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A12_sink1_local6_relativistic_chunk3_sf1k_from_cd2k_openvid_gbs64
run_name: A12_sink1_local6_relativistic_chunk3_sf1k_from_cd2k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -92,7 +92,7 @@ callbacks:
sampling_steps:
- 4
guidance_scale: 3.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A12/sf/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A12/tf.yaml
+4 -4
View File
@@ -23,10 +23,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -46,7 +46,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A12_sink1_local6_relativistic_chunk3_tf3k_openvid_gbs64
run_name: A12_sink1_local6_relativistic_chunk3_tf3k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -59,7 +59,7 @@ callbacks:
sampling_steps:
- 40
guidance_scale: 6.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A12/tf/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A13/cd.yaml
+4 -4
View File
@@ -37,10 +37,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -60,7 +60,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A13_sink0_local6_relativistic_chunk3_cd2k_from_tf3k_openvid_gbs64
run_name: A13_sink0_local6_relativistic_chunk3_cd2k_from_tf3k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -73,7 +73,7 @@ callbacks:
sampling_steps:
- 4
guidance_scale: 3.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A13/cd/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A13/sf.yaml
+4 -4
View File
@@ -53,10 +53,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -76,7 +76,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A13_sink0_local6_relativistic_chunk3_sf1k_from_cd2k_openvid_gbs64
run_name: A13_sink0_local6_relativistic_chunk3_sf1k_from_cd2k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -92,7 +92,7 @@ callbacks:
sampling_steps:
- 4
guidance_scale: 3.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A13/sf/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A13/tf.yaml
+4 -4
View File
@@ -23,10 +23,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -46,7 +46,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A13_sink0_local6_relativistic_chunk3_tf3k_openvid_gbs64
run_name: A13_sink0_local6_relativistic_chunk3_tf3k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -59,7 +59,7 @@ callbacks:
sampling_steps:
- 40
guidance_scale: 6.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A13/tf/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A14/cd.yaml
+4 -4
View File
@@ -37,10 +37,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -60,7 +60,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A14_sink1_local6_absolute_chunk3_cd2k_from_tf3k_openvid_gbs64
run_name: A14_sink1_local6_absolute_chunk3_cd2k_from_tf3k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -73,7 +73,7 @@ callbacks:
sampling_steps:
- 4
guidance_scale: 3.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A14/cd/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A14/sf.yaml
+4 -4
View File
@@ -53,10 +53,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -76,7 +76,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A14_sink1_local6_absolute_chunk3_sf1k_from_cd2k_openvid_gbs64
run_name: A14_sink1_local6_absolute_chunk3_sf1k_from_cd2k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -92,7 +92,7 @@ callbacks:
sampling_steps:
- 4
guidance_scale: 3.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A14/sf/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A14/tf.yaml
+4 -4
View File
@@ -23,10 +23,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -46,7 +46,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A14_sink1_local6_absolute_chunk3_tf3k_openvid_gbs64
run_name: A14_sink1_local6_absolute_chunk3_tf3k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -59,7 +59,7 @@ callbacks:
sampling_steps:
- 40
guidance_scale: 6.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A14/tf/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A15/cd.yaml
+4 -4
View File
@@ -40,10 +40,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -63,7 +63,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A15_sink1_local6_relativistic_framewise_cd2k_from_tf3k_openvid_gbs64
run_name: A15_sink1_local6_relativistic_framewise_cd2k_from_tf3k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -76,7 +76,7 @@ callbacks:
sampling_steps:
- 4
guidance_scale: 3.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A15/cd/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A15/sf.yaml
+4 -4
View File
@@ -54,10 +54,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -77,7 +77,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A15_sink1_local6_relativistic_framewise_sf1k_from_cd2k_openvid_gbs64
run_name: A15_sink1_local6_relativistic_framewise_sf1k_from_cd2k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -93,7 +93,7 @@ callbacks:
sampling_steps:
- 4
guidance_scale: 3.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A15/sf/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/A15/tf.yaml
+4 -4
View File
@@ -24,10 +24,10 @@ training:
train_batch_size: 2
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 30
num_latent_t: 21
num_height: 480
num_width: 832
num_frames: 117
num_frames: 81
optimizer:
learning_rate: 2.0e-06
betas:
@@ -47,7 +47,7 @@ training:
trackers:
- wandb
project_name: causal_forcing_openvid_a12_a15
run_name: A15_sink1_local6_relativistic_framewise_tf3k_openvid_gbs64
run_name: A15_sink1_local6_relativistic_framewise_tf3k_openvid_81f21l_gbs64
model:
enable_gradient_checkpointing_type: full
callbacks:
@@ -60,7 +60,7 @@ callbacks:
sampling_steps:
- 40
guidance_scale: 6.0
num_frames: 249
num_frames: 81
output_dir: /mnt/lustre/vlm-k1kong/experiments/openvid_causal_a12_a15/20260717_014659_openvid_a12_a15/A15/tf/validation
offload_training_state: true
unload_pipeline_after_validation: true
examples/train/configs/ablation/openvid_causal_a12_a15/README.md
+3 -4
View File
@@ -16,10 +16,9 @@ All stages use 4 GPUs, microbatch 2/rank, gradient accumulation 8, hence global
batch = 2 * 4 * 8 = 64. `dataloader_num_workers=0` limits shared-memory and
Lustre prefetch pressure.
All four conditions train on the first 30 of the source's 31 latent frames
(`117` raw frames). This makes chunk-3 TF, CD, and SF use ten identical
3-latent blocks and removes a known T=31 remainder-partition mismatch. A15 is
also cropped to T=30 so framewise vs chunk-3 remains a length-matched ablation.
All four conditions use exactly 21 latent frames / 81 raw frames in TF, CD,
SF, and validation. Chunk-3 conditions therefore use seven identical
three-latent blocks. A15 is length-matched and uses framewise blocks.
A15 "framewise" means `num_frames_per_block=1` on every causal role, plus
`method.chunk_size=1` in TF and SF. Causal CD has no independent-frame timestep
scripts/train/openvid_causal_a12_a15/train_stage.sh
+18
View File
@@ -58,6 +58,8 @@ from __future__ import annotations
import sys
import yaml
from fastvideo.train.utils.config import load_run_config
@@ -72,6 +74,22 @@ expected = {
"dataloader_num_workers": 0,
}
actual = {name: getattr(data, name) for name in expected}
with open(config_path, encoding="utf-8") as handle:
raw = yaml.safe_load(handle)
expected.update({
"num_latent_t": 21,
"num_frames": 81,
})
actual.update({
"num_latent_t": data.num_latent_t,
"num_frames": data.num_frames,
})
validation_frames = raw["callbacks"]["validation"]["num_frames"]
if validation_frames != 81:
raise SystemExit(
f"OpenVid sequence-length preflight failed for {config_path}: "
f"callbacks.validation.num_frames expected 81, got {validation_frames!r}"
)
errors = [
f"{name}: expected {value!r}, got {actual[name]!r}"
for name, value in expected.items()
scripts/train/prepare_openvid_a12_a15.py
+33 -13
View File
@@ -12,6 +12,9 @@ STREAMING_MANIFEST_PATH = "/mnt/lustre/vlm-k1kong/dataset-index/openvid/streamin
MODEL_ID = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
TEACHER_ID = "Wan-AI/Wan2.1-T2V-14B-Diffusers"
REQUIRED_ANCESTOR = "30ada30e4c6b05aa68cd1eb8940a34d149457147"
TRAIN_NUM_LATENTS = 21
TRAIN_NUM_FRAMES = 81
VALIDATION_NUM_FRAMES = 81
CONDITIONS = {
"A12": {"sink": 1, "local": 6, "rope": "relativistic", "framewise": False},
@@ -40,13 +43,13 @@ def training(stage_dir: str, *, steps: int, keep: int) -> dict:
"train_batch_size": 2,
"training_cfg_rate": 0.0,
"seed": 1000,
# The source tensor has T=31. Crop its tail deterministically so
# chunk-3 TF/CD/SF all use exactly ten aligned blocks. Keeping
# T=31 would make TF/CD [3x10,1] but SF [4,3x9].
"num_latent_t": 30,
# Keep every stage on the intended 21-latent / 81-frame training
# distribution. For chunk-3 conditions this is exactly seven
# aligned blocks; A15 is length-matched and uses framewise blocks.
"num_latent_t": TRAIN_NUM_LATENTS,
"num_height": 480,
"num_width": 832,
"num_frames": 117,
"num_frames": TRAIN_NUM_FRAMES,
},
"optimizer": {
"learning_rate": 2.0e-6,
@@ -85,7 +88,7 @@ def validation(stage_dir: str, *, distilled: bool, dmd: bool = False) -> dict:
"every_steps": 200,
"sampling_steps": [4] if distilled else [40],
"guidance_scale": 3.0 if distilled else 6.0,
"num_frames": 249,
"num_frames": VALIDATION_NUM_FRAMES,
"output_dir": f"{stage_dir}/validation",
"offload_training_state": True,
"unload_pipeline_after_validation": True,
@@ -143,7 +146,7 @@ def configs(run_root: str, condition: str, spec: dict, repo: str) -> dict[str, d
},
"pipeline": copy.deepcopy(pipeline),
}
tf["training"]["tracker"]["run_name"] = f"{tag}_tf3k_openvid_gbs64"
tf["training"]["tracker"]["run_name"] = f"{tag}_tf3k_openvid_81f21l_gbs64"
cd = {
"models": {
@@ -168,7 +171,7 @@ def configs(run_root: str, condition: str, spec: dict, repo: str) -> dict[str, d
},
"pipeline": copy.deepcopy(pipeline),
}
cd["training"]["tracker"]["run_name"] = f"{tag}_cd2k_from_tf3k_openvid_gbs64"
cd["training"]["tracker"]["run_name"] = f"{tag}_cd2k_from_tf3k_openvid_81f21l_gbs64"
sf_student = causal_role(trainable=True, override=cd_export, framewise=framewise)
sf = {
@@ -214,7 +217,7 @@ def configs(run_root: str, condition: str, spec: dict, repo: str) -> dict[str, d
},
"pipeline": copy.deepcopy(pipeline),
}
sf["training"]["tracker"]["run_name"] = f"{tag}_sf1k_from_cd2k_openvid_gbs64"
sf["training"]["tracker"]["run_name"] = f"{tag}_sf1k_from_cd2k_openvid_81f21l_gbs64"
for config in (tf, cd, sf):
config["callbacks"]["validation"]["dataset_file"] = config["callbacks"]["validation"]["dataset_file"].replace(
@@ -283,6 +286,8 @@ from __future__ import annotations
import sys
import yaml
from fastvideo.train.utils.config import load_run_config
@@ -297,6 +302,22 @@ expected = {
"dataloader_num_workers": 0,
}
actual = {name: getattr(data, name) for name in expected}
with open(config_path, encoding="utf-8") as handle:
raw = yaml.safe_load(handle)
expected.update({
"num_latent_t": 21,
"num_frames": 81,
})
actual.update({
"num_latent_t": data.num_latent_t,
"num_frames": data.num_frames,
})
validation_frames = raw["callbacks"]["validation"]["num_frames"]
if validation_frames != 81:
raise SystemExit(
f"OpenVid sequence-length preflight failed for {config_path}: "
f"callbacks.validation.num_frames expected 81, got {validation_frames!r}"
)
errors = [
f"{name}: expected {value!r}, got {actual[name]!r}"
for name, value in expected.items()
@@ -501,10 +522,9 @@ All stages use 4 GPUs, microbatch 2/rank, gradient accumulation 8, hence global
batch = 2 * 4 * 8 = 64. `dataloader_num_workers=0` limits shared-memory and
Lustre prefetch pressure.
All four conditions train on the first 30 of the source's 31 latent frames
(`117` raw frames). This makes chunk-3 TF, CD, and SF use ten identical
3-latent blocks and removes a known T=31 remainder-partition mismatch. A15 is
also cropped to T=30 so framewise vs chunk-3 remains a length-matched ablation.
All four conditions use exactly 21 latent frames / 81 raw frames in TF, CD,
SF, and validation. Chunk-3 conditions therefore use seven identical
three-latent blocks. A15 is length-matched and uses framewise blocks.
A15 "framewise" means `num_frames_per_block=1` on every causal role, plus
`method.chunk_size=1` in TF and SF. Causal CD has no independent-frame timestep
tests/local_tests/train/test_openvid_launcher_modes.py
+22
View File
@@ -1,6 +1,7 @@
from __future__ import annotations
import ast
import importlib.util
import os
import py_compile
import re
@@ -34,6 +35,27 @@ BASH = shutil.which("bash")
pytestmark = pytest.mark.skipif(BASH is None, reason="bash is required")
def test_all_stages_are_locked_to_21_latents_and_81_frames() -> None:
spec = importlib.util.spec_from_file_location("openvid_prepare", PREPARE)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
for condition, condition_spec in module.CONDITIONS.items():
generated = module.configs(
f"/tmp/{condition}", condition, condition_spec, "/repo"
)
assert set(generated) == {"tf", "cd", "sf"}
for stage, config in generated.items():
data = config["training"]["data"]
assert data["num_latent_t"] == 21, (condition, stage)
assert data["num_frames"] == 81, (condition, stage)
assert config["callbacks"]["validation"]["num_frames"] == 81, (
condition,
stage,
)
def _bash_path(path: Path) -> str:
"""Return a path understood by bash.exe/WSL as well as native Linux bash."""
path = path.resolve()