Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ba371e33ed | ||
|
|
06c8c5f4b8 | ||
|
|
c535797e3d | ||
|
|
59fddb00f2 | ||
|
|
9f59519959 |
@@ -52,8 +52,8 @@ in one commit.
|
||||
script that supports a global dtype downcast flag MUST own an explicit
|
||||
allowlist of fp32-keep tensors, documented at the top of the file.
|
||||
|
||||
2. **The `add-model-conversion` skill** should enforce two checks for any
|
||||
converter that ships a `--cast-bf16`-style flag:
|
||||
2. **Check two things** for any converter that ships a `--cast-bf16`-style
|
||||
flag:
|
||||
- Run the parity test for **every** variant of the model (base, distill,
|
||||
SR, etc.), not just the headline variant. Different variants exercise
|
||||
different code paths.
|
||||
|
||||
@@ -62,10 +62,10 @@ parity rationale:
|
||||
2. **Use layer-by-layer activation hooks** when DiT parity is close-but-not-
|
||||
bit-exact and the gap grows with depth. The
|
||||
`fastvideo/hooks/activation_trace.py` infra exists exactly for this case
|
||||
(`add-model-trace` skill). In MagiHuman's case it would have localized the
|
||||
(see `docs/contributing/activation_trace.md`). In MagiHuman's case it would have localized the
|
||||
first divergence point in one pass.
|
||||
|
||||
3. **The `add-model-port-dit` skill** should explicitly call out:
|
||||
3. **DiT ports must account for:**
|
||||
- SDPA input dtype must match the upstream kernel's internal cast.
|
||||
- Post-attention upcast happens **before** any per-head gate, not after.
|
||||
- Residual stream dtype across block boundaries is a parity invariant.
|
||||
|
||||
@@ -64,6 +64,5 @@ because both sides of the parity check are now correct.
|
||||
`(pT pH pW C)`) at the call site and assert the layout matches the
|
||||
downstream consumer's expectation.
|
||||
|
||||
4. The `add-model-port-dit` skill's parity gate should require an E2E hash
|
||||
check for any DiT that does video token packing, not just latent
|
||||
bit-exactness.
|
||||
4. DiT parity for any model that does video token packing should include an
|
||||
E2E hash check, not just latent bit-exactness.
|
||||
|
||||
@@ -1,173 +0,0 @@
|
||||
---
|
||||
name: add-model-01-prep
|
||||
description: Use at the start of a FastVideo model port to gather required inputs, inspect/download HF weights, clone and install the official reference repo in the current environment, create a local_tests README skeleton, and produce a handoff before conversion or implementation.
|
||||
---
|
||||
|
||||
# Add Model Prep
|
||||
|
||||
## Goal
|
||||
|
||||
Prepare external assets and the shared parity-test environment for a FastVideo
|
||||
model port. Stop before writing conversion scripts, model components, pipeline
|
||||
code, registry entries, or executable parity tests.
|
||||
|
||||
## Ask First
|
||||
|
||||
Ask once, then proceed if the HF token is already exported:
|
||||
|
||||
```text
|
||||
Before prep: (1) official reference repo or Diffusers pipeline URL, (2) HF repo
|
||||
id or local weights path and whether it has a root model_index.json, (3) target
|
||||
model_family, (4) workload types, (5) which token env var is exported:
|
||||
HF_TOKEN, HUGGINGFACE_HUB_TOKEN, or HF_API_KEY, (6) may I stage clone and
|
||||
weights under the FastVideo repo root, and (7) may I install official reference
|
||||
dependencies into the current FastVideo conda/env for parity tests?
|
||||
```
|
||||
|
||||
Useful optional inputs: `pipeline_class`, `reference_dir`, `hf_revision`,
|
||||
`official_revision`, `reuse_hints`, `download_scope`.
|
||||
|
||||
## Rules
|
||||
|
||||
- Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
|
||||
escape hatches, and skip/pass semantics.
|
||||
- Run from the FastVideo repo root.
|
||||
- Use repo-relative defaults: `<ReferenceDir>/`,
|
||||
`official_weights/<model_family>/`, `converted_weights/<model_family>/`.
|
||||
- Install official reference deps into the current FastVideo environment, not a
|
||||
new venv/conda env, so parity tests run both implementations with one shared
|
||||
numeric stack.
|
||||
- If the reference is a Diffusers class/package instead of a cloneable repo,
|
||||
record import path and version instead of cloning.
|
||||
- Prep may create only the local-test README and `PORT_STATUS.md` skeletons;
|
||||
executable `.py` parity tests belong to `../add-model-02-parity/SKILL.md`.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md`. Prep-specific ask cases include
|
||||
overwriting an existing clone or weight directory, installing untrusted/private
|
||||
deps, choosing between incompatible official references, large downloads outside
|
||||
the agreed scope, or missing gated-repo auth setup by env var name.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Verify the repo:
|
||||
|
||||
```bash
|
||||
git rev-parse --show-toplevel
|
||||
```
|
||||
|
||||
Expected markers: `fastvideo/`, `scripts/checkpoint_conversion/`,
|
||||
`scripts/huggingface/download_hf.py`, `fastvideo/registry.py`.
|
||||
|
||||
2. Inspect HF or local weight layout:
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \
|
||||
"Org/Model" \
|
||||
--revision "<revision>" \
|
||||
--json
|
||||
```
|
||||
|
||||
For a local path, replace `Org/Model` with `/path/to/weights`. Record
|
||||
`source_layout`, `needs_conversion`, `model_index_class`, and
|
||||
`components_seen`.
|
||||
|
||||
3. Download HF weights if needed:
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \
|
||||
"Org/Model" \
|
||||
"official_weights/<model_family>" \
|
||||
--revision "<revision>"
|
||||
```
|
||||
|
||||
For selected files, repeat `--file-name`. For partial snapshots, repeat
|
||||
`--allow-pattern` or `--ignore-pattern`. If the user provided a local path,
|
||||
record it instead of copying large weights by default.
|
||||
|
||||
4. Clone the official reference repo if applicable:
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \
|
||||
"<official_repo_url>" \
|
||||
"<ReferenceDir>" \
|
||||
--branch "<tag-or-branch>" \
|
||||
--commit "<commit-sha>" \
|
||||
--update-gitignore
|
||||
```
|
||||
|
||||
Omit `--branch`, `--commit`, or `--update-gitignore` when not needed. The
|
||||
helper refuses to overwrite existing paths and prints remote/HEAD instead.
|
||||
|
||||
5. Keep prep assets ignored. Ensure `.gitignore` includes relevant entries:
|
||||
|
||||
```gitignore
|
||||
/<ReferenceDir>/
|
||||
/official_weights/
|
||||
/converted_weights/
|
||||
```
|
||||
|
||||
6. Follow the official repo's setup instructions in the current environment.
|
||||
Inspect dependency files and README install docs before installing anything:
|
||||
|
||||
- `README*`, install docs, or model-card instructions.
|
||||
- `requirements*.txt`, `pyproject.toml`, `setup.py`, `environment.yml`.
|
||||
|
||||
Use the current FastVideo conda/env. Do not create a new env even if upstream
|
||||
docs recommend one; translate the needed install commands into the active env.
|
||||
Prefer editable/no-deps first so the official source is importable without
|
||||
changing shared pins:
|
||||
|
||||
```bash
|
||||
uv pip install --no-deps -e ./<ReferenceDir>
|
||||
```
|
||||
|
||||
Then install only missing official deps needed for parity imports. Stop before
|
||||
installing requirements that would change FastVideo's core stack. If upstream
|
||||
requires private/non-PyPI deps, record that parity needs a local stub helper
|
||||
rather than pretending setup is complete.
|
||||
|
||||
7. Create the model-family local test skeleton and top-level port state file:
|
||||
|
||||
```bash
|
||||
mkdir -p tests/local_tests/<model_family>
|
||||
cp ".agents/skills/add-model-01-prep/templates/local_tests_readme.md" \
|
||||
tests/local_tests/<model_family>/README.md
|
||||
cp ".agents/skills/add-model-01-prep/templates/port_status.md" \
|
||||
tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
```
|
||||
|
||||
Edit every placeholder in the README and `PORT_STATUS.md`. The README gives
|
||||
later review agents enough information to reproduce the shared environment and
|
||||
run/review parity work:
|
||||
|
||||
- official code URL or import path, local clone path, and commit/version;
|
||||
- HF URL or local weight path, revision, access notes, and token env var name
|
||||
only;
|
||||
- commands already run and any blocked official dependency installs;
|
||||
- shared-env install commands to re-run without changing core pins;
|
||||
- expected local parity test paths and pytest commands;
|
||||
- private-dependency stubs or known setup gaps;
|
||||
- PR/review notes explaining which parity tests are required before handoff.
|
||||
|
||||
Do not include raw tokens, absolute cache paths that are not repo-reproducible,
|
||||
or large generated outputs. If prep is blocked before imports work, still create
|
||||
the README with `official_env_status=blocked` and the exact blocker.
|
||||
|
||||
`PORT_STATUS.md` must follow `../add-model/contracts/port_state.md`. Record open
|
||||
questions and prep issues immediately, using stable IDs such as `Q001` and
|
||||
`I001`. Keep resolved questions/issues in the table with a resolution instead of
|
||||
deleting them.
|
||||
|
||||
## Handoff
|
||||
|
||||
End with the canonical prep handoff contract from
|
||||
`../add-model/contracts/prep_handoff.md` and update the shared state files before
|
||||
handoff.
|
||||
|
||||
## Helper Scripts
|
||||
|
||||
- `scripts/inspect_hf_layout.py`: classify HF/local layout.
|
||||
- `scripts/download_hf_weights.py`: download HF snapshot or selected files.
|
||||
- `scripts/clone_reference_repo.py`: clone reference repo safely.
|
||||
@@ -1,119 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Clone an official reference repo without overwriting existing paths."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Clone a reference repo for FastVideo parity tests.")
|
||||
parser.add_argument("repo_url", help="Official reference repository URL")
|
||||
parser.add_argument("target_dir", help="Directory to clone into")
|
||||
parser.add_argument("--branch", help="Branch or tag to clone")
|
||||
parser.add_argument("--commit", help="Commit SHA to check out after clone")
|
||||
parser.add_argument(
|
||||
"--update-gitignore",
|
||||
action="store_true",
|
||||
help="Add the target directory to .gitignore if missing",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--gitignore",
|
||||
default=".gitignore",
|
||||
help="Path to gitignore file when --update-gitignore is used",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def run(command: list[str], check: bool = True) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
command,
|
||||
check=check,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
|
||||
def print_existing_repo_info(target: Path) -> int:
|
||||
print(f"target_exists: {target}")
|
||||
if not (target / ".git").exists():
|
||||
print("error: target exists but is not a git repo", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
remote = run(["git", "-C", str(target), "remote", "-v"], check=False)
|
||||
head = run(["git", "-C", str(target), "rev-parse", "HEAD"], check=False)
|
||||
if remote.stdout:
|
||||
print("remote_v:")
|
||||
print(remote.stdout.rstrip())
|
||||
if head.stdout:
|
||||
print(f"head: {head.stdout.strip()}")
|
||||
print("not_overwritten: true")
|
||||
return 0
|
||||
|
||||
|
||||
def gitignore_entry_for(target: Path) -> str:
|
||||
root = Path.cwd().resolve()
|
||||
resolved = target.resolve()
|
||||
try:
|
||||
relative = resolved.relative_to(root)
|
||||
except ValueError as exc:
|
||||
raise ValueError("--update-gitignore requires target_dir to be under the current directory") from exc
|
||||
|
||||
text = relative.as_posix().rstrip("/")
|
||||
return "/" + text + "/"
|
||||
|
||||
|
||||
def update_gitignore(path: Path, target: Path) -> bool:
|
||||
entry = gitignore_entry_for(target)
|
||||
existing = path.read_text().splitlines() if path.exists() else []
|
||||
if entry in existing:
|
||||
return False
|
||||
|
||||
new_text = "\n".join(existing).rstrip("\n")
|
||||
if new_text:
|
||||
new_text += "\n"
|
||||
new_text += entry + "\n"
|
||||
path.write_text(new_text)
|
||||
return True
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
target = Path(args.target_dir)
|
||||
|
||||
if target.exists():
|
||||
return print_existing_repo_info(target)
|
||||
|
||||
command = ["git", "clone", "--depth", "1"]
|
||||
if args.branch:
|
||||
command.extend(["--branch", args.branch])
|
||||
command.extend([args.repo_url, str(target)])
|
||||
|
||||
try:
|
||||
run(command)
|
||||
if args.commit:
|
||||
run(["git", "-C", str(target), "fetch", "--depth", "1", "origin", args.commit])
|
||||
run(["git", "-C", str(target), "checkout", args.commit])
|
||||
except subprocess.CalledProcessError as exc:
|
||||
if exc.stdout:
|
||||
print(exc.stdout, end="")
|
||||
if exc.stderr:
|
||||
print(exc.stderr, end="", file=sys.stderr)
|
||||
return exc.returncode
|
||||
|
||||
head = run(["git", "-C", str(target), "rev-parse", "HEAD"])
|
||||
print(f"cloned: {target}")
|
||||
print(f"head: {head.stdout.strip()}")
|
||||
|
||||
if args.update_gitignore:
|
||||
changed = update_gitignore(Path(args.gitignore), target)
|
||||
print(f"gitignore_updated: {str(changed).lower()}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,103 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Download HF weights using the standard FastVideo token env vars."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Download a HF model snapshot or selected files into a local directory.")
|
||||
parser.add_argument("repo_id", help="HF repo id, for example Org/Model")
|
||||
parser.add_argument("local_dir", help="Destination directory")
|
||||
parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)")
|
||||
parser.add_argument("--revision", help="HF branch, tag, or commit")
|
||||
parser.add_argument(
|
||||
"--file-name",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Download one file; may be repeated. If omitted, download full snapshot.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow-pattern",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Snapshot allow pattern; may be repeated. Ignored when --file-name is used.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ignore-pattern",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Snapshot ignore pattern; may be repeated. Ignored when --file-name is used.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def resolve_token() -> tuple[str | None, str | None]:
|
||||
for key in HF_TOKEN_ENV_KEYS:
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
return key, value
|
||||
return None, None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
token_env, token = resolve_token()
|
||||
local_dir = Path(args.local_dir).expanduser()
|
||||
|
||||
if token_env:
|
||||
print(f"token_env: {token_env}")
|
||||
else:
|
||||
print("token_env: none", file=sys.stderr)
|
||||
|
||||
try:
|
||||
if local_dir.exists() and not local_dir.is_dir():
|
||||
print(
|
||||
f"error: destination exists and is not a directory: {local_dir}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
local_dir.mkdir(parents=True, exist_ok=True)
|
||||
if args.file_name:
|
||||
from huggingface_hub import hf_hub_download
|
||||
|
||||
for file_name in args.file_name:
|
||||
path = hf_hub_download(
|
||||
repo_id=args.repo_id,
|
||||
filename=file_name,
|
||||
repo_type=args.repo_type,
|
||||
revision=args.revision,
|
||||
local_dir=str(local_dir),
|
||||
token=token,
|
||||
)
|
||||
print(f"downloaded_file: {path}")
|
||||
else:
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
path = snapshot_download(
|
||||
repo_id=args.repo_id,
|
||||
repo_type=args.repo_type,
|
||||
revision=args.revision,
|
||||
local_dir=str(local_dir),
|
||||
token=token,
|
||||
allow_patterns=args.allow_pattern or None,
|
||||
ignore_patterns=args.ignore_pattern or None,
|
||||
)
|
||||
print(f"downloaded_snapshot: {path}")
|
||||
except Exception as exc: # noqa: BLE001 - CLI should print concise failures.
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"local_dir: {local_dir.resolve()}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,260 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Inspect a Hugging Face repo or local weight directory layout."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY")
|
||||
RAW_WEIGHT_SUFFIXES = (".safetensors", ".pt", ".pth", ".ckpt", ".bin")
|
||||
KNOWN_COMPONENTS = {
|
||||
"audio_vae",
|
||||
"conditioner",
|
||||
"feature_extractor",
|
||||
"image_encoder",
|
||||
"scheduler",
|
||||
"text_encoder",
|
||||
"text_encoder_2",
|
||||
"tokenizer",
|
||||
"tokenizer_2",
|
||||
"transformer",
|
||||
"transformer_2",
|
||||
"unet",
|
||||
"upsampler",
|
||||
"vae",
|
||||
"vocoder",
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Classify a HF repo or local directory as Diffusers, raw, custom, or unknown.")
|
||||
parser.add_argument("source", help="HF repo id or local weights directory")
|
||||
parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)")
|
||||
parser.add_argument("--revision", help="HF revision to inspect")
|
||||
parser.add_argument(
|
||||
"--max-local-files",
|
||||
type=int,
|
||||
default=20000,
|
||||
help="Maximum local files to scan recursively (default: 20000)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sample-limit",
|
||||
type=int,
|
||||
default=80,
|
||||
help="Number of file paths to print in human output (default: 80)",
|
||||
)
|
||||
parser.add_argument("--json", action="store_true", help="Emit JSON only")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def resolve_token() -> tuple[str | None, str | None]:
|
||||
for key in HF_TOKEN_ENV_KEYS:
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
return key, value
|
||||
return None, None
|
||||
|
||||
|
||||
def load_local_files(root: Path, max_files: int) -> tuple[list[str], bool]:
|
||||
files: list[str] = []
|
||||
truncated = False
|
||||
for path in root.rglob("*"):
|
||||
if not path.is_file():
|
||||
continue
|
||||
files.append(path.relative_to(root).as_posix())
|
||||
if len(files) >= max_files:
|
||||
truncated = True
|
||||
break
|
||||
return sorted(files), truncated
|
||||
|
||||
|
||||
def load_local_model_index(root: Path) -> tuple[dict[str, Any] | None, str | None]:
|
||||
index_path = root / "model_index.json"
|
||||
if not index_path.is_file():
|
||||
return None, None
|
||||
try:
|
||||
return json.loads(index_path.read_text()), None
|
||||
except Exception as exc: # noqa: BLE001 - surface malformed JSON clearly.
|
||||
return None, f"failed to parse local model_index.json: {exc}"
|
||||
|
||||
|
||||
def load_remote_files(
|
||||
repo_id: str,
|
||||
repo_type: str,
|
||||
revision: str | None,
|
||||
token: str | None,
|
||||
) -> list[str]:
|
||||
from huggingface_hub import list_repo_files
|
||||
|
||||
return sorted(list_repo_files(
|
||||
repo_id,
|
||||
repo_type=repo_type,
|
||||
revision=revision,
|
||||
token=token,
|
||||
))
|
||||
|
||||
|
||||
def load_remote_model_index(
|
||||
repo_id: str,
|
||||
repo_type: str,
|
||||
revision: str | None,
|
||||
token: str | None,
|
||||
) -> tuple[dict[str, Any] | None, str | None]:
|
||||
from huggingface_hub import hf_hub_download
|
||||
|
||||
try:
|
||||
path = hf_hub_download(
|
||||
repo_id=repo_id,
|
||||
filename="model_index.json",
|
||||
repo_type=repo_type,
|
||||
revision=revision,
|
||||
token=token,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - missing/inaccessible file is data.
|
||||
return None, f"failed to download model_index.json: {exc}"
|
||||
|
||||
try:
|
||||
return json.loads(Path(path).read_text()), None
|
||||
except Exception as exc: # noqa: BLE001 - surface malformed JSON clearly.
|
||||
return None, f"failed to parse remote model_index.json: {exc}"
|
||||
|
||||
|
||||
def root_file_names(files: list[str]) -> set[str]:
|
||||
return {name for name in files if "/" not in name}
|
||||
|
||||
|
||||
def component_names(files: list[str], model_index: dict[str, Any] | None) -> list[str]:
|
||||
components: set[str] = set()
|
||||
for name in files:
|
||||
parts = name.split("/", 1)
|
||||
if len(parts) != 2:
|
||||
continue
|
||||
top, rest = parts
|
||||
if top in KNOWN_COMPONENTS or rest == "config.json":
|
||||
components.add(top)
|
||||
|
||||
if model_index:
|
||||
for key, value in model_index.items():
|
||||
if key.startswith("_"):
|
||||
continue
|
||||
if isinstance(value, list) and len(value) == 2:
|
||||
components.add(key)
|
||||
|
||||
return sorted(components)
|
||||
|
||||
|
||||
def classify_layout(
|
||||
files: list[str],
|
||||
model_index: dict[str, Any] | None,
|
||||
components: list[str],
|
||||
) -> tuple[str, str]:
|
||||
roots = root_file_names(files)
|
||||
raw_weight_files = [name for name in roots if name.endswith(RAW_WEIGHT_SUFFIXES)]
|
||||
has_model_index = "model_index.json" in roots or model_index is not None
|
||||
|
||||
if has_model_index and components:
|
||||
return "diffusers", "no"
|
||||
if has_model_index:
|
||||
return "custom", "unknown"
|
||||
if raw_weight_files:
|
||||
return "raw_official", "yes"
|
||||
if any(name.endswith(RAW_WEIGHT_SUFFIXES) for name in files):
|
||||
return "custom", "yes"
|
||||
return "unknown", "unknown"
|
||||
|
||||
|
||||
def build_result(args: argparse.Namespace) -> dict[str, Any]:
|
||||
token_env, token = resolve_token()
|
||||
source_path = Path(args.source).expanduser()
|
||||
is_local = source_path.exists()
|
||||
|
||||
if is_local:
|
||||
root = source_path.resolve()
|
||||
if not root.is_dir():
|
||||
raise ValueError(f"local source is not a directory: {root}")
|
||||
files, truncated = load_local_files(root, args.max_local_files)
|
||||
model_index, model_index_error = load_local_model_index(root)
|
||||
source_kind = "local"
|
||||
source = str(root)
|
||||
else:
|
||||
files = load_remote_files(args.source, args.repo_type, args.revision, token)
|
||||
truncated = False
|
||||
model_index, model_index_error = load_remote_model_index(
|
||||
args.source,
|
||||
args.repo_type,
|
||||
args.revision,
|
||||
token,
|
||||
)
|
||||
source_kind = "hf"
|
||||
source = args.source
|
||||
|
||||
components = component_names(files, model_index)
|
||||
source_layout, needs_conversion = classify_layout(files, model_index, components)
|
||||
|
||||
return {
|
||||
"source": source,
|
||||
"source_kind": source_kind,
|
||||
"repo_type": None if is_local else args.repo_type,
|
||||
"revision": args.revision,
|
||||
"token_env": token_env,
|
||||
"source_layout": source_layout,
|
||||
"needs_conversion": needs_conversion,
|
||||
"model_index_class": (model_index or {}).get("_class_name"),
|
||||
"model_index_diffusers_version": (model_index or {}).get("_diffusers_version"),
|
||||
"model_index_error": model_index_error,
|
||||
"components_seen": components,
|
||||
"file_count": len(files),
|
||||
"file_scan_truncated": truncated,
|
||||
"files_sample": files[:args.sample_limit],
|
||||
}
|
||||
|
||||
|
||||
def print_human(result: dict[str, Any]) -> None:
|
||||
for key in (
|
||||
"source",
|
||||
"source_kind",
|
||||
"repo_type",
|
||||
"revision",
|
||||
"token_env",
|
||||
"source_layout",
|
||||
"needs_conversion",
|
||||
"model_index_class",
|
||||
"model_index_diffusers_version",
|
||||
"model_index_error",
|
||||
"file_count",
|
||||
"file_scan_truncated",
|
||||
):
|
||||
value = result.get(key)
|
||||
if value is not None:
|
||||
print(f"{key}: {value}")
|
||||
|
||||
components = result["components_seen"]
|
||||
print("components_seen: " + (", ".join(components) if components else "none"))
|
||||
print("files_sample:")
|
||||
for name in result["files_sample"]:
|
||||
print(f" {name}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
try:
|
||||
result = build_result(args)
|
||||
except Exception as exc: # noqa: BLE001 - CLI should print concise failures.
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(result, indent=2, sort_keys=True))
|
||||
else:
|
||||
print_human(result)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,122 +0,0 @@
|
||||
# <Model Family> Local Tests
|
||||
|
||||
Local-only parity and smoke tests for the `<model_family>` FastVideo port. These
|
||||
tests compare FastVideo against the official reference implementation and are
|
||||
not expected to run in CI unless explicitly promoted later.
|
||||
|
||||
Port progress, open questions, issues, and handoff notes live in
|
||||
`tests/local_tests/<model_family>/PORT_STATUS.md`.
|
||||
|
||||
## Reference Assets
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Model family | `<model_family>` |
|
||||
| Workload types | `<T2V/I2V/V2V/T2I/or compatibility shim with rationale>` |
|
||||
| Official reference | `<url or import path>` |
|
||||
| Local reference dir | `<ReferenceDir or none>` |
|
||||
| Official commit/version | `<sha, tag, package version, or unknown>` |
|
||||
| HF weights | `<HF repo id/url or local path>` |
|
||||
| HF revision | `<revision or default>` |
|
||||
| Local weights dir | `<official_weights/model_family or local path>` |
|
||||
| Source layout | `<diffusers/raw_official/monolithic/separate_components/mixed/custom/unknown>` |
|
||||
| Needs conversion | `<yes/no/unknown>` |
|
||||
|
||||
Do not write token values in this file. Use only the token env var name:
|
||||
`<HF_TOKEN or HUGGINGFACE_HUB_TOKEN or HF_API_KEY>`.
|
||||
|
||||
## Shared Environment Setup
|
||||
|
||||
Run from the FastVideo repo root in the same conda/env used for FastVideo.
|
||||
Do not create a separate upstream environment for parity tests.
|
||||
|
||||
```bash
|
||||
# Official reference source, if cloneable.
|
||||
python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \
|
||||
"<official_repo_url>" \
|
||||
"<ReferenceDir>" \
|
||||
--commit "<commit-sha>" \
|
||||
--update-gitignore
|
||||
|
||||
# Editable install without changing shared core pins.
|
||||
uv pip install --no-deps -e ./<ReferenceDir>
|
||||
|
||||
# Additional official deps installed or required for imports:
|
||||
# <package list or none>
|
||||
```
|
||||
|
||||
Do not change core dependency versions (`torch`, `diffusers`, `transformers`,
|
||||
`flash-attn`, `triton`, CUDA packages) without explicit approval.
|
||||
|
||||
## Official Environment Status
|
||||
|
||||
```text
|
||||
dependency_changes: <none | installed no-deps editable | installed official deps in current env | blocked on user>
|
||||
official_env_status: <imports_ok | private_deps_need_stubs | blocked>
|
||||
private_dep_stubs: <none or tests/local_tests/helpers/<model_family>_upstream.py>
|
||||
blocked_on: <none or exact blocker>
|
||||
```
|
||||
|
||||
## Weight Setup
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \
|
||||
"<Org/Model>" \
|
||||
"official_weights/<model_family>" \
|
||||
--revision "<revision>"
|
||||
```
|
||||
|
||||
If weights are local-only, record the local path and do not copy large files into
|
||||
the repository.
|
||||
|
||||
## Prototype And Conversion Artifacts
|
||||
|
||||
State-dict key/shape dumps are generated after FastVideo native prototypes exist
|
||||
and are used to build the conversion mapping.
|
||||
|
||||
```text
|
||||
official_key_dumps:
|
||||
<component>: converted_weights/<model_family>/_mapping/<component>_official_keys.json
|
||||
fastvideo_key_dumps:
|
||||
<component>: converted_weights/<model_family>/_mapping/<component>_fastvideo_keys.json
|
||||
conversion_script: scripts/checkpoint_conversion/<model_family>_to_diffusers.py
|
||||
conversion_source_layout: <diffusers | separate_components | monolithic | mixed | custom>
|
||||
converted_weights_dir: converted_weights/<model_family>
|
||||
strict_load_status: <not_run | pass | pass_with_documented_exclusions | blocked>
|
||||
```
|
||||
|
||||
For monolithic official checkpoints, record the component prefix split here. For
|
||||
example, a single checkpoint may contain transformer, VAE/pretransform,
|
||||
conditioner, and scheduler/vocoder keys that the conversion script writes into
|
||||
separate FastVideo component subfolders.
|
||||
|
||||
## Expected Parity Tests
|
||||
|
||||
Planned local tests for this family:
|
||||
|
||||
| Component | Official files / args | Test | Concerns | Status |
|
||||
|---|---|---|---|---|
|
||||
| `<component>` | `<definition path; instantiation path + args>` | `tests/local_tests/<bucket>/test_<model_family>_<component>_parity.py` | `<prototype or setup concerns>` | `<planned/scaffold_skip/debug_red/non_skip_pass/blocked>` |
|
||||
| `pipeline` | `<official pipeline call>` | `tests/local_tests/pipelines/test_<model_family>_pipeline_parity.py` | `<pipeline concerns>` | `<planned/scaffold_skip/debug_red/non_skip_pass/blocked>` |
|
||||
|
||||
Include reused components in this table. Reuse is accepted only after the
|
||||
FastVideo component definition and official instantiation arguments have both
|
||||
been checked and the component parity test passes non-skip.
|
||||
|
||||
Run the relevant tests with:
|
||||
|
||||
```bash
|
||||
pytest tests/local_tests/<bucket>/test_<model_family>_<component>_parity.py -v -s
|
||||
pytest tests/local_tests/pipelines/test_<model_family>_pipeline_parity.py -v -s
|
||||
```
|
||||
|
||||
## Review Notes
|
||||
|
||||
- Required before handoff: non-skip PASS for each required component parity
|
||||
test, including reused components that own weights or numerical behavior.
|
||||
- Pipeline parity may start as a scaffold, but final handoff requires non-skip
|
||||
PASS or an explicit blocker accepted through the escape-hatch process.
|
||||
- User decisions and pause points are tracked as `E###` rows in
|
||||
`PORT_STATUS.md`; do not rely on chat history for escape-hatch context.
|
||||
- Review agents should verify this README's setup commands still match the PR,
|
||||
then run the listed parity tests or report the exact blocker.
|
||||
@@ -1,69 +0,0 @@
|
||||
# <Model Family> Port Status
|
||||
|
||||
## Summary
|
||||
|
||||
- model_family: `<model_family>`
|
||||
- workload_types: `<T2V/I2V/V2V/T2I/or compatibility shim with rationale>`
|
||||
- official_ref: `<url or import path>`
|
||||
- official_ref_dir: `<ReferenceDir or none>`
|
||||
- hf_weights_path: `<HF repo id/url or local path>`
|
||||
- local_weights_dir: `<official_weights/model_family or local path>`
|
||||
- source_layout: `<diffusers/raw_official/monolithic/separate_components/mixed/custom/unknown>`
|
||||
- local_tests_readme: `tests/local_tests/<model_family>/README.md`
|
||||
|
||||
## Current Phase
|
||||
|
||||
- phase: `prep`
|
||||
- status: `in_progress`
|
||||
- owner: `prep`
|
||||
- last_updated: `<YYYY-MM-DD>`
|
||||
|
||||
## Component Matrix
|
||||
|
||||
| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| `<component>` | `<dit/vae/encoder/generic>` | `<unknown/reuse/port>` | `<path + symbols>` | `<path + args>` | `<target files>` | `<not_started/in_progress/pass/blocked>` | `<not_started/pass/blocked>` | `<not_started/scaffold_skip/debug_red/non_skip_pass/blocked>` | `<none or IDs>` |
|
||||
|
||||
## Conversion State
|
||||
|
||||
- conversion_script: `scripts/checkpoint_conversion/<model_family>_to_diffusers.py`
|
||||
- converted_weights_dir: `converted_weights/<model_family>`
|
||||
- source_layout: `<diffusers/separate_components/monolithic/mixed/custom/unknown>`
|
||||
- strict_load_status: `not_run`
|
||||
- passthrough_components: `<none or list>`
|
||||
- retry_history: `<none>`
|
||||
|
||||
## Parity Commands
|
||||
|
||||
| Scope | Command | Last Result | Notes |
|
||||
|---|---|---|---|
|
||||
| component | `pytest tests/local_tests/<bucket>/test_<model_family>_<component>_parity.py -v -s` | `not_run` | `<notes>` |
|
||||
| pipeline | `pytest tests/local_tests/pipelines/test_<model_family>_pipeline_parity.py -v -s` | `not_run` | `<notes>` |
|
||||
|
||||
## Open Questions
|
||||
|
||||
| ID | Question | Owner | Needed By Phase | Status | Resolution |
|
||||
|---|---|---|---|---|---|
|
||||
| Q001 | `<question>` | `<owner>` | `<phase>` | `<open/resolved>` | `<resolution or blank>` |
|
||||
|
||||
## Issues And Blockers
|
||||
|
||||
| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| I001 | `<phase>` | `<component or all>` | `<low/medium/high/blocker>` | `<issue>` | `<logs/paths/commands>` | `<owner>` | `<open/resolved>` | `<resolution or blank>` |
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution |
|
||||
|---|---|---|---|---|---|---|
|
||||
| E001 | `<phase>` | `<scope/dependency/auth/cost/destructive/ambiguity/blocker>` | `<one precise question>` | `<safe recommended option>` | `<open/resolved>` | `<resolution or blank>` |
|
||||
|
||||
## Decisions
|
||||
|
||||
| Date | Decision | Rationale | Impact |
|
||||
|---|---|---|---|
|
||||
| `<YYYY-MM-DD>` | `<decision>` | `<why>` | `<affected components/phases>` |
|
||||
|
||||
## Handoff Notes
|
||||
|
||||
- `<short notes for the next agent>`
|
||||
@@ -1,234 +0,0 @@
|
||||
---
|
||||
name: add-model-02-parity
|
||||
description: Use during /add-model after reference/architecture study to scaffold and later activate local FastVideo component parity tests. Emphasizes early test creation, official-reference loading, standardized FastVideo loading, and non-skip handoff gates.
|
||||
---
|
||||
|
||||
# Add Model Parity
|
||||
|
||||
## Goal
|
||||
|
||||
Create parity tests as early as possible in a FastVideo port. The first pass can
|
||||
land before conversion or component implementation as an executable scaffold;
|
||||
handoff is blocked until the same tests become non-skip PASS with real weights.
|
||||
|
||||
## When To Run
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
|
||||
escape hatches, and skip/pass semantics.
|
||||
|
||||
Run immediately after `/add-model` Phase 1 has identified:
|
||||
|
||||
- official component classes and call signatures;
|
||||
- FastVideo target component buckets/classes/configs;
|
||||
- local reference clone or import path from `add-model-01-prep`;
|
||||
- local raw or Diffusers weight path;
|
||||
- `official_env_status=imports_ok`, or private deps that will be stubbed
|
||||
locally in tests;
|
||||
- `local_tests_readme` documenting setup and planned review/test commands;
|
||||
- expected component inputs and output tensors.
|
||||
|
||||
Do not wait for all FastVideo components to be implemented. Write the tests
|
||||
first, then let component-porting subagents make them pass.
|
||||
|
||||
## Outputs
|
||||
|
||||
- One component parity test per required component, including reused components:
|
||||
`tests/local_tests/<bucket>/test_<family>_<component>_parity.py`.
|
||||
- Optional helper for upstream private deps:
|
||||
`tests/local_tests/helpers/<family>_upstream.py`.
|
||||
- Pipeline parity is owned later by `../add-model-09-pipeline/SKILL.md` after all
|
||||
component parity tests pass non-skip.
|
||||
- A parity status block for the `/add-model` parity verification phase.
|
||||
|
||||
## Early Scaffold Rules
|
||||
|
||||
- A scaffold may skip while the FastVideo class, converted weights, or official
|
||||
import is missing.
|
||||
- A scaffold must already encode the real official load path, FastVideo load
|
||||
path, deterministic inputs, expected output extraction, and tolerance target.
|
||||
- Each parity test must declare its coverage scope in the file docstring or a
|
||||
module constant: `production_loader`, `implementation_subcomponent`, or `both`.
|
||||
Implementation/subcomponent parity may bypass production loaders deliberately,
|
||||
but final handoff still needs production-loader coverage somewhere before the
|
||||
pipeline depends on that component.
|
||||
- Official reference imports must run in the current FastVideo environment; do
|
||||
not create or assume a separate upstream venv/conda env.
|
||||
- A scaffold is not evidence of correctness. It becomes evidence only after a
|
||||
local non-skip PASS.
|
||||
- Prefer env-var path overrides with repo-relative defaults.
|
||||
- Keep tests local-only under `tests/local_tests/`; package/CI quality tests are
|
||||
added later.
|
||||
- Update shared state files as described in
|
||||
`../add-model/shared/common_rules.md` whenever adding or activating parity
|
||||
tests.
|
||||
|
||||
## Component Template
|
||||
|
||||
Copy `templates/component_parity_test.py` and fill every `TODO` marker. The
|
||||
template is distilled from:
|
||||
|
||||
- `tests/local_tests/transformers/test_ltx2.py`
|
||||
- `tests/local_tests/transformers/test_gamecraft_parity.py`
|
||||
- `tests/local_tests/encoders/test_ltx2_gemma_parity.py`
|
||||
- `tests/local_tests/vaes/test_oobleck_vae_parity.py`
|
||||
- `tests/local_tests/sd35/test_sd35_component_parity.py`
|
||||
|
||||
The template supports three states:
|
||||
|
||||
| State | Meaning |
|
||||
|---|---|
|
||||
| Scaffold skip | Test is committed early, but official import, FastVideo class, or weights are not available yet. |
|
||||
| Debug red | Both sides load and the test fails numerically. This is useful: porting can chase the first drift. |
|
||||
| Non-skip pass | Required before `/add-model` handoff. |
|
||||
|
||||
## Subagent Dispatch Pattern
|
||||
|
||||
After Phase 1, dispatch one parity subagent per component before or alongside
|
||||
component implementation:
|
||||
|
||||
```text
|
||||
Create a local parity test scaffold for <family> <component>.
|
||||
|
||||
Use the prep handoff:
|
||||
- official_ref_dir/import: <...>
|
||||
- local_weights_dir: <...>
|
||||
- source_layout: <...>
|
||||
- needs_conversion: <yes/no>
|
||||
- official_env_status: <imports_ok | private_deps_need_stubs>
|
||||
- local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
- port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
- official_definition_files: <paths + classes/functions>
|
||||
- official_instantiation_files: <paths + factory/pipeline/config call sites + args>
|
||||
- concerns_or_unknowns: <known ambiguous inputs, outputs, deps, or args>
|
||||
|
||||
The complete per-component packet must match
|
||||
`../add-model/contracts/component_context.md`.
|
||||
|
||||
Read the official component call path and the planned FastVideo component API.
|
||||
Add tests/local_tests/<bucket>/test_<family>_<component>_parity.py based on
|
||||
add-model-02-parity/templates/component_parity_test.py.
|
||||
|
||||
The scaffold must load the official model with real weights when available,
|
||||
load the FastVideo model through the standardized config/class/loader path when
|
||||
available, create deterministic inputs, compare concrete outputs, and skip only
|
||||
when a dependency is genuinely missing. Do not make an unconditional skip or a
|
||||
shape-only test.
|
||||
```
|
||||
|
||||
## FastVideo Load Patterns
|
||||
|
||||
Pick the narrowest load path that matches the component:
|
||||
|
||||
| Component | Preferred FastVideo load path |
|
||||
|---|---|
|
||||
| DiT / transformer | Bucket config + model class, or `TransformerLoader` when testing converted Diffusers component dirs. |
|
||||
| VAE | VAE class `from_pretrained(...)` when implemented, or bucket config + class for local converted dirs. |
|
||||
| Text/image encoder | Bucket config + model class; pass HF subpaths from `local_weights_dir` or converted component dirs. |
|
||||
| Scheduler/conditioner | Native class/config plus exact official kwargs. |
|
||||
|
||||
For early scaffolds, an import of the planned FastVideo class may be inside a
|
||||
helper that calls `pytest.skip` if the class does not exist yet. Replace that
|
||||
skip with a real import once the component PR adds the class.
|
||||
|
||||
Direct class/config construction is allowed for implementation or subcomponent
|
||||
parity, such as connector-only encoder checks or official monolithic-checkpoint
|
||||
mapping tests. Label that scope explicitly and add separate production-loader
|
||||
coverage when converted component dirs are available.
|
||||
|
||||
## Official Load Patterns
|
||||
|
||||
- Clone/reference repo path: add its source dir to `sys.path` before imports.
|
||||
- HF/Diffusers reference: import only inside the test, not production code.
|
||||
- Private deps: add a helper under `tests/local_tests/helpers/` to install
|
||||
stubs before importing upstream modules; do not rely on an external upstream
|
||||
environment.
|
||||
- Gated HF repos: resolve `HF_TOKEN`, `HUGGINGFACE_HUB_TOKEN`, or `HF_API_KEY`
|
||||
under the token rules in `../add-model/shared/common_rules.md`.
|
||||
|
||||
## Non-Skip Activation Checklist
|
||||
|
||||
Before `/add-model` handoff, each scaffolded test must be activated:
|
||||
|
||||
```text
|
||||
[ ] Official side imports and loads real weights.
|
||||
[ ] FastVideo side imports and loads the converted or original weights.
|
||||
[ ] Test executes at least one real forward call on both sides.
|
||||
[ ] Test compares output tensors, not only shapes or state-dict keys.
|
||||
[ ] Local pytest output contains PASSED, not SKIPPED or XFAIL.
|
||||
[ ] Tolerance is justified for the component scope and kernel alignment.
|
||||
```
|
||||
|
||||
## Component Parity Details
|
||||
|
||||
Reference imports:
|
||||
|
||||
- Import from `official_ref_dir` or the recorded package/import path.
|
||||
- If upstream has private deps, add a helper under
|
||||
`tests/local_tests/helpers/<family>_upstream.py` that installs minimal stubs
|
||||
before importing upstream modules.
|
||||
- Common stubs: identity compile/op-registration decorators, CP world size set to
|
||||
1, identity scatter/gather, and test-friendly custom-op kernels.
|
||||
- Stub decorators that register `torch.ops.<ns>.<op>` must preserve the
|
||||
`torch.library` registration side-effect. Identity decorators alone are not
|
||||
enough.
|
||||
- Delete stub helpers and every `install_stubs()` call as soon as the real deps
|
||||
become required installs. No-op shims are dead code.
|
||||
|
||||
Kernel and wrapper pitfalls:
|
||||
|
||||
- If parity routes flash-attn GQA through SDPA, expand KV heads manually on the
|
||||
SDPA side with `repeat_interleave` along the head axis.
|
||||
- If upstream VAE `decode()` denormalizes internally but FastVideo/Diffusers
|
||||
expects pre-denormalized latents, apply `z = z * std + mean` only on the
|
||||
FastVideo side in the parity test.
|
||||
- Per-channel VAE `latents_mean` / `latents_std` must be reshaped explicitly,
|
||||
e.g. `.view(1, z_dim, 1, 1, 1)` for 5D video latents.
|
||||
|
||||
Tolerance guide:
|
||||
|
||||
| Scope | Start `atol` / `rtol` | Notes |
|
||||
|---|---|---|
|
||||
| Single block, same kernel | `1e-4` / `1e-4` | Tight default. |
|
||||
| Full DiT, aligned kernels | `1e-2` / `1e-2` | Cross-layer accumulation. |
|
||||
| Full DiT, cross-kernel bf16 | `0.1` / `0.1` | Also require abs-mean drift below 5% and per-modality diagnostics. |
|
||||
| VAE decode fp32 | `5e-2` / `5e-2` | After normalization alignment. |
|
||||
| Encoder wrapper around same HF class | `1e-3` / `1e-3` | Should be near-zero. |
|
||||
|
||||
Element-wise `assert_close` alone is not enough for deep full-DiT parity. Also
|
||||
log global abs-mean drift and per-modality summaries.
|
||||
|
||||
When a non-skip component parity run is numerically red after weight/input
|
||||
checks, invoke `../add-model-08-trace/SKILL.md` before adding bespoke forward
|
||||
hooks. Use `docs/contributing/activation_trace.md` to keep
|
||||
`FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and `FASTVIDEO_TRACE_STEPS`
|
||||
identical across FastVideo and upstream traces.
|
||||
|
||||
Useful local commands:
|
||||
|
||||
```bash
|
||||
pytest tests/local_tests/<bucket>/test_<family>_*parity*.py -v -s
|
||||
pytest tests/local_tests -k "<family> and parity" -v -s
|
||||
```
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md`. Parity-specific ask cases include
|
||||
private dependency approval, choosing between incompatible official references,
|
||||
accepting a shape-only substitute, or loosening required tolerances.
|
||||
|
||||
## Pipeline Parity
|
||||
|
||||
Pipeline parity is later than component parity because it needs stages, presets,
|
||||
registry wiring, converted weights, and green component parity. Record official
|
||||
pipeline call notes in `local_tests_readme`, but do not treat pipeline parity as
|
||||
owned by this skill.
|
||||
|
||||
Use `../add-model-09-pipeline/SKILL.md` and its
|
||||
`templates/pipeline_parity_test.py` for pipeline parity scaffolding and
|
||||
debugging. Compare denoised latents or decoded media, not just successful
|
||||
generation.
|
||||
|
||||
## Handoff Status Block
|
||||
|
||||
Return `../add-model/contracts/parity_status.md` to `/add-model` and update the
|
||||
shared state files before handoff.
|
||||
@@ -1,188 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Component parity scaffold for <FAMILY> <COMPONENT>.
|
||||
|
||||
This file is intended to be created early in a port. It may skip until the
|
||||
official reference, FastVideo class, and real weights are available, but it must
|
||||
never become an unconditional skip or shape-only test.
|
||||
|
||||
Fill every TODO before considering this test active.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from torch.testing import assert_close
|
||||
|
||||
os.environ.setdefault("MASTER_ADDR", "localhost")
|
||||
os.environ.setdefault("MASTER_PORT", "29519")
|
||||
os.environ.setdefault("DISABLE_SP", "1")
|
||||
os.environ.setdefault("FASTVIDEO_ATTENTION_BACKEND", "TORCH_SDPA")
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
FAMILY = "<family>" # TODO: snake_case family name.
|
||||
COMPONENT = "<component>" # TODO: transformer | vae | encoder | conditioner | ...
|
||||
PARITY_SCOPE = "implementation_subcomponent" # TODO: production_loader | implementation_subcomponent | both
|
||||
OFFICIAL_MODULE = "<official.module>" # TODO: e.g. "ltx_core.model.transformer".
|
||||
OFFICIAL_CLASS = "<OfficialClass>" # TODO: official class/factory name.
|
||||
FASTVIDEO_CONFIG_MODULE = "fastvideo.configs.models.<bucket>" # TODO.
|
||||
FASTVIDEO_CONFIG_CLASS = "<FastVideoConfig>" # TODO.
|
||||
FASTVIDEO_MODEL_MODULE = "fastvideo.models.<bucket>.<module>" # TODO.
|
||||
FASTVIDEO_MODEL_CLASS = "<FastVideoModel>" # TODO.
|
||||
|
||||
OFFICIAL_REF_DIR = Path(os.getenv("<FAMILY_UPPER>_OFFICIAL_REF_DIR", REPO_ROOT / "<ReferenceDir>"))
|
||||
LOCAL_WEIGHTS_DIR = Path(os.getenv("<FAMILY_UPPER>_LOCAL_WEIGHTS_DIR", REPO_ROOT / "official_weights" / FAMILY))
|
||||
CONVERTED_WEIGHTS_DIR = Path(os.getenv("<FAMILY_UPPER>_CONVERTED_WEIGHTS_DIR",
|
||||
REPO_ROOT / "converted_weights" / FAMILY))
|
||||
|
||||
|
||||
def _resolve_hf_token() -> str | None:
|
||||
for key in ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY"):
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _add_official_to_path() -> None:
|
||||
"""Add the official source path before importing upstream modules."""
|
||||
# TODO: adjust for the official repo layout. Common examples:
|
||||
# OFFICIAL_REF_DIR / "src"
|
||||
# OFFICIAL_REF_DIR / "packages" / "<pkg>" / "src"
|
||||
# OFFICIAL_REF_DIR
|
||||
official_src = OFFICIAL_REF_DIR / "src"
|
||||
if not official_src.exists():
|
||||
official_src = OFFICIAL_REF_DIR
|
||||
if official_src.exists() and str(official_src) not in sys.path:
|
||||
sys.path.insert(0, str(official_src))
|
||||
|
||||
|
||||
def _import_or_skip(module_name: str, attr_name: str | None = None):
|
||||
if "<" in module_name or (attr_name is not None and "<" in attr_name):
|
||||
pytest.skip(f"Template import placeholder not filled: {module_name}.{attr_name}")
|
||||
try:
|
||||
module = importlib.import_module(module_name)
|
||||
except Exception as exc: # noqa: BLE001 - local parity should skip missing refs.
|
||||
pytest.skip(f"Cannot import {module_name}: {exc}")
|
||||
if attr_name is None:
|
||||
return module
|
||||
try:
|
||||
return getattr(module, attr_name)
|
||||
except AttributeError:
|
||||
pytest.skip(f"{module_name} has no attribute {attr_name}")
|
||||
|
||||
|
||||
def _load_official_model(device: torch.device, dtype: torch.dtype) -> torch.nn.Module:
|
||||
"""Load the official component with real weights."""
|
||||
_add_official_to_path()
|
||||
if not OFFICIAL_REF_DIR.exists():
|
||||
pytest.skip(f"Official reference missing: {OFFICIAL_REF_DIR}")
|
||||
if not LOCAL_WEIGHTS_DIR.exists():
|
||||
pytest.skip(f"Local weights missing: {LOCAL_WEIGHTS_DIR}")
|
||||
|
||||
# TODO: import official class/factory and load real weights strictly.
|
||||
# Examples in-tree:
|
||||
# - LTX2: SingleGPUModelBuilder(...).build(device=device, dtype=dtype)
|
||||
# - GameCraft: torch.load(...)["module"] -> official_model.load_state_dict(...)
|
||||
# - Oobleck: create_model_from_config(config) + ckpt state_dict
|
||||
OfficialClass = _import_or_skip(OFFICIAL_MODULE, OFFICIAL_CLASS)
|
||||
model = OfficialClass() # TODO: pass official config kwargs.
|
||||
state_dict = {} # TODO: load official state dict from LOCAL_WEIGHTS_DIR.
|
||||
missing, unexpected = model.load_state_dict(state_dict, strict=True)
|
||||
assert not missing and not unexpected, (f"official load mismatch missing={missing[:5]} unexpected={unexpected[:5]}")
|
||||
return model.to(device=device, dtype=dtype).eval()
|
||||
|
||||
|
||||
def _load_fastvideo_model(device: torch.device, dtype: torch.dtype) -> torch.nn.Module:
|
||||
"""Load the FastVideo component with the same tensor content."""
|
||||
if not CONVERTED_WEIGHTS_DIR.exists() and not LOCAL_WEIGHTS_DIR.exists():
|
||||
pytest.skip(f"No FastVideo loadable weights: {CONVERTED_WEIGHTS_DIR} or {LOCAL_WEIGHTS_DIR}")
|
||||
|
||||
# TODO: replace with the bucket-specific FastVideo config/class/loader.
|
||||
# DiT examples:
|
||||
# from fastvideo.configs.models.dits import <Config>
|
||||
# from fastvideo.models.dits.<module> import <Model>
|
||||
# VAE examples:
|
||||
# from fastvideo.models.vaes.<module> import <VAE>
|
||||
# model = <VAE>.from_pretrained(...)
|
||||
FastVideoConfig = _import_or_skip(FASTVIDEO_CONFIG_MODULE, FASTVIDEO_CONFIG_CLASS)
|
||||
FastVideoModel = _import_or_skip(FASTVIDEO_MODEL_MODULE, FASTVIDEO_MODEL_CLASS)
|
||||
|
||||
config = FastVideoConfig()
|
||||
model = FastVideoModel(config=config)
|
||||
state_dict = {} # TODO: load converted or directly mapped state dict.
|
||||
missing, unexpected = model.load_state_dict(state_dict, strict=True)
|
||||
assert not missing and not unexpected, (
|
||||
f"FastVideo load mismatch missing={missing[:5]} unexpected={unexpected[:5]}")
|
||||
return model.to(device=device, dtype=dtype).eval()
|
||||
|
||||
|
||||
def _make_inputs(device: torch.device, dtype: torch.dtype) -> dict[str, torch.Tensor]:
|
||||
"""Create deterministic inputs matching the official component call."""
|
||||
torch.manual_seed(0)
|
||||
# TODO: replace with component-specific tensors and metadata.
|
||||
return {
|
||||
"hidden_states": torch.randn(1, 4, 16, device=device, dtype=dtype),
|
||||
"timestep": torch.tensor([10], device=device),
|
||||
}
|
||||
|
||||
|
||||
def _run_official(model: torch.nn.Module, inputs: dict[str, torch.Tensor]) -> torch.Tensor:
|
||||
"""Run official component and return the tensor to compare."""
|
||||
with torch.inference_mode():
|
||||
output = model(**inputs) # TODO: adapt official call signature.
|
||||
if isinstance(output, dict):
|
||||
sample = output.get("sample")
|
||||
output = sample if sample is not None else output.get("x")
|
||||
elif hasattr(output, "sample"):
|
||||
output = output.sample
|
||||
elif isinstance(output, tuple):
|
||||
output = output[0]
|
||||
assert torch.is_tensor(output), f"official output is not tensor: {type(output)}"
|
||||
return output.detach().float().cpu()
|
||||
|
||||
|
||||
def _run_fastvideo(model: torch.nn.Module, inputs: dict[str, torch.Tensor]) -> torch.Tensor:
|
||||
"""Run FastVideo component and return the tensor to compare."""
|
||||
with torch.inference_mode():
|
||||
output = model(**inputs) # TODO: adapt FastVideo call signature.
|
||||
if isinstance(output, dict):
|
||||
sample = output.get("sample")
|
||||
output = sample if sample is not None else output.get("x")
|
||||
elif hasattr(output, "sample"):
|
||||
output = output.sample
|
||||
elif isinstance(output, tuple):
|
||||
output = output[0]
|
||||
assert torch.is_tensor(output), f"FastVideo output is not tensor: {type(output)}"
|
||||
return output.detach().float().cpu()
|
||||
|
||||
|
||||
@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required for this parity test.")
|
||||
def test_component_parity():
|
||||
"""Compare official and FastVideo outputs on identical inputs."""
|
||||
device = torch.device("cuda:0")
|
||||
dtype = torch.bfloat16
|
||||
|
||||
official = _load_official_model(device, dtype)
|
||||
fastvideo = _load_fastvideo_model(device, dtype)
|
||||
inputs = _make_inputs(device, dtype)
|
||||
|
||||
official_out = _run_official(official, inputs)
|
||||
fastvideo_out = _run_fastvideo(fastvideo, inputs)
|
||||
|
||||
assert official_out.shape == fastvideo_out.shape
|
||||
diff = (official_out - fastvideo_out).abs()
|
||||
print(f"official abs_mean={official_out.abs().mean().item():.6f} "
|
||||
f"fastvideo abs_mean={fastvideo_out.abs().mean().item():.6f} "
|
||||
f"diff_max={diff.max().item():.6f} diff_mean={diff.mean().item():.6f}")
|
||||
|
||||
# TODO: pick tolerance by scope:
|
||||
# - single block / same kernel: 1e-4
|
||||
# - full DiT aligned kernels: 1e-2
|
||||
# - full DiT cross-kernel bf16: 1e-1 + abs_mean drift check
|
||||
# - VAE decode fp32: 5e-2 after normalization alignment
|
||||
assert_close(fastvideo_out, official_out, atol=1e-4, rtol=1e-4)
|
||||
@@ -1,114 +0,0 @@
|
||||
---
|
||||
name: add-model-03-port-dit
|
||||
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native DiT/transformer component.
|
||||
---
|
||||
|
||||
# Add Model Port DiT
|
||||
|
||||
## Goal
|
||||
|
||||
Prototype or parity-debug one diffusion transformer in FastVideo-native code.
|
||||
This skill is for one component only; do not work on the VAE, encoders,
|
||||
pipeline, or unrelated conversion code unless the current component cannot load
|
||||
without a minimal fix there.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/component_skill_common.md` and require the complete
|
||||
packet from `../add-model/contracts/component_context.md`.
|
||||
|
||||
DiT-specific packet fields:
|
||||
|
||||
- `component`: transformer or DiT name.
|
||||
- `parity_test`: `tests/local_tests/<bucket>/test_<family>_<component>_parity.py`.
|
||||
- `weights`: converted transformer dir or local official path.
|
||||
- `target_files`: `fastvideo/models/dits/<family>.py` and
|
||||
`fastvideo/configs/models/dits/<family>.py`.
|
||||
|
||||
## Modes
|
||||
|
||||
Use the common prototype and parity-debug modes from
|
||||
`../add-model/shared/component_skill_common.md`.
|
||||
|
||||
DiT-specific prototype concerns include ambiguous official flags, shape
|
||||
mismatches, missing FastVideo layer equivalents, and dedicated output heads.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
Apply the shared reuse proof. DiT-specific comparison must include attention
|
||||
algorithm, positional embeddings, RoPE/patching, timestep/guidance embeddings,
|
||||
scaling constants, dtype casts, state-dict names, and every output head.
|
||||
|
||||
## Existing FastVideo Patterns
|
||||
|
||||
- Base class: `fastvideo/models/dits/base.py::BaseDiT`.
|
||||
- Config bases: `DiTConfig` and `DiTArchConfig` in
|
||||
`fastvideo/configs/models/dits/base.py`.
|
||||
- Use the matching DiT config bucket. Wrong bucket inheritance can typecheck but
|
||||
fail during pipeline wiring.
|
||||
- Config export: add the config to
|
||||
`fastvideo/configs/models/dits/__init__.py`.
|
||||
- Registry discovery: set `EntryClass = <ClassName>` in the model file.
|
||||
- Loader path: `TransformerLoader` reads `transformer/config.json`, calls
|
||||
`dit_config.update_model_arch(config)`, resolves `_class_name` through
|
||||
`ModelRegistry`, and constructs the class with `config` and `hf_config`.
|
||||
- Reference examples: `stable_audio.py`, `wanvideo.py`, `sd3.py`, `longcat.py`,
|
||||
and `ltx2.py`.
|
||||
- Layer guidance: `fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Use FastVideo-native layers by default: `ReplicatedLinear` for DiT hot-path
|
||||
linears, `DistributedAttention` for standard full-sequence attention, and
|
||||
`LocalAttention` for local/window attention or simple single-GPU parity paths.
|
||||
- Raw SDPA is acceptable for cross-modality flat streams when no FastVideo
|
||||
distributed equivalent exists; document the SP gap in the module docstring.
|
||||
- Mirror official tensor contracts exactly: latent packing, patch ordering,
|
||||
timestep embedding scale, RoPE/positional embedding, guidance embedding,
|
||||
cross-attention context order, output head order, and dtype casts.
|
||||
- Preserve all output heads that the official DiT emits. Do not silently drop
|
||||
audio, depth, pose, mask, or auxiliary heads.
|
||||
- Put architecture fields on `DiTArchConfig`; keep inference steps, CFG scales,
|
||||
FPS, flow shift, and sampling defaults out of the arch config.
|
||||
- Define `_fsdp_shard_conditions`, `_compile_conditions`,
|
||||
`param_names_mapping`, and `reverse_param_names_mapping` where needed.
|
||||
- Follow the production import boundary in
|
||||
`../add-model/shared/common_rules.md`.
|
||||
|
||||
## Prototype Checks
|
||||
|
||||
Follow the shared prototype success criteria. A useful one-off check is:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
# Import the target config/class, instantiate with random weights, and print
|
||||
# state_dict names/shapes for the conversion mapping.
|
||||
PY
|
||||
```
|
||||
|
||||
## Parity-Debug Loop
|
||||
|
||||
Run the shared parity-debug loop. The component test command is:
|
||||
|
||||
```bash
|
||||
pytest <parity_test> -v -s
|
||||
```
|
||||
|
||||
For numerical drift, use `../add-model-08-trace/SKILL.md` before writing bespoke
|
||||
hooks. Start with FastVideo's activation trace (`fastvideo/hooks/activation_trace.py`;
|
||||
`docs/contributing/activation_trace.md`) and a block-level regex such as
|
||||
`FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$"`. Only fall back to custom
|
||||
per-block hooks if the needed boundary or statistic is not exposed by
|
||||
`FASTVIDEO_TRACE_STATS`.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
|
||||
in `../add-model/shared/component_skill_common.md`. DiT-specific ask cases include
|
||||
dropping an output head/modality, accepting an unsupported kernel/private op, or
|
||||
choosing between incompatible official transformer definitions.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/component_skill_handoff.md` following the common
|
||||
handoff rules in `../add-model/shared/component_skill_common.md`.
|
||||
@@ -1,100 +0,0 @@
|
||||
---
|
||||
name: add-model-04-port-vae
|
||||
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native VAE component.
|
||||
---
|
||||
|
||||
# Add Model Port VAE
|
||||
|
||||
## Goal
|
||||
|
||||
Prototype or parity-debug one VAE or autoencoder in FastVideo-native code. This
|
||||
skill covers video, image, and audio VAEs.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/component_skill_common.md` and require the complete
|
||||
packet from `../add-model/contracts/component_context.md`.
|
||||
|
||||
VAE-specific packet fields:
|
||||
|
||||
- `component`: VAE or autoencoder name.
|
||||
- `parity_test`: `tests/local_tests/vaes/test_<family>_<component>_parity.py`.
|
||||
- `weights`: converted VAE dir, HF subfolder, or local official path.
|
||||
- `target_files`: `fastvideo/models/vaes/<arch_or_family>.py` and
|
||||
`fastvideo/configs/models/vaes/<arch_or_family>.py`.
|
||||
|
||||
## Modes
|
||||
|
||||
Use the common prototype and parity-debug modes from
|
||||
`../add-model/shared/component_skill_common.md`.
|
||||
|
||||
VAE-specific prototype concerns include latent normalization, stochastic
|
||||
posterior behavior, tiling incompatibility, temporal/spatial/audio layout, and
|
||||
decode output containers.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
Apply the shared reuse proof. VAE-specific comparison must include latent layout,
|
||||
temporal/spatial/audio compression, scaling factor, mean/std normalization,
|
||||
posterior behavior, encode/decode output objects, tiling flags, and cropping.
|
||||
|
||||
## Existing FastVideo Patterns
|
||||
|
||||
- Shared tiling wrapper: `fastvideo/models/vaes/common.py::ParallelTiledVAE`.
|
||||
- Config bases: `VAEConfig` and `VAEArchConfig` in
|
||||
`fastvideo/configs/models/vaes/base.py`.
|
||||
- Use the matching VAE config bucket. Wrong bucket inheritance can typecheck but
|
||||
fail during pipeline wiring.
|
||||
- Config export: add the config to
|
||||
`fastvideo/configs/models/vaes/__init__.py`.
|
||||
- Registry discovery: set `EntryClass = <ClassName>` in the model file.
|
||||
- Loader path: VAE loaders resolve `_class_name` through `ModelRegistry` and
|
||||
load converted component weights from the VAE subdir.
|
||||
- Reference examples: `oobleck.py`, `autoencoder_kl.py`, `wanvae.py`,
|
||||
`ltx2vae.py`, and `gamecraftvae.py`.
|
||||
- Layer guidance: `fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Name reusable VAE architectures by architecture (`oobleck.py`,
|
||||
`autoencoder_kl.py`); name family-specific VAEs by family.
|
||||
- Match official encode/decode contracts exactly: input layout, latent layout,
|
||||
temporal/spatial/audio compression, scaling factor, mean/std normalization,
|
||||
posterior sampling behavior, decode output object, and frame/sample cropping.
|
||||
- Compare deterministic outputs in parity: decode outputs, encode mean/mode, or
|
||||
round-trip tensors. Do not compare stochastic samples unless the RNG path is
|
||||
explicitly controlled.
|
||||
- Use FastVideo tiling only when it preserves official numerics for the tested
|
||||
shape; disable it in config for audio or unsupported dimensions.
|
||||
- Put architecture constants on `VAEArchConfig`; put `load_encoder`,
|
||||
`load_decoder`, tiling, dtype, and pretrained path fields on `VAEConfig`.
|
||||
- Follow the production import boundary in
|
||||
`../add-model/shared/common_rules.md`.
|
||||
|
||||
## Prototype Checks
|
||||
|
||||
Follow the shared prototype success criteria.
|
||||
|
||||
## Parity-Debug Loop
|
||||
|
||||
Run the shared parity-debug loop. The component test command is:
|
||||
|
||||
```bash
|
||||
pytest <parity_test> -v -s
|
||||
```
|
||||
|
||||
For numerical drift, check normalization, latent scaling, posterior mode vs
|
||||
sample, channel order, and temporal/spatial/audio cropping before changing
|
||||
layers.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
|
||||
in `../add-model/shared/component_skill_common.md`. VAE-specific ask cases include
|
||||
dropping an encode/decode path, accepting an unsupported private op, or choosing
|
||||
between incompatible official VAE definitions.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/component_skill_handoff.md` following the common
|
||||
handoff rules in `../add-model/shared/component_skill_common.md`.
|
||||
@@ -1,119 +0,0 @@
|
||||
---
|
||||
name: add-model-05-port-encoder
|
||||
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native text, image, audio, or compound encoder component.
|
||||
---
|
||||
|
||||
# Add Model Port Encoder
|
||||
|
||||
## Goal
|
||||
|
||||
Prototype or parity-debug one encoder or encoder-like conditioner in
|
||||
FastVideo-native code. Use this for text encoders, image encoders, audio
|
||||
encoders, and compound conditioners that fit the encoder config/loader bucket.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/component_skill_common.md` and require the complete
|
||||
packet from `../add-model/contracts/component_context.md`.
|
||||
|
||||
Encoder-specific packet fields:
|
||||
|
||||
- `component`: encoder or encoder-like conditioner name.
|
||||
- `parity_test`: `tests/local_tests/encoders/test_<family>_<component>_parity.py`.
|
||||
- `weights`: converted encoder dir, HF subfolder, or external HF id.
|
||||
- `target_files`: `fastvideo/models/encoders/<arch_or_family>.py` and
|
||||
`fastvideo/configs/models/encoders/<arch_or_family>.py`.
|
||||
|
||||
## Modes
|
||||
|
||||
Use the common prototype and parity-debug modes from
|
||||
`../add-model/shared/component_skill_common.md`.
|
||||
|
||||
Encoder-specific prototype concerns include tokenizer kwargs, hidden-state
|
||||
extraction, output packing, connector order, and external/passthrough weight
|
||||
needs.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
Apply the shared reuse proof. Encoder-specific comparison must include tokenizer
|
||||
contracts, hidden-state extraction, masks, positional IDs, output packing,
|
||||
connector/projection ordering, passthrough paths, and returned dataclass shape.
|
||||
|
||||
## Existing FastVideo Patterns
|
||||
|
||||
- Base classes: `TextEncoder` and `ImageEncoder` in
|
||||
`fastvideo/models/encoders/base.py`.
|
||||
- Output type: `BaseEncoderOutput`.
|
||||
- Config bases: `TextEncoderConfig`, `ImageEncoderConfig`,
|
||||
`TextEncoderArchConfig`, and `ImageEncoderArchConfig` in
|
||||
`fastvideo/configs/models/encoders/base.py`.
|
||||
- Use the matching encoder config bucket. Wrong bucket inheritance can typecheck
|
||||
but fail during pipeline wiring.
|
||||
- Config export: add the config to
|
||||
`fastvideo/configs/models/encoders/__init__.py`.
|
||||
- Registry discovery: set `EntryClass = <ClassName>` or a list of class names in
|
||||
the model file.
|
||||
- Reference examples: native `t5.py`, `clip.py`, `siglip.py`, `llama.py`,
|
||||
`qwen2_5.py`, `gemma.py`, and compound `stable_audio_conditioner.py`.
|
||||
- Layer guidance: `fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Reuse tokenizers and pure data utilities when needed, but do not add runtime
|
||||
third-party model-class imports as a placeholder for a component that owns
|
||||
weights or numerical behavior.
|
||||
- For LLM-style encoders, follow existing tensor-parallel patterns such as
|
||||
`QKVParallelLinear`, `MergedColumnParallelLinear`, `RowParallelLinear`,
|
||||
`VocabParallelEmbedding`, and `RMSNorm` when matching native examples.
|
||||
- Match official hidden-state extraction exactly: layer index, pooled output,
|
||||
attention mask dtype, padding side, truncation, special tokens, final norm,
|
||||
output_hidden_states, and returned tuple/dataclass shape.
|
||||
- For connector or conditioner modules, preserve sub-conditioner order and the
|
||||
exact packing of cross-attention tokens, masks, and global conditioning.
|
||||
- Put tokenizer kwargs and architecture constants on the arch config when they
|
||||
affect numerical behavior.
|
||||
- If an external HF encoder is explicitly accepted as a lazy wrapper, keep it
|
||||
isolated, document why it is not a native port, and still require parity for
|
||||
the wrapper's output contract.
|
||||
|
||||
Hybrid external-HF encoder checklist:
|
||||
|
||||
- Put external model folders in passthrough subfolders such as
|
||||
`text_encoder/<external_name>/`, or record a root `model_index.json` path field
|
||||
that the loader resolves to a local directory.
|
||||
- Keep external model parameters out of the FastVideo-owned state-dict surface
|
||||
when the external model is loaded lazily from its own HF files.
|
||||
- Convert and strict-check only the FastVideo-owned connector/projection weights;
|
||||
document external model weights as passthrough.
|
||||
- Add parity for the wrapper's final output contract and, when useful, a narrower
|
||||
connector-only parity test that labels its scope as
|
||||
`implementation_subcomponent`.
|
||||
- Verify the production loader resolves the same external path used by the
|
||||
pipeline, not just the direct class used in the parity test.
|
||||
|
||||
## Prototype Checks
|
||||
|
||||
Follow the shared prototype success criteria.
|
||||
|
||||
## Parity-Debug Loop
|
||||
|
||||
Run the shared parity-debug loop. The component test command is:
|
||||
|
||||
```bash
|
||||
pytest <parity_test> -v -s
|
||||
```
|
||||
|
||||
For numerical drift, check tokenization, masks, hidden-state selection,
|
||||
positional IDs, dtype/autocast, and output packing before changing layers.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
|
||||
in `../add-model/shared/component_skill_common.md`. Encoder-specific ask cases
|
||||
include accepting private model-code execution, choosing between incompatible
|
||||
tokenizer/encoder references, or dropping a required conditioning stream.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/component_skill_handoff.md` following the common
|
||||
handoff rules in `../add-model/shared/component_skill_common.md`.
|
||||
@@ -1,111 +0,0 @@
|
||||
---
|
||||
name: add-model-06-port-generic
|
||||
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one non-DiT, non-VAE, non-encoder FastVideo component.
|
||||
---
|
||||
|
||||
# Add Model Port Generic
|
||||
|
||||
## Goal
|
||||
|
||||
Prototype or parity-debug one scheduler, conditioner, upsampler, vocoder,
|
||||
adapter, preprocessor, or unknown component in FastVideo-native code.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/component_skill_common.md` and require the complete
|
||||
packet from `../add-model/contracts/component_context.md`.
|
||||
|
||||
Generic-component packet fields:
|
||||
|
||||
- `component`: component name.
|
||||
- `component_type`: scheduler, conditioner, upsampler, vocoder, adapter,
|
||||
preprocessor, or unknown.
|
||||
- `parity_test`: `tests/local_tests/<bucket>/test_<family>_<component>_parity.py`.
|
||||
- `weights`: converted component dir, HF subfolder, or none.
|
||||
- `target_files`: matching `fastvideo/models/` and `fastvideo/configs/models/`
|
||||
bucket files when applicable.
|
||||
|
||||
## Modes
|
||||
|
||||
Use the common prototype and parity-debug modes from
|
||||
`../add-model/shared/component_skill_common.md`.
|
||||
|
||||
Generic-component prototype concerns include stateless/stateful ambiguity,
|
||||
missing loader buckets, source prefixes, mutable scheduler state, and output
|
||||
container shape.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
Apply the shared reuse proof. Generic-component comparison must include mutable
|
||||
state, scaling constants, scheduler/conditioner semantics, output containers, and
|
||||
whether the component owns state or is stateless.
|
||||
|
||||
## Existing FastVideo Patterns
|
||||
|
||||
- Schedulers live under `fastvideo/models/schedulers/` and expose `EntryClass`.
|
||||
- Upsamplers use `fastvideo/models/upsamplers/` plus configs under
|
||||
`fastvideo/configs/models/upsamplers/`; see `hunyuan15.py`.
|
||||
- Vocoders and audio-specific modules can live under `fastvideo/models/audio/`
|
||||
with configs under `fastvideo/configs/models/audio/`; see `ltx2_audio_vae.py`.
|
||||
- Compound conditioners may fit the encoder bucket when the pipeline loader uses
|
||||
`ConditionerLoader`; see `stable_audio_conditioner.py`.
|
||||
- Registry discovery uses `EntryClass`; config bucket exports are required when
|
||||
pipeline configs import them by bucket.
|
||||
- Use the narrowest matching config bucket. Wrong bucket inheritance can typecheck
|
||||
but fail during pipeline wiring.
|
||||
- Layer guidance: `fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Bucket Decision
|
||||
|
||||
- If the component is a transformer/DiT, stop and use `add-model-03-port-dit`.
|
||||
- If the component is a VAE/autoencoder, stop and use `add-model-04-port-vae`.
|
||||
- If the component is a text/image/audio encoder or encoder-like conditioner,
|
||||
stop and use `add-model-05-port-encoder` unless the loader requires a different
|
||||
bucket.
|
||||
- Otherwise choose the narrowest existing bucket. Add a new bucket only when no
|
||||
existing loader/config shape can represent the component without misleading
|
||||
names or unsafe runtime behavior.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Match official behavior, not just shapes: constructor args, default values,
|
||||
runtime flags, RNG use, dtype/autocast, scaling constants, masks, and output
|
||||
containers all matter.
|
||||
- Keep the implementation minimal and native. Do not keep a runtime import of
|
||||
the official implementation as the production component.
|
||||
- For schedulers, compare timesteps, sigmas/noise levels, step outputs, shift
|
||||
handling, prediction type, and any mutable internal state.
|
||||
- For upsamplers, compare resize mode, align_corners, residual branches,
|
||||
causal padding, normalization, and exact target-shape behavior.
|
||||
- For vocoders/audio components, compare waveform shape, sample-rate contract,
|
||||
channel order, hop length, normalization, and dtype.
|
||||
- If private upstream deps are required only for tests, keep stubs under
|
||||
`tests/local_tests/helpers/` and do not import them from production code.
|
||||
|
||||
## Prototype Checks
|
||||
|
||||
Follow the shared prototype success criteria.
|
||||
|
||||
## Parity-Debug Loop
|
||||
|
||||
Run the shared parity-debug loop. The component test command is:
|
||||
|
||||
```bash
|
||||
pytest <parity_test> -v -s
|
||||
```
|
||||
|
||||
For numerical drift, add targeted intermediate comparisons in the test to
|
||||
identify the first divergent operation.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
|
||||
in `../add-model/shared/component_skill_common.md`. Generic-component ask cases
|
||||
include creating a new loader bucket, accepting an unsupported private op,
|
||||
choosing between incompatible official definitions, or dropping a required
|
||||
component.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/component_skill_handoff.md` following the common
|
||||
handoff rules in `../add-model/shared/component_skill_common.md`.
|
||||
@@ -1,186 +0,0 @@
|
||||
---
|
||||
name: add-model-07-conversion
|
||||
description: Use during /add-model Phase 5 to write and verify a FastVideo checkpoint conversion script after native component prototypes expose FastVideo state-dict keys/shapes.
|
||||
---
|
||||
|
||||
# Add Model Conversion
|
||||
|
||||
## Goal
|
||||
|
||||
Convert official weights into a FastVideo-loadable component layout after Phase 4
|
||||
native prototypes exist. The conversion script owns parameter mapping, component
|
||||
splitting, passthrough assets, config emission, and strict-load verification.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
|
||||
escape hatches, production boundaries, and skip/pass semantics.
|
||||
|
||||
Require the initial request from
|
||||
`../add-model/contracts/conversion_request.md`.
|
||||
|
||||
If the FastVideo key/shape dump is missing, return to `/add-model` Phase 4. Do
|
||||
not write a final mapping against an unimplemented component.
|
||||
|
||||
For Phase 6 retry requests from component skills, also require the retry shape
|
||||
from `../add-model/contracts/conversion_request.md`.
|
||||
|
||||
## Output
|
||||
|
||||
- `scripts/checkpoint_conversion/<family>_to_diffusers.py`.
|
||||
- `converted_weights/<family>/` with `model_index.json` and per-component
|
||||
subfolders.
|
||||
- Updated `tests/local_tests/<model_family>/README.md` with conversion command,
|
||||
source layout, output path, and strict-load status.
|
||||
- Updated `tests/local_tests/<model_family>/PORT_STATUS.md` with conversion
|
||||
state, retry history, open questions, and issues/blockers.
|
||||
|
||||
## Reference Scripts
|
||||
|
||||
- `scripts/checkpoint_conversion/convert_ltx2_weights.py`: component prefix
|
||||
splitting, metadata config extraction, passthrough Gemma/tokenizer assets, and
|
||||
optional component-only output.
|
||||
- `scripts/checkpoint_conversion/stable_audio_to_diffusers.py`: monolithic
|
||||
`model.safetensors` split into transformer/VAE/conditioner, plus copied
|
||||
passthrough subfolders. Use this shape for single-checkpoint official repos.
|
||||
- `scripts/checkpoint_conversion/convert_gamecraft_full.py`: separate official
|
||||
sources for transformer, VAE, text encoders, tokenizers, scheduler, and root
|
||||
`model_index.json`.
|
||||
- `scripts/checkpoint_conversion/longcat_to_fastvideo.py`: fused QKV/KV split,
|
||||
renamed native transformer weights, and copied existing Diffusers components.
|
||||
- `scripts/checkpoint_conversion/pt_to_safetensors.py`: simple `.pt` extraction
|
||||
helper for nested checkpoint dictionaries.
|
||||
|
||||
## Source Layout Decision
|
||||
|
||||
Choose exactly one primary layout:
|
||||
|
||||
| Layout | Conversion behavior |
|
||||
|---|---|
|
||||
| `diffusers` | Usually no tensor remap; verify configs/classes and copy or update `_class_name` only when needed. |
|
||||
| `raw_official` | Convert a raw official checkpoint file or directory. Choose explicit component ownership before writing output. |
|
||||
| `separate_components` | Convert/copy each component from its own file or directory. |
|
||||
| `monolithic` | Load one model checkpoint and split state dict by authoritative prefixes into component buckets. |
|
||||
| `mixed` | Convert some components and copy passthrough components such as tokenizers, text encoders, schedulers, or already-Diffusers VAE dirs. |
|
||||
| `custom` | Document why none of the above fits before writing conversion code. |
|
||||
|
||||
Monolithic checkpoints need explicit prefix ownership. For example, Stable Audio
|
||||
uses one `model.safetensors` with DiT, pretransform/VAE, and conditioner keys;
|
||||
the converter splits those keys into FastVideo component subfolders and writes
|
||||
per-component configs.
|
||||
|
||||
## Script Shape
|
||||
|
||||
Start from `templates/family_to_diffusers.py` or the closest reference script.
|
||||
Keep the script explicit and reviewable:
|
||||
|
||||
- `COMPONENT_SPECS` or `COMPONENT_PREFIXES` declares component ownership.
|
||||
- `PARAM_NAME_MAP` declares key renames.
|
||||
- `SKIP_PATTERNS` declares intentionally dropped training-only keys.
|
||||
- tensor split/fuse helpers are named by operation, e.g. `split_qkv`.
|
||||
- `build_component_configs(...)` writes loader-compatible config files. Most
|
||||
model components use `config.json`; schedulers use `scheduler_config.json`.
|
||||
- `build_model_index(...)` writes a root `model_index.json` matching the target
|
||||
FastVideo pipeline and component classes.
|
||||
- verification reports missing, unexpected, skipped, unchanged, renamed, and
|
||||
shape-mismatched keys.
|
||||
|
||||
`model_index.json` library tokens must match FastVideo loaders:
|
||||
|
||||
- standard native DiT/VAE/audio/vocoder/upsampler components loaded by existing
|
||||
Diffusers-style loaders usually use `"diffusers"` with a FastVideo
|
||||
`_class_name` in the component `config.json`;
|
||||
- text encoders, tokenizers, image encoders, processors, and feature extractors
|
||||
usually use `"transformers"`;
|
||||
- `conditioner` currently expects `"fastvideo"`;
|
||||
- use fully qualified `"fastvideo.<module>"` only when intentionally relying on
|
||||
the custom fastvideo-library escape path;
|
||||
- do not write bare `"fastvideo"` for transformer, VAE, or other loaders that
|
||||
expect `"diffusers"` unless the loader explicitly expects it.
|
||||
|
||||
## Mapping Rules
|
||||
|
||||
- Use Phase 4 key/shape dumps to derive mappings. Do not guess from official key
|
||||
names alone.
|
||||
- Preserve each component's official file paths, parity test path, and prototype
|
||||
concerns in comments or structured constants near the mapping that uses them.
|
||||
- Every official inference parameter should be mapped, copied through, or listed
|
||||
as intentionally skipped with a reason.
|
||||
- Every FastVideo prototype parameter should receive a tensor or be listed as an
|
||||
intentional external/passthrough parameter.
|
||||
- Shape matches are necessary but not sufficient; check semantic pairing for
|
||||
Q/K/V, gate/up/down, norm scale/bias, LoRA/base, and modality-specific heads.
|
||||
- If official and FastVideo fuse or split tensors differently, convert tensors in
|
||||
the script rather than changing production code to match checkpoint quirks.
|
||||
|
||||
## Verification
|
||||
|
||||
Run conversion locally, then verify before returning to Phase 6. For retry
|
||||
requests, update the mapping, rerun conversion, and refresh only the implicated
|
||||
converted component when safe; otherwise rerun the full conversion.
|
||||
|
||||
```bash
|
||||
python scripts/checkpoint_conversion/<family>_to_diffusers.py \
|
||||
--src <official_weights> \
|
||||
--revision <hf_revision> \
|
||||
--dst converted_weights/<model_family>
|
||||
```
|
||||
|
||||
Omit `--revision` for local sources or when prep recorded `default` / `none`.
|
||||
|
||||
Minimum output layout:
|
||||
|
||||
```text
|
||||
converted_weights/<family>/
|
||||
model_index.json
|
||||
transformer/config.json
|
||||
transformer/*.safetensors
|
||||
vae/config.json
|
||||
vae/*.safetensors
|
||||
scheduler/scheduler_config.json as needed
|
||||
text_encoder/... as needed
|
||||
```
|
||||
|
||||
Required checks:
|
||||
|
||||
- `model_index.json` exists and lists every required component.
|
||||
- Each converted component has the config filename its loader expects and
|
||||
safetensors weights when it owns weights. Scheduler dirs require
|
||||
`scheduler_config.json`; most other native model dirs use `config.json`.
|
||||
- Weight filenames may vary by loader: transformer and VAE loaders glob all
|
||||
`*.safetensors`; text encoders may load `*.safetensors`, `*.bin`, and
|
||||
sometimes `*.pt`; `conditioner` currently expects
|
||||
`diffusion_pytorch_model.safetensors`. Use the loader's actual accepted layout
|
||||
rather than assuming one global filename.
|
||||
- Passthrough components are copied or referenced deliberately.
|
||||
- Each emitted component config validates through the same path production
|
||||
loaders use. Instantiate the relevant config and call `update_model_arch(...)`
|
||||
or `update_model_config(...)` with the emitted JSON so unknown keys fail during
|
||||
conversion, not at pipeline load time.
|
||||
- Record production loader strictness for every stateful component. If the loader
|
||||
intentionally uses non-strict loading, add explicit missing/unexpected-key
|
||||
assertions in the parity test and document exactly which keys are allowed.
|
||||
- Each new FastVideo component strict-loads converted weights where its production
|
||||
loader is strict. If strict loading is impossible, record the exact allowed
|
||||
missing/unexpected keys and why they are not inference weights.
|
||||
- Retry fixes include the original component parity evidence and the new
|
||||
strict-load result in `local_tests_readme` so the component subagent can resume
|
||||
without rediscovering context.
|
||||
- `local_tests_readme` records the command, output directory, and strict-load
|
||||
result.
|
||||
|
||||
Do not chase numerical parity in this skill except to identify a conversion
|
||||
mapping bug. Long parity-debug loops belong to `/add-model` Phase 6.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md`. Conversion-specific ask cases
|
||||
include selecting between incompatible official checkpoints, publishing/uploading
|
||||
weights, overwriting an existing converted repo not created by this run,
|
||||
accepting non-strict missing inference weights, or dropping a component/output
|
||||
from scope.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/conversion_handoff.md` and update the shared state
|
||||
files before handoff.
|
||||
@@ -1,293 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Convert <model_family> official weights to a FastVideo Diffusers-style tree.
|
||||
|
||||
This template supports both separate component sources and a monolithic pipeline
|
||||
checkpoint that must be split by component prefix. Replace every TODO before
|
||||
using it for a real port.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
from collections import OrderedDict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
from safetensors import safe_open
|
||||
from safetensors.torch import load_file, save_file
|
||||
|
||||
try:
|
||||
from huggingface_hub import snapshot_download
|
||||
except ImportError: # pragma: no cover - optional local conversion dependency
|
||||
snapshot_download = None
|
||||
|
||||
# TODO: fill with authoritative component prefixes for monolithic checkpoints.
|
||||
# Example: {"model.model.": "transformer", "pretransform.model.": "vae"}
|
||||
COMPONENT_PREFIXES: dict[str, str] = {}
|
||||
|
||||
# TODO: fill with component-specific source paths for separate-component repos.
|
||||
# Example: {"transformer": "transformer/model.safetensors", "vae": "vae/"}
|
||||
SEPARATE_COMPONENT_PATHS: dict[str, str] = {}
|
||||
|
||||
# TODO: copy passthrough dirs that are already loadable by FastVideo/Diffusers.
|
||||
PASSTHROUGH_SUBFOLDERS: tuple[str, ...] = ("tokenizer", "scheduler")
|
||||
|
||||
# TODO: add regex renames derived from Phase 4 key/shape dumps.
|
||||
PARAM_NAME_MAP: dict[str, str] = {}
|
||||
|
||||
# TODO: include training-only or dynamically-computed keys that must not load.
|
||||
SKIP_PATTERNS: tuple[str, ...] = ()
|
||||
|
||||
|
||||
def _hf_token() -> str | None:
|
||||
return (os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACE_HUB_TOKEN") or os.environ.get("HF_API_KEY"))
|
||||
|
||||
|
||||
def resolve_src(src: str, revision: str | None) -> Path:
|
||||
if os.path.exists(src):
|
||||
return Path(src)
|
||||
if snapshot_download is None:
|
||||
raise RuntimeError("huggingface_hub is required when --src is a repo id")
|
||||
return Path(snapshot_download(repo_id=src, revision=revision, token=_hf_token()))
|
||||
|
||||
|
||||
def load_checkpoint(path: Path) -> dict[str, torch.Tensor]:
|
||||
if path.is_dir():
|
||||
weights: dict[str, torch.Tensor] = {}
|
||||
for shard in sorted(path.glob("*.safetensors")):
|
||||
weights.update(load_file(str(shard)))
|
||||
if weights:
|
||||
return weights
|
||||
raise FileNotFoundError(f"No safetensors found in {path}")
|
||||
|
||||
if path.suffix == ".safetensors":
|
||||
return load_file(str(path))
|
||||
|
||||
checkpoint = torch.load(path, map_location="cpu", weights_only=True)
|
||||
if isinstance(checkpoint, dict):
|
||||
for key in ("state_dict", "model_state_dict", "model", "module", "ema"):
|
||||
if key in checkpoint and isinstance(checkpoint[key], dict):
|
||||
return checkpoint[key]
|
||||
return checkpoint
|
||||
raise TypeError(f"Unsupported checkpoint type: {type(checkpoint)!r}")
|
||||
|
||||
|
||||
def should_skip_key(key: str) -> bool:
|
||||
return any(re.search(pattern, key) for pattern in SKIP_PATTERNS)
|
||||
|
||||
|
||||
def apply_mapping(key: str) -> str | None:
|
||||
if should_skip_key(key):
|
||||
return None
|
||||
for pattern, replacement in PARAM_NAME_MAP.items():
|
||||
if re.match(pattern, key):
|
||||
return re.sub(pattern, replacement, key)
|
||||
return key
|
||||
|
||||
|
||||
def split_monolithic(state: dict[str, torch.Tensor], ) -> dict[str, OrderedDict[str, torch.Tensor]]:
|
||||
components: dict[str, OrderedDict[str, torch.Tensor]] = {
|
||||
name: OrderedDict()
|
||||
for name in set(COMPONENT_PREFIXES.values())
|
||||
}
|
||||
intentionally_skipped: list[str] = []
|
||||
unowned: list[str] = []
|
||||
for key, value in state.items():
|
||||
if should_skip_key(key):
|
||||
intentionally_skipped.append(key)
|
||||
continue
|
||||
for prefix, component in COMPONENT_PREFIXES.items():
|
||||
if key.startswith(prefix):
|
||||
mapped = apply_mapping(key[len(prefix):])
|
||||
if mapped is not None:
|
||||
components[component][mapped] = value
|
||||
break
|
||||
else:
|
||||
unowned.append(key)
|
||||
if unowned:
|
||||
sample = ", ".join(unowned[:10])
|
||||
raise ValueError(f"Unowned monolithic keys: {len(unowned)}. "
|
||||
f"Add COMPONENT_PREFIXES or SKIP_PATTERNS entries. Sample: {sample}")
|
||||
if intentionally_skipped:
|
||||
print(f"Intentionally skipped {len(intentionally_skipped)} keys")
|
||||
return {name: weights for name, weights in components.items() if weights}
|
||||
|
||||
|
||||
def load_separate_components(src_dir: Path) -> dict[str, OrderedDict[str, torch.Tensor]]:
|
||||
components: dict[str, OrderedDict[str, torch.Tensor]] = {}
|
||||
for component, rel_path in SEPARATE_COMPONENT_PATHS.items():
|
||||
state = load_checkpoint(src_dir / rel_path)
|
||||
converted: OrderedDict[str, torch.Tensor] = OrderedDict()
|
||||
for key, value in state.items():
|
||||
mapped = apply_mapping(key)
|
||||
if mapped is not None:
|
||||
converted[mapped] = value
|
||||
components[component] = converted
|
||||
return components
|
||||
|
||||
|
||||
def build_component_configs(_src_dir: Path) -> dict[str, dict[str, Any]]:
|
||||
# TODO: emit config content accepted by FastVideo loaders. Most components use
|
||||
# config.json; schedulers use scheduler_config.json.
|
||||
return {
|
||||
"transformer": {
|
||||
"_class_name": "<FastVideoTransformerClass>"
|
||||
},
|
||||
"vae": {
|
||||
"_class_name": "<FastVideoVAEClass>"
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def config_filename(component: str) -> str:
|
||||
if component == "scheduler":
|
||||
return "scheduler_config.json"
|
||||
return "config.json"
|
||||
|
||||
|
||||
def source_label(src: str) -> str:
|
||||
if os.path.exists(src):
|
||||
return Path(src).name
|
||||
return src
|
||||
|
||||
|
||||
def build_model_index(
|
||||
src: str,
|
||||
revision: str | None,
|
||||
available_components: set[str],
|
||||
) -> dict[str, Any]:
|
||||
# TODO: match the target pipeline and every required component.
|
||||
index: dict[str, Any] = {
|
||||
"_class_name": "<FastVideoPipelineClass>",
|
||||
"_diffusers_version": "0.30.0",
|
||||
"_fastvideo_converted_from": source_label(src),
|
||||
# Existing transformer/VAE loaders expect "diffusers" even when
|
||||
# _class_name names a FastVideo-native class registered in FastVideo.
|
||||
"transformer": ["diffusers", "<FastVideoTransformerClass>"],
|
||||
"vae": ["diffusers", "<FastVideoVAEClass>"],
|
||||
}
|
||||
if revision:
|
||||
index["_fastvideo_converted_revision"] = revision
|
||||
return {key: value for key, value in index.items() if key.startswith("_") or key in available_components}
|
||||
|
||||
|
||||
def validate_component_configs(configs: dict[str, dict[str, Any]]) -> None:
|
||||
# TODO: instantiate each FastVideo config and call update_model_arch(...) or
|
||||
# update_model_config(...) with this JSON so unknown emitted keys fail here.
|
||||
placeholder_configs = [name for name, config in configs.items() if "<" in json.dumps(config)]
|
||||
if placeholder_configs:
|
||||
raise ValueError(f"Replace config placeholders for: {placeholder_configs}")
|
||||
|
||||
|
||||
def verify_conversion(
|
||||
dst_dir: Path,
|
||||
components: dict[str, OrderedDict[str, torch.Tensor]],
|
||||
) -> None:
|
||||
del dst_dir, components
|
||||
# TODO: load each emitted stateful component through its production loader and
|
||||
# assert strict load, or document exact allowed missing/unexpected keys.
|
||||
raise NotImplementedError("Implement production config validation and strict-load checks")
|
||||
|
||||
|
||||
def write_component(
|
||||
dst_dir: Path,
|
||||
name: str,
|
||||
state: dict[str, torch.Tensor],
|
||||
config: dict[str, Any] | None,
|
||||
) -> None:
|
||||
component_dir = dst_dir / name
|
||||
if component_dir.exists() and any(component_dir.iterdir()):
|
||||
shutil.rmtree(component_dir)
|
||||
component_dir.mkdir(parents=True, exist_ok=True)
|
||||
save_file(dict(state), str(component_dir / "diffusion_pytorch_model.safetensors"))
|
||||
if config is not None:
|
||||
config_path = component_dir / config_filename(name)
|
||||
with config_path.open("w", encoding="utf-8") as f:
|
||||
json.dump(config, f, indent=2)
|
||||
f.write("\n")
|
||||
print(f"Wrote {name}: {len(state)} tensors")
|
||||
|
||||
|
||||
def copy_passthrough(src_dir: Path, dst_dir: Path) -> list[str]:
|
||||
copied: list[str] = []
|
||||
for subfolder in PASSTHROUGH_SUBFOLDERS:
|
||||
src = src_dir / subfolder
|
||||
if not src.is_dir():
|
||||
continue
|
||||
dst = dst_dir / subfolder
|
||||
if dst.exists():
|
||||
shutil.rmtree(dst)
|
||||
shutil.copytree(src, dst)
|
||||
copied.append(subfolder)
|
||||
print(f"Copied {subfolder}/")
|
||||
return copied
|
||||
|
||||
|
||||
def default_monolithic_checkpoint(src_path: Path) -> Path:
|
||||
if src_path.is_file():
|
||||
return src_path
|
||||
return src_path / "model.safetensors"
|
||||
|
||||
|
||||
def convert(
|
||||
src: str,
|
||||
dst: str,
|
||||
layout: str,
|
||||
revision: str | None,
|
||||
) -> None:
|
||||
src_path = resolve_src(src, revision)
|
||||
dst_dir = Path(dst)
|
||||
dst_dir.mkdir(parents=True, exist_ok=True)
|
||||
model_index_path = dst_dir / "model_index.json"
|
||||
|
||||
if layout in {"monolithic", "raw_official"}:
|
||||
# TODO: replace model.safetensors with the official monolithic file name.
|
||||
components = split_monolithic(load_checkpoint(default_monolithic_checkpoint(src_path)))
|
||||
elif layout in {"separate_components", "mixed"}:
|
||||
if not src_path.is_dir():
|
||||
raise ValueError(f"{layout} layout requires a source directory: {src_path}")
|
||||
components = load_separate_components(src_path)
|
||||
else:
|
||||
raise ValueError(f"Unsupported template layout: {layout}")
|
||||
|
||||
copied = (copy_passthrough(src_path, dst_dir) if src_path.is_dir() else [])
|
||||
configs = build_component_configs(src_path if src_path.is_dir() else src_path.parent)
|
||||
validate_component_configs(configs)
|
||||
for name, state in components.items():
|
||||
write_component(dst_dir, name, state, configs.get(name))
|
||||
|
||||
available = set(components) | set(copied)
|
||||
with model_index_path.open("w", encoding="utf-8") as f:
|
||||
json.dump(build_model_index(src, revision, available), f, indent=2)
|
||||
f.write("\n")
|
||||
print(f"Wrote {dst_dir / 'model_index.json'}")
|
||||
verify_conversion(dst_dir, components)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--src", required=True, help="HF repo id, local dir, or checkpoint path")
|
||||
parser.add_argument("--revision", help="HF branch, tag, or commit for repo sources")
|
||||
parser.add_argument(
|
||||
"--dst",
|
||||
required=True,
|
||||
help="Output converted_weights/<model_family> directory",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--layout",
|
||||
choices=("raw_official", "monolithic", "separate_components", "mixed"),
|
||||
required=True,
|
||||
help="Official source layout",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
convert(args.src, args.dst, args.layout, args.revision)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,214 +0,0 @@
|
||||
---
|
||||
name: add-model-08-trace
|
||||
description: Use during /add-model Phase 6 when component parity has failed and root cause requires layer-by-layer divergence analysis. Uses FastVideo activation trace first, falling back to custom hooks only for boundaries or stats the utility cannot observe.
|
||||
---
|
||||
|
||||
# Add-Model Trace
|
||||
|
||||
## Manual Invocation
|
||||
|
||||
Load this skill when `/add-model` Phase 6 component parity has failed and the
|
||||
root cause requires layer-by-layer divergence analysis. This skill is not
|
||||
auto-fired. The calling subagent (DiT, VAE, encoder, or generic port skill)
|
||||
loads it when its standard parity-debug loop hits a wall and cannot isolate
|
||||
the divergence from end-to-end tensor comparisons alone.
|
||||
|
||||
Do not load this skill for first-pass parity failures. Try weight-diff and
|
||||
end-to-end tensor comparison first. Load this skill only when those do not
|
||||
isolate the cause.
|
||||
|
||||
## Goal
|
||||
|
||||
Find the first numerical divergence point between FastVideo's port and the
|
||||
official reference, layer by layer, by instrumenting both sides at matching
|
||||
tensor boundaries. The investigation must leave zero source residue in
|
||||
production code when it closes.
|
||||
|
||||
## When To Run
|
||||
|
||||
After a component parity test FAILS at a bf16-noise-realistic tolerance AND
|
||||
the calling subagent's first-pass debug (weight-diff, end-to-end tensor
|
||||
compare) does not isolate the cause.
|
||||
|
||||
Required inputs before starting:
|
||||
|
||||
- A working FastVideo loader for the component under investigation.
|
||||
- A working official loader, typically via
|
||||
`tests/local_tests/helpers/<family>_upstream.py::load_upstream_<component>`.
|
||||
- Shared deterministic test inputs (same tensors on both sides).
|
||||
- The component parity test file path and its current failure output.
|
||||
|
||||
## Primary Path: FastVideo Activation Trace
|
||||
|
||||
Use FastVideo's first-class activation trace before writing custom hooks:
|
||||
`fastvideo/hooks/activation_trace.py`, documented in
|
||||
`docs/contributing/activation_trace.md`.
|
||||
|
||||
Pipeline runs attach trace to the transformer during pipeline initialization.
|
||||
Component-only parity harnesses may call `attach_activation_trace(model)` from
|
||||
local test/debug code; do not add trace calls to production model code.
|
||||
|
||||
Prefix the failing parity command with a tight layer regex:
|
||||
|
||||
```bash
|
||||
FASTVIDEO_TRACE_ACTIVATIONS=1 \
|
||||
FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$" \
|
||||
FASTVIDEO_TRACE_STATS="abs_mean,sum,max,shape" \
|
||||
FASTVIDEO_TRACE_STEPS="0" \
|
||||
FASTVIDEO_TRACE_OUTPUT="/tmp/opencode/fv_trace.jsonl" \
|
||||
pytest tests/local_tests -k "parity" -v -s
|
||||
```
|
||||
|
||||
Match the layer regex to the actual `model.named_modules()` names. Empty or
|
||||
broad regexes are expensive; prefer block-level names first, then narrow to
|
||||
submodules after the first divergent block is known.
|
||||
|
||||
## Trace Compare Contract
|
||||
|
||||
One JSONL file per side. FastVideo output should use `FASTVIDEO_TRACE_OUTPUT`;
|
||||
the upstream harness should emit the same JSONL shape:
|
||||
|
||||
```json
|
||||
{"module":"block.layers.0","tensor":"out","step":0,"abs_mean":0.0123,"sum":1.0,"max":0.5,"shape":[1,16,32]}
|
||||
```
|
||||
|
||||
Compare rows by `(module, step, tensor)`. The first row whose `shape`,
|
||||
`abs_mean`, or `max` diverges beyond the component tolerance is the first broken
|
||||
boundary. Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
|
||||
`FASTVIDEO_TRACE_STEPS` identical between sides; if row order differs, sort or
|
||||
normalize before diffing.
|
||||
|
||||
## Drill-Down Loop
|
||||
|
||||
**Initial run:** trace every top-level block (`^block\.layers\.[0-9]+$` or the
|
||||
family's equivalent). Identify the first block index where `abs_mean` or `max`
|
||||
drifts beyond tolerance while earlier blocks match.
|
||||
|
||||
**Drill run:** tighten `FASTVIDEO_TRACE_LAYERS` to submodules inside the first
|
||||
divergent block: attention output, MLP projections, norm outputs, modality
|
||||
adapters, or other named boundaries exposed by `named_modules()`.
|
||||
|
||||
**Iterate:** if the first divergent operation is a free function or tensor op not
|
||||
visible as an `nn.Module`, use the fallback instrumentation hierarchy below.
|
||||
|
||||
The loop ends when the first divergent submodule or operation is identified with
|
||||
a file:line citation in the official source.
|
||||
|
||||
## Fallback Instrumentation Hierarchy
|
||||
|
||||
Use these only when activation trace cannot observe the needed boundary or
|
||||
statistic.
|
||||
|
||||
### (1) Custom forward hooks
|
||||
|
||||
`module.register_forward_hook(...)` and `register_forward_pre_hook(...)`.
|
||||
Always within `try/finally` with `handle.remove()`. Zero source residue.
|
||||
|
||||
### (2) Runtime monkey-patch
|
||||
|
||||
`module.attr = wrapped_func` or `cls.method = wrapped_method`, restored via
|
||||
`try/finally` (save original first). Use for free functions and non-Module sites
|
||||
such as activation functions (`swiglu`, `apply_rotary_emb`).
|
||||
|
||||
### (3) Source edits in FastVideo's own code
|
||||
|
||||
Only when (1) and (2) are insufficient. Track all edits within a single named
|
||||
`git stash` boundary OR a temporary branch. Run `git diff` before closing the
|
||||
investigation to confirm cleanup.
|
||||
|
||||
### (4) Source edits in official repo source
|
||||
|
||||
Allowed only when hook and monkey-patch approaches cannot capture the site.
|
||||
For git-tracked or editable official clones, use `git diff` in the clone path to
|
||||
verify cleanup. For non-editable site-packages, back up the target file before
|
||||
editing and restore it before handoff.
|
||||
|
||||
## Hypothesis Toggles
|
||||
|
||||
Use env-var-gated monkey-patches to A/B test suspect implementations without
|
||||
source edits. Pattern: `<FAMILY>_DEBUG_PATCH_<HYPOTHESIS>=1`.
|
||||
|
||||
Example from the magi-human investigation:
|
||||
|
||||
```
|
||||
MAGI_DEBUG_PATCH_LINEAR=1
|
||||
```
|
||||
|
||||
This patched `PackedExpertLinear.forward` to mirror upstream's
|
||||
`_BF16ComputeLinear` explicit-cast pattern, isolating a dtype-cast difference
|
||||
as the root cause.
|
||||
|
||||
Document all toggles in the script docstring. Each toggle must:
|
||||
|
||||
- save the original before patching;
|
||||
- restore the original in a `try/finally` block;
|
||||
- print a `[debug] Patched <ClassName>.<method>` line to stdout when active.
|
||||
|
||||
## Cleanup Gate
|
||||
|
||||
The calling agent MUST report `[cleanup-gate] PASS` on all five items before
|
||||
handoff. Do not hand off with any item unresolved.
|
||||
|
||||
1. `git diff` in the FastVideo repo: empty. No stray prints, hooks, or
|
||||
monkey-patches in production code.
|
||||
2. `git diff` in the official-repo clone (if used): empty. For non-editable
|
||||
site-packages installs: `diff original.py original.py.trace-backup` is
|
||||
empty OR `pip install --force-reinstall <pkg>` succeeded and the installed
|
||||
file matches the original.
|
||||
3. `git stash list`: only the named investigation stash (or empty). No
|
||||
unnamed stashes left from this session.
|
||||
4. No new untracked files outside `/tmp/opencode/` (logs) and the existing
|
||||
debug script directory (`tests/local_tests/transformers/` or equivalent).
|
||||
5. `mypy` clean on any production files touched during the investigation.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Escalate to the calling bucket skill when:
|
||||
|
||||
- A forward hook on an official module raises because of a custom `forward`
|
||||
signature or varlen handler args that the hook closure cannot satisfy. The
|
||||
bucket skill has component-specific knowledge to work around this.
|
||||
- The first divergent layer is `block[0]`, meaning the divergence is in the
|
||||
adapter, modality dispatcher, coordinate embedding, or packing step before
|
||||
any block runs. Check those sites first; the bug is not in attention or MLP.
|
||||
- Per-block drift is never zero anywhere across all blocks. This usually means
|
||||
the inputs are not bit-identical between sides. Verify with a state-dict
|
||||
compare (weight-diff script) AND confirm the input tensors are the same
|
||||
object or have identical values before the forward call.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return to the calling subagent with:
|
||||
|
||||
- FastVideo trace JSONL path and upstream trace JSONL path.
|
||||
- Trace settings used: `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
|
||||
`FASTVIDEO_TRACE_STEPS`.
|
||||
- The first divergent `(module, step, tensor)` row and observed drift.
|
||||
- The upstream file:line citation where the divergence originates.
|
||||
- Fallback hook/patch verdict if activation trace could not observe the boundary.
|
||||
- Hypothesis verdict if an A/B toggle was used, for example `PATCH_LINEAR=1`.
|
||||
- Cleanup-gate status: `[cleanup-gate] PASS` or a list of unresolved items.
|
||||
|
||||
The calling agent uses this to scope the production fix in the FastVideo
|
||||
component file.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/contributing/activation_trace.md` for canonical activation-trace env vars,
|
||||
JSONL output, cost model, and troubleshooting.
|
||||
- `fastvideo/hooks/activation_trace.py` for the implementation and
|
||||
`attach_activation_trace(model)` entry point.
|
||||
- `templates/block_trace_debug.py` in this skill directory: fallback custom-hook
|
||||
template when activation trace cannot observe the needed boundary or stat.
|
||||
- `tests/local_tests/transformers/_debug_magi_human_block_parity.py` in the
|
||||
FastVideo3 repo: historical worked example for custom hook/patch debugging.
|
||||
- `add-model/SKILL.md` Phase 6: the calling context for this skill.
|
||||
- `add-model-03-port-dit/SKILL.md`, `add-model-04-port-vae/SKILL.md`,
|
||||
`add-model-05-port-encoder/SKILL.md`, `add-model-06-port-generic/SKILL.md`:
|
||||
bucket-specific debug language and component-specific escape-hatch knowledge.
|
||||
|
||||
## Changelog
|
||||
|
||||
| Date | Change |
|
||||
|---|---|
|
||||
| 2026-05-01 | Initial skill extracted from `_debug_magi_human_block_parity.py` pattern. |
|
||||
@@ -1,334 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Per-block divergence debugger template for FastVideo model ports.
|
||||
|
||||
Run directly (not a pytest test):
|
||||
python tests/local_tests/transformers/_debug_<family>_<component>_parity.py
|
||||
|
||||
Generalizes: tests/local_tests/transformers/_debug_magi_human_block_parity.py
|
||||
|
||||
Fill FAMILY, COMPONENT, and the two loader functions. Run once for the initial
|
||||
drift table, then set <FAMILY>_DEBUG_DRILL_LAYER=NN to drill into submodules.
|
||||
Add <FAMILY>_DEBUG_PATCH_<HYPOTHESIS>=1 to A/B test a suspect implementation.
|
||||
|
||||
CLEANUP: all hooks removed in try/finally; monkey-patches restored in
|
||||
try/finally; source edits tracked in a named git stash. Zero source residue.
|
||||
See add-model-08-trace/SKILL.md for the full cleanup gate checklist.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import gc
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
FAMILY: str = "<family>" # e.g. "magi_human", "ltx2", "wan"
|
||||
COMPONENT: str = "<component>" # e.g. "dit", "vae", "encoder"
|
||||
DRILL_LAYER_ENV: str = "<FAMILY>_DEBUG_DRILL_LAYER"
|
||||
HYPOTHESIS_ENV: str = "<FAMILY>_DEBUG_PATCH_<HYPOTHESIS>"
|
||||
REL_THRESHOLD: float = 0.005 # 0.5% abs_mean drift flags a block as divergent
|
||||
LOG_DIR: Path = Path("/tmp/opencode")
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
|
||||
def load_official(device: torch.device) -> torch.nn.Module:
|
||||
"""Load the official upstream model. TODO: implement for your family.
|
||||
|
||||
Example (magi-human):
|
||||
from tests.local_tests.helpers.magi_human_upstream import install_stubs, load_upstream_dit
|
||||
install_stubs()
|
||||
return load_upstream_dit(base_shard_dir, device=device, dtype=None)
|
||||
"""
|
||||
raise NotImplementedError(f"Fill load_official() for {FAMILY}/{COMPONENT}.")
|
||||
|
||||
|
||||
def load_fastvideo(device: torch.device) -> torch.nn.Module:
|
||||
"""Load the FastVideo-native model. TODO: implement for your family.
|
||||
|
||||
Example (magi-human):
|
||||
from fastvideo.configs.models.dits.magi_human import MagiHumanVideoConfig
|
||||
from fastvideo.models.dits.magi_human import MagiHumanDiT
|
||||
from safetensors.torch import load_file; import glob
|
||||
fv = MagiHumanDiT(MagiHumanVideoConfig())
|
||||
state = {}
|
||||
for shard in sorted(glob.glob(str(transformer_dir / "*.safetensors"))): state.update(load_file(shard))
|
||||
fv.load_state_dict(state, strict=False); return fv.to(device).eval()
|
||||
"""
|
||||
raise NotImplementedError(f"Fill load_fastvideo() for {FAMILY}/{COMPONENT}.")
|
||||
|
||||
|
||||
def build_inputs(device: torch.device) -> dict[str, Any]:
|
||||
"""Return deterministic inputs shared by both sides. TODO: replace.
|
||||
|
||||
Both sides must receive the SAME tensors (clone before each forward call).
|
||||
Non-identical inputs cause non-zero drift everywhere.
|
||||
"""
|
||||
torch.manual_seed(0)
|
||||
return {"x": torch.randn(64, 1024, dtype=torch.bfloat16, device=device)}
|
||||
|
||||
|
||||
def _stat(name: str, t: torch.Tensor) -> dict:
|
||||
f = t.detach().float()
|
||||
return {
|
||||
"name": name,
|
||||
"shape": tuple(t.shape),
|
||||
"abs_mean": f.abs().mean().item(),
|
||||
"sum": f.sum().item(),
|
||||
"min": f.min().item(),
|
||||
"max": f.max().item(),
|
||||
}
|
||||
|
||||
|
||||
def _attach_block_hooks(
|
||||
model: torch.nn.Module,
|
||||
label: str,
|
||||
log: list[dict],
|
||||
tensors: dict[str, torch.Tensor] | None = None,
|
||||
drill_layer: int | None = None,
|
||||
) -> list[Any]:
|
||||
"""Return hook handles. Caller MUST remove them in try/finally."""
|
||||
handles: list[Any] = []
|
||||
|
||||
def _hook(name: str):
|
||||
|
||||
def fn(_module, _inputs, outputs):
|
||||
t = outputs[0] if isinstance(outputs, tuple) else outputs
|
||||
if not torch.is_tensor(t):
|
||||
return
|
||||
log.append({"side": label, **_stat(name, t)})
|
||||
if tensors is not None:
|
||||
tensors[name] = t.detach().float().cpu()
|
||||
|
||||
return fn
|
||||
|
||||
def _pre_hook(name: str):
|
||||
# Pre-hooks observe a free function's output by intercepting the next
|
||||
# module's input (useful when the activation is not an nn.Module).
|
||||
def fn(_module, inputs):
|
||||
t = inputs[0] if isinstance(inputs, tuple) else inputs
|
||||
if not torch.is_tensor(t):
|
||||
return
|
||||
key = f"{name}<in>"
|
||||
log.append({"side": label, **_stat(key, t)})
|
||||
if tensors is not None:
|
||||
tensors[key] = t.detach().float().cpu()
|
||||
|
||||
return fn
|
||||
|
||||
# TODO: adapt attribute paths to your model. Remove adapter block if absent.
|
||||
if hasattr(model, "adapter"):
|
||||
handles.append(model.adapter.register_forward_hook(_hook("adapter")))
|
||||
|
||||
# TODO: adapt model.block.layers to your block container.
|
||||
# Alternatives: model.transformer.layers, model.blocks, model.layers
|
||||
block_layers = model.block.layers # type: ignore[attr-defined]
|
||||
for i, layer in enumerate(block_layers):
|
||||
handles.append(layer.register_forward_hook(_hook(f"block[{i:02d}]")))
|
||||
if drill_layer is not None and i == drill_layer:
|
||||
tag = f"L{i:02d}"
|
||||
# TODO: adapt submodule names to your layer's attributes.
|
||||
# magi-human uses: attention, mlp.pre_norm, mlp.up_gate_proj,
|
||||
# mlp.down_proj (pre+post), mlp, attn_post_norm, mlp_post_norm.
|
||||
if hasattr(layer, "attention"):
|
||||
handles.append(layer.attention.register_forward_hook(_hook(f"{tag}.attention")))
|
||||
if hasattr(layer, "mlp"):
|
||||
mlp = layer.mlp
|
||||
if hasattr(mlp, "pre_norm"):
|
||||
handles.append(mlp.pre_norm.register_forward_hook(_hook(f"{tag}.mlp.pre_norm")))
|
||||
if hasattr(mlp, "up_gate_proj"):
|
||||
handles.append(mlp.up_gate_proj.register_forward_hook(_hook(f"{tag}.mlp.up_gate_proj")))
|
||||
if hasattr(mlp, "down_proj"):
|
||||
handles.append(mlp.down_proj.register_forward_pre_hook(_pre_hook(f"{tag}.mlp.down_proj")))
|
||||
handles.append(mlp.down_proj.register_forward_hook(_hook(f"{tag}.mlp.down_proj")))
|
||||
handles.append(mlp.register_forward_hook(_hook(f"{tag}.mlp")))
|
||||
if hasattr(layer, "attn_post_norm"):
|
||||
handles.append(layer.attn_post_norm.register_forward_hook(_hook(f"{tag}.attn_post_norm")))
|
||||
if hasattr(layer, "mlp_post_norm"):
|
||||
handles.append(layer.mlp_post_norm.register_forward_hook(_hook(f"{tag}.mlp_post_norm")))
|
||||
return handles
|
||||
|
||||
|
||||
def _apply_hypothesis_patch() -> bool:
|
||||
"""Apply an optional monkey-patch gated by HYPOTHESIS_ENV. TODO: implement.
|
||||
|
||||
Pattern: save original on the class, patch, restore in _restore_hypothesis_patch().
|
||||
"""
|
||||
if os.getenv(HYPOTHESIS_ENV) != "1":
|
||||
return False
|
||||
# TODO: import FastVideo class, save original, apply patch.
|
||||
print(f"[debug] Hypothesis patch {HYPOTHESIS_ENV}=1 applied.")
|
||||
return True
|
||||
|
||||
|
||||
def _restore_hypothesis_patch() -> None:
|
||||
if os.getenv(HYPOTHESIS_ENV) != "1":
|
||||
return
|
||||
# TODO: restore original, e.g.: _mod.TargetClass.method = _mod._ORIGINAL_METHOD
|
||||
|
||||
|
||||
def _write_log(entries: list[dict], path: Path) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(path, "w") as f:
|
||||
for e in entries:
|
||||
f.write(f"{e['name']} {e['shape']} "
|
||||
f"{e['abs_mean']:.8f} {e['sum']:.4f} "
|
||||
f"{e['min']:.6f} {e['max']:.6f}\n")
|
||||
|
||||
|
||||
def _sort_key(name: str, drill_layer: int) -> tuple:
|
||||
if name == "adapter":
|
||||
return (0, "")
|
||||
if name.startswith(f"L{drill_layer:02d}."):
|
||||
sub_order = {
|
||||
"attention": 0,
|
||||
"attn_post_norm": 1,
|
||||
"mlp.pre_norm": 2,
|
||||
"mlp.up_gate_proj": 3,
|
||||
"mlp.down_proj<in>": 4,
|
||||
"mlp.down_proj": 5,
|
||||
"mlp": 6,
|
||||
"mlp_post_norm": 7,
|
||||
}.get(name.split(".", 1)[1], 9)
|
||||
return (1, f"block[{drill_layer:02d}]", sub_order)
|
||||
if name.startswith("block["):
|
||||
return (1, name, 99)
|
||||
return (2, name, 0)
|
||||
|
||||
|
||||
def _print_table(by_name: dict[str, dict], drill_layer: int) -> int | None:
|
||||
hdr = (f"{'name':<18} {'up_shape':<22} {'up_absmean':>12} {'fv_absmean':>12} "
|
||||
f"{'absmean_diff':>14} {'rel%':>8} {'up_sum':>14} {'fv_sum':>14} {'sum_diff':>12}")
|
||||
print(f"\n{hdr}\n{'-' * len(hdr)}")
|
||||
first_div: int | None = None
|
||||
for name in sorted(by_name.keys(), key=lambda n: _sort_key(n, drill_layer)):
|
||||
d = by_name[name]
|
||||
up, fv = d.get("up"), d.get("fv")
|
||||
if up is None or fv is None:
|
||||
continue
|
||||
am_diff = abs(up["abs_mean"] - fv["abs_mean"])
|
||||
am_rel = am_diff / max(up["abs_mean"], 1e-9)
|
||||
sum_diff = abs(up["sum"] - fv["sum"])
|
||||
flag = ""
|
||||
if name.startswith("block[") and am_rel > REL_THRESHOLD:
|
||||
flag = " <<< DIVERGE"
|
||||
if first_div is None:
|
||||
first_div = int(name[len("block["):-1])
|
||||
print(f"{name:<18} {str(up['shape']):<22} {up['abs_mean']:>12.6f} "
|
||||
f"{fv['abs_mean']:>12.6f} {am_diff:>14.6f} {am_rel * 100:>7.3f}% "
|
||||
f"{up['sum']:>14.4f} {fv['sum']:>14.4f} {sum_diff:>12.4f}{flag}")
|
||||
return first_div
|
||||
|
||||
|
||||
def _print_elementwise(up_t: dict[str, torch.Tensor], fv_t: dict[str, torch.Tensor], drill_layer: int) -> None:
|
||||
common = set(up_t.keys()) & set(fv_t.keys())
|
||||
if not common:
|
||||
return
|
||||
hdr = f"{'name':<30} {'shape':<22} {'diff_max':>12} {'diff_mean':>12} {'diff_rel%':>10}"
|
||||
print(f"\nElement-wise diffs for drilled L{drill_layer:02d} submodules:\n{hdr}\n{'-' * len(hdr)}")
|
||||
for name in sorted(common):
|
||||
a, b = up_t[name], fv_t[name]
|
||||
if a.shape != b.shape:
|
||||
continue
|
||||
diff = (a - b).abs()
|
||||
rel = (diff.mean().item() / max(a.abs().mean().item(), 1e-9)) * 100
|
||||
print(f"{name:<30} {str(tuple(a.shape)):<22} "
|
||||
f"{diff.max().item():>12.6f} {diff.mean().item():>12.6f} {rel:>9.4f}%")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if not torch.cuda.is_available():
|
||||
print("Need CUDA. Skipping.")
|
||||
return
|
||||
|
||||
# TODO: add precondition checks (official clone present, weights available).
|
||||
|
||||
drill_layer = int(os.getenv(DRILL_LAYER_ENV, "0"))
|
||||
device = torch.device("cuda:0")
|
||||
patched = _apply_hypothesis_patch()
|
||||
try:
|
||||
inputs = build_inputs(device)
|
||||
|
||||
print("Loading official model...")
|
||||
official = load_official(device)
|
||||
up_log: list[dict] = []
|
||||
up_t: dict[str, torch.Tensor] = {}
|
||||
up_handles = _attach_block_hooks(official, "up", up_log, up_t, drill_layer)
|
||||
print("Running official forward (with hooks)...")
|
||||
try:
|
||||
with torch.inference_mode():
|
||||
# TODO: adapt forward call signature to your component.
|
||||
ref_out = official(**{k: v.clone() for k, v in inputs.items()})
|
||||
if isinstance(ref_out, dict):
|
||||
sample = ref_out.get("sample")
|
||||
ref_out = sample if sample is not None else ref_out.get("x")
|
||||
elif hasattr(ref_out, "sample"):
|
||||
ref_out = ref_out.sample
|
||||
elif isinstance(ref_out, tuple):
|
||||
ref_out = ref_out[0]
|
||||
assert torch.is_tensor(ref_out), f"official output is not tensor: {type(ref_out)}"
|
||||
ref_out = ref_out.detach().float().cpu()
|
||||
finally:
|
||||
for h in up_handles:
|
||||
h.remove()
|
||||
del official
|
||||
gc.collect()
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
print("Loading FastVideo model...")
|
||||
fv = load_fastvideo(device)
|
||||
fv_log: list[dict] = []
|
||||
fv_t: dict[str, torch.Tensor] = {}
|
||||
fv_handles = _attach_block_hooks(fv, "fv", fv_log, fv_t, drill_layer)
|
||||
print("Running FastVideo forward (with hooks)...")
|
||||
try:
|
||||
with torch.inference_mode():
|
||||
# TODO: adapt forward call signature to your component.
|
||||
fv_out = fv(**{k: v.clone() for k, v in inputs.items()})
|
||||
if isinstance(fv_out, dict):
|
||||
sample = fv_out.get("sample")
|
||||
fv_out = sample if sample is not None else fv_out.get("x")
|
||||
elif hasattr(fv_out, "sample"):
|
||||
fv_out = fv_out.sample
|
||||
elif isinstance(fv_out, tuple):
|
||||
fv_out = fv_out[0]
|
||||
assert torch.is_tensor(fv_out), f"FastVideo output is not tensor: {type(fv_out)}"
|
||||
fv_out = fv_out.detach().float().cpu()
|
||||
finally:
|
||||
for h in fv_handles:
|
||||
h.remove()
|
||||
finally:
|
||||
_restore_hypothesis_patch()
|
||||
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
up_path = LOG_DIR / f"{FAMILY}_{COMPONENT}_up_layers.log"
|
||||
fv_path = LOG_DIR / f"{FAMILY}_{COMPONENT}_fv_layers.log"
|
||||
_write_log(up_log, up_path)
|
||||
_write_log(fv_log, fv_path)
|
||||
print(f"\nLogs: {up_path} {fv_path}\nDiff: diff {up_path} {fv_path}")
|
||||
|
||||
by_name: dict[str, dict] = {}
|
||||
for entry in up_log + fv_log:
|
||||
by_name.setdefault(entry["name"], {})[entry["side"]] = entry
|
||||
first_div = _print_table(by_name, drill_layer)
|
||||
|
||||
print()
|
||||
if first_div is not None:
|
||||
print(f"First block exceeding {REL_THRESHOLD * 100:.2f}% drift: block[{first_div:02d}]")
|
||||
print(f"Re-run with {DRILL_LAYER_ENV}={first_div} to drill submodules.")
|
||||
else:
|
||||
print(f"No block exceeded {REL_THRESHOLD * 100:.2f}% -- divergence is amortized or pre-block.")
|
||||
|
||||
diff = (ref_out - fv_out).abs()
|
||||
print(f"\nFinal ref_abs={ref_out.abs().mean():.6f} fv_abs={fv_out.abs().mean():.6f} "
|
||||
f"diff_max={diff.max():.6f} diff_mean={diff.mean():.6f}")
|
||||
_print_elementwise(up_t, fv_t, drill_layer)
|
||||
if patched:
|
||||
print(f"\n[debug] Hypothesis {HYPOTHESIS_ENV}=1 was active this run.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,255 +0,0 @@
|
||||
---
|
||||
name: add-model-09-pipeline
|
||||
description: Use during /add-model Phase 7 after all required component parity tests pass to define FastVideo pipeline wiring, configs, presets, registry entries, examples, smoke tests, and pipeline parity tests.
|
||||
---
|
||||
|
||||
# Add Model Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Implement and verify the end-to-end FastVideo pipeline after the native
|
||||
components and converted weights have passed non-skip component parity. This
|
||||
skill owns pipeline class/stage wiring, pipeline configs, presets, registry
|
||||
entries, examples, smoke tests, and pipeline parity-debug.
|
||||
|
||||
FastVideo has one pipeline architecture: stage-based composition through
|
||||
`ComposedPipelineBase`. Add or specialize stages only when existing stages cannot
|
||||
represent the official behavior safely.
|
||||
|
||||
## Hard Gate
|
||||
|
||||
Do not start pipeline work until every required component, including reused
|
||||
components, has a non-skip local parity PASS.
|
||||
|
||||
If any component row is missing, skipped, red, or blocked, return to `/add-model`
|
||||
Phase 6. Pipeline parity cannot distinguish stage wiring mistakes from broken
|
||||
component numerics when component parity is still unresolved.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
|
||||
escape hatches, production boundaries, and skip/pass semantics.
|
||||
|
||||
Require a complete packet matching
|
||||
`../add-model/contracts/pipeline_context.md`.
|
||||
|
||||
The packet must include:
|
||||
|
||||
- official pipeline files and official call/default sources;
|
||||
- workload types, input/output modalities, and output contract;
|
||||
- converted or source `model_index.json` path;
|
||||
- component parity rows, all `non_skip_pass`;
|
||||
- target FastVideo pipeline/config/preset/registry/example/test paths;
|
||||
- `local_tests_readme` and `port_state_file` paths.
|
||||
|
||||
## Outputs
|
||||
|
||||
- Pipeline package under `fastvideo/pipelines/basic/<family>/`.
|
||||
- Pipeline config under `fastvideo/configs/pipelines/<family>.py` or a documented
|
||||
family-local config file when that matches existing project style.
|
||||
- Presets under `fastvideo/pipelines/basic/<family>/presets.py`.
|
||||
- Registry updates in `fastvideo/registry.py`.
|
||||
- Basic example under `examples/inference/basic/basic_<family>*.py`.
|
||||
- Local smoke and parity tests under `tests/local_tests/pipelines/`.
|
||||
- Updated `tests/local_tests/<model_family>/README.md`.
|
||||
- Updated `tests/local_tests/<model_family>/PORT_STATUS.md`.
|
||||
- Handoff matching `../add-model/contracts/pipeline_handoff.md`.
|
||||
|
||||
## Mode: Pipeline Definition
|
||||
|
||||
Use this mode first.
|
||||
|
||||
1. Read the official pipeline call path before editing FastVideo code.
|
||||
2. Compare official defaults against the planned FastVideo config and presets:
|
||||
steps, CFG scales, secondary CFG, flow shift, schedulers, sigmas, seed/RNG,
|
||||
resolution, frames, FPS, duration, VAE scaling, decode slicing, negative
|
||||
prompt defaults, and output heads.
|
||||
3. Create or update the pipeline class with `_required_config_modules` matching
|
||||
the emitted `model_index.json` and `ComposedPipelineBase.load_modules`.
|
||||
Runtime pipeline resolution is exact: `model_index.json["_class_name"]` must
|
||||
match a registered `EntryClass.__name__`, or a wrapper/alias class in
|
||||
`EntryClass`. Registry detectors do not select the executable pipeline class.
|
||||
4. Add new public generation kwargs to `fastvideo/api/sampling_param.py` before
|
||||
examples or presets use them. `SamplingParam.update()` ignores unknown keys
|
||||
except for logging, and preset defaults apply only to declared fields. Add CLI
|
||||
args when the option should be available from command-line entrypoints.
|
||||
5. Put loader-time changes in `load_modules()` or earlier, not
|
||||
`initialize_pipeline()`. `ComposedPipelineBase.__init__` loads modules before
|
||||
`post_init()` calls `initialize_pipeline()`, so process-global flags, loader
|
||||
path rewrites, dtype overrides, and tokenizer path changes needed for loading
|
||||
cannot be introduced there.
|
||||
6. Use `self.get_module("transformer_2", None)` and similar optional accessors
|
||||
for truly optional modules. Do not hard-require optional modules by accident.
|
||||
7. Avoid mutating class-level `_required_config_modules` in custom code. If a
|
||||
pipeline needs dynamic modules, copy the list to an instance-owned value or
|
||||
pass `required_config_modules` explicitly so one pipeline instance cannot leak
|
||||
module requirements into another.
|
||||
8. Create the stage chain in official execution order. Prefer existing shared
|
||||
stages for standard text encoding, timestep preparation, latent preparation,
|
||||
denoising, and decoding.
|
||||
9. Add model-specific stages only for family-specific behavior that does not fit
|
||||
the shared stage contracts.
|
||||
10. Add pipeline config classes for wiring and runtime defaults. Do not duplicate
|
||||
component architecture fields unless a loader requires them in the subconfig.
|
||||
Family-local config files such as
|
||||
`fastvideo/pipelines/basic/<family>/pipeline_configs.py` are valid only when
|
||||
`fastvideo/registry.py` imports and registers the classes explicitly.
|
||||
11. Add `InferencePreset` objects with `model_family`, `name`, `version`,
|
||||
`defaults`, optional validation-only `stage_schemas`, and an `ALL_PRESETS`
|
||||
tuple. `stage_schemas` validates user-facing `stage_overrides` names; it does
|
||||
not drive `create_pipeline_stages()` execution.
|
||||
12. Register config classes and presets in `fastvideo/registry.py`: add
|
||||
`register_configs(...)`, import the family's `ALL_PRESETS`, and append it to
|
||||
`_register_presets()`. Detectors should cover HF paths and `_class_name`
|
||||
strings for config/preset lookup, but not as a replacement for exact pipeline
|
||||
class-name resolution.
|
||||
13. Add a basic example with a user-story docstring and normal file-path inputs
|
||||
for image, audio, or video references. Keep orchestration glue in the
|
||||
pipeline or a helper, not in the example.
|
||||
14. Add a separate smoke test
|
||||
`tests/local_tests/pipelines/test_<family>_pipeline_smoke.py` that proves
|
||||
imports, `EntryClass`, registry, presets, config defaults, and at least one
|
||||
real load/generate path when weights are local. Older local tests sometimes
|
||||
colocate smoke checks in parity files; new ports should use the separate file
|
||||
convention.
|
||||
15. Add or update pipeline parity test scaffolding with
|
||||
`templates/pipeline_parity_test.py`.
|
||||
16. Update `local_tests_readme` and `port_state_file` with commands, statuses,
|
||||
default sources, decisions, and blockers.
|
||||
|
||||
Production import boundaries are defined in
|
||||
`../add-model/shared/common_rules.md`.
|
||||
|
||||
## Mode: Pipeline Parity Debug
|
||||
|
||||
Run after pipeline definition and after smoke can execute far enough to load the
|
||||
pipeline. Loop until pipeline parity is a non-skip PASS or a precise blocker is
|
||||
returned.
|
||||
|
||||
Mandatory order:
|
||||
|
||||
```bash
|
||||
pytest tests/local_tests/pipelines/test_<family>_pipeline_smoke.py -v -s
|
||||
DISABLE_SP=1 pytest tests/local_tests/pipelines/test_<family>_pipeline_parity.py -v -s
|
||||
python examples/inference/basic/basic_<family>.py
|
||||
```
|
||||
|
||||
Pipeline parity must compare real outputs, not only successful generation:
|
||||
|
||||
- denoised latents when decode parity is expensive or nondeterministic;
|
||||
- decoded videos/images when visual output should be deterministic enough;
|
||||
- decoded waveform or audio features for audio pipelines;
|
||||
- separate video and audio targets for joint AV pipelines unless a validated
|
||||
joint metric exists.
|
||||
|
||||
Debug pipeline drift in this order:
|
||||
|
||||
1. Confirm both sides use the same component weights and component parity PASS
|
||||
results are still valid.
|
||||
2. Align official and FastVideo call arguments, presets, and default values.
|
||||
3. Align scheduler timesteps, sigmas/noise levels, prediction type, flow shift,
|
||||
guidance math, and secondary-guidance branches.
|
||||
4. Align RNG: initial latents/noise, generator device, seed, per-step noise, VAE
|
||||
sampling, and any official `+1 frame` or crop/slice behavior.
|
||||
5. Align conditioning: prompt templates, negative prompts, masks, image/audio
|
||||
preprocessing, modality packing, text truncation, and dtype/autocast.
|
||||
6. Align decode: latent scaling, per-channel mean/std, tiling flags, output
|
||||
channel order, sample rate, FPS, and final slicing.
|
||||
7. Add targeted stage-level diagnostics to identify the first divergent stage.
|
||||
|
||||
If stage diagnostics show the first bad stage is transformer/denoising or a
|
||||
mid-DiT block, enable activation trace before adding ad hoc pipeline prints; see
|
||||
`docs/contributing/activation_trace.md` and `../add-model-08-trace/SKILL.md`.
|
||||
Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
|
||||
`FASTVIDEO_TRACE_STEPS` identical across reruns so pipeline parity traces diff
|
||||
one-to-one.
|
||||
|
||||
If the first divergence belongs to component implementation, strict loading, or
|
||||
conversion mapping, stop pipeline edits and return `next_step=return_to_phase_6`
|
||||
with the exact failing evidence. Do not patch conversion from this skill.
|
||||
|
||||
## Stage And Variant Rules
|
||||
|
||||
- Canonical video T2V order: `InputValidationStage`, `TextEncodingStage`,
|
||||
`ConditioningStage`, `TimestepPreparationStage`, `LatentPreparationStage`,
|
||||
`DenoisingStage`, `DecodingStage`.
|
||||
- Canonical video I2V delta adds image loading/encoding and image VAE encoding in
|
||||
the official order, commonly: `TextEncodingStage`, `ImageEncodingStage`,
|
||||
`ConditioningStage`, `TimestepPreparationStage`, `LatentPreparationStage`,
|
||||
`ImageVAEEncodingStage`, `DenoisingStage`, `DecodingStage`.
|
||||
- Treat `ConditioningStage` as default-present for Wan-style pipelines, but still
|
||||
follow the reference if another family truly skips or replaces it.
|
||||
- T2V video pipelines usually use validation, text encoding, conditioning,
|
||||
timestep preparation, latent preparation, denoising, and decoding.
|
||||
- I2V adds image loading/encoding and image-latent preparation according to the
|
||||
official pipeline, not by assuming CLIP or Wan-specific branches.
|
||||
- Pick image, audio, and video encoders from the reference. Do not assume CLIP or
|
||||
any other common encoder unless the reference uses it.
|
||||
- Cross-attention class names are not prescribed; match the family style and
|
||||
preserve the official tensor contract.
|
||||
- `WorkloadType` currently has no `T2A`, `A2A`, or `AV` values. Until that enum
|
||||
is extended, audio-only pipelines may register with `WorkloadType.T2V` and
|
||||
preset `workload_type="t2v"` as a compatibility shim, but must document the
|
||||
rationale in code and `PORT_STATUS.md`.
|
||||
- Audio-only pipelines should not force real video semantics into presets. Use
|
||||
minimal video-shaped placeholders such as small `height`/`width` and
|
||||
`num_frames=1` only when shared `VideoGenerator`/validation paths require them,
|
||||
and document that the real output is audio.
|
||||
- Record modality-specific shape knobs and output contract in the pipeline
|
||||
handoff: video uses `height`, `width`, `num_frames`, and `fps`; audio uses
|
||||
`audio_seconds` and `sampling_rate`; joint AV records both plus whether output
|
||||
is muxed or paired files.
|
||||
- Use sibling pipeline classes/configs when required modules, HF repo layout,
|
||||
stage chains, or inputs differ materially.
|
||||
- Use one kwargs-driven pipeline class only when variants share weights,
|
||||
modules, stage chain, and safe call semantics.
|
||||
- Split later if components diverge, workload tags require separate discovery,
|
||||
signatures become unsafe, or stage branches become substantial.
|
||||
- If the DiT branches on `added_kv_proj_dim`, document the T2V/I2V split.
|
||||
- If the reference uses `transformer_2`, `boundary_ratio`, `guidance_scale_2`, or
|
||||
DMD step lists, keep those on config, presets, or stages deliberately.
|
||||
- Support every official output head in scope. If a head is out of scope, record
|
||||
explicit user approval in `PORT_STATUS.md`.
|
||||
|
||||
Pipeline verification order:
|
||||
|
||||
```bash
|
||||
pytest tests/local_tests/pipelines/test_<family>_pipeline_smoke.py -v -s
|
||||
DISABLE_SP=1 pytest -v -s tests/local_tests/pipelines/test_<family>_pipeline_parity.py
|
||||
python examples/inference/basic/basic_<family>.py
|
||||
```
|
||||
|
||||
Smoke tests prove loadability only. They are not a substitute for numerical
|
||||
component or pipeline parity.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md`. Pipeline-specific ask cases include
|
||||
dropping a public mode, modality, or output head; adding a new workload enum;
|
||||
changing official defaults for user-facing behavior; accepting a known pipeline
|
||||
parity blocker; running GPU-heavy quality work outside the agreed scope; or
|
||||
publishing/uploading generated references or converted weights.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/pipeline_handoff.md` and update the shared state
|
||||
files before handoff.
|
||||
|
||||
Do not hand back a green pipeline if smoke or parity skipped locally. A skip is a
|
||||
setup gap, not a pass.
|
||||
|
||||
## References
|
||||
|
||||
- `fastvideo/pipelines/composed_pipeline_base.py` for module loading and stage
|
||||
execution.
|
||||
- `fastvideo/pipelines/basic/wan/` for standard video T2V/I2V/DMD variants.
|
||||
- `fastvideo/pipelines/basic/stable_audio/` for audio-specific stage composition.
|
||||
- `fastvideo/configs/pipelines/stable_audio.py` and
|
||||
`fastvideo/pipelines/basic/stable_audio/presets.py` for config/preset shape.
|
||||
- `fastvideo/registry.py` for `register_configs(...)` and preset registration.
|
||||
- `tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py` for latent
|
||||
parity structure.
|
||||
- `tests/local_tests/pipelines/test_stable_audio_pipeline_parity.py` for audio
|
||||
parity structure.
|
||||
- `tests/local_tests/pipelines/test_stable_audio_pipeline_smoke.py` for no-GPU
|
||||
import/registry/preset preflight shape.
|
||||
@@ -1,147 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Pipeline parity scaffold for TODO_MODEL_FAMILY.
|
||||
|
||||
Copy this file to
|
||||
`tests/local_tests/pipelines/test_<family>_pipeline_parity.py` and replace every
|
||||
TODO before treating it as an executable scaffold.
|
||||
|
||||
The filled test should compare denoised latents, decoded media, audio waveform,
|
||||
or another concrete output from the official pipeline against FastVideo. A
|
||||
successful generation without tensor/media comparison is not parity.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from torch.testing import assert_close
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
_MODEL_FAMILY = "TODO_MODEL_FAMILY"
|
||||
_OFFICIAL_REF_ENV = "TODO_OFFICIAL_REF_PATH"
|
||||
_OFFICIAL_REF_DEFAULT = _REPO_ROOT / "TODO_OFFICIAL_REF_DIR"
|
||||
_FASTVIDEO_MODEL_ENV = "TODO_FASTVIDEO_MODEL_PATH"
|
||||
_FASTVIDEO_MODEL_DEFAULT = _REPO_ROOT / "converted_weights" / _MODEL_FAMILY
|
||||
|
||||
|
||||
def _path_from_env(env_name: str, default: Path) -> Path:
|
||||
return Path(os.getenv(env_name, str(default))).expanduser()
|
||||
|
||||
|
||||
def _add_official_to_path() -> Path:
|
||||
official_path = _path_from_env(_OFFICIAL_REF_ENV, _OFFICIAL_REF_DEFAULT)
|
||||
if not official_path.exists():
|
||||
pytest.skip(f"Official reference not found at {official_path}")
|
||||
if str(official_path) not in sys.path:
|
||||
sys.path.insert(0, str(official_path))
|
||||
return official_path
|
||||
|
||||
|
||||
def _log_tensor_stats(label: str, tensor: torch.Tensor) -> None:
|
||||
value = tensor.detach().float()
|
||||
print(f"[{_MODEL_FAMILY} PIPELINE] {label}: shape={tuple(tensor.shape)} "
|
||||
f"dtype={tensor.dtype} device={tensor.device} "
|
||||
f"min={value.min().item():.6f} max={value.max().item():.6f} "
|
||||
f"mean={value.mean().item():.6f} std={value.std().item():.6f}")
|
||||
|
||||
|
||||
def _extract_tensor(output: Any, key: str) -> torch.Tensor:
|
||||
if isinstance(output, dict):
|
||||
value = output.get(key)
|
||||
else:
|
||||
value = getattr(output, key, None)
|
||||
if value is None:
|
||||
raise AssertionError(f"Pipeline output did not contain {key!r}")
|
||||
if not torch.is_tensor(value):
|
||||
try:
|
||||
import numpy as np
|
||||
value = torch.from_numpy(np.asarray(value))
|
||||
except Exception as exc: # pragma: no cover - scaffold guard
|
||||
raise AssertionError(f"Could not convert {key!r} to tensor") from exc
|
||||
return value.detach().float().cpu()
|
||||
|
||||
|
||||
def _run_official_pipeline(
|
||||
official_path: Path,
|
||||
params: dict[str, Any],
|
||||
device: torch.device,
|
||||
) -> Any:
|
||||
del official_path, params, device
|
||||
pytest.skip("TODO: import the official pipeline/factory, load official weights, "
|
||||
"run with params, and return the comparison target.")
|
||||
|
||||
|
||||
def _run_fastvideo_pipeline(model_path: Path, params: dict[str, Any]) -> Any:
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
str(model_path),
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False,
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=False,
|
||||
text_encoder_cpu_offload=False,
|
||||
)
|
||||
try:
|
||||
return generator.generate_video(
|
||||
prompt=params["prompt"],
|
||||
negative_prompt=params.get("negative_prompt"),
|
||||
output_path=f"outputs_{_MODEL_FAMILY}/pipeline_parity",
|
||||
save_video=False,
|
||||
height=params.get("height"),
|
||||
width=params.get("width"),
|
||||
num_frames=params.get("num_frames"),
|
||||
fps=params.get("fps"),
|
||||
num_inference_steps=params["num_inference_steps"],
|
||||
guidance_scale=params.get("guidance_scale"),
|
||||
seed=params["seed"],
|
||||
)
|
||||
finally:
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not torch.cuda.is_available(),
|
||||
reason="TODO_MODEL_FAMILY pipeline parity requires CUDA.",
|
||||
)
|
||||
def test_todo_model_family_pipeline_official_parity() -> None:
|
||||
official_path = _add_official_to_path()
|
||||
fastvideo_model_path = _path_from_env(
|
||||
_FASTVIDEO_MODEL_ENV,
|
||||
_FASTVIDEO_MODEL_DEFAULT,
|
||||
)
|
||||
if not fastvideo_model_path.exists():
|
||||
pytest.skip(f"FastVideo model path not found at {fastvideo_model_path}")
|
||||
|
||||
device = torch.device("cuda:0")
|
||||
params = {
|
||||
"prompt": "TODO: stable parity prompt",
|
||||
"negative_prompt": "",
|
||||
"height": 64,
|
||||
"width": 64,
|
||||
"num_frames": 9,
|
||||
"fps": 8,
|
||||
"num_inference_steps": 4,
|
||||
"guidance_scale": 1.0,
|
||||
"seed": 0,
|
||||
}
|
||||
|
||||
official_output = _run_official_pipeline(official_path, params, device)
|
||||
fastvideo_output = _run_fastvideo_pipeline(fastvideo_model_path, params)
|
||||
|
||||
comparison_key = "TODO_COMPARISON_KEY"
|
||||
official_tensor = _extract_tensor(official_output, comparison_key)
|
||||
fastvideo_tensor = _extract_tensor(fastvideo_output, comparison_key)
|
||||
|
||||
_log_tensor_stats("official", official_tensor)
|
||||
_log_tensor_stats("fastvideo", fastvideo_tensor)
|
||||
assert official_tensor.shape == fastvideo_tensor.shape
|
||||
|
||||
diff = (official_tensor - fastvideo_tensor).abs()
|
||||
print(f"diff max={diff.max().item():.6f} "
|
||||
f"mean={diff.mean().item():.6f} median={diff.median().item():.6f}")
|
||||
assert_close(fastvideo_tensor, official_tensor, atol=1e-2, rtol=1e-2)
|
||||
@@ -1,141 +0,0 @@
|
||||
---
|
||||
name: add-model-10-pr-review
|
||||
description: Review rubric for FastVideo PRs that add or modify model families, variants, first-class components, checkpoint conversion, pipelines, parity coverage, or generated-media quality baselines. Use when reviewing a PR whose diff touches fastvideo/models/, fastvideo/pipelines/basic/, fastvideo/registry.py, scripts/checkpoint_conversion/, fastvideo/tests/ssim/, or related model-port surfaces. Pairs with review-pr-link as a project-scoped review pass; produces findings, not fixes.
|
||||
---
|
||||
|
||||
# Add-Model PR Review
|
||||
|
||||
Use this skill when a reviewed PR appears to add, port, or substantially modify
|
||||
a FastVideo model family, model variant, first-class model component,
|
||||
checkpoint conversion, model pipeline, or local parity coverage.
|
||||
|
||||
This is a review skill, not an implementation workflow. Do not run `/add-model`
|
||||
or start writing missing port code during review. Use the add-model skill stack
|
||||
as a rubric for findings.
|
||||
|
||||
## Trigger Paths
|
||||
|
||||
Trigger this skill if `git diff --name-only <base>...HEAD` includes any of:
|
||||
|
||||
- `fastvideo/models/dits/`, `fastvideo/configs/models/dits/`
|
||||
- `fastvideo/models/vaes/`, `fastvideo/configs/models/vaes/`
|
||||
- `fastvideo/models/encoders/`, `fastvideo/configs/models/encoders/`
|
||||
- `fastvideo/models/schedulers/`, `fastvideo/configs/models/schedulers/`
|
||||
- `fastvideo/models/upsamplers/`, `fastvideo/configs/models/upsamplers/`
|
||||
- `fastvideo/models/audio/`, `fastvideo/configs/models/audio/`
|
||||
- `fastvideo/pipelines/basic/`, `fastvideo/configs/pipelines/`
|
||||
- `fastvideo/registry.py`, `fastvideo/api/sampling_param.py`
|
||||
- `scripts/checkpoint_conversion/`
|
||||
- `examples/inference/basic/`
|
||||
- `tests/local_tests/`, especially component or pipeline parity tests
|
||||
- `fastvideo/tests/ssim/` or other quality-regression tests for generated media
|
||||
|
||||
Also trigger when the PR title/body claims a new model, model variant, VAE,
|
||||
encoder, scheduler, conditioner, pipeline, conversion script, or generated-media
|
||||
quality baseline even if the path list is incomplete.
|
||||
|
||||
## Review Inputs
|
||||
|
||||
Read these add-model references as review checklists:
|
||||
|
||||
- `../add-model/SKILL.md`: phase gates and final handoff requirements.
|
||||
- `../add-model/shared/common_rules.md`: token/auth safety, production import
|
||||
boundaries, state files, and skip/pass semantics.
|
||||
- `../add-model/contracts/final_handoff.md`: final evidence expected from a
|
||||
complete port.
|
||||
- `../add-model/contracts/component_context.md` and
|
||||
`../add-model/contracts/component_skill_handoff.md`: component evidence and
|
||||
parity-debug expectations.
|
||||
- `../add-model/contracts/conversion_request.md` and
|
||||
`../add-model/contracts/conversion_handoff.md`: conversion evidence,
|
||||
strict-load status, config validation, and retry context.
|
||||
- `../add-model/contracts/pipeline_context.md` and
|
||||
`../add-model/contracts/pipeline_handoff.md`: pipeline class/stage/config/
|
||||
preset/registry/example evidence.
|
||||
|
||||
Then read only the satellite skill(s) that match touched areas:
|
||||
|
||||
- DiT/transformer changes: `../add-model-03-port-dit/SKILL.md`.
|
||||
- VAE changes: `../add-model-04-port-vae/SKILL.md`.
|
||||
- Encoder/conditioner changes: `../add-model-05-port-encoder/SKILL.md`.
|
||||
- Scheduler/upsampler/vocoder/other components:
|
||||
`../add-model-06-port-generic/SKILL.md`.
|
||||
- Component parity tests: `../add-model-02-parity/SKILL.md`.
|
||||
- Checkpoint conversion: `../add-model-07-conversion/SKILL.md`.
|
||||
- Pipeline/config/presets/registry/examples:
|
||||
`../add-model-09-pipeline/SKILL.md`.
|
||||
- Prep/state docs: `../add-model-01-prep/SKILL.md`.
|
||||
|
||||
## Required Review Lanes
|
||||
|
||||
For a full model-family or model-variant PR, cover all lanes. For a
|
||||
component-only PR, cover the component, conversion/parity as applicable, and the
|
||||
documented downstream consumer.
|
||||
|
||||
1. Scope and source-of-truth lane:
|
||||
Verify the PR clearly identifies the official reference, weights/revision,
|
||||
supported variants, modalities, output heads, and any approved scope cuts.
|
||||
|
||||
2. Component lane:
|
||||
Verify each required component is FastVideo-native or has a documented and
|
||||
accepted lazy-wrapper exception. Check bucket/config inheritance, `EntryClass`,
|
||||
state-dict surface, reused-component evidence, and output heads.
|
||||
|
||||
3. Conversion lane:
|
||||
Verify mappings are derived from prototype key/shape dumps, source layout is
|
||||
supported, skipped keys are intentional, emitted configs validate through
|
||||
production paths, component strict-load status is recorded, `model_index.json`
|
||||
library tokens match loaders, and revisions are pinned when converting from
|
||||
HF.
|
||||
|
||||
4. Component parity lane:
|
||||
Verify local parity tests exist for every required component, including reused
|
||||
components. Scaffolds may skip in CI, but the PR must provide local non-skip
|
||||
PASS evidence or an explicit accepted blocker.
|
||||
|
||||
5. Pipeline lane:
|
||||
Verify stage order, required modules, `_class_name` / `EntryClass.__name__`
|
||||
resolution, config defaults, presets, `SamplingParam` fields, registry
|
||||
registration, examples, smoke tests, and pipeline parity.
|
||||
|
||||
6. Quality and evidence lane:
|
||||
Verify media quality regression is added or explicitly deferred, examples run,
|
||||
generated outputs are non-corrupt, `tests/local_tests/<family>/README.md` and
|
||||
`PORT_STATUS.md` are current, and final blockers are surfaced in the review.
|
||||
|
||||
## Findings To Prioritize
|
||||
|
||||
Prioritize review findings in this order:
|
||||
|
||||
- Missing or skipped required component parity without accepted blocker.
|
||||
- Pipeline parity/smoke/example missing or skipped for a pipeline PR.
|
||||
- Conversion emits unloadable or unvalidated configs/weights.
|
||||
- Wrong `model_index.json` `_class_name`, component library token, or registry
|
||||
class resolution.
|
||||
- Runtime diffusers/transformers model-class imports for components that own
|
||||
weights or numerical behavior.
|
||||
- Dropped modalities, output heads, variants, or conditioning streams without
|
||||
explicit approval.
|
||||
- Reused FastVideo component lacks exact definition/instantiation proof or
|
||||
non-skip parity.
|
||||
- Public generation kwargs/preset defaults missing from `SamplingParam`.
|
||||
- Tests only check shapes, importability, or successful generation without
|
||||
numerical/media comparison.
|
||||
- Tokens, credentials, reference clones, staged weights, or generated bulk assets
|
||||
committed to the PR.
|
||||
|
||||
## Output Format
|
||||
|
||||
Write normal code-review findings first, ordered by severity. Include file and
|
||||
line references from the PR diff when possible.
|
||||
|
||||
Use this phrasing for missing add-model evidence:
|
||||
|
||||
```text
|
||||
This PR does not satisfy the add-model <component|conversion|pipeline|final>
|
||||
gate because <specific required evidence> is missing. The risk is <runtime load,
|
||||
numerical parity, dropped output, registry resolution, etc.>.
|
||||
```
|
||||
|
||||
Keep the summary short. Mention which lanes were reviewed and which could not be
|
||||
verified because assets, GPU time, or external credentials were unavailable.
|
||||
@@ -1,438 +0,0 @@
|
||||
---
|
||||
name: add-model
|
||||
description: Manual /add-model workflow for implementing a FastVideo model or first-class component port after add-model-01-prep has staged reference code and weights. Organizes the port into numbered phases with conversion rules, component policies, parity gates, and handoff checks.
|
||||
---
|
||||
|
||||
# Add Model
|
||||
|
||||
## Manual Invocation
|
||||
|
||||
This skill is for explicit `/add-model` use only. Do not auto-start it from a
|
||||
casual model-port mention. The setup-only workflow is
|
||||
`../add-model-01-prep/SKILL.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
Port a new FastVideo model family, model variant, or first-class reusable
|
||||
component so it can be loaded through FastVideo's native model, config, stage,
|
||||
registry, preset, and test infrastructure.
|
||||
|
||||
FastVideo has one pipeline architecture: stage-based composition via
|
||||
`ComposedPipelineBase`. Vary the stages and modules, not the architecture.
|
||||
|
||||
## Scope Shapes
|
||||
|
||||
Use this skill for either shape:
|
||||
|
||||
| Shape | Required output |
|
||||
|---|---|
|
||||
| Full model family or variant | Native components, conversion if needed, pipeline config/class, presets, registry, smoke test, local parity tests, example, quality regression. |
|
||||
| First-class component contribution | Native component class/config, bucket export, component parity test, and a documented downstream pipeline that will consume it. Skip pipeline/preset/registry rows only when the contribution is intentionally component-only. |
|
||||
|
||||
If upstream ships many variants, lock scope before coding. "Base model" means
|
||||
checkpoint variant, not a modality subset. If the base checkpoint produces
|
||||
audio, pose, depth, masks, or other output heads, either support those outputs
|
||||
or get explicit user agreement to drop them.
|
||||
|
||||
## Required Input
|
||||
|
||||
Start from an `add-model-01-prep` handoff, or equivalent fields matching
|
||||
`contracts/prep_handoff.md`.
|
||||
|
||||
Before Phase 0, read the shared rules and all relevant schemas:
|
||||
|
||||
- `shared/common_rules.md`
|
||||
- `contracts/prep_handoff.md`
|
||||
- `contracts/port_state.md`
|
||||
- `contracts/escape_hatch.md`
|
||||
- `contracts/component_context.md`
|
||||
- `contracts/parity_status.md`
|
||||
- `contracts/conversion_request.md`
|
||||
- `contracts/conversion_handoff.md`
|
||||
- `contracts/component_skill_handoff.md`
|
||||
- `contracts/pipeline_context.md`
|
||||
- `contracts/pipeline_handoff.md`
|
||||
- `contracts/final_handoff.md`
|
||||
|
||||
## Hard Rules
|
||||
|
||||
- Follow `shared/common_rules.md` for token/auth safety, state files, escape
|
||||
hatches, production import boundaries, and skip/pass semantics.
|
||||
- If the prep handoff is missing or ambiguous, stop and run
|
||||
`../add-model-01-prep/SKILL.md`.
|
||||
- If a needed component is not ported, do not ship the pipeline that needs it.
|
||||
- Wan is grandfathered for missing local parity; do not copy its missing-test
|
||||
precedent for new work.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `shared/common_rules.md` and `contracts/escape_hatch.md`. The main
|
||||
orchestrator should ask only when no phase skill can safely continue under the
|
||||
shared rules.
|
||||
|
||||
## Files Map
|
||||
|
||||
| Area | Paths |
|
||||
|---|---|
|
||||
| DiT | `fastvideo/models/dits/<family>.py`, `fastvideo/configs/models/dits/<family>.py`, bucket `__init__.py`. |
|
||||
| VAE | `fastvideo/models/vaes/<arch_or_family>.py`, `fastvideo/configs/models/vaes/<arch_or_family>.py`, bucket `__init__.py`. Name by shared arch when reusable (`oobleck.py`, `autoencoder_kl.py`), otherwise by family (`wanvae.py`). |
|
||||
| Encoder / conditioner / scheduler / upsampler | Native class/config in the matching `fastvideo/models/<bucket>/` and `fastvideo/configs/models/<bucket>/` bucket. |
|
||||
| Lazy loader wrapper | Optional `fastvideo/models/<bucket>/<family>_loader.py` or similar thin `nn.Module` wrapper when a component is fetched from an external HF repo and should be hidden from host-pipeline state-dict matching. |
|
||||
| Conversion | `scripts/checkpoint_conversion/<family>_to_diffusers.py` only when `needs_conversion=yes`. |
|
||||
| Pipeline | `fastvideo/pipelines/basic/<family>/<family>_pipeline.py` plus sibling files for variants whose components or required modules differ. |
|
||||
| Pipeline config | `fastvideo/configs/pipelines/<family>.py` or `fastvideo/pipelines/basic/<family>/pipeline_configs.py`. |
|
||||
| Stages | `fastvideo/pipelines/basic/<family>/stages/` only for model-specific stage subclasses. |
|
||||
| Presets / registry | `fastvideo/pipelines/basic/<family>/presets.py`, `fastvideo/registry.py`. |
|
||||
| Tests | Component parity under `tests/local_tests/<bucket>/`; pipeline smoke/parity under `tests/local_tests/pipelines/`; CI-backed quality tests under `fastvideo/tests/`. |
|
||||
| Example | `examples/inference/basic/basic_<family>*.py`, one per public mode/variant. |
|
||||
|
||||
## Phase 0: Scope And Handoff Gate
|
||||
|
||||
1. Validate every required handoff field.
|
||||
2. Resolve `needs_conversion=unknown` before component work:
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \
|
||||
"<hf-or-local-path>" \
|
||||
--json
|
||||
```
|
||||
|
||||
3. List first-PR scope across both axes:
|
||||
- Variant axis: base, distill, SR/refine, causal, DMD, I2V, V2V, etc.
|
||||
- Modality axis: video, image, audio, pose, depth, masks, text, etc.
|
||||
4. For component-only work, explicitly name the downstream full-pipeline PR or
|
||||
planned consumer.
|
||||
5. Confirm `official_env_status` is `imports_ok` or
|
||||
`private_deps_need_stubs`. If it is `blocked`, return to
|
||||
`../add-model-01-prep/SKILL.md` before parity scaffolding.
|
||||
6. Confirm `local_tests_readme` exists and records official setup, HF weights,
|
||||
dependency changes, and planned parity commands for reviewers.
|
||||
7. Confirm `port_state_file` exists, follows `contracts/port_state.md`, and has
|
||||
rows for open questions/issues found during prep.
|
||||
8. If there are multiple official implementations, choose the one whose
|
||||
architecture matches the published weights. A blessed library port can be a
|
||||
better parity reference than a highly configurable research repo; document
|
||||
the choice in tests.
|
||||
|
||||
## Phase 1: Reference And Architecture Study
|
||||
|
||||
Read the official pipeline call path before writing code.
|
||||
|
||||
Record:
|
||||
|
||||
- Required modules from `model_index.json` or equivalent: transformer, VAE,
|
||||
text encoders, tokenizers, scheduler, image encoders, audio VAE, vocoder,
|
||||
conditioners, upsamplers.
|
||||
- Input/output modalities and every dedicated DiT output head.
|
||||
- Text/image/audio encoding flow, latent shape, dtype, scaling, packing,
|
||||
scheduler/timestep math, guidance math, VAE normalization, and decode flow.
|
||||
- Whether the official code relies on private deps, custom ops, or special
|
||||
kernels that parity tests must stub.
|
||||
|
||||
Arch config rule:
|
||||
|
||||
- `ArchConfig` fields must match the emitted per-component config, especially
|
||||
`transformer/config.json`, one-to-one.
|
||||
- Pipeline knobs do not belong on the DiT arch config: inference steps, CFG
|
||||
scales, flow shift, FPS, VAE stride, text target length, data-proxy knobs,
|
||||
eval defaults, and sampling defaults go on `PipelineConfig`, presets, or
|
||||
stages.
|
||||
- If the HF repo is raw or has empty configs, synthesize
|
||||
`transformer/config.json` from the official Python model-config class, not
|
||||
from data/eval config classes.
|
||||
|
||||
## Phase 2: Early Parity Scaffolding
|
||||
|
||||
Create component parity tests before or alongside implementation. Use
|
||||
`../add-model-02-parity/SKILL.md` and its `templates/component_parity_test.py`.
|
||||
The official reference must import in the current FastVideo environment, or the
|
||||
prep handoff must identify private deps that will be stubbed locally for tests.
|
||||
Use `local_tests_readme` as the reviewer-facing source for setup commands and
|
||||
update its planned test table as parity scaffolds are added.
|
||||
|
||||
This phase is early by design:
|
||||
|
||||
- Official loading can be implemented from the reference study.
|
||||
- FastVideo loading can target planned standardized class/config/loader paths.
|
||||
- Tests may initially skip because the FastVideo class or converted weights do
|
||||
not exist yet.
|
||||
- The scaffold must still contain real official loading, deterministic inputs,
|
||||
output extraction, and concrete tensor comparisons. No unconditional skips,
|
||||
no shape-only tests.
|
||||
|
||||
Use subagents here: dispatch one parity-test subagent per required component,
|
||||
including components that may be reused. Their output becomes the red/skip
|
||||
target that porting or reuse-verification subagents make pass later.
|
||||
|
||||
## Phase 3: Reuse Gate And Component Dispatch
|
||||
|
||||
Build a component inventory before implementation:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| Component | transformer, VAE, text encoder, image encoder, scheduler, conditioner, upsampler, vocoder, etc. |
|
||||
| Official definition | Repo-relative source file, class/function name, and relevant line/range if known. |
|
||||
| Official instantiation | Repo-relative pipeline/config/factory call site plus constructor args and runtime flags. |
|
||||
| FastVideo target | Existing class to reuse or new bucket/file/config to add. |
|
||||
| Parity test | Required local test path, including reused components. |
|
||||
| Status | `reuse_pending`, `reuse_proven`, `port_pending`, `non_skip_pass`, or `blocked`. |
|
||||
|
||||
Reuse is allowed only from the checked-out FastVideo tree. Do not wait for or
|
||||
depend on an open PR adding a native class; add the native port directly in this
|
||||
PR if the current tree cannot be reused.
|
||||
|
||||
Reuse decision:
|
||||
|
||||
1. Record exact official definition and instantiation evidence for every
|
||||
component.
|
||||
2. If an existing FastVideo class and config match both definition and
|
||||
instantiation, pass that reused target to the bucket-specific skill in
|
||||
`mode=prototype` and require reuse evidence plus key/shape dumps.
|
||||
3. If either definition or instantiation differs, port the component directly as
|
||||
FastVideo-native code through the bucket-specific skill.
|
||||
4. Reused components still require non-skip component parity against the exact
|
||||
official instantiation used by the target pipeline.
|
||||
|
||||
Porting subagent dispatch:
|
||||
|
||||
- Dispatch one subagent per component after Phase 2 parity scaffolds exist.
|
||||
- Use `../add-model-03-port-dit/SKILL.md` for DiTs/transformers.
|
||||
- Use `../add-model-04-port-vae/SKILL.md` for VAEs.
|
||||
- Use `../add-model-05-port-encoder/SKILL.md` for text, image, audio, or compound
|
||||
encoders/conditioners that fit the encoder config bucket.
|
||||
- Use `../add-model-06-port-generic/SKILL.md` for schedulers, upsamplers,
|
||||
vocoders, adapters, preprocessors, or unknown components.
|
||||
- Each subagent owns one component only and must loop on that component's local
|
||||
parity test until it produces a non-skip PASS or returns a precise blocker.
|
||||
|
||||
Every component subagent must receive a complete packet matching
|
||||
`contracts/component_context.md`. If any required path is unknown, pass `unknown`
|
||||
plus the exact search already performed. Do not silently omit ambiguous official
|
||||
files or prototype concerns.
|
||||
|
||||
Bucket, layer, and attention rules live in the bucket-specific skills and
|
||||
`fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Phase 4: Native Component Prototype
|
||||
|
||||
Conversion needs a FastVideo state-dict surface. Use the Phase 3
|
||||
bucket-specific skill in `mode=prototype` for every required component, including
|
||||
reused components.
|
||||
|
||||
Prototype success criteria:
|
||||
|
||||
- the FastVideo-native or reused class/config can import and instantiate with the
|
||||
exact official architecture args;
|
||||
- official and FastVideo key/shape dumps exist for every stateful component;
|
||||
- `local_tests_readme` and `port_state_file` record prototype status and concerns;
|
||||
- the returned handoff matches `contracts/component_skill_handoff.md`.
|
||||
|
||||
Do not chase numerical parity in Phase 4. Prototype mode ends when conversion has
|
||||
the key/shape surface it needs, or when the component skill returns a precise
|
||||
blocker or escape hatch.
|
||||
|
||||
## Phase 5: Param Mapping And Weight Conversion
|
||||
|
||||
Use `../add-model-07-conversion/SKILL.md` after Phase 4 prototypes exist.
|
||||
Send a request matching `contracts/conversion_request.md`; consume the returned
|
||||
`contracts/conversion_handoff.md` update before Phase 6.
|
||||
|
||||
Use the prep handoff's `needs_conversion` value:
|
||||
|
||||
- `no`: verify the source already has the component layout FastVideo loaders can
|
||||
consume, then record any passthrough components.
|
||||
- `yes`: write `scripts/checkpoint_conversion/<family>_to_diffusers.py` and
|
||||
output `converted_weights/<family>/`.
|
||||
- `unknown`: return to Phase 0.
|
||||
|
||||
The conversion skill owns source-layout handling, mapping derivation, config and
|
||||
`model_index.json` emission, passthrough assets, strict-load verification, and
|
||||
Phase 6 retry requests. Component skills must not patch conversion scripts or
|
||||
converted weights ad hoc.
|
||||
|
||||
## Phase 6: Component Parity Debug
|
||||
|
||||
This is the expected expensive loop. Dispatch one subagent per required
|
||||
component, including reused components, using the bucket-specific skill in
|
||||
`mode=parity-debug`.
|
||||
|
||||
Each subagent gets:
|
||||
|
||||
- the complete component context packet from Phase 3/4;
|
||||
- updated conversion mapping notes and strict-load result from Phase 5;
|
||||
- any prototype concerns or unknowns that were not resolved before conversion.
|
||||
|
||||
The bucket-specific skills own parity-debug tactics. If a failure belongs to
|
||||
conversion, route it through `../add-model-07-conversion/SKILL.md` with a retry
|
||||
request matching `contracts/conversion_request.md`, then resume the component
|
||||
skill with the updated conversion handoff.
|
||||
|
||||
When a component failure narrows to layer-by-layer numerical drift, load
|
||||
`../add-model-08-trace/SKILL.md` before writing custom hooks. It uses
|
||||
`fastvideo/hooks/activation_trace.py`; canonical env vars and JSONL format are
|
||||
documented in `docs/contributing/activation_trace.md`.
|
||||
|
||||
Phase 6 ends only when every required component handoff reports
|
||||
`parity_status=non_skip_pass`, or when a precise blocker or escape hatch is
|
||||
recorded in `port_state_file`.
|
||||
|
||||
## Phase 7: Pipeline, Stages, And Variants
|
||||
|
||||
Do not start Phase 7 until every required component, reused or ported, has a
|
||||
non-skip local parity PASS from Phase 6. If any component parity test is still
|
||||
`scaffold_skip`, `debug_red`, `blocked`, or missing, resume Phase 6 first.
|
||||
|
||||
Use `../add-model-09-pipeline/SKILL.md` for pipeline definition and parity-debug.
|
||||
Send a complete packet matching `contracts/pipeline_context.md`; consume the
|
||||
returned `contracts/pipeline_handoff.md` before moving to quality regression or
|
||||
final handoff.
|
||||
|
||||
The pipeline skill owns:
|
||||
|
||||
- pipeline class, stage chain, and optional model-specific stages;
|
||||
- pipeline config, presets, registry updates, and examples;
|
||||
- official args/defaults/presets comparison before setting FastVideo defaults;
|
||||
- pipeline smoke and parity tests;
|
||||
- continuous pipeline parity-debug until non-skip PASS or precise blocker;
|
||||
- updates to `local_tests_readme` and `port_state_file`.
|
||||
|
||||
The pipeline handoff must explicitly cover stage order, variants, modality and
|
||||
output-head handling, config/preset/registry/example status, smoke/parity tests,
|
||||
and any return-to-Phase-6 evidence.
|
||||
|
||||
## Phase 8: PipelineConfig, Presets, Registry, Examples
|
||||
|
||||
This phase is implemented through `../add-model-09-pipeline/SKILL.md` after the
|
||||
Phase 7 component-parity gate passes. Accept the pipeline handoff only if it
|
||||
covers configs, presets, registry detection/exact class resolution, examples,
|
||||
new `SamplingParam` fields for public kwargs/defaults, and local smoke/parity
|
||||
status. Detailed rules live in `../add-model-09-pipeline/SKILL.md`.
|
||||
|
||||
## Phase 9: Parity Activation And Local Verification
|
||||
|
||||
Local parity is author-run, not CI-enforced. CI may only run package-level
|
||||
quality tests later. Before handoff, Phase 2 scaffolds must be activated into
|
||||
non-skip PASS results.
|
||||
|
||||
Order is mandatory:
|
||||
|
||||
1. Run conversion if needed.
|
||||
2. Run component parity for every required component, including reused ones.
|
||||
3. Run pipeline smoke.
|
||||
4. Run pipeline parity.
|
||||
5. Run the basic example.
|
||||
|
||||
If pipeline smoke or parity points back to component implementation,
|
||||
strict-load, or conversion mapping, return to Phase 6 or Phase 5 rather than
|
||||
patching around the issue in the pipeline.
|
||||
|
||||
Skip policy:
|
||||
|
||||
- Follow `shared/common_rules.md`: a committed local test may skip for absent
|
||||
clones/weights, but a local skip is not a verified pass.
|
||||
|
||||
Use the commands and tolerance guidance from `../add-model-02-parity/SKILL.md` for
|
||||
component checks and from `../add-model-09-pipeline/SKILL.md` for pipeline smoke,
|
||||
pipeline parity, and examples. Record exact commands, status, and blockers in
|
||||
`local_tests_readme` and `port_state_file`.
|
||||
|
||||
## Phase 10: Quality Regression
|
||||
|
||||
Video outputs:
|
||||
|
||||
- Add `fastvideo/tests/ssim/test_<family>_similarity.py` when output video
|
||||
quality must be preserved.
|
||||
- Seed references through `seed-ssim-references` after the test exists.
|
||||
|
||||
Audio outputs:
|
||||
|
||||
- SSIM does not apply. Use an audio-specific regression metric such as
|
||||
mel-spectrogram L1, multi-resolution STFT, CLAP cosine, or a project-approved
|
||||
learned metric.
|
||||
- Document the metric and hardware/runtime assumptions in the test.
|
||||
|
||||
Joint AV outputs:
|
||||
|
||||
- Keep video and audio regression checks separate unless there is a validated
|
||||
joint metric.
|
||||
|
||||
## Phase 11: Post-Parity Review And Handoff
|
||||
|
||||
After parity is green, run a hot-path review before handoff:
|
||||
|
||||
- Hoist constant tensor allocations out of sampler/denoising loops.
|
||||
- Replace per-step `randn_like` churn with preallocated buffers plus
|
||||
`.normal_()` when safe.
|
||||
- Move `torch.backends.*` flag changes to one-shot setup/load paths.
|
||||
- Delete `batch.extra` writes that nothing reads.
|
||||
- Derive magic constants from configs when possible.
|
||||
|
||||
Pre-handoff checklist:
|
||||
|
||||
```text
|
||||
[ ] Prep handoff is complete and committed nowhere with token values.
|
||||
[ ] Conversion was run if needed and output loads with real weights.
|
||||
[ ] Every required component, reused or newly ported, has a non-skip local parity PASS.
|
||||
[ ] `local_tests_readme` lists every component parity test, command, status, and blocker if any.
|
||||
[ ] `port_state_file` has every open question/issue either resolved or listed as an explicit blocker.
|
||||
[ ] Any `next_step=ask_user` has a matching `escape_hatch` block and `E###` row.
|
||||
[ ] Pipeline smoke has a non-skip local PASS.
|
||||
[ ] Pipeline parity has a non-skip local PASS against the official reference.
|
||||
[ ] Basic example runs and writes a non-corrupt output.
|
||||
[ ] Video SSIM or audio-specific quality regression is added or explicitly deferred.
|
||||
[ ] Runtime production code has no diffusers/transformers model-class imports.
|
||||
[ ] Production comments are WHY-focused; examples have user-story docstrings.
|
||||
[ ] Post-parity hot-path pass is complete.
|
||||
```
|
||||
|
||||
Ask before deleting any reference clone or staged weights created by
|
||||
`add-model-01-prep`. Leave `.gitignore` entries so future parity assets stay
|
||||
untracked. Never commit the clone, weights, `.env`, credentials, or anything
|
||||
matching `*secret*`.
|
||||
|
||||
## References
|
||||
|
||||
- `../add-model-01-prep/SKILL.md` for user-input collection, HF inspection,
|
||||
weight staging, reference cloning, and setup handoff.
|
||||
- `contracts/` for canonical handoff schemas used by prep, parity, conversion,
|
||||
component porting, escape hatches, and final handoff.
|
||||
- `../add-model-02-parity/SKILL.md` for early component parity scaffolds and
|
||||
activation templates.
|
||||
- `../add-model-07-conversion/SKILL.md` for Phase 5 mapping, conversion scripts,
|
||||
monolithic checkpoint splitting, and strict-load checks.
|
||||
- `../add-model-03-port-dit/SKILL.md`, `../add-model-04-port-vae/SKILL.md`,
|
||||
`../add-model-05-port-encoder/SKILL.md`, and
|
||||
`../add-model-06-port-generic/SKILL.md` for component subagent implementation
|
||||
and parity-debug loops.
|
||||
- `../add-model-09-pipeline/SKILL.md` for pipeline definition, config/preset/
|
||||
registry/example wiring, smoke tests, and pipeline parity-debug.
|
||||
- `fastvideo/layers/AGENTS.md` for native layer selection and state-dict surface
|
||||
guidance.
|
||||
- `docs/contributing/coding_agents.md` for narrative context.
|
||||
- `docs/design/overview.md` for pipeline/config/registry architecture.
|
||||
- `fastvideo/pipelines/basic/wan/` for standard T2V/I2V/DMD/Causal variants.
|
||||
- `fastvideo/pipelines/basic/ltx2/` for non-standard stages and audio/video
|
||||
patterns.
|
||||
- `tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py` for pipeline
|
||||
parity shape.
|
||||
- `tests/local_tests/transformers/test_ltx2.py`,
|
||||
`tests/local_tests/vaes/test_ltx2_vae.py`, and
|
||||
`tests/local_tests/encoders/test_ltx2_gemma_parity.py` for component parity.
|
||||
- `scripts/checkpoint_conversion/convert_ltx2_weights.py` for modern conversion
|
||||
script shape.
|
||||
- `scripts/checkpoint_conversion/wan_to_diffusers.py` for legacy regex mapping
|
||||
reference only.
|
||||
|
||||
## Changelog
|
||||
|
||||
| Date | Change |
|
||||
|---|---|
|
||||
| 2026-04-24 | Initial FastVideo add-model workflow. |
|
||||
| 2026-04-30 | Split external setup into `add-model-01-prep`. |
|
||||
| 2026-04-30 | Rewrote as manual `/add-model` phase workflow and incorporated prior review decisions. |
|
||||
| 2026-04-30 | Extracted early parity scaffolding into `add-model-02-parity` and moved it before conversion/component implementation. |
|
||||
| 2026-04-30 | Added component reuse proof gate, bucket-specific porting skills, and parity PASS requirement for reused components. |
|
||||
| 2026-04-30 | Split prototype, conversion, and parity-debug phases; added conversion skill for monolithic and separate checkpoint layouts. |
|
||||
| 2026-04-30 | Extracted handoff schemas into `contracts/` for shared use across skills. |
|
||||
| 2026-04-30 | Added pipeline skill contract and Phase 7 component-parity gate. |
|
||||
| 2026-04-30 | Added escape-hatch contract for user decisions and `ask_user` handoffs. |
|
||||
@@ -1,29 +0,0 @@
|
||||
# Add Model Contracts
|
||||
|
||||
Canonical handoff schemas for the `/add-model` workflow. When a skill needs to
|
||||
send or receive structured context, use these files instead of inventing a local
|
||||
schema.
|
||||
|
||||
| Contract | Use |
|
||||
|---|---|
|
||||
| `prep_handoff.md` | `add-model-01-prep` output and `/add-model` Phase 0 input. |
|
||||
| `port_state.md` | Per-port `PORT_STATUS.md` file tracking progress, open questions, and issues. |
|
||||
| `escape_hatch.md` | Shared pause-and-ask schema for user decisions the workflow cannot safely choose. |
|
||||
| `component_context.md` | Per-component packet passed to parity, prototype, conversion, and parity-debug subagents. |
|
||||
| `parity_status.md` | `add-model-02-parity` scaffold/activation status returned to `/add-model`. |
|
||||
| `conversion_request.md` | Phase 5 conversion input and Phase 6 conversion retry request. |
|
||||
| `conversion_handoff.md` | `add-model-07-conversion` output back to `/add-model` and component subagents. |
|
||||
| `component_skill_handoff.md` | Component porting skill output in prototype or parity-debug mode. |
|
||||
| `pipeline_context.md` | Phase 7 packet passed to `add-model-09-pipeline` after component parity is green. |
|
||||
| `pipeline_handoff.md` | `add-model-09-pipeline` output back to `/add-model` after pipeline definition or parity-debug. |
|
||||
| `final_handoff.md` | Final `/add-model` pre-handoff checklist summary. |
|
||||
|
||||
Rules:
|
||||
|
||||
- Do not omit required fields. Use `unknown` plus the search already performed
|
||||
when the value is not known yet.
|
||||
- Do not include raw token values. Use env var names only.
|
||||
- Keep model-specific mapping details in conversion scripts and the local tests
|
||||
README/status notes, not in generic skill docs.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block matching
|
||||
`escape_hatch.md`.
|
||||
@@ -1,55 +0,0 @@
|
||||
# Component Context Contract
|
||||
|
||||
Canonical per-component packet passed from `/add-model` to parity, prototype,
|
||||
conversion, and parity-debug subagents.
|
||||
|
||||
```text
|
||||
component_context:
|
||||
model_family: <snake_case>
|
||||
component: <name>
|
||||
component_type: <dit|vae|encoder|scheduler|conditioner|upsampler|vocoder|generic>
|
||||
mode: parity-scaffold | prototype | parity-debug
|
||||
official_ref_dir: <path or import path>
|
||||
official_definition_files:
|
||||
- path: <repo-relative or absolute path in official repo>
|
||||
symbols: <class/function names>
|
||||
notes: <layer graph, output contract, state-dict owner>
|
||||
official_instantiation_files:
|
||||
- path: <repo-relative or absolute path in official repo>
|
||||
symbols: <factory/pipeline/config names>
|
||||
args: <constructor args, config values, runtime flags>
|
||||
official_weight_source: <checkpoint file, subfolder, prefix, or passthrough source>
|
||||
fastvideo_target_files:
|
||||
- fastvideo/models/<bucket>/<file>.py
|
||||
- fastvideo/configs/models/<bucket>/<file>.py
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
parity_test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py
|
||||
prototype_key_dumps:
|
||||
official: converted_weights/<family>/_mapping/<component>_official_keys.json | planned | unknown
|
||||
fastvideo: converted_weights/<family>/_mapping/<component>_fastvideo_keys.json | planned | unknown
|
||||
conversion:
|
||||
script: scripts/checkpoint_conversion/<family>_to_diffusers.py | not_created | not_needed | unknown
|
||||
converted_component_dir: converted_weights/<family>/<component> | not_created | not_needed | unknown
|
||||
model_index_library: <diffusers|transformers|fastvideo|fastvideo.*|unknown|none>
|
||||
config_file: <config.json|scheduler_config.json|none|unknown>
|
||||
mapping_notes: <key prefixes, split/fuse concerns, skipped keys, not_created, not_needed, or unknown>
|
||||
production_loader_strictness: <strict|non_strict_with_allowed_keys|stateless|unknown>
|
||||
strict_load: <not_run | pass | pass_with_documented_exclusions | blocked>
|
||||
concerns_or_unknowns:
|
||||
- <prototype mismatch, ambiguous arg, missing op, dtype concern, output head, etc.>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- If any required path is unknown, pass `unknown` plus the exact search already
|
||||
performed.
|
||||
- Do not silently omit ambiguous official files, instantiation args, or prototype
|
||||
concerns.
|
||||
- For reused components, still fill every field and set `fastvideo_target_files`
|
||||
to the reused class/config.
|
||||
- In `mode=parity-scaffold`, prototype and conversion fields may be `planned`,
|
||||
`not_created`, `not_needed`, or `unknown`; do not invent paths or statuses that
|
||||
do not exist yet.
|
||||
- Update `port_state_file` when concerns, issues, conversion status, or parity
|
||||
status change.
|
||||
@@ -1,41 +0,0 @@
|
||||
# Component Skill Handoff Contract
|
||||
|
||||
Returned by `add-model-03-port-dit`, `add-model-04-port-vae`,
|
||||
`add-model-05-port-encoder`, and `add-model-06-port-generic`.
|
||||
|
||||
```text
|
||||
component: <name>
|
||||
mode: prototype | parity-debug
|
||||
files_changed: <model/config/export/test/readme paths>
|
||||
official_files_used: <definition files, instantiation files>
|
||||
prototype_key_dumps: <official path, fastvideo path, or none>
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
concerns_or_unknowns: <remaining or newly discovered concerns>
|
||||
parity_test: <path>
|
||||
parity_status: scaffold_skip | debug_red | non_skip_pass | blocked
|
||||
production_loader_strictness: strict | non_strict_with_allowed_keys | stateless
|
||||
strict_load: pass | pass_with_documented_exclusions | blocked | not_run
|
||||
pytest_output: <command + short result>
|
||||
blocker: <none or exact missing dependency/weights/numeric mismatch>
|
||||
conversion_retry_request: <none or failing keys/shapes/prefixes/evidence for add-model-07-conversion>
|
||||
readme_updated: yes | no
|
||||
next_step: phase_5_conversion | phase_5_conversion_retry | phase_6_continue | ask_user | blocked
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- In `mode=prototype`, parity may be `scaffold_skip` or `blocked`; key dumps are
|
||||
the required artifact. Successful prototype handoff should use
|
||||
`next_step=phase_5_conversion`.
|
||||
- In `mode=parity-debug`, final success requires `parity_status=non_skip_pass`.
|
||||
- If conversion is implicated, return `conversion_retry_request` and do not edit
|
||||
conversion scripts or converted weights directly.
|
||||
- If production loading is non-strict, list allowed missing/unexpected keys in
|
||||
the parity test or handoff and mark `strict_load=pass_with_documented_exclusions`.
|
||||
- Update `port_state_file` before returning: component row, open questions,
|
||||
issues/blockers, decisions, and handoff notes.
|
||||
- Return an `escape_hatch` only for user decisions, not for normal component
|
||||
implementation or parity-debug failures.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
|
||||
`PORT_STATUS.md` row.
|
||||
@@ -1,41 +0,0 @@
|
||||
# Conversion Handoff Contract
|
||||
|
||||
Returned by `../add-model-07-conversion/SKILL.md` to `/add-model` and component
|
||||
parity-debug subagents.
|
||||
|
||||
```text
|
||||
conversion_script: scripts/checkpoint_conversion/<family>_to_diffusers.py
|
||||
source_layout: <diffusers|raw_official|separate_components|monolithic|mixed|custom>
|
||||
converted_weights_dir: converted_weights/<model_family>
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
components_written: <list>
|
||||
passthrough_components: <list>
|
||||
strict_load: pass | pass_with_documented_exclusions | blocked
|
||||
component_context_updates:
|
||||
- component: <name>
|
||||
converted_component_dir: <path>
|
||||
model_index_library: <diffusers|transformers|fastvideo|fastvideo.*>
|
||||
config_file: <path or none>
|
||||
config_validation: pass | blocked | not_applicable
|
||||
mapping_notes: <prefixes, split/fuse ops, skipped keys>
|
||||
production_loader_strictness: strict | non_strict_with_allowed_keys | stateless
|
||||
strict_load: pass | pass_with_documented_exclusions | blocked | not_run
|
||||
retry_resolved: <yes | no | not_a_retry>
|
||||
concerns_or_unknowns: <remaining list>
|
||||
blocked_on: <none or exact blocker>
|
||||
next_step: phase_6_component_parity_debug | ask_user
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Include strict-load evidence for every stateful converted component.
|
||||
- If a component intentionally loads non-strictly, list the exact missing or
|
||||
unexpected keys and why they are safe.
|
||||
- Include the actual `model_index.json` library token, config filename, and config
|
||||
validation result for every emitted component.
|
||||
- Preserve retry evidence so the requesting component subagent can resume with
|
||||
updated context.
|
||||
- Keep `port_state_file` synchronized with `component_context_updates`.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
|
||||
`PORT_STATUS.md` row.
|
||||
@@ -1,54 +0,0 @@
|
||||
# Conversion Request Contract
|
||||
|
||||
Consumed by `../add-model-07-conversion/SKILL.md` in Phase 5 and during Phase 6
|
||||
conversion retries.
|
||||
|
||||
Initial conversion request:
|
||||
|
||||
```text
|
||||
model_family: <snake_case>
|
||||
source_layout: diffusers | raw_official | monolithic | separate_components | mixed | custom
|
||||
official_weights: <HF repo, local dir, or checkpoint file>
|
||||
hf_revision: <revision | default | none>
|
||||
converted_weights_dir: converted_weights/<model_family>
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
components:
|
||||
- name: <transformer|vae|text_encoder|conditioner|scheduler|...>
|
||||
component_type: <dit|vae|encoder|scheduler|conditioner|upsampler|vocoder|generic>
|
||||
official_definition_files: <paths + symbols>
|
||||
official_instantiation_files: <paths + call sites + args>
|
||||
official_weight_source: <checkpoint file, prefix, subfolder, or passthrough source>
|
||||
official_keys: <path to official key/shape dump>
|
||||
fastvideo_keys: <path to FastVideo prototype key/shape dump>
|
||||
fastvideo_class: <class name>
|
||||
model_index_library: <diffusers|transformers|fastvideo|fastvideo.*>
|
||||
config_filename: <config.json|scheduler_config.json|none>
|
||||
production_loader_strictness: <strict|non_strict_with_allowed_keys|stateless>
|
||||
source_prefix_or_path: <prefix or path>
|
||||
parity_test: <component parity test path>
|
||||
prototype_concerns_or_unknowns: <short list>
|
||||
```
|
||||
|
||||
Retry request from a component skill:
|
||||
|
||||
```text
|
||||
conversion_retry_request:
|
||||
component: <name>
|
||||
parity_test: <path>
|
||||
failing_keys: <official and FastVideo keys, if known>
|
||||
expected_actual_shapes: <expected vs actual shapes, if known>
|
||||
source_prefix_or_path: <prefix/path implicated by the failure>
|
||||
evidence: <strict-load error, first divergent tensor, parity log excerpt>
|
||||
suspected_fix: <rename | split | fuse | skip | component bucket | config | unknown>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Phase 5 conversion requires Phase 4 official/FastVideo key dumps.
|
||||
- Component skills must use the retry request instead of editing conversion
|
||||
scripts or converted weights directly.
|
||||
- Conversion must update `port_state_file` with conversion status, retry history,
|
||||
strict-load status, new issues, and resolved issues.
|
||||
- Conversion must validate emitted config keys through the production config
|
||||
update path and record the config filename expected by each loader.
|
||||
@@ -1,51 +0,0 @@
|
||||
# Escape Hatch Contract
|
||||
|
||||
Canonical pause-and-ask schema for `/add-model` skills. Use this when the next
|
||||
action requires user input instead of autonomous debugging.
|
||||
|
||||
```text
|
||||
escape_hatch:
|
||||
needs_user_input: yes | no
|
||||
decision_type: scope | dependency | auth | cost | destructive | ambiguity | blocker
|
||||
question: <one precise question>
|
||||
recommended_option: <safe recommended choice>
|
||||
options:
|
||||
- <option + consequence>
|
||||
safe_default: <what the agent will do after approval, or none>
|
||||
blocked_until_answered: yes | no
|
||||
state_snapshot:
|
||||
phase: <phase or skill mode>
|
||||
files_changed:
|
||||
- <paths>
|
||||
command_or_test: <last relevant command, or not_run>
|
||||
evidence: <short logs, paths, error text, or blocker ID>
|
||||
```
|
||||
|
||||
Use `needs_user_input=no` when the handoff is green or the next step is already
|
||||
specified by the workflow.
|
||||
|
||||
Ask the user only for decisions the workflow cannot safely choose:
|
||||
|
||||
- product or PR scope changes, including dropping a modality, output head, or
|
||||
variant;
|
||||
- core dependency changes, version pin changes, or installing untrusted/private
|
||||
dependencies;
|
||||
- auth setup for gated repos, using env var names only and never token values;
|
||||
- large downloads, publishing weights, SSIM/reference uploads, or GPU-heavy work
|
||||
where cost/runtime approval is needed;
|
||||
- destructive file/git operations, overwriting existing clones/weights, or
|
||||
deleting staged assets;
|
||||
- ambiguous official sources of truth with incompatible behavior;
|
||||
- accepting a known blocker, loosening parity/quality tolerances, or shipping
|
||||
without required non-skip parity.
|
||||
|
||||
Do not ask for normal recoverable failures:
|
||||
|
||||
- missing imports, missing local paths, skipped tests, failing parity, conversion
|
||||
mapping errors, strict-load failures, format/lint failures, or implementation
|
||||
bugs covered by the skill workflow.
|
||||
|
||||
Before returning `next_step=ask_user`, update
|
||||
`tests/local_tests/<model_family>/PORT_STATUS.md` with the blocker/question ID,
|
||||
include the exact evidence, and provide one recommended option plus at most three
|
||||
alternatives.
|
||||
@@ -1,38 +0,0 @@
|
||||
# Final Handoff Contract
|
||||
|
||||
Completed by `/add-model` before handing work back to the user or opening a PR.
|
||||
|
||||
```text
|
||||
final_handoff:
|
||||
prep_handoff_complete: yes | no
|
||||
conversion_status: not_needed | pass | blocked
|
||||
components:
|
||||
- name: <component>
|
||||
reuse_or_port: reused | ported
|
||||
parity_test: <path>
|
||||
parity_status: non_skip_pass | blocked
|
||||
concerns_or_unknowns: <none or list>
|
||||
pipeline_smoke: pass | blocked | not_run
|
||||
pipeline_parity: pass | blocked | not_run
|
||||
example_status: pass | blocked | not_run
|
||||
quality_regression: added | deferred_with_reason | not_applicable
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
token_values_committed: no
|
||||
runtime_third_party_model_imports: none | listed_with_rationale
|
||||
blockers: <none or list>
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Required before handoff:
|
||||
|
||||
- Every required component, reused or ported, has non-skip local parity PASS.
|
||||
- Pipeline smoke and pipeline parity are non-skip PASS, or a blocker is explicit.
|
||||
- Basic example runs and writes a non-corrupt output.
|
||||
- `local_tests_readme` lists every component parity command/status/blocker.
|
||||
- `port_state_file` has no unresolved blocker that is omitted from the final
|
||||
response or PR notes.
|
||||
- No raw HF token values, credentials, `.env`, reference clone, or staged weight
|
||||
blobs are committed.
|
||||
- If final handoff is blocked on user input, include an `escape_hatch` block and
|
||||
matching `PORT_STATUS.md` row.
|
||||
@@ -1,44 +0,0 @@
|
||||
# Parity Status Contract
|
||||
|
||||
Returned by `../add-model-02-parity/SKILL.md` to `/add-model` and later updated by
|
||||
component parity-debug subagents.
|
||||
|
||||
```text
|
||||
component_parity:
|
||||
- component: <name>
|
||||
test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py
|
||||
status: scaffold_skip | debug_red | non_skip_pass | blocked
|
||||
missing: <none | fastvideo_class | converted_weights | official_import | ...>
|
||||
coverage_scope: production_loader | implementation_subcomponent | both
|
||||
official_definition_files: <paths>
|
||||
official_instantiation_files: <paths>
|
||||
concerns_or_unknowns: <short list>
|
||||
pipeline_parity:
|
||||
test: <path or not-created>
|
||||
status: not_started | scaffold_skip | debug_red | non_skip_pass | blocked
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
notes: <short list>
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Status meanings:
|
||||
|
||||
- `scaffold_skip`: test is present but skips for a specific missing dependency,
|
||||
FastVideo class, or weights.
|
||||
- `debug_red`: both sides load and the test fails numerically.
|
||||
- `non_skip_pass`: required before final handoff for every required component,
|
||||
including reused components.
|
||||
- `blocked`: a precise missing dependency, weight, official call path, or
|
||||
component/conversion regression prevents local activation.
|
||||
|
||||
Coverage meanings:
|
||||
|
||||
- `production_loader`: FastVideo side loads through the same loader/path used by
|
||||
a pipeline.
|
||||
- `implementation_subcomponent`: FastVideo side constructs classes or remaps
|
||||
tensors directly to isolate implementation behavior.
|
||||
- `both`: the test covers both production loading and implementation behavior.
|
||||
|
||||
Use `escape_hatch` only when blocked status requires a user decision. Normal
|
||||
skips or red parity should be debugged by the workflow without asking.
|
||||
@@ -1,82 +0,0 @@
|
||||
# Pipeline Context Contract
|
||||
|
||||
Canonical packet passed from `/add-model` to `add-model-09-pipeline` for pipeline
|
||||
definition and pipeline parity-debug work.
|
||||
|
||||
```text
|
||||
pipeline_context:
|
||||
model_family: <snake_case>
|
||||
mode: pipeline-definition | pipeline-parity-debug
|
||||
workload_types:
|
||||
- <T2V|I2V|V2V|T2I|compatibility-shim-with-rationale>
|
||||
modalities:
|
||||
inputs: <text/image/video/audio/pose/depth/mask/etc.>
|
||||
outputs: <video/image/audio/joint-av/latents/etc.>
|
||||
official_ref_dir: <path or import path>
|
||||
official_pipeline_files:
|
||||
- path: <repo-relative or absolute path in official repo>
|
||||
symbols: <pipeline/factory/sample functions>
|
||||
notes: <stage order, mutable state, output contract>
|
||||
official_call:
|
||||
command_or_api: <official CLI, Python call, or package entrypoint>
|
||||
args_and_defaults: <height, width, frames, fps, duration, steps, CFG, scheduler, seeds, etc.>
|
||||
preset_source: <model card, config file, official script, or unknown>
|
||||
scheduler_and_rng: <timestep/sigma/noise/generator behavior>
|
||||
output_contract: <decoded media, denoised latents, waveform, dict keys, etc.>
|
||||
model_index:
|
||||
class_name: <FastVideo pipeline class name to emit in model_index.json>
|
||||
entry_class_names: <registered EntryClass.__name__ values that must include class_name>
|
||||
required_modules: <text_encoder, tokenizer, vae, transformer, scheduler, etc.>
|
||||
passthrough_modules: <tokenizer, scheduler, processor, external HF dirs, or none>
|
||||
sampling_param:
|
||||
new_fields: <none or list of public kwargs/preset defaults to add to SamplingParam>
|
||||
cli_fields: <none or list of fields that need CLI args>
|
||||
placeholder_fields: <none or video-shaped compatibility placeholders with rationale>
|
||||
components:
|
||||
- name: <component>
|
||||
component_type: <dit|vae|encoder|scheduler|conditioner|upsampler|vocoder|generic>
|
||||
parity_test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py
|
||||
parity_status: non_skip_pass
|
||||
fastvideo_target_files: <model/config/export files>
|
||||
converted_component_dir: converted_weights/<family>/<component>
|
||||
conversion:
|
||||
converted_weights_dir: converted_weights/<family>
|
||||
source_layout: <diffusers|raw_official|monolithic|separate_components|mixed|custom>
|
||||
model_index_path: converted_weights/<family>/model_index.json
|
||||
fastvideo_targets:
|
||||
pipeline_files:
|
||||
- fastvideo/pipelines/basic/<family>/<family>_pipeline.py
|
||||
stage_files:
|
||||
- fastvideo/pipelines/basic/<family>/stages/<stage>.py
|
||||
pipeline_config_files:
|
||||
- fastvideo/configs/pipelines/<family>.py
|
||||
preset_file: fastvideo/pipelines/basic/<family>/presets.py
|
||||
registry_file: fastvideo/registry.py
|
||||
example_files:
|
||||
- examples/inference/basic/basic_<family>.py
|
||||
smoke_test: tests/local_tests/pipelines/test_<family>_pipeline_smoke.py
|
||||
parity_test: tests/local_tests/pipelines/test_<family>_pipeline_parity.py
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
concerns_or_unknowns:
|
||||
- <pipeline branch, unsupported workload, output head, preset ambiguity, etc.>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Start only after every required component, reused or ported, has
|
||||
`parity_status=non_skip_pass`. If any row is missing or skipped, return to
|
||||
`/add-model` Phase 6.
|
||||
- Record official call arguments and default sources before writing FastVideo
|
||||
presets. Do not invent inference defaults from memory.
|
||||
- `model_index.class_name` must match a registered pipeline `EntryClass.__name__`;
|
||||
registry detectors are not sufficient for executable pipeline resolution.
|
||||
- Every public generation kwarg or preset default must be represented in
|
||||
`SamplingParam`, or documented as an intentional internal-only field.
|
||||
- `T2A`, `A2A`, and `AV` may be used only after `WorkloadType` supports them;
|
||||
otherwise record the compatibility shim and rationale explicitly.
|
||||
- Keep token values out of the packet. Use only token environment variable names.
|
||||
- If a target path is unknown, use `unknown` plus the exact search already
|
||||
performed.
|
||||
- Update `local_tests_readme` and `port_state_file` whenever pipeline smoke,
|
||||
parity, presets, registry, examples, or blockers change.
|
||||
@@ -1,71 +0,0 @@
|
||||
# Pipeline Handoff Contract
|
||||
|
||||
Returned by `add-model-09-pipeline` to `/add-model` after pipeline definition or
|
||||
pipeline parity-debug work.
|
||||
|
||||
```text
|
||||
pipeline_handoff:
|
||||
model_family: <snake_case>
|
||||
mode: pipeline-definition | pipeline-parity-debug
|
||||
files_changed:
|
||||
- <pipeline/config/preset/registry/stage/example/test/readme/status paths>
|
||||
official_files_used:
|
||||
- <definition/call/default source paths>
|
||||
required_config_modules:
|
||||
emitted: <list from pipeline class>
|
||||
model_index: <list from converted or source model_index.json>
|
||||
status: match | mismatch | blocked
|
||||
pipeline_class_resolution:
|
||||
model_index_class_name: <_class_name>
|
||||
entry_class_names: <registered EntryClass.__name__ values>
|
||||
status: exact_match | alias_added | blocked
|
||||
sampling_param:
|
||||
fields_added: <none or list>
|
||||
cli_fields_added: <none or list>
|
||||
unknown_kwargs_checked: yes | no | blocked
|
||||
stage_chain:
|
||||
- <stage names in execution order>
|
||||
pipeline_config:
|
||||
file: <path>
|
||||
classes: <class names>
|
||||
official_defaults_checked: yes | no | blocked
|
||||
presets:
|
||||
file: <path>
|
||||
names: <preset names>
|
||||
status: pass | blocked | not_run
|
||||
registry:
|
||||
status: pass | blocked | not_run
|
||||
detectors: <HF paths and model_index _class_name strings covered>
|
||||
smoke_test:
|
||||
path: tests/local_tests/pipelines/test_<family>_pipeline_smoke.py
|
||||
status: non_skip_pass | blocked | not_run
|
||||
pytest_output: <command + short result>
|
||||
pipeline_parity:
|
||||
path: tests/local_tests/pipelines/test_<family>_pipeline_parity.py
|
||||
status: scaffold_skip | debug_red | non_skip_pass | blocked
|
||||
pytest_output: <command + short result>
|
||||
comparison_target: <latents|decoded video|audio|joint outputs>
|
||||
example:
|
||||
path: examples/inference/basic/basic_<family>.py
|
||||
status: pass | blocked | not_run
|
||||
output: <path or none>
|
||||
readme_updated: yes | no
|
||||
port_state_updated: yes | no
|
||||
blockers: <none or exact blocker list>
|
||||
next_step: phase_10_quality_regression | return_to_phase_6 | ask_user
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `pipeline-definition` may return with parity still `scaffold_skip` only if the
|
||||
exact missing dependency, weight, or call-path blocker is recorded.
|
||||
- Final `/add-model` handoff requires `smoke_test.status=non_skip_pass` and
|
||||
`pipeline_parity.status=non_skip_pass`, unless the user explicitly accepts a
|
||||
documented blocker.
|
||||
- If parity failure traces to a component, conversion, or strict-load issue,
|
||||
return `next_step=return_to_phase_6` and include the exact failing evidence.
|
||||
- Keep `local_tests_readme` and `port_state_file` synchronized with this
|
||||
handoff before returning.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
|
||||
`PORT_STATUS.md` row.
|
||||
@@ -1,89 +0,0 @@
|
||||
# Port State Contract
|
||||
|
||||
Canonical per-port state file created during prep and updated by every
|
||||
`/add-model` phase.
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
```
|
||||
|
||||
Purpose:
|
||||
|
||||
- Single source of truth for resumable port progress.
|
||||
- Tracks component status, conversion status, parity status, open questions,
|
||||
blockers, escape hatches, and issue history.
|
||||
- Lets review agents run the same setup/tests without reconstructing handoffs
|
||||
from conversation history.
|
||||
|
||||
Required sections:
|
||||
|
||||
```text
|
||||
# <Model Family> Port Status
|
||||
|
||||
## Summary
|
||||
- model_family:
|
||||
- workload_types:
|
||||
- official_ref:
|
||||
- official_ref_dir:
|
||||
- hf_weights_path:
|
||||
- local_weights_dir:
|
||||
- source_layout:
|
||||
- local_tests_readme:
|
||||
|
||||
## Current Phase
|
||||
- phase:
|
||||
- status: not_started | in_progress | blocked | complete
|
||||
- owner: orchestrator | prep | parity | conversion | component:<name> | pipeline
|
||||
- last_updated:
|
||||
|
||||
## Component Matrix
|
||||
| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
|
||||
## Conversion State
|
||||
- conversion_script:
|
||||
- converted_weights_dir:
|
||||
- source_layout:
|
||||
- strict_load_status:
|
||||
- passthrough_components:
|
||||
- retry_history:
|
||||
|
||||
## Parity Commands
|
||||
| Scope | Command | Last Result | Notes |
|
||||
|---|---|---|---|
|
||||
|
||||
## Open Questions
|
||||
| ID | Question | Owner | Needed By Phase | Status | Resolution |
|
||||
|---|---|---|---|---|---|
|
||||
|
||||
## Issues And Blockers
|
||||
| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
|
||||
## Escape Hatches
|
||||
| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution |
|
||||
|---|---|---|---|---|---|---|
|
||||
|
||||
## Decisions
|
||||
| Date | Decision | Rationale | Impact |
|
||||
|---|---|---|---|
|
||||
|
||||
## Handoff Notes
|
||||
- <short notes for the next agent>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Update this file whenever a phase starts, blocks, resolves an issue, or hands
|
||||
off to another skill.
|
||||
- Record open questions and issues immediately. Do not leave blockers only in
|
||||
chat history or subagent responses.
|
||||
- Use stable IDs: `Q001`, `Q002`, `I001`, `I002`, etc.
|
||||
- Use stable escape-hatch IDs: `E001`, `E002`, etc. Link them from handoff
|
||||
`escape_hatch.state_snapshot.evidence` when returning `next_step=ask_user`.
|
||||
- Do not include raw token values, machine-local cache internals, or large output
|
||||
dumps. Use repo-relative paths when possible.
|
||||
- If a question or issue is resolved, keep the row and fill `Resolution` instead
|
||||
of deleting it.
|
||||
@@ -1,42 +0,0 @@
|
||||
# Prep Handoff Contract
|
||||
|
||||
Produced by `../add-model-01-prep/SKILL.md` and consumed by `/add-model` Phase 0.
|
||||
|
||||
```text
|
||||
model_family: <snake_case>
|
||||
workload_types: <T2V/I2V/V2V/T2I/or compatibility shim with rationale>
|
||||
official_ref: <url or import path>
|
||||
official_ref_dir: <ReferenceDir or none>
|
||||
official_ref_commit: <sha or unknown>
|
||||
hf_weights_path: <HF id or local path>
|
||||
hf_revision: <revision or default>
|
||||
local_weights_dir: official_weights/<model_family> or <local path>
|
||||
source_layout: diffusers | raw_official | monolithic | separate_components | mixed | custom | unknown
|
||||
model_index_class: <_class_name or none>
|
||||
components_seen: <components>
|
||||
needs_conversion: yes | no | unknown
|
||||
hf_token_env: <env var name only>
|
||||
dependency_changes: none | installed no-deps editable | installed official deps in current env | blocked on user
|
||||
official_env_status: imports_ok | private_deps_need_stubs | blocked
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
gitignore_entries_added: <list>
|
||||
next_step: add-model | ask_user
|
||||
open_questions: <short list>
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Validation:
|
||||
|
||||
- `official_env_status` must be `imports_ok` or `private_deps_need_stubs` before
|
||||
component parity scaffolding.
|
||||
- `local_tests_readme` must exist and describe official setup, HF weights,
|
||||
dependency changes, planned parity commands, and review notes.
|
||||
- `port_state_file` must exist and follow `contracts/port_state.md`.
|
||||
- Prep does not go directly to conversion; `/add-model` must run component
|
||||
prototype/key-dump Phase 4 before Phase 5 conversion.
|
||||
- `T2A`, `A2A`, and `AV` may be used only after `WorkloadType` supports them;
|
||||
otherwise record the compatibility shim and rationale explicitly.
|
||||
- Never include HF token values.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
|
||||
`PORT_STATUS.md` row.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Shared Add-Model Rules
|
||||
|
||||
These rules apply to every `add-model` related skill: prep, parity, conversion,
|
||||
component porting, pipeline, and the main `/add-model` orchestrator.
|
||||
|
||||
## Token And Auth Safety
|
||||
|
||||
- Never accept, print, echo, log, hard-code, or commit raw HF token values.
|
||||
- Refer only to token environment variable names: `HF_TOKEN`,
|
||||
`HUGGINGFACE_HUB_TOKEN`, or `HF_API_KEY`.
|
||||
- Scripts may read those environment variables but must not print their values.
|
||||
- Ask for auth setup only by env var name. Do not ask the user to paste a token.
|
||||
- Read scope is needed for gated repos during conversion/load. Write scope is
|
||||
needed for publishing converted weights or seeding generated references.
|
||||
|
||||
## Shared State Files
|
||||
|
||||
- `tests/local_tests/<model_family>/README.md` is the reviewer-facing setup and
|
||||
verification log. Keep it current with setup commands, dependency blockers,
|
||||
parity commands, conversion commands, and pass/blocker status.
|
||||
- `tests/local_tests/<model_family>/PORT_STATUS.md` is the per-port state file.
|
||||
It must follow `../contracts/port_state.md` and keep stable `Q###`, `I###`, and
|
||||
`E###` IDs.
|
||||
- Keep resolved questions/issues in `PORT_STATUS.md` with the resolution instead
|
||||
of deleting them.
|
||||
- Before returning a handoff, update both state files when the skill changed
|
||||
setup, tests, conversion, parity status, blockers, or decisions.
|
||||
- Do not include raw tokens, non-reproducible absolute cache paths, large
|
||||
generated outputs, `.env`, credentials, or anything matching `*secret*`.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Continue autonomously for recoverable setup, implementation, conversion,
|
||||
strict-load, smoke, parity-debug, lint, or test failures. Stop and ask the user
|
||||
only when the next action requires a product, cost, safety, auth, dependency, or
|
||||
scope decision the workflow cannot safely choose.
|
||||
|
||||
Use `../contracts/escape_hatch.md` whenever returning `next_step=ask_user`.
|
||||
|
||||
Ask for user input only for:
|
||||
|
||||
- scope changes, such as dropping a modality, output head, variant, component, or
|
||||
public mode;
|
||||
- core dependency changes, untrusted/private dependency installs, or version pin
|
||||
changes;
|
||||
- auth setup for gated repos, using env var names only;
|
||||
- large downloads, publishing weights, SSIM/reference uploads, or GPU-heavy work
|
||||
where cost/runtime approval is needed;
|
||||
- destructive file/git operations, overwriting existing clones/weights, or
|
||||
deleting staged assets;
|
||||
- incompatible official sources of truth where no reference can be chosen from
|
||||
published weights and docs;
|
||||
- accepting a blocker, loosening tolerances, using shape-only substitutes, or
|
||||
shipping without required non-skip parity.
|
||||
|
||||
Do not ask for normal recoverable failures: missing imports, missing local paths,
|
||||
skipped tests, failing parity, conversion mapping bugs, strict-load errors,
|
||||
format/lint failures, smoke failures, registry import issues, example failures,
|
||||
or implementation bugs covered by the phase workflow.
|
||||
|
||||
Before asking:
|
||||
|
||||
- update `PORT_STATUS.md` with an `E###` escape-hatch row plus any linked `Q###`
|
||||
or `I###` row;
|
||||
- include exact evidence: command, path, short error text, parity or strict-load
|
||||
excerpt, or blocker ID;
|
||||
- provide one recommended option and at most three alternatives;
|
||||
- set the relevant handoff `next_step=ask_user` and include the `escape_hatch`
|
||||
block.
|
||||
|
||||
Skill-specific escape-hatch sections may add extra examples, but they must not
|
||||
weaken these shared rules.
|
||||
|
||||
## Production Boundary
|
||||
|
||||
- No runtime `from diffusers import <model class>` or
|
||||
`from transformers import <model class>` in `fastvideo/` production code.
|
||||
- Components that own weights or numerical behavior must be FastVideo-native
|
||||
unless the user explicitly accepts a documented lazy-wrapper exception.
|
||||
- Allowed third-party runtime exceptions are tokenizers and pure data utilities
|
||||
when they match existing project patterns.
|
||||
- Tests may import diffusers/transformers as parity references.
|
||||
- Production comments explain why, not what or provenance. Avoid narrative
|
||||
comments like `vendored from`, `matches upstream`, `REVIEW`, or session-history
|
||||
commentary.
|
||||
|
||||
## Verification Semantics
|
||||
|
||||
- A committed local test may skip when clones, weights, or private deps are absent
|
||||
so CI and other contributors are not blocked.
|
||||
- On the porter's machine, a skip is not a pass. Fix the missing import, weights,
|
||||
or path before claiming verification.
|
||||
- New ports require local non-skip parity for required components and pipeline
|
||||
parity when a pipeline is in scope.
|
||||
- Smoke tests prove loadability only. They are not a substitute for numerical
|
||||
component or pipeline parity.
|
||||
@@ -1,117 +0,0 @@
|
||||
# Component Skill Common Instructions
|
||||
|
||||
These instructions apply to `add-model-03-port-dit`, `add-model-04-port-vae`,
|
||||
`add-model-05-port-encoder`, and `add-model-06-port-generic`. Bucket-specific skills
|
||||
add target paths, implementation patterns, drift checks, and scope questions.
|
||||
|
||||
## Required Context
|
||||
|
||||
Require the complete packet from `../contracts/component_context.md`.
|
||||
|
||||
Do not start if the official definition files, official instantiation files, or
|
||||
parity test path are missing. Ask the `/add-model` orchestrator for the complete
|
||||
component context packet instead of rediscovering broad scope silently.
|
||||
|
||||
If the parity scaffold is missing, create it first with
|
||||
`../../add-model-02-parity/templates/component_parity_test.py`.
|
||||
|
||||
## Prototype Mode
|
||||
|
||||
Prototype mode runs before conversion:
|
||||
|
||||
- implement or prove reuse for the minimal native component, config, export, and
|
||||
`EntryClass` surface needed by the relevant loader;
|
||||
- instantiate with random weights using the exact official architecture args, or
|
||||
instantiate/document stateless components with no weights;
|
||||
- dump official and FastVideo `state_dict()` names/shapes for every stateful
|
||||
component so conversion can derive mappings from real surfaces;
|
||||
- return concerns discovered during prototype work, such as ambiguous official
|
||||
flags, shape mismatches, private ops, missing loader buckets, passthrough
|
||||
weights, or output heads;
|
||||
- update `local_tests_readme` with prototype status and key-dump paths;
|
||||
- update `port_state_file` with prototype status, open questions, issues, and
|
||||
handoff notes;
|
||||
- do not chase numerical parity and do not block on converted weights.
|
||||
|
||||
Prototype mode succeeds when the component imports, instantiates with official
|
||||
args, and required key/shape dumps exist. Converted weights and parity PASS are
|
||||
not required yet.
|
||||
|
||||
## Parity-Debug Mode
|
||||
|
||||
Parity-debug mode runs after conversion:
|
||||
|
||||
- strict-load converted weights through the same path the pipeline will use, or
|
||||
document that the component is stateless or an approved passthrough;
|
||||
- use `conversion_context` and `concerns_or_unknowns` to decide whether a failure
|
||||
belongs to mapping, loading, implementation, tokenization, normalization,
|
||||
scheduler semantics, or the parity test;
|
||||
- run only the component parity test first with `pytest <parity_test> -v -s`;
|
||||
- if it skips, fix the missing official import, FastVideo class, tokenizer,
|
||||
converted weights, or path;
|
||||
- if it fails numerically, add targeted intermediate comparisons to identify the
|
||||
first divergent operation or tensor;
|
||||
- update component implementation only when the failure is a component
|
||||
layer/config/forward/contract bug;
|
||||
- update `local_tests_readme` with the command, result, and blocker or PASS;
|
||||
- update `port_state_file` with parity status, resolved/new issues, open
|
||||
questions, and handoff notes;
|
||||
- keep iterating until the test is a non-skip PASS or return a precise blocker.
|
||||
|
||||
## Conversion Boundary
|
||||
|
||||
Component skills must not patch conversion scripts or converted weights ad hoc.
|
||||
If the first drift or strict-load failure points to wrong keys, missing tensors,
|
||||
shape mismatches, component prefixes, split/fuse logic, skipped-key policy, or
|
||||
config emission, return a conversion retry request for
|
||||
`../../add-model-07-conversion/SKILL.md` matching
|
||||
`../contracts/conversion_request.md`.
|
||||
Resume parity-debug only after conversion returns an updated handoff.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
When `fastvideo_target_files` point to existing FastVideo code instead of a new
|
||||
port:
|
||||
|
||||
- compare the official definition against the FastVideo target: graph/operation
|
||||
structure, parameter or state shapes, normalization, activation, positional or
|
||||
temporal behavior, scaling constants, dtype behavior, state-dict names, output
|
||||
containers, and output tensors;
|
||||
- compare the official instantiation against the FastVideo config and loader
|
||||
args: constructor args, config values, defaults, variant flags, optional
|
||||
submodules, checkpoint metadata, tokenizer/media paths, and loader path;
|
||||
- treat a matching class instantiated with different args as not reusable;
|
||||
- record reuse evidence in `local_tests_readme` and keep the reused component in
|
||||
`prototype_key_dumps` when it owns state so conversion and parity-debug use the
|
||||
same surface;
|
||||
- still run parity-debug to a non-skip PASS. If mismatch is found, return the
|
||||
concern so `/add-model` can switch the component to a native port.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../contracts/component_skill_handoff.md`.
|
||||
|
||||
Mode-specific expectations:
|
||||
|
||||
- In `mode=prototype`, `parity_status` may be `scaffold_skip` or `blocked`; key
|
||||
dumps are the required artifact and successful prototype handoff should use
|
||||
`next_step=phase_5_conversion`.
|
||||
- In `mode=parity-debug`, final success requires
|
||||
`parity_status=non_skip_pass`.
|
||||
- If conversion is implicated, return `conversion_retry_request` and leave
|
||||
conversion edits to `add-model-07-conversion`.
|
||||
- If production loading is non-strict, list allowed missing/unexpected keys in
|
||||
the parity test or handoff and mark
|
||||
`strict_load=pass_with_documented_exclusions`.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `common_rules.md`. Do not ask for normal prototype or parity-debug
|
||||
failures such as missing imports, tokenizer/path issues, red parity,
|
||||
strict-load failures, key mismatches, shape mismatches, or implementation bugs.
|
||||
Return conversion retry requests or precise blockers as directed by the workflow.
|
||||
|
||||
Ask only when component work requires a scope or safety decision, such as
|
||||
dropping a required stream/output/path, changing core dependencies, accepting
|
||||
private model code or unsupported private ops, choosing between incompatible
|
||||
official definitions, creating a new loader bucket, or loosening required parity.
|
||||
@@ -1,338 +0,0 @@
|
||||
---
|
||||
name: decompose-pipeline-pr
|
||||
description: Decompose an oversized FastVideo pipeline PR into a stack of independently-reviewable PRs. Tiers the diff by blast radius (invisible / dead code / cross-cutting infra / activation), produces a branch graph and worktree bootstrap, drafts the AGENTS.md manifest, flags missing tests on cross-cutting infra changes, and extracts lessons from the PR body.
|
||||
---
|
||||
|
||||
# Decompose Pipeline PR
|
||||
|
||||
## Purpose
|
||||
|
||||
When a PR adds a new pipeline (or first-class component port) and crosses
|
||||
~3,000 LOC, single-shot review converges to rubber-stamping. This skill
|
||||
decomposes such a PR into a stack of independently-reviewable PRs without
|
||||
disturbing `main`.
|
||||
|
||||
It is the inverse of `add-model`: where `add-model` walks adding a new
|
||||
pipeline as a fresh PR, this skill walks decomposing an existing oversized
|
||||
pipeline PR.
|
||||
|
||||
**Worked example:** PR #1280 (daVinci-MagiHuman, 9,812 LOC, 56 files) →
|
||||
2 prerequisite PRs off main + 8-PR stack:
|
||||
- #1293 `will/activation-trace` (prerequisite)
|
||||
- #1294 `will/loader-infra` (prerequisite)
|
||||
- #1295 (1/8) housekeeping
|
||||
- #1296 (2/8) t5gemma encoder
|
||||
- #1297 (3/8) DiT
|
||||
- #1298 (4/8) pipeline stages
|
||||
- #1299 (5/8) pipeline orchestrator
|
||||
- #1300 (6/8) provenance (AGENTS.md, JOURNAL.md, lessons)
|
||||
- #1301 (7/8) conversion scripts
|
||||
- #1302 (8/8) registry activation
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Open PR number on `hao-ai-lab/FastVideo` (or any FastVideo fork)
|
||||
- `gh` CLI authenticated against the target remote
|
||||
- Local git worktree support (`git worktree`)
|
||||
- Git config `user.name` / `user.email` set
|
||||
- Pre-commit installed (`pre-commit install --hook-type pre-commit --hook-type commit-msg`)
|
||||
- The target PR's branch fetched locally as `origin/<feature-branch>`
|
||||
|
||||
## Inputs
|
||||
|
||||
| Parameter | Required | Description |
|
||||
|-----------|----------|-------------|
|
||||
| PR number or URL | Yes | E.g. `1280` or `https://github.com/hao-ai-lab/FastVideo/pull/1280` |
|
||||
| Max desired PR size | No | Defaults to ~2,500 LOC of code per stack PR (excluding generated/journal files) |
|
||||
| Output directory | No | Defaults to `.agents/tmp/decompose-<pr-number>/` (gitignored) |
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Verify ground truth (do not trust `gh pr diff --name-only`)
|
||||
|
||||
`gh pr diff <N> --name-only` has been observed to emit phantom file entries.
|
||||
Always cross-check against the authoritative `git diff`:
|
||||
|
||||
```bash
|
||||
mkdir -p .agents/tmp/decompose-<N>
|
||||
git fetch origin pull/<N>/head:<feature-branch>
|
||||
git diff origin/main..origin/<feature-branch> --name-status \
|
||||
> .agents/tmp/decompose-<N>/files.txt
|
||||
git diff origin/main..origin/<feature-branch> --stat
|
||||
```
|
||||
|
||||
Use the `--name-status` output as the authoritative file list. If it
|
||||
disagrees with `gh pr diff --name-only`, trust the git diff.
|
||||
|
||||
### 2. Tier the diff by blast radius
|
||||
|
||||
Classify every changed file into one of four tiers:
|
||||
|
||||
| Tier | Description | Examples |
|
||||
|---|---|---|
|
||||
| **Tier 0 — Invisible** | Lint/style/CI configs that don't affect runtime | `.gitignore`, `pyproject.toml` (codespell only), agent documentation |
|
||||
| **Tier 1 — Dead code** | New files in their own dirs; aggregator one-liners | `fastvideo/models/dits/<new>/`, `fastvideo/pipelines/basic/<new>/`, `examples/inference/basic/basic_<new>*.py`, `tests/local_tests/<new>/`, `__init__.py` exports |
|
||||
| **Tier 2 — Cross-cutting infra** | Modifications to files used by every pipeline | See protected-paths list below |
|
||||
| **Tier 3 — Activation switch** | `register_configs(...)` calls + the example scripts that demo them | `fastvideo/registry.py` |
|
||||
|
||||
**FastVideo Tier 2 protected paths:**
|
||||
```
|
||||
fastvideo/utils.py
|
||||
fastvideo/pipelines/composed_pipeline_base.py
|
||||
fastvideo/models/loader/component_loader.py
|
||||
fastvideo/configs/models/dits/__init__.py
|
||||
fastvideo/configs/models/encoders/__init__.py
|
||||
fastvideo/configs/models/vaes/__init__.py
|
||||
fastvideo/envs.py
|
||||
fastvideo/fastvideo_args.py
|
||||
fastvideo/distributed/**
|
||||
fastvideo/layers/**
|
||||
fastvideo/attention/**
|
||||
fastvideo/registry.py # treat as Tier 3 if change is the activation
|
||||
```
|
||||
|
||||
Tier 3 detection (mechanical):
|
||||
```bash
|
||||
git diff origin/main..origin/<feature-branch> -- fastvideo/registry.py | \
|
||||
grep -E "^\+.*register_configs\("
|
||||
```
|
||||
|
||||
If `registry.py` only contains `register_configs` additions, treat it as
|
||||
Tier 3. If it modifies existing behavior, treat it as Tier 2 (rare).
|
||||
|
||||
### 3. Identify reusable Tier-1 components
|
||||
|
||||
Within Tier 1, look for sub-trees that are **not** model-specific and could
|
||||
land separately:
|
||||
|
||||
- Encoders matching a known multi-model base (T5/T5-Gemma/Llama/Gemma/CLIP variants)
|
||||
- New stage classes that subclass shared bases without referencing the new model
|
||||
- Hook/profiler/debug infra under `fastvideo/hooks/`
|
||||
- New helpers that have no model-specific dependencies
|
||||
|
||||
These get split into their own PRs (e.g. PR 4 `t5gemma-encoder` in the
|
||||
MagiHuman example).
|
||||
|
||||
### 4. Hunt for missing test coverage on Tier 2 changes
|
||||
|
||||
For every Tier-2 file modified, check whether the original PR added unit
|
||||
tests for the new behavior:
|
||||
|
||||
```bash
|
||||
for f in <list-of-tier-2-files>; do
|
||||
echo "=== Tests for $f ==="
|
||||
git diff origin/main..origin/<feature-branch> -- \
|
||||
"$(echo $f | sed 's|fastvideo/|fastvideo/tests/|; s|\.py|*|')"
|
||||
done
|
||||
```
|
||||
|
||||
If a Tier-2 PR has no accompanying tests, **emit a "must-add tests" list**
|
||||
with a sketch of the case grid. Tier-2 PRs do not ship without those tests.
|
||||
|
||||
The MagiHuman example required this for PR-B (`utils.py`): the original PR
|
||||
shipped no `test_utils_loader.py`, so the decomposition added 9 unit-test
|
||||
cases covering the umbrella-detector boundary, the optional-component-dirs
|
||||
relaxation, and regression coverage on every existing 2-segment HF id.
|
||||
|
||||
### 5. Build the dependency DAG and topo-sort
|
||||
|
||||
Edges:
|
||||
- Tier 2 infra → Tier 1 code that imports it
|
||||
- Reusable Tier 1 components → model-specific Tier 1 code that uses them
|
||||
(encoder before DiT before pipeline)
|
||||
- Tier 1 → Tier 3 (activation always last)
|
||||
- Tier 0 has no dependents (lands first as a freebie)
|
||||
|
||||
Topo-sort produces the stack ordering. Pull Tier-2 PRs **out of the stack**
|
||||
when they have no model-specific dependency — they should land off main
|
||||
with their own focused review, not buried in a model port.
|
||||
|
||||
Render as a tree (markdown):
|
||||
|
||||
```
|
||||
main
|
||||
├─ <prereq-A>
|
||||
│ └─ <prereq-B>
|
||||
│ ├─ <stack-01-housekeeping>
|
||||
│ │ └─ <stack-02-encoder>
|
||||
│ │ └─ <stack-03-dit>
|
||||
│ │ └─ ...
|
||||
│ │ └─ <stack-N-activate>
|
||||
│ └─ (parallel) <skill-pr> off main
|
||||
```
|
||||
|
||||
### 6. Detect mis-shelved docs and debug scratch
|
||||
|
||||
Two categories to flag:
|
||||
|
||||
- **Mis-shelved docs**: Markdown files under `tests/local_tests/` are
|
||||
journals, not tests. Flag for relocation to the package dir as
|
||||
`JOURNAL.md`.
|
||||
- **Debug scratch**: files starting with `_debug_`, `_scratch_`, or
|
||||
`_explore_`. Flag for drop (do not carry into any output PR).
|
||||
|
||||
For MagiHuman: `tests/local_tests/magi-human.md` → relocate. Two
|
||||
`_debug_magi_human_*.py` files → drop.
|
||||
|
||||
### 7. Author the AGENTS.md manifest skeleton
|
||||
|
||||
For the new pipeline package, generate a 6-section `AGENTS.md` scaffold
|
||||
with the file table pre-populated from the diff:
|
||||
|
||||
1. **Manifest** — file table by role
|
||||
2. **Parity invariants** — load-bearing rules with one-paragraph each + lesson refs
|
||||
3. **Cross-refs** — "If you change X, re-run Y" matrix
|
||||
4. **Run book** — single pytest command + prereqs (HF tokens, GPU, wall-time)
|
||||
5. **Open questions** — known issues (e.g. tolerance carve-outs)
|
||||
6. **Provenance** — PR table with branch names and source SHA
|
||||
|
||||
The provenance section is filled incrementally during stack execution and
|
||||
finalized in the activation PR.
|
||||
|
||||
### 8. Extract lessons from the PR body
|
||||
|
||||
Scan the PR body for sections titled "Key implementation work", "Bug hunt",
|
||||
"Lessons", or sentences with patterns like "took N waves to localize",
|
||||
"silent regression", "investigation revealed". Each becomes a candidate
|
||||
`.agents/lessons/<YYYY-MM-DD>_<slug>.md` draft.
|
||||
|
||||
Lessons MUST follow the existing template in
|
||||
`.agents/lessons/README.md`:
|
||||
- YAML frontmatter: `date`, `experiment`, `category`, `severity`
|
||||
- Sections: What Happened, Root Cause, Fix / Workaround, Prevention
|
||||
- Filename: `<YYYY-MM-DD>_<short-slug>.md`
|
||||
|
||||
Lessons co-locate with the code they concern: a conversion-script lesson
|
||||
lands in the same PR as the conversion script, not in the docs PR.
|
||||
|
||||
### 9. Emit the commit-footer convention
|
||||
|
||||
Every commit in the stack ends with:
|
||||
|
||||
```
|
||||
<Feature>-Stack: N/M
|
||||
```
|
||||
|
||||
E.g. `Magi-Stack: 5/8`. Use the package directory name as the feature key.
|
||||
After all PRs squash-merge, `git log --grep='^<Feature>-Stack:'` reconstructs
|
||||
the lineage even if PR numbers later get renumbered.
|
||||
|
||||
### 10. Produce the worktree bootstrap
|
||||
|
||||
Generate a runnable bash script:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
REPO=/home/<user>/FastVideo
|
||||
WORKTREE=/home/<user>/FastVideoMagi # NB: directory name must be a valid
|
||||
# Python identifier (no hyphens) so
|
||||
# mypy doesn't choke
|
||||
SOURCE_PR=<N>
|
||||
SOURCE_BRANCH=will/<feature>
|
||||
SOURCE_SHA=$(git -C "$REPO" rev-parse "origin/$SOURCE_BRANCH")
|
||||
|
||||
git -C "$REPO" fetch origin main:main
|
||||
git -C "$REPO" fetch "origin/$SOURCE_BRANCH"
|
||||
git -C "$REPO" worktree add "$WORKTREE" origin/main
|
||||
|
||||
# Capture baseline for provenance. Everything under .agents/tmp is transient
|
||||
# and ignored by git.
|
||||
OUTPUT_DIR="$REPO/.agents/tmp/decompose-$SOURCE_PR"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
cat > "$OUTPUT_DIR/<feature>-baseline-${SOURCE_SHA:0:8}.txt" <<EOF
|
||||
Source PR: <repo>#$SOURCE_PR
|
||||
Source SHA: $SOURCE_SHA
|
||||
Authoritative file count: $(git -C "$REPO" diff origin/main..origin/$SOURCE_BRANCH --name-only | wc -l)
|
||||
Date captured: $(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
EOF
|
||||
```
|
||||
|
||||
### 11. Author preserve via `git checkout`, not `cherry-pick`
|
||||
|
||||
For each stack PR:
|
||||
|
||||
```bash
|
||||
git -C "$WORKTREE" switch -c <new-branch> <base-branch>
|
||||
git -C "$WORKTREE" checkout origin/<source-branch> -- <file1> <file2> ...
|
||||
git -C "$WORKTREE" commit -m "[<scope>]: <subject>
|
||||
|
||||
<body>
|
||||
|
||||
<Feature>-Stack: N/M"
|
||||
git -C "$WORKTREE" push -u origin <new-branch>
|
||||
gh pr create --base <base-branch> --head <new-branch> --title "..." --body "$(cat <<EOF ... EOF)"
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `git checkout origin/<source> -- <files>` extracts only the named files,
|
||||
preserving the diff. The original PR's author is **not** preserved on the
|
||||
new commit (it's authored by whoever runs the script). Reference the
|
||||
original PR + source SHA in every commit body and PR description for
|
||||
authorship attribution.
|
||||
- **Never use `git cherry-pick`** for this workflow — cherry-pick applies
|
||||
whole commits, which mixes concerns across PR boundaries.
|
||||
|
||||
## Outputs
|
||||
|
||||
The skill produces all transient planning artifacts under
|
||||
`.agents/tmp/decompose-<pr>/`:
|
||||
|
||||
1. A markdown decomposition plan (`plan.md`)
|
||||
2. A proposed branch graph
|
||||
3. A worktree-bootstrap script (`bootstrap.sh`)
|
||||
4. Per-PR file allocation lists (under `stack/`)
|
||||
5. AGENTS.md scaffolds for any new pipeline packages
|
||||
6. Draft lesson files (placed alongside the PR that owns the code they concern)
|
||||
7. A finalized provenance table for the package AGENTS.md
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
The skill should warn against:
|
||||
|
||||
- **"Just rebase the megaPR into smaller commits."** Doesn't help review;
|
||||
reviewer still sees one PR.
|
||||
- **Co-locating tests under the new package.** FastVideo's convention is
|
||||
by-kind under `fastvideo/tests/` and `tests/local_tests/<family>/`. Don't
|
||||
invent a new layout per pipeline.
|
||||
- **Splitting Tier 2 changes into "one file per PR."** Tier 2 PRs are
|
||||
about semantic units (e.g., "loader umbrella + optional component dirs"
|
||||
together because they jointly define the new diffusers-format contract),
|
||||
not file-count.
|
||||
- **Landing the activation switch first** ("just register, the code can
|
||||
be empty"). The skill enforces activation-last so every intermediate
|
||||
state is dead code, not broken code.
|
||||
- **Trusting `gh pr diff --name-only`.** Cross-check against
|
||||
`git diff origin/main..origin/<feature-branch> --name-status` —
|
||||
`gh`'s output has been observed to include phantom entries.
|
||||
- **Worktree dir names with hyphens.** mypy interprets them as invalid
|
||||
Python package names and refuses to run. Use CamelCase or underscores.
|
||||
- **Skipping the lesson-extraction step.** PR bodies contain the most
|
||||
expensive learnings of the original implementation. Losing them to a
|
||||
squash-merge is the silent decay of institutional knowledge.
|
||||
|
||||
## Example Usage
|
||||
|
||||
```
|
||||
User: split PR 1280
|
||||
Agent: [invokes decompose-pipeline-pr]
|
||||
→ produces .agents/tmp/decompose-1280/plan.md with:
|
||||
- tiered file table (56 files: 3 tier-0, 35 tier-1, 9 tier-2,
|
||||
9 tier-3)
|
||||
- branch graph (PR-A + PR-B + 8-PR stack)
|
||||
- worktree bootstrap script
|
||||
- per-PR file lists
|
||||
- AGENTS.md scaffold for fastvideo/pipelines/basic/magi_human/
|
||||
- 3 draft lessons extracted from the PR body
|
||||
→ asks user to confirm before opening branches
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- The MagiHuman decomposition (worked example):
|
||||
`fastvideo/pipelines/basic/magi_human/AGENTS.md` (after PR #1302 merges)
|
||||
- Existing skill: `.agents/skills/add-model/SKILL.md` (the inverse — adding
|
||||
a new pipeline as a fresh PR)
|
||||
- Lesson template: `.agents/lessons/README.md`
|
||||
- Skill template: `.agents/skills/SKILL_TEMPLATE.md`
|
||||
@@ -105,8 +105,8 @@ Detect artefact type by inspecting the file's imports / helper call:
|
||||
- **pixel** (`.mp4`) — file imports
|
||||
`run_text_to_video_similarity_test` / `run_image_to_video_similarity_test`
|
||||
from `fastvideo.tests.ssim.inference_similarity_utils`, OR uses the
|
||||
legacy custom-inline helper pattern (see `test_gamecraft`,
|
||||
`test_longcat`, etc.). Default to pixel when both heuristics fail.
|
||||
legacy custom-inline helper pattern (see `test_gen3c`). Default to pixel
|
||||
when both heuristics fail.
|
||||
|
||||
Record `ARTEFACT_TYPE ∈ {pixel, latent}` for use in step 4. Steps 2, 3, 5,
|
||||
and 6 are artefact-type-agnostic — `_iter_reference_files`,
|
||||
|
||||
@@ -205,40 +205,6 @@ pull_request_rules:
|
||||
label:
|
||||
add: ["scope: model"]
|
||||
|
||||
# ============================================================
|
||||
# Pre-commit failure help comment
|
||||
# ============================================================
|
||||
|
||||
- name: comment on pre-commit failure
|
||||
conditions:
|
||||
- check-failure~=pre-commit
|
||||
- -closed
|
||||
actions:
|
||||
comment:
|
||||
message: |
|
||||
## Pre-commit checks failed
|
||||
|
||||
Hi @{{author}}, the pre-commit checks have failed. To fix them locally:
|
||||
|
||||
```bash
|
||||
# Install pre-commit if you haven't already
|
||||
uv pip install pre-commit
|
||||
pre-commit install
|
||||
|
||||
# Run all checks and auto-fix what's possible
|
||||
pre-commit run --all-files
|
||||
```
|
||||
|
||||
Common fixes:
|
||||
- **yapf**: `yapf -i <file>` (formatting)
|
||||
- **ruff**: `ruff check --fix <file>` (linting)
|
||||
- **codespell**: `codespell --write-changes <file>` (spelling)
|
||||
|
||||
After fixing, commit and push the changes. The checks will re-run automatically.
|
||||
|
||||
For future commits, `pre-commit` will run automatically on changed files before each commit.
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Merge conflict detection
|
||||
# ============================================================
|
||||
@@ -251,16 +217,6 @@ pull_request_rules:
|
||||
actions:
|
||||
label:
|
||||
add: [needs-rebase]
|
||||
comment:
|
||||
message: |
|
||||
This PR has merge conflicts with the base branch. Please rebase:
|
||||
|
||||
```bash
|
||||
git fetch origin main
|
||||
git rebase origin/main
|
||||
# Resolve any conflicts, then:
|
||||
git push --force-with-lease
|
||||
```
|
||||
|
||||
- name: remove conflict label when resolved
|
||||
conditions:
|
||||
@@ -290,35 +246,5 @@ pull_request_rules:
|
||||
merge:
|
||||
method: squash
|
||||
|
||||
# ============================================================
|
||||
# PR title format help
|
||||
# ============================================================
|
||||
|
||||
- name: comment on invalid PR title format
|
||||
conditions:
|
||||
- -closed
|
||||
- -draft
|
||||
- "-title~=(?i)^\\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\\]"
|
||||
actions:
|
||||
comment:
|
||||
message: |
|
||||
## ⚠️ PR title format required
|
||||
|
||||
Your PR title must start with a type tag in brackets. Examples:
|
||||
- `[feat] Add new model support`
|
||||
- `[bugfix] Fix VAE tiling corruption`
|
||||
- `[refactor] Restructure training pipeline`
|
||||
- `[perf] Optimize attention kernel`
|
||||
- `[ci] Update test infrastructure`
|
||||
- `[infra] Add activation trace hooks`
|
||||
- `[docs] Add inference guide`
|
||||
- `[misc] Clean up configs`
|
||||
- `[new-model] Port Flux2 to FastVideo`
|
||||
- `[skill] Add add-model agent skill`
|
||||
|
||||
Valid tags: `feat`, `feature`, `bugfix`, `fix`, `refactor`, `perf`, `ci`, `infra`, `doc`, `docs`, `misc`, `chore`, `kernel`, `new-model`, `skill`, `skills`
|
||||
|
||||
Please update your PR title and the merge protection check will pass automatically.
|
||||
|
||||
merge_protections_settings:
|
||||
reporting_method: check-runs
|
||||
|
||||
@@ -12,6 +12,7 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import fnmatch
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import TextIO
|
||||
@@ -130,11 +131,6 @@ class FamilyCoverage:
|
||||
|
||||
|
||||
FAMILY_COVERAGE = (
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])dreamx(_world)?([/_.-]|$)"),
|
||||
("test_dreamx.py", ),
|
||||
("test_dreamx_world_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])flux[_-]?2([/_.-]|$)"),
|
||||
("test_flux2_klein.py", ),
|
||||
@@ -145,21 +141,11 @@ FAMILY_COVERAGE = (
|
||||
("test_flux.py", ),
|
||||
("test_flux_t2i_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])(hunyuan)?gamecraft([/_.-]|$)"),
|
||||
("test_gamecraft.py", ),
|
||||
("test_gamecraft_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])gen3c([/_.-]|$)"),
|
||||
("test_gen3c.py", ),
|
||||
("test_gen3c_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])glm[_-]?image([/_.-]|$)"),
|
||||
("test_glm_image.py", ),
|
||||
("test_glm_image_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])hunyuan(video)?15([a-z0-9_-]*)([/_.-]|$)"),
|
||||
(),
|
||||
@@ -175,11 +161,6 @@ FAMILY_COVERAGE = (
|
||||
("test_lingbot.py", ),
|
||||
("test_lingbot_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])longcat([/_.-]|$)"),
|
||||
("test_longcat.py", ),
|
||||
("test_longcat_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])ltx[_-]?2([/_.-]|$)"),
|
||||
("test_ltx2.py", ),
|
||||
@@ -210,11 +191,6 @@ FAMILY_COVERAGE = (
|
||||
("test_stable_audio.py", ),
|
||||
("test_stable_audio_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])turbo(diffusion)?([/_.-]|$)"),
|
||||
(),
|
||||
("test_turbodiffusion_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])wan(video|vae)?([/_.-]|$)"),
|
||||
("test_wan_t2v.py", "test_wan_vae.py", "test_wan_causal.py", "test_wan_denoising.py"),
|
||||
@@ -225,11 +201,6 @@ FAMILY_COVERAGE = (
|
||||
"test_wan_ti2v_similarity.py",
|
||||
),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])z[_-]?image([/_.-]|$)"),
|
||||
("test_zimage.py", ),
|
||||
("test_zimage_similarity.py", ),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@@ -325,16 +296,22 @@ def _select_output_coverage(plan: MergePlan, path: str) -> None:
|
||||
plan.add_ssim(SSIM_SMOKE_TESTS, reason=f"shared output SSIM smoke coverage: {path}")
|
||||
|
||||
|
||||
def classify_paths(paths: list[str]) -> MergePlan:
|
||||
def _normalize_path(raw_path: str) -> str:
|
||||
path = raw_path.strip()
|
||||
while path.startswith("./"):
|
||||
path = path[2:]
|
||||
return path
|
||||
|
||||
|
||||
def classify_paths(paths: list[str], removed_paths: Iterable[str] = ()) -> MergePlan:
|
||||
"""Plan merge lanes for ``paths``.
|
||||
|
||||
``removed_paths`` lists changed paths that no longer exist at the PR head
|
||||
(deleted files and rename sources); removed golden/SSIM tests are not run.
|
||||
"""
|
||||
plan = MergePlan()
|
||||
normalized_paths: list[str] = []
|
||||
for raw_path in paths:
|
||||
path = raw_path.strip()
|
||||
while path.startswith("./"):
|
||||
path = path[2:]
|
||||
if path:
|
||||
normalized_paths.append(path)
|
||||
normalized_paths = sorted(set(normalized_paths))
|
||||
normalized_paths = sorted({path for path in map(_normalize_path, paths) if path})
|
||||
removed = {path for path in map(_normalize_path, removed_paths) if path}
|
||||
if not normalized_paths:
|
||||
plan.require_all("changed-file list was empty; failing closed")
|
||||
return plan
|
||||
@@ -374,7 +351,10 @@ def classify_paths(paths: list[str]) -> MergePlan:
|
||||
if path.startswith("fastvideo/tests/golden_gate/"):
|
||||
name = Path(path).name
|
||||
if name.startswith("test_") and name.endswith(".py"):
|
||||
plan.add_golden((name, ), reason=f"changed golden test: {path}")
|
||||
if path in removed:
|
||||
plan.reasons.append(f"removed golden test has nothing to run: {path}")
|
||||
else:
|
||||
plan.add_golden((name, ), reason=f"changed golden test: {path}")
|
||||
elif name in {"AGENTS.md", "README.md"}:
|
||||
plan.reasons.append(f"golden documentation only: {path}")
|
||||
else:
|
||||
@@ -385,7 +365,10 @@ def classify_paths(paths: list[str]) -> MergePlan:
|
||||
if path.startswith("fastvideo/tests/ssim/"):
|
||||
name = Path(path).name
|
||||
if name.startswith("test_") and name.endswith(".py"):
|
||||
plan.add_ssim((name, ), reason=f"changed SSIM test: {path}")
|
||||
if path in removed:
|
||||
plan.reasons.append(f"removed SSIM test has nothing to run: {path}")
|
||||
else:
|
||||
plan.add_ssim((name, ), reason=f"changed SSIM test: {path}")
|
||||
elif path.endswith((".py", ".json", ".pt", ".png", ".mp4")):
|
||||
plan.ssim_all = True
|
||||
plan.add_lanes("ssim", reason=f"shared SSIM harness/reference: {path}")
|
||||
@@ -565,6 +548,7 @@ def _write_summary(output: TextIO, plan: MergePlan) -> None:
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--paths-file", type=Path, required=True)
|
||||
parser.add_argument("--removed-paths-file", type=Path)
|
||||
parser.add_argument("--github-output", type=Path)
|
||||
parser.add_argument("--summary-file", type=Path)
|
||||
return parser.parse_args()
|
||||
@@ -573,7 +557,9 @@ def parse_args() -> argparse.Namespace:
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
paths = args.paths_file.read_text(encoding="utf-8").splitlines()
|
||||
plan = classify_paths(paths)
|
||||
removed_paths = (args.removed_paths_file.read_text(encoding="utf-8").splitlines()
|
||||
if args.removed_paths_file else [])
|
||||
plan = classify_paths(paths, removed_paths)
|
||||
print(f"MERGE_TEST_PLAN={plan.encoded_lanes()}")
|
||||
print(f"MERGE_GOLDEN_TESTS={plan.encoded_golden_tests()}")
|
||||
print(f"MERGE_SSIM_TESTS={plan.encoded_ssim_tests()}")
|
||||
|
||||
@@ -180,6 +180,8 @@ jobs:
|
||||
set -euo pipefail
|
||||
changed_json="$RUNNER_TEMP/merge-changed-files.json"
|
||||
changed_paths="$RUNNER_TEMP/merge-changed-paths.txt"
|
||||
removed_paths="$RUNNER_TEMP/merge-removed-paths.txt"
|
||||
: > "$removed_paths"
|
||||
if gh api --paginate --slurp \
|
||||
"repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/files?per_page=100" \
|
||||
> "$changed_json"; then
|
||||
@@ -187,6 +189,10 @@ jobs:
|
||||
if [ "$observed" = "$EXPECTED_CHANGED_FILES" ]; then
|
||||
jq -r '.[][] | .filename, (.previous_filename // empty)' "$changed_json" \
|
||||
| sort -u > "$changed_paths"
|
||||
# Paths absent from the PR head, so the planner never selects a deleted test.
|
||||
jq -r '.[][] | if .status == "removed" then .filename
|
||||
elif .status == "renamed" then (.previous_filename // empty) else empty end' \
|
||||
"$changed_json" | sort -u > "$removed_paths"
|
||||
else
|
||||
echo "::warning::Changed-file API returned $observed of $EXPECTED_CHANGED_FILES paths; selecting all merge lanes."
|
||||
echo '__FASTVIDEO_CI_PLAN_ALL__' > "$changed_paths"
|
||||
@@ -202,6 +208,7 @@ jobs:
|
||||
run: |
|
||||
python3 .github/scripts/plan_merge_ci.py \
|
||||
--paths-file "$RUNNER_TEMP/merge-changed-paths.txt" \
|
||||
--removed-paths-file "$RUNNER_TEMP/merge-removed-paths.txt" \
|
||||
--github-output "$GITHUB_OUTPUT" \
|
||||
--summary-file "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
name: Close Stale Issues and PRs
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily at 1:30 AM UTC
|
||||
- cron: '30 1 * * *'
|
||||
|
||||
jobs:
|
||||
stale:
|
||||
if: github.repository == 'hao-ai-lab/FastVideo'
|
||||
permissions:
|
||||
issues: write
|
||||
pull-requests: write
|
||||
actions: write
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/stale@997185467fa4f803885201cee163a9f38240193d # v10.1.1
|
||||
with:
|
||||
operations-per-run: 500
|
||||
|
||||
exempt-draft-pr: true
|
||||
exempt-issue-labels: 'keep-open,pinned,security,Bug,RFC'
|
||||
exempt-pr-labels: 'keep-open,pinned'
|
||||
|
||||
labels-to-add-when-unstale: 'unstale'
|
||||
labels-to-remove-when-stale: 'unstale'
|
||||
|
||||
days-before-issue-stale: 90
|
||||
days-before-issue-close: 30
|
||||
stale-issue-label: 'stale'
|
||||
stale-issue-message: >
|
||||
This issue has been automatically marked as stale because it has not
|
||||
had any activity within 90 days. It will be automatically closed if
|
||||
no further activity occurs within 30 days. Leave a comment if you
|
||||
feel this issue should remain open. Thank you!
|
||||
close-issue-message: >
|
||||
This issue has been automatically closed due to inactivity. Please
|
||||
feel free to reopen if you feel it is still relevant. Thank you!
|
||||
|
||||
days-before-pr-stale: 60
|
||||
days-before-pr-close: 14
|
||||
stale-pr-label: 'stale'
|
||||
stale-pr-message: >
|
||||
This pull request has been automatically marked as stale because it
|
||||
has not had any activity within 60 days. It will be automatically
|
||||
closed if no further activity occurs within 14 days. Leave a comment
|
||||
if you feel this pull request should remain open. Thank you!
|
||||
close-pr-message: >
|
||||
This pull request has been automatically closed due to inactivity.
|
||||
Please feel free to reopen if you intend to continue working on it.
|
||||
Thank you!
|
||||
@@ -1,56 +0,0 @@
|
||||
name: Welcome First-Time Contributors
|
||||
|
||||
on:
|
||||
issues:
|
||||
types: [opened]
|
||||
pull_request_target:
|
||||
types: [opened]
|
||||
|
||||
permissions:
|
||||
issues: write
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
welcome:
|
||||
if: github.repository == 'hao-ai-lab/FastVideo'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/first-interaction@34f15e814fe48ac9312ccf29db4e74fa767cbab7 # v1.3.0
|
||||
with:
|
||||
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
issue-message: |
|
||||
Welcome to FastVideo! Thanks for opening your first issue.
|
||||
|
||||
To help us investigate, please include:
|
||||
- **FastVideo version**: `pip show fastvideo`
|
||||
- **GPU**: `nvidia-smi` output (GPU model, driver, CUDA version)
|
||||
- **Python version**: `python --version`
|
||||
- **OS**: e.g., Ubuntu 22.04
|
||||
|
||||
If this is a bug, a minimal reproduction script helps us fix it faster.
|
||||
|
||||
Useful links:
|
||||
- [Documentation](https://hao-ai-lab.github.io/FastVideo)
|
||||
- [Contributing Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
|
||||
- [Slack](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ)
|
||||
pr-message: |
|
||||
Welcome to FastVideo! Thanks for your first pull request.
|
||||
|
||||
**How our CI works:**
|
||||
|
||||
PRs run a three-tier CI system:
|
||||
1. **Pre-commit** — formatting (yapf), linting (ruff), type checking (mypy). Runs immediately on every PR.
|
||||
2. **Fastcheck** — six core GPU lanes run automatically via Buildkite (~10-15 min).
|
||||
3. **Merge gate** — a reviewer adds `ready`; changed paths select only the relevant integration, training, golden, or SSIM coverage.
|
||||
|
||||
**Before your PR is reviewed:**
|
||||
- [ ] `pre-commit run --all-files` passes locally
|
||||
- [ ] You've added or updated tests for your changes
|
||||
- [ ] The PR description explains what and why
|
||||
|
||||
If pre-commit fails, a bot comment will explain how to fix it. Fastcheck and merge-gate results appear in the Checks section below.
|
||||
|
||||
**Useful links:**
|
||||
- [Contributing Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
|
||||
- [Development Roadmap](https://github.com/hao-ai-lab/FastVideo/issues/899)
|
||||
- [Slack](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ)
|
||||
@@ -36,7 +36,6 @@ env
|
||||
*.log
|
||||
weights/
|
||||
logs/
|
||||
/Z-Image/
|
||||
official_weights/
|
||||
converted_weights/
|
||||
|
||||
|
||||
@@ -160,8 +160,8 @@ class FastH3Backend:
|
||||
Each clip length is a torch.compile shape, and the fullgraph regional
|
||||
route treats exceeding dynamo's recompile limit as a crash rather than
|
||||
a fallback. FastVideo's own imports lower it (`layers/lora/linear.py`
|
||||
to 16, longcat's `bsa_interface.py` to 32), so this raises it again
|
||||
and turns overflow back into a recompile.
|
||||
to 16), so this raises it again and turns overflow back into a
|
||||
recompile.
|
||||
|
||||
One pinned clip length keeps every process far under even the lowered
|
||||
limit; a varied `warmup_lengths` is where this becomes the seatbelt it
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"version": 15,
|
||||
"version": 16,
|
||||
"recipes": [
|
||||
{
|
||||
"id": "fastwan21-t2v",
|
||||
@@ -161,53 +161,6 @@
|
||||
"Native MLX FastMetal T2V on 36 GB+ unified memory. Same --fast, --fast-spatial, and --refine flags as the 1.3B script."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "turbodiffusion-wan21-1-3b-t2v",
|
||||
"family": "turbodiffusion",
|
||||
"stage": "inference",
|
||||
"task": "Text to video",
|
||||
"label": "TurboWan2.1 1.3B",
|
||||
"summary": "A TurboDiffusion-accelerated Wan2.1 1.3B text-to-video run from its maintained single-GPU example.",
|
||||
"model": "loayrashid/TurboWan2.1-T2V-1.3B-Diffusers",
|
||||
"source": "examples/inference/basic/basic_turbodiffusion.py",
|
||||
"command": "python examples/inference/basic/basic_turbodiffusion.py",
|
||||
"gpu_types": ["NVIDIA"],
|
||||
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
|
||||
"evidence": "Source-backed",
|
||||
"expected_artifact": "MP4 videos under video_samples_turbodiffusion/",
|
||||
"related": ["turbodiffusion-wan21-14b-t2v", "turbowan22-i2v"]
|
||||
},
|
||||
{
|
||||
"id": "turbodiffusion-wan21-14b-t2v",
|
||||
"family": "turbodiffusion",
|
||||
"stage": "inference",
|
||||
"task": "Text to video",
|
||||
"label": "TurboWan2.1 14B",
|
||||
"summary": "TurboDiffusion acceleration applied to the 14B Wan2.1 text-to-video checkpoint; the checked-in source is configured for two GPUs.",
|
||||
"model": "loayrashid/TurboWan2.1-T2V-14B-Diffusers",
|
||||
"source": "examples/inference/basic/basic_turbodiffusion_14b.py",
|
||||
"command": "python examples/inference/basic/basic_turbodiffusion_14b.py",
|
||||
"gpu_types": ["NVIDIA"],
|
||||
"hardware": {"gpu_count": 2, "evidence": "source-configured"},
|
||||
"evidence": "Source-backed",
|
||||
"expected_artifact": "MP4 videos under video_samples_turbodiffusion_14B/",
|
||||
"related": ["turbodiffusion-wan21-1-3b-t2v", "turbowan22-i2v"]
|
||||
},
|
||||
{
|
||||
"id": "turbowan22-i2v",
|
||||
"family": "turbodiffusion",
|
||||
"stage": "inference",
|
||||
"task": "Image to video",
|
||||
"label": "TurboWan2.2 A14B",
|
||||
"summary": "A one-to-four-step image-to-video path using TurboDiffusion and the SLA attention backend from its maintained example.",
|
||||
"model": "loayrashid/TurboWan2.2-I2V-A14B-Diffusers",
|
||||
"source": "examples/inference/basic/basic_turbodiffusion_i2v.py",
|
||||
"command": "python examples/inference/basic/basic_turbodiffusion_i2v.py",
|
||||
"gpu_types": ["NVIDIA"],
|
||||
"hardware": {"gpu_count": 2, "evidence": "source-configured"},
|
||||
"evidence": "Source-backed",
|
||||
"related": ["turbodiffusion-wan21-14b-t2v"]
|
||||
},
|
||||
{
|
||||
"id": "ltx2-distilled-t2v",
|
||||
"family": "ltx2",
|
||||
@@ -476,53 +429,6 @@
|
||||
"expected_artifact": "PNG images under outputs/flux_dev/samples/",
|
||||
"limitations": ["FLUX.1 is loadable by ID but registers no model_family in fastvideo/registry.py; it is grouped under FLUX for documentation only."]
|
||||
},
|
||||
{
|
||||
"id": "glm-image-t2i",
|
||||
"family": "glm_image",
|
||||
"stage": "inference",
|
||||
"task": "Text to image",
|
||||
"label": "GLM-Image",
|
||||
"summary": "GLM-Image text-to-image generation from its maintained example.",
|
||||
"model": "zai-org/GLM-Image",
|
||||
"source": "examples/inference/basic/basic_glm_image.py",
|
||||
"command": "python examples/inference/basic/basic_glm_image.py",
|
||||
"gpu_types": ["NVIDIA"],
|
||||
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
|
||||
"evidence": "Source-backed",
|
||||
"expected_artifact": "PNG image at image_output/landscape.png",
|
||||
"related": ["glm-image-edit"]
|
||||
},
|
||||
{
|
||||
"id": "glm-image-edit",
|
||||
"family": "glm_image",
|
||||
"stage": "inference",
|
||||
"task": "Image editing",
|
||||
"label": "GLM-Image editing",
|
||||
"summary": "Edit an input image with an instruction prompt using GLM-Image, from its maintained editing example.",
|
||||
"model": "zai-org/GLM-Image",
|
||||
"source": "examples/inference/basic/edit_glm_image.py",
|
||||
"command": "python examples/inference/basic/edit_glm_image.py",
|
||||
"gpu_types": ["NVIDIA"],
|
||||
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
|
||||
"evidence": "Source-backed",
|
||||
"expected_artifact": "PNG image at image_output/edited.png (input: assets/images/couple.jpg)",
|
||||
"related": ["glm-image-t2i"]
|
||||
},
|
||||
{
|
||||
"id": "zimage-turbo-t2i",
|
||||
"family": "zimage",
|
||||
"stage": "inference",
|
||||
"task": "Text to image",
|
||||
"label": "Z-Image Turbo",
|
||||
"summary": "Z-Image Turbo text-to-image on a single GPU from its maintained example.",
|
||||
"model": "Tongyi-MAI/Z-Image-Turbo",
|
||||
"source": "examples/inference/basic/basic_zimage.py",
|
||||
"command": "python examples/inference/basic/basic_zimage.py",
|
||||
"gpu_types": ["NVIDIA"],
|
||||
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
|
||||
"evidence": "Source-backed",
|
||||
"expected_artifact": "PNG image at outputs/zimage/zimage_turbo.png"
|
||||
},
|
||||
{
|
||||
"id": "sd35-medium-t2i",
|
||||
"family": "sd35",
|
||||
@@ -914,38 +820,6 @@
|
||||
],
|
||||
"limitations": ["Supply a compatible FastH3 adapter. The script infers dense or VSA attention from the adapter payload unless you override it."]
|
||||
},
|
||||
{
|
||||
"id": "longcat-t2v",
|
||||
"family": "longcat",
|
||||
"stage": "inference",
|
||||
"task": "Text to video",
|
||||
"label": "LongCat Video T2V",
|
||||
"summary": "LongCat Video text-to-video at 480p (50 steps), with distilled and 720p refinement passes included in the same maintained script.",
|
||||
"model": "FastVideo/LongCat-Video-T2V-Diffusers",
|
||||
"source": "examples/inference/basic/basic_longcat_t2v.py",
|
||||
"command": "python examples/inference/basic/basic_longcat_t2v.py",
|
||||
"gpu_types": ["NVIDIA"],
|
||||
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
|
||||
"evidence": "Source-backed",
|
||||
"expected_artifact": "MP4 videos under outputs_video/longcat_t2v_basic/, longcat_t2v_distill/, and longcat_t2v_refine_720p/",
|
||||
"related": ["longcat-i2v"]
|
||||
},
|
||||
{
|
||||
"id": "longcat-i2v",
|
||||
"family": "longcat",
|
||||
"stage": "inference",
|
||||
"task": "Image to video",
|
||||
"label": "LongCat Video I2V",
|
||||
"summary": "LongCat Video image-to-video with optional distilled and refinement passes, from its maintained example.",
|
||||
"model": "FastVideo/LongCat-Video-I2V-Diffusers",
|
||||
"source": "examples/inference/basic/basic_longcat_i2v.py",
|
||||
"command": "python examples/inference/basic/basic_longcat_i2v.py",
|
||||
"gpu_types": ["NVIDIA"],
|
||||
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
|
||||
"evidence": "Source-backed",
|
||||
"expected_artifact": "MP4 videos under outputs_video/longcat_i2v_basic/ and longcat_i2v_distill/",
|
||||
"related": ["longcat-t2v"]
|
||||
},
|
||||
{
|
||||
"id": "stable-audio-open-t2a",
|
||||
"family": "stable_audio",
|
||||
|
||||
@@ -9,18 +9,15 @@ plain typographic tile instead — that tile is a UI placeholder, not a logo.
|
||||
| --- | --- | --- |
|
||||
| `wan-ai.webp` | [Official Wan-AI Hugging Face organization avatar](https://huggingface.co/Wan-AI) | Wan family card and page header |
|
||||
| `ltx.webp` | [Official Lightricks Hugging Face organization avatar](https://huggingface.co/Lightricks) | LTX family card and page header |
|
||||
| `tencent-hunyuan.webp` | [Official Tencent Hunyuan Hugging Face organization avatar](https://huggingface.co/Tencent-Hunyuan) | Hunyuan and GameCraft cards, Hunyuan page header |
|
||||
| `tencent-hunyuan.webp` | [Official Tencent Hunyuan Hugging Face organization avatar](https://huggingface.co/Tencent-Hunyuan) | Hunyuan family card and page header |
|
||||
| `nvidia.webp` | [Official NVIDIA Hugging Face organization avatar](https://huggingface.co/nvidia) | Cosmos and GEN3C cards, Cosmos page header |
|
||||
| `kandinsky.webp` | [Official Kandinsky Lab Hugging Face organization avatar](https://huggingface.co/kandinskylab) | Kandinsky 5 and Kandinsky 6 cards and page headers |
|
||||
| `black-forest-labs.webp` | [Official Black Forest Labs Hugging Face organization avatar](https://huggingface.co/black-forest-labs) | FLUX family card and page header |
|
||||
| `minimax.webp` | [Official MiniMax Hugging Face organization avatar](https://huggingface.co/MiniMaxAI) | MiniMax H3 family card and page header |
|
||||
| `tongyi.webp` | [Official Tongyi MAI Hugging Face organization avatar](https://huggingface.co/Tongyi-MAI) | Z-Image family card and page header |
|
||||
| `zai.webp` | [Official Z.ai Hugging Face organization avatar](https://huggingface.co/zai-org) | GLM-Image family card and page header |
|
||||
| `stabilityai.webp` | [Official Stability AI Hugging Face organization avatar](https://huggingface.co/stabilityai) | Stable Diffusion and Stable Audio cards and page headers |
|
||||
| `meituan-longcat.webp` | [Official Meituan LongCat Hugging Face organization avatar](https://huggingface.co/meituan-longcat) | LongCat family card and page header |
|
||||
| `fastvideo.webp` | [Official FastVideo Hugging Face organization avatar](https://huggingface.co/FastVideo) | Matrix Game and MMAudio cards (converted weights published by this org), DreamX card |
|
||||
| `fastvideo.webp` | [Official FastVideo Hugging Face organization avatar](https://huggingface.co/FastVideo) | Matrix Game and MMAudio cards (converted weights published by this org) |
|
||||
|
||||
Typographic tiles (no vendored image): TurboDiffusion ("Turbo"), HY-World
|
||||
("HY"), LingBot ("LB"), MMAudio page header ("MMA"). These publishers have no
|
||||
Typographic tiles (no vendored image): HY-World ("HY"), LingBot ("LB"),
|
||||
MMAudio page header ("MMA"). These publishers have no
|
||||
single official mark appropriate for reuse in the catalog; add a licensed
|
||||
asset here if one becomes available.
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 5.1 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 6.0 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.4 KiB |
@@ -0,0 +1,14 @@
|
||||
# Contact and Maintainers
|
||||
|
||||
## Contact
|
||||
|
||||
- **Contact email:** wlsaidhi@gmail.com
|
||||
- **Bug reports and feature requests:** open a
|
||||
[GitHub issue](https://github.com/hao-ai-lab/FastVideo/issues).
|
||||
- **Documentation:** see the [Developer Guide](contributing/overview.md).
|
||||
|
||||
## Maintainers
|
||||
|
||||
FastVideo is an open-source project of the
|
||||
[hao-ai-lab](https://github.com/hao-ai-lab) GitHub organization. The source code
|
||||
lives in [hao-ai-lab/FastVideo](https://github.com/hao-ai-lab/FastVideo).
|
||||
@@ -247,9 +247,6 @@ upstream model, walks its `named_modules()`, and attaches hooks externally.
|
||||
This is the same pattern used in
|
||||
`tests/local_tests/transformers/_debug_magi_human_block_parity.py`.
|
||||
|
||||
The `add-model-08-trace` skill at `.agents/skills/add-model-08-trace/`
|
||||
provides a script template for this purpose.
|
||||
|
||||
## Future extensions (design only, not yet implemented)
|
||||
|
||||
### Extension 1: FX/Dynamo backend graph rewrite
|
||||
@@ -319,7 +316,6 @@ granularity isn't enough.
|
||||
- Env vars: `fastvideo/envs.py` (`FASTVIDEO_TRACE_ACTIVATIONS` and friends)
|
||||
- Pipeline integration: `fastvideo/pipelines/composed_pipeline_base.py`
|
||||
- Tests: `fastvideo/tests/hooks/test_activation_trace.py`
|
||||
- Companion skill (for ad-hoc port investigations): `.agents/skills/add-model-08-trace/`
|
||||
|
||||
## Changelog
|
||||
|
||||
|
||||
@@ -461,8 +461,11 @@ The reusable implementation lives in
|
||||
| Workflow | Trigger | Purpose |
|
||||
|---|---|---|
|
||||
| `community-issue-labeler.yml` | Issue opened or edited | Adds scope/platform labels from issue keywords. |
|
||||
| `community-welcome.yml` | First contribution | Posts a welcome comment for first-time contributors. |
|
||||
| `community-stale.yml` | Daily schedule | Marks and closes stale issues and PRs, with exemption labels. |
|
||||
|
||||
None of these automations post comments or messages on issues or pull
|
||||
requests; they only set labels. Issues and pull requests are never marked
|
||||
stale or closed automatically. Keep it that way: new automation must report
|
||||
through checks, commit statuses, or labels instead of PR/issue comments.
|
||||
|
||||
## File Reference
|
||||
|
||||
|
||||
@@ -405,6 +405,9 @@ Action:
|
||||
causal masks, and broadcast shapes).
|
||||
- **Scheduler / sigma schedule mismatch**: even small differences in schedules
|
||||
or timestep shapes can cause noticeable drift.
|
||||
- **Skipped parity tests**: local parity tests skip when the reference clone,
|
||||
weights, or a GPU are missing. A skip is not a pass; run them where the
|
||||
reference and weights are available before claiming parity.
|
||||
|
||||
## Diffusers vs manual conversion
|
||||
|
||||
|
||||
@@ -304,36 +304,44 @@ longer exists also fails the test, so the fixing pull request deletes its entry.
|
||||
| `FASTVIDEO_TEST_WAN22_5B_ALLOW_LOW_MEMORY` | bool | `0` | test | Run the MLX Wan2.2 5B real-weights parity test on hosts with little memory. Deprecated names: `FASTVIDEO_WAN22_5B_ALLOW_LOW_MEMORY`. |
|
||||
| `FASTVIDEO_TEST_WAN22_5B_ROOT` | str | unset | test | Local Wan2.2 5B checkpoint for the MLX real-weights parity test. Deprecated names: `FASTVIDEO_WAN22_5B_ROOT`. |
|
||||
| `FASTVIDEO_TEST_GRADNORM_UPDATE` | bool | `0` | test | Gradient-norm regression tests update their references. Deprecated names: `FASTVIDEO_GRADNORM_UPDATE`. |
|
||||
| `FASTVIDEO_TEST_DREAMX_WORLD_SSIM_MODEL_PATH` | str | `FastVideo/DreamX-World-5B-Cam-Diffusers` | test | Model for the DreamX-World camera SSIM test. Deprecated names: `DREAMX_WORLD_SSIM_MODEL_PATH`. |
|
||||
| `FASTVIDEO_TEST_DREAMX_WORLD_AR_SSIM_MODEL_PATH` | str | `FastVideo/DreamX-World-5B-Diffusers` | test | Model for the DreamX-World autoregressive SSIM test. Deprecated names: `DREAMX_WORLD_AR_SSIM_MODEL_PATH`. |
|
||||
| `FASTVIDEO_TEST_FLUX_T2I_MODEL_DIR` | str | `black-forest-labs/FLUX.1-dev` | test | Model for the Flux text-to-image SSIM test. Deprecated names: `FLUX_T2I_MODEL_DIR`. |
|
||||
| `FASTVIDEO_TEST_FLUX_TRANSFORMER_PATH` | str | unset | test | Local Flux transformer for the Flux transformer test. Deprecated names: `FLUX_TRANSFORMER_PATH`. |
|
||||
| `FASTVIDEO_TEST_GAMECRAFT_MODEL_PATH` | str | `FastVideo/HunyuanGameCraft-Diffusers` | test | Model for the HunyuanGameCraft SSIM test. Deprecated names: `GAMECRAFT_MODEL_PATH`. |
|
||||
| `FASTVIDEO_TEST_GEN3C_MODEL_PATH` | str | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | test | Model for the GEN3C SSIM test. Deprecated names: `GEN3C_MODEL_PATH`. |
|
||||
| `FASTVIDEO_TEST_GEN3C_IMAGE_PATH` | str | unset | test | Input image for the GEN3C SSIM test. Deprecated names: `GEN3C_TEST_IMAGE_PATH`. |
|
||||
| `FASTVIDEO_TEST_GLM_IMAGE_LOCAL_WEIGHTS_DIR` | str | unset | test | Local official GLM-Image weights for the GLM-Image SSIM test. Deprecated names: `GLM_IMAGE_LOCAL_WEIGHTS_DIR`. |
|
||||
| `FASTVIDEO_TEST_GLM_IMAGE_MODEL_DIR` | str | unset | test | Model for the GLM-Image SSIM test. Deprecated names: `GLM_IMAGE_MODEL_DIR`. |
|
||||
| `FASTVIDEO_TEST_KANDINSKY5_E2E_NUM_GPUS` | int | `1` | test | GPUs for the Kandinsky5 nightly end-to-end overfit test. Deprecated names: `KANDINSKY5_E2E_NUM_GPUS`. |
|
||||
| `FASTVIDEO_TEST_KANDINSKY5_E2E_WRITE_REFERENCE` | bool | `0` | test | The Kandinsky5 nightly end-to-end test writes a missing reference video. Deprecated names: `KANDINSKY5_E2E_WRITE_REFERENCE`. |
|
||||
| `FASTVIDEO_TEST_LONGCAT_MODEL_ROOT` | str | unset | test | Local LongCat-Video checkpoint for the golden-gate test. Deprecated names: `LONGCAT_MODEL_ROOT`. |
|
||||
| `FASTVIDEO_TEST_MINIMAX_H3_GATE_GOLDEN_DIR` | str | unset | test | Local directory of MiniMax-H3 golden-gate tensors. Deprecated names: `MINIMAX_H3_GATE_GOLDEN_DIR`. |
|
||||
| `FASTVIDEO_TEST_MINIMAX_H3_GATE_LAYER` | int | `0` | test | Transformer layer that the MiniMax-H3 golden-gate test checks. Deprecated names: `MINIMAX_H3_GATE_LAYER`. |
|
||||
| `FASTVIDEO_TEST_MINIMAX_H3_MODEL_ROOT` | str | unset | test | Local MiniMax-H3 checkpoint for the golden-gate test. Deprecated names: `MINIMAX_H3_MODEL_ROOT`. |
|
||||
| `FASTVIDEO_TEST_SD35_MODEL_DIR` | str | `stabilityai/stable-diffusion-3.5-medium` | test | Model for the Stable Diffusion 3.5 SSIM test. Deprecated names: `SD35_MODEL_DIR`. |
|
||||
| `FASTVIDEO_TEST_TAEH3_REFERENCE_DIR` | str | unset | test | Upstream taehv checkout for the MLX TAEH3 parity test. Deprecated names: `TAEH3_REFERENCE_DIR`. |
|
||||
| `FASTVIDEO_TEST_WAN_ANIMATE_MODEL_DIR` | str | unset | test | Local Wan2.2-Animate-14B checkpoint for the Wan-Animate weight tests. Deprecated names: `WAN_ANIMATE_MODEL_PATH`. |
|
||||
| `FASTVIDEO_TEST_ZIMAGE_MODEL_DIR` | str | `Tongyi-MAI/Z-Image-Turbo` | test | Model for the Z-Image SSIM test. Deprecated names: `ZIMAGE_MODEL_DIR`. |
|
||||
| `FASTVIDEO_TEST_ZIMAGE_MODEL_REVISION` | str | `f332072aa78be7aecdf3ee76d5c247082da564a6` | test | Hugging Face revision of the Z-Image model for its SSIM test. Deprecated names: `ZIMAGE_MODEL_REVISION`. |
|
||||
|
||||
Variables that FastVideo no longer reads; setting one logs a warning:
|
||||
|
||||
| Deprecated variable | Reason |
|
||||
| ----------------------------------------- | ---------------- |
|
||||
| `FASTVIDEO_TARGET_DEVICE` | no code reads it |
|
||||
| `FASTVIDEO_USE_PRECOMPILED` | no code reads it |
|
||||
| `FASTVIDEO_RINGBUFFER_WARNING_INTERVAL` | no code reads it |
|
||||
| `FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S` | no code reads it |
|
||||
| `FASTVIDEO_SERVER_DEV_MODE` | no code reads it |
|
||||
| `FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE` | no code reads it |
|
||||
| `FASTVIDEO_TRACE_FUNCTION` | no code reads it |
|
||||
| Deprecated variable | Reason |
|
||||
| ------------------------------------------------ | ---------------------------- |
|
||||
| `FASTVIDEO_TARGET_DEVICE` | no code reads it |
|
||||
| `FASTVIDEO_USE_PRECOMPILED` | no code reads it |
|
||||
| `FASTVIDEO_RINGBUFFER_WARNING_INTERVAL` | no code reads it |
|
||||
| `FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S` | no code reads it |
|
||||
| `FASTVIDEO_SERVER_DEV_MODE` | no code reads it |
|
||||
| `FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE` | no code reads it |
|
||||
| `FASTVIDEO_TRACE_FUNCTION` | no code reads it |
|
||||
| `FASTVIDEO_TEST_DREAMX_WORLD_SSIM_MODEL_PATH` | DreamX World was removed |
|
||||
| `DREAMX_WORLD_SSIM_MODEL_PATH` | DreamX World was removed |
|
||||
| `FASTVIDEO_TEST_DREAMX_WORLD_AR_SSIM_MODEL_PATH` | DreamX World was removed |
|
||||
| `DREAMX_WORLD_AR_SSIM_MODEL_PATH` | DreamX World was removed |
|
||||
| `FASTVIDEO_TEST_GAMECRAFT_MODEL_PATH` | HunyuanGameCraft was removed |
|
||||
| `GAMECRAFT_MODEL_PATH` | HunyuanGameCraft was removed |
|
||||
| `FASTVIDEO_TEST_GLM_IMAGE_LOCAL_WEIGHTS_DIR` | GLM-Image was removed |
|
||||
| `GLM_IMAGE_LOCAL_WEIGHTS_DIR` | GLM-Image was removed |
|
||||
| `FASTVIDEO_TEST_GLM_IMAGE_MODEL_DIR` | GLM-Image was removed |
|
||||
| `GLM_IMAGE_MODEL_DIR` | GLM-Image was removed |
|
||||
| `FASTVIDEO_TEST_LONGCAT_MODEL_ROOT` | LongCat was removed |
|
||||
| `LONGCAT_MODEL_ROOT` | LongCat was removed |
|
||||
| `FASTVIDEO_TEST_ZIMAGE_MODEL_DIR` | Z-Image was removed |
|
||||
| `ZIMAGE_MODEL_DIR` | Z-Image was removed |
|
||||
| `FASTVIDEO_TEST_ZIMAGE_MODEL_REVISION` | Z-Image was removed |
|
||||
| `ZIMAGE_MODEL_REVISION` | Z-Image was removed |
|
||||
<!-- END GENERATED ENV TABLE -->
|
||||
|
||||
@@ -21,7 +21,7 @@ Common examples:
|
||||
[docs] Add LoRA finetuning guide
|
||||
[new-model] Port HunyuanVideo 1.5 to FastVideo
|
||||
[infra] Add activation trace hooks for pipeline debugging
|
||||
[skill] Add add-model agent skill
|
||||
[skill] Add reseed-ssim-references agent skill
|
||||
```
|
||||
|
||||
Accepted tags are `feat`, `feature`, `bugfix`, `fix`, `refactor`, `perf`,
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Cosmos recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="cosmos" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="cosmos" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# FLUX recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="flux" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="flux" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
---
|
||||
hide:
|
||||
- toc
|
||||
---
|
||||
|
||||
# GLM-Image recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="glm_image" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
<span class="cookbook-family-header__logo">
|
||||
<img class="off-glb" src="../../assets/logos/zai.webp" alt="Z.ai" width="112" height="112">
|
||||
</span>
|
||||
<div>
|
||||
<p class="cookbook-eyebrow">Maintained family · Inference</p>
|
||||
<h2>GLM-Image inference recipes</h2>
|
||||
<p>GLM-Image from Z.ai supports both text-to-image generation and instruction-based image editing, each with a maintained example.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
|
||||
<span class="cookbook-lifecycle__stage cookbook-lifecycle__stage--active">Inference <small>live</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Distillation <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Fine-tuning <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Training <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Evaluation <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Optimization <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Deployment <small>planned</small></span>
|
||||
</div>
|
||||
</header>
|
||||
<nav class="cookbook-jumpnav" aria-label="Recipe page sections">
|
||||
<a href="#recipe-builder">Builder</a>
|
||||
<a href="#cookbook-setup">Setup</a>
|
||||
<a href="#cookbook-troubleshooting">Troubleshooting</a>
|
||||
<a href="#cookbook-evidence">Evidence</a>
|
||||
</nav>
|
||||
|
||||
<section class="cookbook-builder" id="recipe-builder" aria-labelledby="builder-heading">
|
||||
<div class="cookbook-builder__intro">
|
||||
<h2 id="builder-heading">Pick a recipe and runtime</h2>
|
||||
<p>Start with the result you want, then choose one of the runtimes FastVideo actually maintains for it.</p>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-builder__layout">
|
||||
<div class="cookbook-controls">
|
||||
<div class="cookbook-selection-row">
|
||||
<div class="cookbook-selection-row__label">
|
||||
<strong>Recipe</strong>
|
||||
<span>Task and checkpoint</span>
|
||||
</div>
|
||||
<div class="cookbook-option-grid cookbook-option-grid--models" data-cookbook-model-options role="group" aria-label="Recipe">
|
||||
<button type="button" disabled>Loading GLM-Image recipes...</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-selection-row">
|
||||
<div class="cookbook-selection-row__label">
|
||||
<strong>Runtime</strong>
|
||||
<span>Maintained paths only</span>
|
||||
</div>
|
||||
<div class="cookbook-option-grid cookbook-option-grid--hardware" data-cookbook-hardware-options role="group" aria-label="Runtime">
|
||||
<button type="button" disabled>Loading runtimes...</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p class="cookbook-selection-description" data-cookbook-description>Loading recipe details...</p>
|
||||
<p class="cookbook-hardware-note">Exact device and memory details appear only when a recorded run supports them.</p>
|
||||
|
||||
<div class="cookbook-hardware-state" data-cookbook-hardware-state role="status" aria-live="polite">
|
||||
Reading recipe evidence...
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<article class="cookbook-result">
|
||||
<div class="cookbook-result__header">
|
||||
<h3 data-cookbook-label>Loading...</h3>
|
||||
<div class="cookbook-result__badges">
|
||||
<span class="cookbook-badge">Maintained</span>
|
||||
<span class="cookbook-badge" data-cookbook-evidence>Source-backed</span>
|
||||
<span class="cookbook-badge cookbook-badge--neutral" data-cookbook-hardware-badge>Source config</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<dl class="cookbook-result__facts">
|
||||
<div><dt>Model</dt><dd data-cookbook-model>Loading...</dd></div>
|
||||
<div><dt>Workload</dt><dd data-cookbook-task>Loading...</dd></div>
|
||||
<div><dt>Source configuration</dt><dd data-cookbook-gpus>Loading...</dd></div>
|
||||
<div><dt>Expected output</dt><dd data-cookbook-artifact>Loading...</dd></div>
|
||||
</dl>
|
||||
|
||||
<div class="cookbook-command">
|
||||
<div class="cookbook-command__bar">
|
||||
<span>Terminal</span>
|
||||
</div>
|
||||
<pre><code class="language-bash" data-cookbook-command>Loading...</code></pre>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-result__footer">
|
||||
<a data-cookbook-source href="../../inference/examples/basic/">Open example source</a>
|
||||
<a data-cookbook-model-link href="https://huggingface.co/zai-org">View model card</a>
|
||||
</div>
|
||||
<p class="cookbook-picker__status" role="status" aria-live="polite" data-cookbook-status></p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<noscript>
|
||||
<div class="cookbook-noscript">
|
||||
JavaScript is needed for the guided selector. You can still browse the
|
||||
<a href="../../inference/examples/examples_inference_index/">maintained inference examples</a>.
|
||||
</div>
|
||||
</noscript>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-setup">
|
||||
<summary>Setup</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<p>The generated commands expect a local clone:</p>
|
||||
<pre><code>git clone https://github.com/hao-ai-lab/FastVideo.git
|
||||
cd FastVideo</code></pre>
|
||||
<p>Use <a href="../../inference/configuration/">Configuration</a> for supported Python and CLI settings, <a href="../../inference/optimizations/">Optimizations</a> for attention and memory tradeoffs, and the <a href="../../inference/support_matrix/">support matrix</a> for the supported model and optimization surface.</p>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-troubleshooting">
|
||||
<summary>Troubleshooting</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<ul>
|
||||
<li>The editing example reads <code>assets/images/couple.jpg</code> from the repository root, so run it from a repo checkout rather than an arbitrary working directory.</li>
|
||||
<li>Output paths default under <code>image_output/</code>; pass <code>--output</code> to change them.</li>
|
||||
</ul>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-evidence">
|
||||
<summary>Evidence status</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<p>All recipes on this page are <strong>Source-backed</strong>: their commands, model IDs, and flags were validated against the checked-in FastVideo sources listed above (static validation). No runtime GPU validation is recorded for these recipes, so GPU model fit, memory use, throughput, and runtime duration are <strong>Unknown</strong> and deliberately not claimed. Runtime buttons show only the GPU counts configured in checked-in sources.</p>
|
||||
</div>
|
||||
</details>
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Hunyuan recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="hunyuan" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="hunyuan" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -163,50 +163,6 @@ hide:
|
||||
</ul>
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./longcat/" aria-label="Open LongCat recipes">
|
||||
<span class="cookbook-family-tile__visual" data-evervault>
|
||||
<span class="cookbook-evervault" aria-hidden="true">
|
||||
<span class="cookbook-evervault__gradient"></span>
|
||||
<span class="cookbook-evervault__noise" data-cookbook-pattern></span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__logo-wrap">
|
||||
<img class="off-glb" src="../assets/logos/meituan-longcat.webp" alt="" width="132" height="132" loading="lazy">
|
||||
</span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__footer">
|
||||
<span class="cookbook-family-tile__footer-top">
|
||||
<span><strong>LongCat</strong><small>T2V, I2V, optional refine</small></span>
|
||||
<span class="cookbook-count">2 recipes</span>
|
||||
</span>
|
||||
<ul class="cookbook-mode-row">
|
||||
<li>T2V</li>
|
||||
<li>I2V</li>
|
||||
</ul>
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./turbodiffusion/" aria-label="Open TurboDiffusion recipes">
|
||||
<span class="cookbook-family-tile__visual" data-evervault>
|
||||
<span class="cookbook-evervault" aria-hidden="true">
|
||||
<span class="cookbook-evervault__gradient"></span>
|
||||
<span class="cookbook-evervault__noise" data-cookbook-pattern></span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__logo-wrap">
|
||||
<span class="cookbook-family-tile__monogram" aria-hidden="true">Turbo</span>
|
||||
</span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__footer">
|
||||
<span class="cookbook-family-tile__footer-top">
|
||||
<span><strong>TurboDiffusion</strong><small>Accelerated Wan profiles</small></span>
|
||||
<span class="cookbook-count">3 recipes</span>
|
||||
</span>
|
||||
<ul class="cookbook-mode-row">
|
||||
<li>T2V</li>
|
||||
<li>I2V</li>
|
||||
</ul>
|
||||
</span>
|
||||
</a>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -236,49 +192,6 @@ hide:
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./glm-image/" aria-label="Open GLM-Image recipes">
|
||||
<span class="cookbook-family-tile__visual" data-evervault>
|
||||
<span class="cookbook-evervault" aria-hidden="true">
|
||||
<span class="cookbook-evervault__gradient"></span>
|
||||
<span class="cookbook-evervault__noise" data-cookbook-pattern></span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__logo-wrap">
|
||||
<img class="off-glb" src="../assets/logos/zai.webp" alt="" width="132" height="132" loading="lazy">
|
||||
</span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__footer">
|
||||
<span class="cookbook-family-tile__footer-top">
|
||||
<span><strong>GLM-Image</strong><small>Generate and edit</small></span>
|
||||
<span class="cookbook-count">2 recipes</span>
|
||||
</span>
|
||||
<ul class="cookbook-mode-row">
|
||||
<li>T2I</li>
|
||||
<li>Edit</li>
|
||||
</ul>
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./z-image/" aria-label="Open Z-Image recipes">
|
||||
<span class="cookbook-family-tile__visual" data-evervault>
|
||||
<span class="cookbook-evervault" aria-hidden="true">
|
||||
<span class="cookbook-evervault__gradient"></span>
|
||||
<span class="cookbook-evervault__noise" data-cookbook-pattern></span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__logo-wrap">
|
||||
<img class="off-glb" src="../assets/logos/tongyi.webp" alt="" width="132" height="132" loading="lazy">
|
||||
</span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__footer">
|
||||
<span class="cookbook-family-tile__footer-top">
|
||||
<span><strong>Z-Image</strong><small>Turbo text to image</small></span>
|
||||
<span class="cookbook-count">1 recipe</span>
|
||||
</span>
|
||||
<ul class="cookbook-mode-row">
|
||||
<li>T2I</li>
|
||||
</ul>
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./stable-diffusion/" aria-label="Open Stable Diffusion recipes">
|
||||
<span class="cookbook-family-tile__visual" data-evervault>
|
||||
<span class="cookbook-evervault" aria-hidden="true">
|
||||
@@ -407,20 +320,6 @@ hide:
|
||||
<p>These families already have runnable examples. The cookbook page is not ready, so the cards are not links.</p>
|
||||
</div>
|
||||
<div class="cookbook-family-grid">
|
||||
<article class="cookbook-family-tile cookbook-family-tile--coming" aria-label="GameCraft cookbook page planned; runnable examples exist">
|
||||
<span class="cookbook-family-tile__visual">
|
||||
<span class="cookbook-family-tile__logo-wrap">
|
||||
<img class="off-glb" src="../assets/logos/tencent-hunyuan.webp" alt="" width="132" height="132" loading="lazy">
|
||||
</span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__footer">
|
||||
<span class="cookbook-family-tile__footer-top">
|
||||
<span><strong>GameCraft</strong><small>Game world generation</small></span>
|
||||
<span class="cookbook-count">Page planned</span>
|
||||
</span>
|
||||
</span>
|
||||
</article>
|
||||
|
||||
<article class="cookbook-family-tile cookbook-family-tile--coming" aria-label="GEN3C cookbook page planned; runnable examples exist">
|
||||
<span class="cookbook-family-tile__visual">
|
||||
<span class="cookbook-family-tile__logo-wrap">
|
||||
@@ -449,20 +348,6 @@ hide:
|
||||
</span>
|
||||
</article>
|
||||
|
||||
<article class="cookbook-family-tile cookbook-family-tile--coming" aria-label="DreamX cookbook page planned; runnable examples exist">
|
||||
<span class="cookbook-family-tile__visual">
|
||||
<span class="cookbook-family-tile__logo-wrap">
|
||||
<img class="off-glb" src="../assets/logos/fastvideo.webp" alt="" width="132" height="132" loading="lazy">
|
||||
</span>
|
||||
</span>
|
||||
<span class="cookbook-family-tile__footer">
|
||||
<span class="cookbook-family-tile__footer-top">
|
||||
<span><strong>DreamX</strong><small>World generation</small></span>
|
||||
<span class="cookbook-count">Page planned</span>
|
||||
</span>
|
||||
</span>
|
||||
</article>
|
||||
|
||||
<article class="cookbook-family-tile cookbook-family-tile--coming" aria-label="LingBot cookbook page planned; runnable examples exist">
|
||||
<span class="cookbook-family-tile__visual">
|
||||
<span class="cookbook-family-tile__logo-wrap">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Kandinsky 5 recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky5" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky5" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Kandinsky 6 recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky6" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky6" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
---
|
||||
hide:
|
||||
- toc
|
||||
---
|
||||
|
||||
# LongCat recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="longcat" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
<span class="cookbook-family-header__logo">
|
||||
<img class="off-glb" src="../../assets/logos/meituan-longcat.webp" alt="Meituan LongCat" width="112" height="112">
|
||||
</span>
|
||||
<div>
|
||||
<p class="cookbook-eyebrow">Maintained family · Inference</p>
|
||||
<h2>LongCat inference recipes</h2>
|
||||
<p>LongCat Video from Meituan covers text-to-video and image-to-video, and its maintained examples chain optional distilled and 720p refinement passes.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
|
||||
<span class="cookbook-lifecycle__stage cookbook-lifecycle__stage--active">Inference <small>live</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Distillation <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Fine-tuning <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Training <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Evaluation <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Optimization <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Deployment <small>planned</small></span>
|
||||
</div>
|
||||
</header>
|
||||
<nav class="cookbook-jumpnav" aria-label="Recipe page sections">
|
||||
<a href="#recipe-builder">Builder</a>
|
||||
<a href="#cookbook-setup">Setup</a>
|
||||
<a href="#cookbook-troubleshooting">Troubleshooting</a>
|
||||
<a href="#cookbook-evidence">Evidence</a>
|
||||
</nav>
|
||||
|
||||
<section class="cookbook-builder" id="recipe-builder" aria-labelledby="builder-heading">
|
||||
<div class="cookbook-builder__intro">
|
||||
<h2 id="builder-heading">Pick a recipe and runtime</h2>
|
||||
<p>Start with the result you want, then choose one of the runtimes FastVideo actually maintains for it.</p>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-builder__layout">
|
||||
<div class="cookbook-controls">
|
||||
<div class="cookbook-selection-row">
|
||||
<div class="cookbook-selection-row__label">
|
||||
<strong>Recipe</strong>
|
||||
<span>Task and checkpoint</span>
|
||||
</div>
|
||||
<div class="cookbook-option-grid cookbook-option-grid--models" data-cookbook-model-options role="group" aria-label="Recipe">
|
||||
<button type="button" disabled>Loading LongCat recipes...</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-selection-row">
|
||||
<div class="cookbook-selection-row__label">
|
||||
<strong>Runtime</strong>
|
||||
<span>Maintained paths only</span>
|
||||
</div>
|
||||
<div class="cookbook-option-grid cookbook-option-grid--hardware" data-cookbook-hardware-options role="group" aria-label="Runtime">
|
||||
<button type="button" disabled>Loading runtimes...</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p class="cookbook-selection-description" data-cookbook-description>Loading recipe details...</p>
|
||||
<p class="cookbook-hardware-note">Exact device and memory details appear only when a recorded run supports them.</p>
|
||||
|
||||
<div class="cookbook-hardware-state" data-cookbook-hardware-state role="status" aria-live="polite">
|
||||
Reading recipe evidence...
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<article class="cookbook-result">
|
||||
<div class="cookbook-result__header">
|
||||
<h3 data-cookbook-label>Loading...</h3>
|
||||
<div class="cookbook-result__badges">
|
||||
<span class="cookbook-badge">Maintained</span>
|
||||
<span class="cookbook-badge" data-cookbook-evidence>Source-backed</span>
|
||||
<span class="cookbook-badge cookbook-badge--neutral" data-cookbook-hardware-badge>Source config</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<dl class="cookbook-result__facts">
|
||||
<div><dt>Model</dt><dd data-cookbook-model>Loading...</dd></div>
|
||||
<div><dt>Workload</dt><dd data-cookbook-task>Loading...</dd></div>
|
||||
<div><dt>Source configuration</dt><dd data-cookbook-gpus>Loading...</dd></div>
|
||||
<div><dt>Expected output</dt><dd data-cookbook-artifact>Loading...</dd></div>
|
||||
</dl>
|
||||
|
||||
<div class="cookbook-command">
|
||||
<div class="cookbook-command__bar">
|
||||
<span>Terminal</span>
|
||||
</div>
|
||||
<pre><code class="language-bash" data-cookbook-command>Loading...</code></pre>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-result__footer">
|
||||
<a data-cookbook-source href="../../inference/examples/basic/">Open example source</a>
|
||||
<a data-cookbook-model-link href="https://huggingface.co/meituan-longcat">View model card</a>
|
||||
</div>
|
||||
<p class="cookbook-picker__status" role="status" aria-live="polite" data-cookbook-status></p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<noscript>
|
||||
<div class="cookbook-noscript">
|
||||
JavaScript is needed for the guided selector. You can still browse the
|
||||
<a href="../../inference/examples/examples_inference_index/">maintained inference examples</a>.
|
||||
</div>
|
||||
</noscript>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-setup">
|
||||
<summary>Setup</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<p>The generated commands expect a local clone:</p>
|
||||
<pre><code>git clone https://github.com/hao-ai-lab/FastVideo.git
|
||||
cd FastVideo</code></pre>
|
||||
<p>Use <a href="../../inference/configuration/">Configuration</a> for supported Python and CLI settings, <a href="../../inference/optimizations/">Optimizations</a> for attention and memory tradeoffs, and the <a href="../../inference/support_matrix/">support matrix</a> for the supported model and optimization surface.</p>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-troubleshooting">
|
||||
<summary>Troubleshooting</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<ul>
|
||||
<li>Each LongCat script runs multiple passes (basic, distilled, refine); total runtime scales accordingly, and every pass prints its own output directory.</li>
|
||||
<li>Out of memory: the sources already enable VAE and text-encoder CPU offload; further options are covered in <a href="../../inference/offloading/">Offloading</a>.</li>
|
||||
</ul>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-evidence">
|
||||
<summary>Evidence status</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<p>All recipes on this page are <strong>Source-backed</strong>: their commands, model IDs, and flags were validated against the checked-in FastVideo sources listed above (static validation). No runtime GPU validation is recorded for these recipes, so GPU model fit, memory use, throughput, and runtime duration are <strong>Unknown</strong> and deliberately not claimed. Runtime buttons show only the GPU counts configured in checked-in sources.</p>
|
||||
</div>
|
||||
</details>
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# LTX recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="ltx2" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="ltx2" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Matrix Game recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="matrixgame" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="matrixgame" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -18,7 +18,7 @@ eight-forward T2VA command](../getting_started/installation/mlx.md#pruned-eight-
|
||||
It reads `fastvideo_inference.json` for the trained schedule. The command
|
||||
uses native 832x480 resolution and all requested frames.
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="minimax_h3" data-default-recipe="fasth3-preview-cuda" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="minimax_h3" data-default-recipe="fasth3-preview-cuda" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# MMAudio recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="mmaudio" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="mmaudio" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Stable Audio recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="stable_audio" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="stable_audio" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Stable Diffusion recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="sd35" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="sd35" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
---
|
||||
hide:
|
||||
- toc
|
||||
---
|
||||
|
||||
# TurboDiffusion recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="turbodiffusion" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
<span class="cookbook-family-header__logo">
|
||||
<span class="cookbook-family-tile__monogram" aria-hidden="true">Turbo</span>
|
||||
</span>
|
||||
<div>
|
||||
<p class="cookbook-eyebrow">Maintained family · Inference</p>
|
||||
<h2>TurboDiffusion inference recipes</h2>
|
||||
<p>TurboDiffusion profiles accelerate Wan checkpoints with step-distilled sampling and the SLA attention backend. These recipes follow the registry's `turbodiffusion` model family.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
|
||||
<span class="cookbook-lifecycle__stage cookbook-lifecycle__stage--active">Inference <small>live</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Distillation <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Fine-tuning <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Training <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Evaluation <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Optimization <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Deployment <small>planned</small></span>
|
||||
</div>
|
||||
</header>
|
||||
<nav class="cookbook-jumpnav" aria-label="Recipe page sections">
|
||||
<a href="#recipe-builder">Builder</a>
|
||||
<a href="#cookbook-setup">Setup</a>
|
||||
<a href="#cookbook-troubleshooting">Troubleshooting</a>
|
||||
<a href="#cookbook-evidence">Evidence</a>
|
||||
</nav>
|
||||
|
||||
<section class="cookbook-builder" id="recipe-builder" aria-labelledby="builder-heading">
|
||||
<div class="cookbook-builder__intro">
|
||||
<h2 id="builder-heading">Pick a recipe and runtime</h2>
|
||||
<p>Start with the result you want, then choose one of the runtimes FastVideo actually maintains for it.</p>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-builder__layout">
|
||||
<div class="cookbook-controls">
|
||||
<div class="cookbook-selection-row">
|
||||
<div class="cookbook-selection-row__label">
|
||||
<strong>Recipe</strong>
|
||||
<span>Task and checkpoint</span>
|
||||
</div>
|
||||
<div class="cookbook-option-grid cookbook-option-grid--models" data-cookbook-model-options role="group" aria-label="Recipe">
|
||||
<button type="button" disabled>Loading TurboDiffusion recipes...</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-selection-row">
|
||||
<div class="cookbook-selection-row__label">
|
||||
<strong>Runtime</strong>
|
||||
<span>Maintained paths only</span>
|
||||
</div>
|
||||
<div class="cookbook-option-grid cookbook-option-grid--hardware" data-cookbook-hardware-options role="group" aria-label="Runtime">
|
||||
<button type="button" disabled>Loading runtimes...</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p class="cookbook-selection-description" data-cookbook-description>Loading recipe details...</p>
|
||||
<p class="cookbook-hardware-note">Exact device and memory details appear only when a recorded run supports them.</p>
|
||||
|
||||
<div class="cookbook-hardware-state" data-cookbook-hardware-state role="status" aria-live="polite">
|
||||
Reading recipe evidence...
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<article class="cookbook-result">
|
||||
<div class="cookbook-result__header">
|
||||
<h3 data-cookbook-label>Loading...</h3>
|
||||
<div class="cookbook-result__badges">
|
||||
<span class="cookbook-badge">Maintained</span>
|
||||
<span class="cookbook-badge" data-cookbook-evidence>Source-backed</span>
|
||||
<span class="cookbook-badge cookbook-badge--neutral" data-cookbook-hardware-badge>Source config</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<dl class="cookbook-result__facts">
|
||||
<div><dt>Model</dt><dd data-cookbook-model>Loading...</dd></div>
|
||||
<div><dt>Workload</dt><dd data-cookbook-task>Loading...</dd></div>
|
||||
<div><dt>Source configuration</dt><dd data-cookbook-gpus>Loading...</dd></div>
|
||||
<div><dt>Expected output</dt><dd data-cookbook-artifact>Loading...</dd></div>
|
||||
</dl>
|
||||
|
||||
<div class="cookbook-command">
|
||||
<div class="cookbook-command__bar">
|
||||
<span>Terminal</span>
|
||||
</div>
|
||||
<pre><code class="language-bash" data-cookbook-command>Loading...</code></pre>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-result__footer">
|
||||
<a data-cookbook-source href="../../inference/examples/basic/">Open example source</a>
|
||||
<a data-cookbook-model-link href="https://huggingface.co/loayrashid">View model card</a>
|
||||
</div>
|
||||
<p class="cookbook-picker__status" role="status" aria-live="polite" data-cookbook-status></p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<noscript>
|
||||
<div class="cookbook-noscript">
|
||||
JavaScript is needed for the guided selector. You can still browse the
|
||||
<a href="../../inference/examples/examples_inference_index/">maintained inference examples</a>.
|
||||
</div>
|
||||
</noscript>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-setup">
|
||||
<summary>Setup</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<p>The generated commands expect a local clone:</p>
|
||||
<pre><code>git clone https://github.com/hao-ai-lab/FastVideo.git
|
||||
cd FastVideo</code></pre>
|
||||
<p>Use <a href="../../inference/configuration/">Configuration</a> for supported Python and CLI settings, <a href="../../inference/optimizations/">Optimizations</a> for attention and memory tradeoffs, and the <a href="../../inference/support_matrix/">support matrix</a> for the supported model and optimization surface.</p>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-troubleshooting">
|
||||
<summary>Troubleshooting</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<ul>
|
||||
<li>TurboDiffusion paths load community-published <code>loayrashid/TurboWan*</code> checkpoints; availability is governed by those repos.</li>
|
||||
<li>The SLA attention backend used by the I2V recipe is selected inside the example source; do not combine it with another <code>FASTVIDEO_ATTENTION_BACKEND</code> override in the same shell.</li>
|
||||
</ul>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-evidence">
|
||||
<summary>Evidence status</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<p>All recipes on this page are <strong>Source-backed</strong>: their commands, model IDs, and flags were validated against the checked-in FastVideo sources listed above (static validation). No runtime GPU validation is recorded for these recipes, so GPU model fit, memory use, throughput, and runtime duration are <strong>Unknown</strong> and deliberately not claimed. Runtime buttons show only the GPU counts configured in checked-in sources.</p>
|
||||
</div>
|
||||
</details>
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Wan recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="wan" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="wan" data-recipes="../../assets/cookbook-recipes.json?v=16">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
---
|
||||
hide:
|
||||
- toc
|
||||
---
|
||||
|
||||
# Z-Image recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="zimage" data-recipes="../../assets/cookbook-recipes.json?v=15">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
<span class="cookbook-family-header__logo">
|
||||
<img class="off-glb" src="../../assets/logos/tongyi.webp" alt="Tongyi MAI" width="112" height="112">
|
||||
</span>
|
||||
<div>
|
||||
<p class="cookbook-eyebrow">Maintained family · Inference</p>
|
||||
<h2>Z-Image inference recipes</h2>
|
||||
<p>Z-Image Turbo from Tongyi MAI is a fast text-to-image model. The maintained example runs it on a single GPU.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
|
||||
<span class="cookbook-lifecycle__stage cookbook-lifecycle__stage--active">Inference <small>live</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Distillation <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Fine-tuning <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Training <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Evaluation <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Optimization <small>planned</small></span>
|
||||
<span class="cookbook-lifecycle__stage">Deployment <small>planned</small></span>
|
||||
</div>
|
||||
</header>
|
||||
<nav class="cookbook-jumpnav" aria-label="Recipe page sections">
|
||||
<a href="#recipe-builder">Builder</a>
|
||||
<a href="#cookbook-setup">Setup</a>
|
||||
<a href="#cookbook-troubleshooting">Troubleshooting</a>
|
||||
<a href="#cookbook-evidence">Evidence</a>
|
||||
</nav>
|
||||
|
||||
<section class="cookbook-builder" id="recipe-builder" aria-labelledby="builder-heading">
|
||||
<div class="cookbook-builder__intro">
|
||||
<h2 id="builder-heading">Pick a recipe and runtime</h2>
|
||||
<p>Start with the result you want, then choose one of the runtimes FastVideo actually maintains for it.</p>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-builder__layout">
|
||||
<div class="cookbook-controls">
|
||||
<div class="cookbook-selection-row">
|
||||
<div class="cookbook-selection-row__label">
|
||||
<strong>Recipe</strong>
|
||||
<span>Task and checkpoint</span>
|
||||
</div>
|
||||
<div class="cookbook-option-grid cookbook-option-grid--models" data-cookbook-model-options role="group" aria-label="Recipe">
|
||||
<button type="button" disabled>Loading Z-Image recipes...</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-selection-row">
|
||||
<div class="cookbook-selection-row__label">
|
||||
<strong>Runtime</strong>
|
||||
<span>Maintained paths only</span>
|
||||
</div>
|
||||
<div class="cookbook-option-grid cookbook-option-grid--hardware" data-cookbook-hardware-options role="group" aria-label="Runtime">
|
||||
<button type="button" disabled>Loading runtimes...</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p class="cookbook-selection-description" data-cookbook-description>Loading recipe details...</p>
|
||||
<p class="cookbook-hardware-note">Exact device and memory details appear only when a recorded run supports them.</p>
|
||||
|
||||
<div class="cookbook-hardware-state" data-cookbook-hardware-state role="status" aria-live="polite">
|
||||
Reading recipe evidence...
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<article class="cookbook-result">
|
||||
<div class="cookbook-result__header">
|
||||
<h3 data-cookbook-label>Loading...</h3>
|
||||
<div class="cookbook-result__badges">
|
||||
<span class="cookbook-badge">Maintained</span>
|
||||
<span class="cookbook-badge" data-cookbook-evidence>Source-backed</span>
|
||||
<span class="cookbook-badge cookbook-badge--neutral" data-cookbook-hardware-badge>Source config</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<dl class="cookbook-result__facts">
|
||||
<div><dt>Model</dt><dd data-cookbook-model>Loading...</dd></div>
|
||||
<div><dt>Workload</dt><dd data-cookbook-task>Loading...</dd></div>
|
||||
<div><dt>Source configuration</dt><dd data-cookbook-gpus>Loading...</dd></div>
|
||||
<div><dt>Expected output</dt><dd data-cookbook-artifact>Loading...</dd></div>
|
||||
</dl>
|
||||
|
||||
<div class="cookbook-command">
|
||||
<div class="cookbook-command__bar">
|
||||
<span>Terminal</span>
|
||||
</div>
|
||||
<pre><code class="language-bash" data-cookbook-command>Loading...</code></pre>
|
||||
</div>
|
||||
|
||||
<div class="cookbook-result__footer">
|
||||
<a data-cookbook-source href="../../inference/examples/basic/">Open example source</a>
|
||||
<a data-cookbook-model-link href="https://huggingface.co/Tongyi-MAI">View model card</a>
|
||||
</div>
|
||||
<p class="cookbook-picker__status" role="status" aria-live="polite" data-cookbook-status></p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<noscript>
|
||||
<div class="cookbook-noscript">
|
||||
JavaScript is needed for the guided selector. You can still browse the
|
||||
<a href="../../inference/examples/examples_inference_index/">maintained inference examples</a>.
|
||||
</div>
|
||||
</noscript>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-setup">
|
||||
<summary>Setup</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<p>The generated commands expect a local clone:</p>
|
||||
<pre><code>git clone https://github.com/hao-ai-lab/FastVideo.git
|
||||
cd FastVideo</code></pre>
|
||||
<p>Use <a href="../../inference/configuration/">Configuration</a> for supported Python and CLI settings, <a href="../../inference/optimizations/">Optimizations</a> for attention and memory tradeoffs, and the <a href="../../inference/support_matrix/">support matrix</a> for the supported model and optimization surface.</p>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-troubleshooting">
|
||||
<summary>Troubleshooting</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<ul>
|
||||
<li>Output defaults to <code>outputs/zimage/zimage_turbo.png</code>; pass <code>--output</code> to redirect.</li>
|
||||
<li>Gated or missing checkpoints: run <code>huggingface-cli login</code> and confirm you accepted the model's license on Hugging Face.</li>
|
||||
</ul>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<details class="cookbook-collapsible" id="cookbook-evidence">
|
||||
<summary>Evidence status</summary>
|
||||
<div class="cookbook-collapsible__body">
|
||||
<p>All recipes on this page are <strong>Source-backed</strong>: their commands, model IDs, and flags were validated against the checked-in FastVideo sources listed above (static validation). No runtime GPU validation is recorded for these recipes, so GPU model fit, memory use, throughput, and runtime duration are <strong>Unknown</strong> and deliberately not claimed. Runtime buttons show only the GPU counts configured in checked-in sources.</p>
|
||||
</div>
|
||||
</details>
|
||||
@@ -219,9 +219,6 @@ surfaces:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.gen3c.Gen3CConfig
|
||||
- fastvideo.configs.pipelines.gen3c.Gen3CInferenceConfig
|
||||
color_correction_strength:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.dreamx_world.DreamXWorld5BARPipelineConfig
|
||||
default_camera_rotation:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.gen3c.Gen3CConfig
|
||||
@@ -296,11 +293,6 @@ surfaces:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.lingbotworld.LingBotWorldI2V480PConfig
|
||||
- fastvideo.configs.pipelines.lingbotworld.Wan2_2_I2V_A14B_Config
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionI2VConfig
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionI2V_A14B_Config
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2VConfig
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_14B_Config
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_1_3B_Config
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_1_T2V_480P_Config
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_2_TI2V_5B_Config
|
||||
- fastvideo.configs.pipelines.matrixgame2.MatrixGame2BaseI2V480PConfig
|
||||
@@ -319,11 +311,6 @@ surfaces:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.lingbotworld.LingBotWorldI2V480PConfig
|
||||
- fastvideo.configs.pipelines.lingbotworld.Wan2_2_I2V_A14B_Config
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionI2VConfig
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionI2V_A14B_Config
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2VConfig
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_14B_Config
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_1_3B_Config
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_1_T2V_480P_Config
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_2_TI2V_5B_Config
|
||||
- fastvideo.configs.pipelines.matrixgame2.MatrixGame2BaseI2V480PConfig
|
||||
@@ -338,48 +325,21 @@ surfaces:
|
||||
- fastvideo.configs.pipelines.wan.WanI2V720PConfig
|
||||
- fastvideo.configs.pipelines.wan.WanT2V480PConfig
|
||||
- fastvideo.configs.pipelines.wan.WanT2V720PConfig
|
||||
bsa_cdf_threshold:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
bsa_chunk_k:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
bsa_chunk_q:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
bsa_params:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
bsa_sparsity:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
enable_bsa:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
enable_kv_cache:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
enhance_hf:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
offload_kv_cache:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
t_thresh:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
use_distill:
|
||||
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
|
||||
scheduler_arch:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.sd35.SD35Config
|
||||
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
|
||||
text_encoder_archs:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.sd35.SD35Config
|
||||
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
|
||||
tokenizer_archs:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.sd35.SD35Config
|
||||
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
|
||||
transformer_arch:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.sd35.SD35Config
|
||||
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
|
||||
vae_arch:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.sd35.SD35Config
|
||||
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
|
||||
expand_timesteps:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_2_TI2V_5B_Config
|
||||
@@ -494,8 +454,6 @@ surfaces:
|
||||
image_conditioning: "MagiHuman preset variant marker for reference-image conditioning."
|
||||
pdd_step_indices: "MiniMax-H3 PDD fine-grid partition; resolve_checkpoint_settings sets it from fastvideo_inference.json."
|
||||
ref_audio_offset: "MagiHuman internal data-proxy audio alignment offset."
|
||||
scheduler_sigma_min: "Z-Image scheduler parity invariant; not part of the public typed inference API."
|
||||
scheduler_use_reference_discrete_timesteps: "Z-Image scheduler parity invariant; not part of the public typed inference API."
|
||||
sr_local_attn_layers: "MagiHuman SR internal sparse-attention layer selection."
|
||||
text_offset: "MagiHuman internal data-proxy text alignment offset."
|
||||
vae_stride: "MagiHuman internal VAE/data-proxy stride setting."
|
||||
@@ -527,8 +485,6 @@ surfaces:
|
||||
pose: request.inputs.pose
|
||||
c2ws_plucker_emb: request.inputs.c2ws_plucker_emb
|
||||
action_path: request.inputs.action_path
|
||||
refine_from: request.inputs.refine_from
|
||||
stage1_video: request.inputs.stage1_video
|
||||
prompt: request.prompt
|
||||
negative_prompt: request.negative_prompt
|
||||
prompt_path: request.inputs.prompt_path
|
||||
@@ -548,8 +504,6 @@ surfaces:
|
||||
guidance_scale: request.sampling.guidance_scale
|
||||
batch_cfg: request.sampling.batch_cfg
|
||||
guidance_scale_2: request.sampling.guidance_scale_2
|
||||
cfg_normalization: request.sampling.cfg_normalization
|
||||
cfg_truncation: request.sampling.cfg_truncation
|
||||
guidance_rescale: request.sampling.guidance_rescale
|
||||
use_embedded_guidance: request.sampling.use_embedded_guidance
|
||||
true_cfg_scale: request.sampling.true_cfg_scale
|
||||
@@ -573,19 +527,12 @@ surfaces:
|
||||
return_continuation_state: request.output.return_state
|
||||
preset_owned:
|
||||
t_thresh: request.stage_overrides.refine.t_thresh
|
||||
spatial_refine_only: request.stage_overrides.refine.spatial_refine_only
|
||||
num_cond_frames: request.stage_overrides.refine.num_cond_frames
|
||||
trajectory_type: request.extensions.gen3c.trajectory_type
|
||||
movement_distance: request.extensions.gen3c.movement_distance
|
||||
camera_rotation: request.extensions.gen3c.camera_rotation
|
||||
prompt_attention_mask: request.extensions.hyworld.prompt_attention_mask
|
||||
negative_attention_mask: request.extensions.hyworld.negative_attention_mask
|
||||
camera_states: request.extensions.hunyuangamecraft.camera_states
|
||||
camera_trajectory: request.extensions.hunyuangamecraft.camera_trajectory
|
||||
action_list: request.extensions.hunyuangamecraft.action_list
|
||||
action_speed_list: request.extensions.hunyuangamecraft.action_speed_list
|
||||
gt_latents: request.extensions.hunyuangamecraft.gt_latents
|
||||
conditioning_mask: request.extensions.hunyuangamecraft.conditioning_mask
|
||||
ltx2_cfg_scale_video: request.extensions.ltx2.cfg_scale_video
|
||||
ltx2_cfg_scale_audio: request.extensions.ltx2.cfg_scale_audio
|
||||
ltx2_modality_scale_video: request.extensions.ltx2.modality_scale_video
|
||||
|
||||
@@ -38,18 +38,14 @@ COOKBOOK_SOURCE_ROOTS = (
|
||||
# (e.g. black-forest-labs/FLUX.1-dev) and are grouped for documentation only.
|
||||
COOKBOOK_FAMILIES = {
|
||||
"wan",
|
||||
"turbodiffusion",
|
||||
"ltx2",
|
||||
"hunyuan",
|
||||
"cosmos",
|
||||
"kandinsky5",
|
||||
"kandinsky6",
|
||||
"flux",
|
||||
"glm_image",
|
||||
"zimage",
|
||||
"sd35",
|
||||
"minimax_h3",
|
||||
"longcat",
|
||||
"stable_audio",
|
||||
"mmaudio",
|
||||
"matrixgame",
|
||||
|
||||
@@ -232,8 +232,7 @@ Dataclass carrying all pipeline state between stages. Key field groups:
|
||||
- **Scheduler**: `timesteps`, `num_inference_steps`, `guidance_scale`,
|
||||
`sigmas`.
|
||||
- **Task-specific**: `mouse_cond`/`keyboard_cond` (Matrix-Game 2.0), `pose`
|
||||
(HYWorld), `camera_states` (GameCraft), `c2ws_plucker_emb`
|
||||
(LingBotWorld).
|
||||
(HYWorld), `c2ws_plucker_emb` (LingBotWorld).
|
||||
- **Output**: `output: Tensor | None`.
|
||||
- **Logging**: `logging_info: PipelineLoggingInfo`.
|
||||
|
||||
@@ -256,7 +255,6 @@ Standard stages (typical execution order):
|
||||
| `DecodingStage` | `stages/decoding.py` | Decodes latents to video via VAE |
|
||||
|
||||
Specialized variants: `CausalDenoisingStage`, `LTX2DenoisingStage`,
|
||||
`LongCatDenoisingStage`, `GameCraftDenoisingStage`,
|
||||
`HYWorldDenoisingStage`, `MatrixGame2CausalDenoisingStage`,
|
||||
`SRDenoisingStage`, `LTX2AudioDecodingStage`, `SD35ConditioningStage`,
|
||||
`LTX2TextEncodingStage`, `LTX2LatentPreparationStage`.
|
||||
|
||||
@@ -168,8 +168,8 @@ and logs a warning if the flag is set.
|
||||
|
||||
Deferral is opt-in per pipeline. Releasing a component and loading it again is
|
||||
only safe when nothing outside the loader has changed it, and two common habits
|
||||
break that without raising: mutating a component after load, as LongCat does
|
||||
when it enables block-sparse attention, and reading a component's attributes
|
||||
break that without raising: mutating a component after load, such as enabling
|
||||
a sparse-attention mode in `initialize_pipeline`, and reading a component's attributes
|
||||
while stages are built, as the shared denoising stage does to pick an attention
|
||||
backend. A pipeline therefore lists the components it has checked in
|
||||
`_lazy_module_names`, which is empty in the base class. MiniMax-H3 opts in. On
|
||||
|
||||
@@ -31,14 +31,10 @@ column links a runnable script in `examples/inference/basic/` where one exists.
|
||||
| cosmos | `nvidia/Cosmos-Predict2-2B-Video2World` | T2V | — |
|
||||
| cosmos25 | `KyleShao/Cosmos-Predict2.5-2B-Diffusers` | T2V | [basic_cosmos2_5_t2w.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_cosmos2_5_t2w.py) |
|
||||
| cosmos25 | `nvidia/Cosmos-Predict2.5-14B` | T2V | [basic_cosmos2_5_t2w.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_cosmos2_5_t2w.py) |
|
||||
| dreamx_world | `FastVideo/DreamX-World-5B-Cam-Diffusers` | I2V | [basic_dreamx_world.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_dreamx_world.py) |
|
||||
| dreamx_world | `FastVideo/DreamX-World-5B-Diffusers` | I2V | [basic_dreamx_world.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_dreamx_world.py) |
|
||||
| flux | `black-forest-labs/FLUX.1-dev` | T2I | [basic_flux_dev.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_flux_dev.py) |
|
||||
| flux2 | `black-forest-labs/FLUX.2-klein-4B`<br>`black-forest-labs/FLUX.2-klein-9B` | T2I | [basic_flux2_klein.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_flux2_klein.py) |
|
||||
| flux2 | `black-forest-labs/FLUX.2-dev` | T2I | [basic_flux2.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_flux2.py) |
|
||||
| gamecraft | `FastVideo/HunyuanGameCraft-Diffusers` | I2V | [basic_gamecraft.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_gamecraft.py) |
|
||||
| gen3c | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | T2V | [basic_gen3c.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_gen3c.py) |
|
||||
| glm_image | `zai-org/GLM-Image` | T2I | [basic_glm_image.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_glm_image.py) |
|
||||
| hunyuan | `hunyuanvideo-community/HunyuanVideo` | T2V | — |
|
||||
| hunyuan | `FastVideo/FastHunyuan-diffusers` | T2V | — |
|
||||
| hunyuan15 | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | T2V | [basic_hy15.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_hy15.py) |
|
||||
@@ -62,9 +58,6 @@ column links a runnable script in `examples/inference/basic/` where one exists.
|
||||
| lingbotworld | `FastVideo/LingBot-World-Base-Cam-Diffusers` | I2V | [basic_lingbotworld_base_cam.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_lingbotworld_base_cam.py) |
|
||||
| lingbotworld2 | `robbyant/lingbot-world-v2-14b-causal-fast` | I2V | [basic_lingbotworld2_causal_fast.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_lingbotworld2_causal_fast.py) |
|
||||
| lingbotworld_fast | `FastVideo/LingBot-World-Fast-Diffusers` | I2V | [basic_lingbotworld_fast.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_lingbotworld_fast.py) |
|
||||
| longcat | `FastVideo/LongCat-Video-T2V-Diffusers` | T2V | [basic_longcat_t2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_longcat_t2v.py) |
|
||||
| longcat | `FastVideo/LongCat-Video-I2V-Diffusers` | I2V | [basic_longcat_i2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_longcat_i2v.py) |
|
||||
| longcat | `FastVideo/LongCat-Video-VC-Diffusers` | — | [basic_longcat_vc.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_longcat_vc.py) |
|
||||
| ltx2 | `FastVideo/LTX2-Distilled-Diffusers`<br>`FastVideo/LTX2.3-Distilled-Diffusers`<br>`FastVideo/LTX-2.3-Distilled-Diffusers` | T2V | [basic_ltx2_distilled.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_distilled.py) |
|
||||
| ltx2 | `FastVideo/LTX-2.5-Distilled-Diffusers` | T2V, I2V | [T2AV](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_5_t2av.py)<br>[I2AV](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_5_i2av.py) |
|
||||
| ltx2 | `FastVideo/LTX-2.5-Dev-Diffusers` | T2V, I2V | [T2AV](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_5_t2av.py)<br>[I2AV](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_5_i2av.py) |
|
||||
@@ -77,9 +70,6 @@ column links a runnable script in `examples/inference/basic/` where one exists.
|
||||
| sd35 | `stabilityai/stable-diffusion-3.5-medium` | T2I | [basic_sd35_t2i.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_sd35_t2i.py) |
|
||||
| stable_audio | `FastVideo/stable-audio-open-1.0-Diffusers` | T2V | [basic_stable_audio.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_stable_audio.py) |
|
||||
| stable_audio | `FastVideo/stable-audio-open-small-Diffusers` | T2V | [basic_stable_audio_small.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_stable_audio_small.py) |
|
||||
| turbodiffusion | `loayrashid/TurboWan2.1-T2V-1.3B-Diffusers` | T2V | [basic_turbodiffusion.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_turbodiffusion.py) |
|
||||
| turbodiffusion | `loayrashid/TurboWan2.1-T2V-14B-Diffusers` | T2V | [basic_turbodiffusion_14b.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_turbodiffusion_14b.py) |
|
||||
| turbodiffusion | `loayrashid/TurboWan2.2-I2V-A14B-Diffusers` | I2V | [basic_turbodiffusion_i2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_turbodiffusion_i2v.py) |
|
||||
| wan | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | T2V | [basic.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic.py) |
|
||||
| wan | `Wan-AI/Wan2.1-T2V-14B-Diffusers`<br>`FastVideo/Wan2.1-VSA-T2V-14B-720P-Diffusers` | T2V | — |
|
||||
| wan | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | I2V | — |
|
||||
@@ -97,7 +87,6 @@ column links a runnable script in `examples/inference/basic/` where one exists.
|
||||
| wan | `wlsaidhi/SFWan2.1-T2V-1.3B-Diffusers` | T2V | [basic_self_forcing_causal.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_self_forcing_causal.py) |
|
||||
| wan | `rand0nmr/SFWan2.2-T2V-A14B-Diffusers` | T2V | [basic_self_forcing_causal_wan2_2_t2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_self_forcing_causal_wan2_2_t2v.py) |
|
||||
| wan | `FastVideo/SFWan2.2-I2V-A14B-Preview-Diffusers` | I2V | [basic_self_forcing_causal_wan2_2_i2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_self_forcing_causal_wan2_2_i2v.py) |
|
||||
| zimage | `Tongyi-MAI/Z-Image-Turbo` | T2I | [basic_zimage.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_zimage.py) |
|
||||
|
||||
**Note (stable_audio)**: the Stable Audio Open pipelines generate audio
|
||||
(`StableAudioT2AConfig` / `StableAudioOpenSmallConfig`); they are registered
|
||||
@@ -184,33 +173,27 @@ optimizations: absence means **untested**, not incompatible.
|
||||
}
|
||||
</style>
|
||||
|
||||
| Model Name | HuggingFace Model ID | Resolutions | TeaCache | Sliding Tile Attn (Legacy Branch) | Sage Attn | VSA | BSA |
|
||||
|------------|---------------------|-------------|----------|-------------------|-----------|-----|-----|
|
||||
| FastWan2.1 T2V 1.3B | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ✅ | ⭕ |
|
||||
| FastWan2.2 TI2V 5B Full Attn | `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | 720P | ⭕ | ⭕ | ⭕ | ✅ | ⭕ |
|
||||
| Wan2.2 TI2V 5B | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | 720P | ⭕ | ⭕ | ✅ | ⭕ | ⭕ |
|
||||
| DreamX-World 5B Cam | `FastVideo/DreamX-World-5B-Cam-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| DreamX-World 5B AR | `FastVideo/DreamX-World-5B-Diffusers` | 704px1280p | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Lucy Edit Dev 5B*** | `decart-ai/Lucy-Edit-Dev` | 480P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Wan2.2 S2V 14B**** | `FastVideo/Wan2.2-S2V-14B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Wan2.2 T2V A14B | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 480P<br>720P | ❌ | ❌ | ✅ | ⭕ | ⭕ |
|
||||
| Wan2.2 I2V A14B | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | 480P<br>720P | ❌ | ❌ | ✅ | ⭕ | ⭕ |
|
||||
| Wan2.2 Animate 14B | `Wan-AI/Wan2.2-Animate-14B-Diffusers` | 720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| HunyuanVideo | `hunyuanvideo-community/HunyuanVideo` | 720px1280p<br>544px960p | ❌ | ✅ | ✅ | ⭕ | ⭕ |
|
||||
| FastHunyuan | `FastVideo/FastHunyuan-diffusers` | 720px1280p<br>544px960p | ❌ | ✅ | ✅ | ⭕ | ⭕ |
|
||||
| Wan2.1 T2V 1.3B | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | 480P | ✅ | ✅ | ✅ | ⭕ | ⭕ |
|
||||
| Wan2.1 T2V 14B | `Wan-AI/Wan2.1-T2V-14B-Diffusers` | 480P, 720P | ✅ | ✅ | ✅ | ⭕ | ⭕ |
|
||||
| Wan2.1 I2V 480P | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | 480P | ✅ | ✅ | ✅ | ⭕ | ⭕ |
|
||||
| Wan2.1 I2V 720P | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers` | 720P | ✅ | ✅ | ✅ | ⭕ | ⭕ |
|
||||
| TurboWan2.1 T2V 1.3B | `loayrashid/TurboWan2.1-T2V-1.3B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| TurboWan2.1 T2V 14B | `loayrashid/TurboWan2.1-T2V-14B-Diffusers` | 480P, 720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| TurboWan2.2 I2V A14B | `loayrashid/TurboWan2.2-I2V-A14B-Diffusers` | 480P<br>720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| LongCat T2V 13.6B | `FastVideo/LongCat-Video-T2V-Diffusers` | 480P<br>720P | ❌ | ❌ | ❌ | ⭕ | ✅ |
|
||||
| Matrix Game 2.0 Base Distilled | `FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 2.0 GTA Distilled | `FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 2.0 TempleRun Distilled | `FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 3.0 Base Distilled | `FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers` | 720x1280 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| GEN3C Cosmos 7B | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | 704px1280p | ❌ | ❌ | ❌ | ⭕ | ⭕ |
|
||||
| Model Name | HuggingFace Model ID | Resolutions | TeaCache | Sliding Tile Attn (Legacy Branch) | Sage Attn | VSA |
|
||||
|------------|---------------------|-------------|----------|-------------------|-----------|-----|
|
||||
| FastWan2.1 T2V 1.3B | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ✅ |
|
||||
| FastWan2.2 TI2V 5B Full Attn | `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | 720P | ⭕ | ⭕ | ⭕ | ✅ |
|
||||
| Wan2.2 TI2V 5B | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | 720P | ⭕ | ⭕ | ✅ | ⭕ |
|
||||
| Lucy Edit Dev 5B*** | `decart-ai/Lucy-Edit-Dev` | 480P | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Wan2.2 S2V 14B**** | `FastVideo/Wan2.2-S2V-14B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Wan2.2 T2V A14B | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 480P<br>720P | ❌ | ❌ | ✅ | ⭕ |
|
||||
| Wan2.2 I2V A14B | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | 480P<br>720P | ❌ | ❌ | ✅ | ⭕ |
|
||||
| Wan2.2 Animate 14B | `Wan-AI/Wan2.2-Animate-14B-Diffusers` | 720P | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| HunyuanVideo | `hunyuanvideo-community/HunyuanVideo` | 720px1280p<br>544px960p | ❌ | ✅ | ✅ | ⭕ |
|
||||
| FastHunyuan | `FastVideo/FastHunyuan-diffusers` | 720px1280p<br>544px960p | ❌ | ✅ | ✅ | ⭕ |
|
||||
| Wan2.1 T2V 1.3B | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | 480P | ✅ | ✅ | ✅ | ⭕ |
|
||||
| Wan2.1 T2V 14B | `Wan-AI/Wan2.1-T2V-14B-Diffusers` | 480P, 720P | ✅ | ✅ | ✅ | ⭕ |
|
||||
| Wan2.1 I2V 480P | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | 480P | ✅ | ✅ | ✅ | ⭕ |
|
||||
| Wan2.1 I2V 720P | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers` | 720P | ✅ | ✅ | ✅ | ⭕ |
|
||||
| Matrix Game 2.0 Base Distilled | `FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 2.0 GTA Distilled | `FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 2.0 TempleRun Distilled | `FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 3.0 Base Distilled | `FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers` | 720x1280 | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| GEN3C Cosmos 7B | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | 704px1280p | ❌ | ❌ | ❌ | ⭕ |
|
||||
|
||||
## Apple Silicon native runtime
|
||||
|
||||
@@ -284,11 +267,6 @@ listed under [Special requirements](#special-requirements).
|
||||
https://github.com/hao-ai-lab/FastVideo/tree/sta_do_not_delete
|
||||
- STA currently requires Hopper GPUs (H100s).
|
||||
|
||||
### TurboWan2.1 (TurboDiffusion)
|
||||
- Uses TurboDiffusionPipeline with RCM scheduler for 1-4 step generation
|
||||
- Requires SLA attention backend: `export FASTVIDEO_ATTENTION_BACKEND=SLA_ATTN`
|
||||
- Uses `guidance_scale=1.0` (no classifier-free guidance)
|
||||
|
||||
### Matrix Game 2.0
|
||||
- Image-to-video game world models with keyboard/mouse control input
|
||||
- Three variants available: Base (universal), GTA, and TempleRun
|
||||
|
||||
@@ -1,61 +0,0 @@
|
||||
import os
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
OUTPUT_PATH = os.getenv("DREAMX_WORLD_OUTPUT_PATH", "video_samples_dreamx_world")
|
||||
|
||||
|
||||
def _env_int(name: str, default: int) -> int:
|
||||
return int(os.getenv(name, str(default)))
|
||||
|
||||
|
||||
def _env_float(name: str, default: float) -> float:
|
||||
return float(os.getenv(name, str(default)))
|
||||
|
||||
|
||||
def main():
|
||||
model_name = os.getenv("DREAMX_WORLD_MODEL_DIR", "FastVideo/DreamX-World-5B-Cam-Diffusers")
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
model_name,
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False,
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
override_pipeline_cls_name="DreamXWorldPipeline",
|
||||
)
|
||||
|
||||
prompt = os.getenv(
|
||||
"DREAMX_WORLD_PROMPT",
|
||||
"A cinematic first-person drive through a futuristic coastal city at "
|
||||
"sunrise, reflective glass towers, clean streets, soft volumetric light.",
|
||||
)
|
||||
image_path = os.getenv(
|
||||
"DREAMX_WORLD_IMAGE_PATH",
|
||||
"https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/wan_i2v_input.JPG",
|
||||
)
|
||||
|
||||
kwargs = {
|
||||
"output_path": OUTPUT_PATH,
|
||||
"save_video": os.getenv("DREAMX_WORLD_SAVE_VIDEO", "1") != "0",
|
||||
"height": _env_int("DREAMX_WORLD_HEIGHT", 480),
|
||||
"width": _env_int("DREAMX_WORLD_WIDTH", 832),
|
||||
"num_frames": _env_int("DREAMX_WORLD_NUM_FRAMES", 161),
|
||||
"num_inference_steps": _env_int("DREAMX_WORLD_STEPS", 30),
|
||||
"guidance_scale": _env_float("DREAMX_WORLD_GUIDANCE", 5.0),
|
||||
"action_list": os.getenv("DREAMX_WORLD_ACTIONS", "w,d,w").split(","),
|
||||
"action_speed_list":
|
||||
[float(value) for value in os.getenv("DREAMX_WORLD_ACTION_SPEEDS", "4.0,2.0,4.0").split(",")],
|
||||
}
|
||||
if image_path:
|
||||
kwargs["image_path"] = image_path
|
||||
|
||||
try:
|
||||
generator.generate_video(prompt, **kwargs)
|
||||
finally:
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,118 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""
|
||||
Basic inference script for HunyuanGameCraft video generation.
|
||||
|
||||
HunyuanGameCraft generates game-like videos with camera/action control.
|
||||
It takes an optional image input and generates video with camera motion
|
||||
based on simple action commands (forward, left, right, backward, rotations).
|
||||
|
||||
Available actions:
|
||||
- forward (w): Move camera forward
|
||||
- backward (s): Move camera backward
|
||||
- left (a): Move camera left (strafe)
|
||||
- right (d): Move camera right (strafe)
|
||||
- left_rot: Rotate camera left (pan)
|
||||
- right_rot: Rotate camera right (pan)
|
||||
- up_rot: Rotate camera up (tilt)
|
||||
- down_rot: Rotate camera down (tilt)
|
||||
|
||||
T2V vs I2V:
|
||||
- Default: I2V (uses a default reference image). Set GAMECRAFT_I2V_IMAGE to a
|
||||
URL or path to use a different image.
|
||||
- T2V only (no reference image): run with GAMECRAFT_I2V_IMAGE= (empty).
|
||||
"""
|
||||
import os
|
||||
|
||||
import torch
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.models.camera import create_camera_trajectory
|
||||
|
||||
# Model configuration (use GAMECRAFT_MODEL_PATH for local weights)
|
||||
MODEL_PATH = os.environ.get("GAMECRAFT_MODEL_PATH", "FastVideo/HunyuanGameCraft-Diffusers")
|
||||
|
||||
# Default prompts for demo
|
||||
DEFAULT_PROMPTS = {
|
||||
"village":
|
||||
"A charming medieval village with cobblestone streets, thatched-roof houses, and vibrant flower gardens under a bright blue sky.",
|
||||
"temple":
|
||||
"A majestic ancient temple stands under a clear blue sky, its grandeur highlighted by towering Doric columns and intricate architectural details.",
|
||||
"forest":
|
||||
"A lush green forest with tall trees, dappled sunlight filtering through the leaves, and a winding dirt path.",
|
||||
"beach": "A tropical beach with crystal clear turquoise water, white sand, and palm trees swaying in the breeze.",
|
||||
}
|
||||
|
||||
# I2V: default reference image (URL). Can override with a local path.
|
||||
DEFAULT_I2V_IMAGE_URL = ("https://huggingface.co/datasets/huggingface/documentation-images/"
|
||||
"resolve/main/diffusers/astronaut.jpg")
|
||||
DEFAULT_I2V_PROMPT = ("An astronaut hatching from an egg, on the surface of the moon, "
|
||||
"the darkness and depth of space realised in the background.")
|
||||
|
||||
OUTPUT_PATH = "video_samples_gamecraft"
|
||||
|
||||
|
||||
def main():
|
||||
# Initialize generator
|
||||
# FastVideo will automatically download weights from HuggingFace
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
MODEL_PATH,
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=True,
|
||||
dit_cpu_offload=True,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=True,
|
||||
)
|
||||
|
||||
# Video parameters
|
||||
height = 704
|
||||
width = 1280
|
||||
num_frames = 33
|
||||
action = "forward"
|
||||
action_speed = 0.2
|
||||
|
||||
# Create camera trajectory (Plücker coordinates)
|
||||
camera_states = create_camera_trajectory(
|
||||
action=action,
|
||||
height=height,
|
||||
width=width,
|
||||
num_frames=num_frames,
|
||||
action_speed=action_speed,
|
||||
dtype=torch.bfloat16,
|
||||
)
|
||||
print(f"Camera states shape: {camera_states.shape}")
|
||||
|
||||
# I2V vs T2V: unset GAMECRAFT_I2V_IMAGE -> I2V (default image). Set to "" -> T2V.
|
||||
env_image = os.environ.get("GAMECRAFT_I2V_IMAGE")
|
||||
if env_image is None:
|
||||
image_path = DEFAULT_I2V_IMAGE_URL # default: I2V
|
||||
elif env_image.strip() == "":
|
||||
image_path = None # T2V
|
||||
else:
|
||||
image_path = env_image.strip() # I2V with given URL/path
|
||||
|
||||
is_i2v = image_path is not None
|
||||
prompt = DEFAULT_I2V_PROMPT if is_i2v else DEFAULT_PROMPTS["temple"]
|
||||
print(f"Mode: {'I2V' if is_i2v else 'T2V'}, prompt: {prompt[:60]}...")
|
||||
|
||||
gen_kw = dict(
|
||||
prompt=prompt,
|
||||
negative_prompt="",
|
||||
camera_states=camera_states,
|
||||
height=height,
|
||||
width=width,
|
||||
num_frames=num_frames,
|
||||
num_inference_steps=50,
|
||||
guidance_scale=6.0,
|
||||
seed=42,
|
||||
fps=24,
|
||||
output_path=OUTPUT_PATH,
|
||||
save_video=True,
|
||||
)
|
||||
if is_i2v:
|
||||
gen_kw["image_path"] = image_path
|
||||
generator.generate_video(**gen_kw)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,107 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Run GLM-Image text-to-image generation through FastVideo.
|
||||
|
||||
User story:
|
||||
"I have the HF `zai-org/GLM-Image` checkpoint and want a minimal
|
||||
text-to-image generation command, saved as a PNG."
|
||||
"""
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.api import (
|
||||
EngineConfig,
|
||||
GenerationRequest,
|
||||
GeneratorConfig,
|
||||
OutputConfig,
|
||||
ParallelismConfig,
|
||||
PipelineSelection,
|
||||
SamplingConfig,
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Run GLM-Image text-to-image generation.")
|
||||
parser.add_argument(
|
||||
"--model-path",
|
||||
default="zai-org/GLM-Image",
|
||||
help="HF id or local diffusers-format GLM-Image weights directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
default="image_output/landscape.png",
|
||||
help="Output PNG path.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt",
|
||||
default=("A beautiful landscape photography with rolling hills, "
|
||||
"a winding river, and a vibrant sunset in the background. "
|
||||
"Warm golden light, photorealistic style."),
|
||||
help="Text prompt.",
|
||||
)
|
||||
parser.add_argument("--height", type=int, default=1024)
|
||||
parser.add_argument("--width", type=int, default=1024)
|
||||
parser.add_argument("--steps", type=int, default=50)
|
||||
parser.add_argument("--guidance-scale", type=float, default=1.5)
|
||||
parser.add_argument("--seed", type=int, default=1024)
|
||||
parser.add_argument("--num-gpus", type=int, default=1)
|
||||
parser.add_argument("--tp-size", type=int, default=None)
|
||||
parser.add_argument("--sp-size", type=int, default=None)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
|
||||
output = Path(args.output)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
tp_size = args.tp_size if args.tp_size is not None else (args.num_gpus if args.num_gpus > 1 else 1)
|
||||
sp_size = args.sp_size if args.sp_size is not None else (1 if args.num_gpus > 1 else args.num_gpus)
|
||||
|
||||
# GLM-Image needs trust_remote_code for its AR encoder; offload and the
|
||||
# pipeline class come from the model's registered defaults — don't override.
|
||||
generator_config = GeneratorConfig(
|
||||
model_path=args.model_path,
|
||||
trust_remote_code=True,
|
||||
engine=EngineConfig(
|
||||
num_gpus=args.num_gpus,
|
||||
parallelism=ParallelismConfig(tp_size=tp_size, sp_size=sp_size),
|
||||
),
|
||||
pipeline=PipelineSelection(workload_type="t2i"),
|
||||
)
|
||||
|
||||
generator = VideoGenerator.from_config(generator_config)
|
||||
try:
|
||||
request = GenerationRequest(
|
||||
prompt=args.prompt,
|
||||
sampling=SamplingConfig(
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
num_frames=1,
|
||||
fps=1,
|
||||
num_inference_steps=args.steps,
|
||||
guidance_scale=args.guidance_scale,
|
||||
seed=args.seed,
|
||||
),
|
||||
output=OutputConfig(
|
||||
output_path=str(output.parent),
|
||||
save_video=False,
|
||||
return_frames=True,
|
||||
),
|
||||
)
|
||||
result = generator.generate(request)
|
||||
if isinstance(result, list):
|
||||
result = result[0]
|
||||
|
||||
frames = result.frames
|
||||
if frames is not None and len(frames):
|
||||
Image.fromarray(frames[0]).save(output)
|
||||
print(f"Saved image to {output}")
|
||||
finally:
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,204 +0,0 @@
|
||||
"""
|
||||
LongCat Image-to-Video (I2V) Example Script
|
||||
|
||||
This script demonstrates LongCat I2V inference using the FastVideo Python API.
|
||||
LongCat I2V takes an input image and generates a video from it.
|
||||
|
||||
It runs both basic generation (50 steps) and distill+refine generation
|
||||
(16 steps distill + 50 steps refinement to 720p with BSA).
|
||||
|
||||
Usage:
|
||||
python examples/inference/basic/basic_longcat_i2v.py
|
||||
|
||||
Note:
|
||||
Refinement uses 768x768 dimensions where latent (48x48) is divisible by 8,
|
||||
compatible with BSA chunks [4, 4, 8].
|
||||
"""
|
||||
|
||||
import glob
|
||||
import os
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
# Common prompts and settings matching the shell script examples
|
||||
PROMPT = ("A woman sits at a wooden table by the window in a cozy café. She reaches out "
|
||||
"with her right hand, picks up the white coffee cup from the saucer, and gently "
|
||||
"brings it to her lips to take a sip. After drinking, she places the cup back on "
|
||||
"the table and looks out the window, enjoying the peaceful atmosphere.")
|
||||
|
||||
NEGATIVE_PROMPT = ("Bright tones, overexposed, static, blurred details, subtitles, style, works, "
|
||||
"paintings, images, static, overall gray, worst quality, low quality, JPEG compression "
|
||||
"residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, "
|
||||
"deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, "
|
||||
"three legs, many people in the background, walking backwards")
|
||||
|
||||
# Input image path
|
||||
IMAGE_PATH = "assets/girl.png"
|
||||
|
||||
SEED = 42
|
||||
|
||||
|
||||
def basic_generation():
|
||||
"""
|
||||
Run basic LongCat I2V generation (50 steps at 480p).
|
||||
|
||||
This uses the full 50-step denoising process for highest quality.
|
||||
"""
|
||||
print("=" * 60)
|
||||
print("LongCat I2V: Basic Generation (50 steps, 480p)")
|
||||
print("=" * 60)
|
||||
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/LongCat-Video-I2V-Diffusers",
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False, # set to True if GPU is out of memory
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
enable_bsa=False,
|
||||
)
|
||||
|
||||
output_path = "outputs_video/longcat_i2v_basic"
|
||||
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
negative_prompt=NEGATIVE_PROMPT,
|
||||
image_path=IMAGE_PATH,
|
||||
output_path=output_path,
|
||||
save_video=True,
|
||||
height=480,
|
||||
width=480, # Square
|
||||
num_frames=93,
|
||||
num_inference_steps=50,
|
||||
fps=15,
|
||||
guidance_scale=4.0,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
print(f"\nBasic generation complete! Video saved to: {output_path}")
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
def distill_refine_generation():
|
||||
"""
|
||||
Run LongCat I2V with distill+refine pipeline (16 steps + refinement to 768p).
|
||||
|
||||
This uses the distilled LoRA for fast 480p generation (16 steps),
|
||||
then refines to 768p using the refinement LoRA with BSA enabled.
|
||||
"""
|
||||
print("\n" + "=" * 60)
|
||||
print("LongCat I2V: Distill + Refine Pipeline")
|
||||
print("=" * 60)
|
||||
|
||||
# Stage 1: Distilled generation (16 steps at 480p)
|
||||
print("\n[Stage 1] Distilled generation (16 steps, 480p)")
|
||||
print("-" * 40)
|
||||
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/LongCat-Video-I2V-Diffusers",
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=True,
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
enable_bsa=False,
|
||||
lora_path="FastVideo/LongCat-Video-T2V-Distilled-LoRA",
|
||||
lora_nickname="distilled",
|
||||
)
|
||||
|
||||
distill_output_path = "outputs_video/longcat_i2v_distill"
|
||||
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
negative_prompt=NEGATIVE_PROMPT,
|
||||
image_path=IMAGE_PATH,
|
||||
output_path=distill_output_path,
|
||||
save_video=True,
|
||||
height=480,
|
||||
width=480, # Square
|
||||
num_frames=93,
|
||||
num_inference_steps=16,
|
||||
fps=15,
|
||||
guidance_scale=1.0,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
print(f"Distilled generation complete! Video saved to: {distill_output_path}")
|
||||
generator.shutdown()
|
||||
|
||||
# Stage 2: Refinement (480p -> 768p)
|
||||
print("\n[Stage 2] Refinement (480p -> 768p with BSA)")
|
||||
print("-" * 40)
|
||||
|
||||
# Find the actual saved video file from stage 1
|
||||
video_files = glob.glob(os.path.join(distill_output_path, "*.mp4"))
|
||||
if not video_files:
|
||||
raise FileNotFoundError(f"No video file found in {distill_output_path}")
|
||||
# Use the most recently created video file
|
||||
distill_video_path = max(video_files, key=os.path.getmtime)
|
||||
print(f"Using stage 1 video: {distill_video_path}")
|
||||
|
||||
# Create a new generator with refinement LoRA and BSA enabled
|
||||
# Note: Refinement uses the T2V model (not I2V) since it's upscaling the generated video
|
||||
# For BSA [4, 4, 8]: latent must be divisible by 8
|
||||
# 768x768: latent 48x48, 48%8=0 ✓
|
||||
refine_generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/LongCat-Video-T2V-Diffusers",
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=True,
|
||||
dit_cpu_offload=True,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
enable_bsa=True,
|
||||
bsa_sparsity=0.875,
|
||||
bsa_chunk_q=[4, 4, 4],
|
||||
bsa_chunk_k=[4, 4, 4],
|
||||
lora_path="FastVideo/LongCat-Video-T2V-Refinement-LoRA",
|
||||
lora_nickname="refinement",
|
||||
)
|
||||
|
||||
refine_output_path = "outputs_video/longcat_i2v_refine_720p"
|
||||
|
||||
refine_generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
negative_prompt=NEGATIVE_PROMPT,
|
||||
output_path=refine_output_path,
|
||||
save_video=True,
|
||||
refine_from=distill_video_path,
|
||||
t_thresh=0.5,
|
||||
spatial_refine_only=False,
|
||||
num_cond_frames=0,
|
||||
height=720,
|
||||
width=720,
|
||||
num_inference_steps=50,
|
||||
fps=30,
|
||||
guidance_scale=1.0,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
print(f"Refinement complete! Video saved to: {refine_output_path}")
|
||||
refine_generator.shutdown()
|
||||
|
||||
|
||||
def main():
|
||||
"""Run both basic and distill+refine generation pipelines."""
|
||||
print("\n" + "=" * 60)
|
||||
print("LongCat Image-to-Video Example")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
# Run basic generation
|
||||
basic_generation()
|
||||
|
||||
# Run distill+refine pipeline
|
||||
distill_refine_generation()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("All generations complete!")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,192 +0,0 @@
|
||||
"""
|
||||
LongCat Text-to-Video (T2V) Example Script
|
||||
|
||||
This script demonstrates LongCat T2V inference using the FastVideo Python API.
|
||||
It runs both basic generation (50 steps) and distill+refine generation
|
||||
(16 steps distill + 50 steps refinement to 720p).
|
||||
|
||||
Usage:
|
||||
python examples/inference/basic/basic_longcat_t2v.py
|
||||
"""
|
||||
|
||||
import glob
|
||||
import os
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
# Common prompts and settings matching the shell script examples
|
||||
PROMPT = ("In a realistic photography style, a white boy around seven or eight years old "
|
||||
"sits on a park bench, wearing a light blue T-shirt, denim shorts, and white sneakers. "
|
||||
"He holds an ice cream cone with vanilla and chocolate flavors, and beside him is a "
|
||||
"medium-sized golden Labrador. Smiling, the boy offers the ice cream to the dog, "
|
||||
"who eagerly licks it with its tongue. The sun is shining brightly, and the background "
|
||||
"features a green lawn and several tall trees, creating a warm and loving scene.")
|
||||
|
||||
NEGATIVE_PROMPT = ("Bright tones, overexposed, static, blurred details, subtitles, style, works, "
|
||||
"paintings, images, static, overall gray, worst quality, low quality, JPEG compression "
|
||||
"residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, "
|
||||
"deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, "
|
||||
"three legs, many people in the background, walking backwards")
|
||||
|
||||
SEED = 42
|
||||
|
||||
|
||||
def basic_generation():
|
||||
"""
|
||||
Run basic LongCat T2V generation (50 steps at 480p).
|
||||
|
||||
This uses the full 50-step denoising process for highest quality.
|
||||
"""
|
||||
print("=" * 60)
|
||||
print("LongCat T2V: Basic Generation (50 steps, 480p)")
|
||||
print("=" * 60)
|
||||
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/LongCat-Video-T2V-Diffusers",
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False, # set to True if GPU is out of memory
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
enable_bsa=False,
|
||||
)
|
||||
|
||||
output_path = "outputs_video/longcat_t2v_basic"
|
||||
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
negative_prompt=NEGATIVE_PROMPT,
|
||||
output_path=output_path,
|
||||
save_video=True,
|
||||
height=480,
|
||||
width=832,
|
||||
num_frames=93,
|
||||
num_inference_steps=50,
|
||||
fps=15,
|
||||
guidance_scale=4.0,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
print(f"\nBasic generation complete! Video saved to: {output_path}")
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
def distill_refine_generation():
|
||||
"""
|
||||
Run LongCat T2V with distill+refine pipeline (16 steps + refinement to 720p).
|
||||
|
||||
This uses the distilled LoRA for fast 480p generation (16 steps),
|
||||
then refines to 720p using the refinement LoRA with BSA enabled.
|
||||
"""
|
||||
print("\n" + "=" * 60)
|
||||
print("LongCat T2V: Distill + Refine Pipeline")
|
||||
print("=" * 60)
|
||||
|
||||
# Stage 1: Distilled generation (16 steps at 480p)
|
||||
print("\n[Stage 1] Distilled generation (16 steps, 480p)")
|
||||
print("-" * 40)
|
||||
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/LongCat-Video-T2V-Diffusers",
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=True,
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
enable_bsa=False,
|
||||
lora_path="FastVideo/LongCat-Video-T2V-Distilled-LoRA",
|
||||
lora_nickname="distilled",
|
||||
)
|
||||
|
||||
distill_output_path = "outputs_video/longcat_t2v_distill"
|
||||
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
negative_prompt=NEGATIVE_PROMPT,
|
||||
output_path=distill_output_path,
|
||||
save_video=True,
|
||||
height=480,
|
||||
width=832,
|
||||
num_frames=93,
|
||||
num_inference_steps=16,
|
||||
fps=15,
|
||||
guidance_scale=1.0,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
print(f"Distilled generation complete! Video saved to: {distill_output_path}")
|
||||
generator.shutdown()
|
||||
|
||||
# Stage 2: Refinement (480p -> 720p)
|
||||
print("\n[Stage 2] Refinement (480p -> 720p with BSA)")
|
||||
print("-" * 40)
|
||||
|
||||
# Find the actual saved video file from stage 1
|
||||
video_files = glob.glob(os.path.join(distill_output_path, "*.mp4"))
|
||||
if not video_files:
|
||||
raise FileNotFoundError(f"No video file found in {distill_output_path}")
|
||||
# Use the most recently created video file
|
||||
distill_video_path = max(video_files, key=os.path.getmtime)
|
||||
print(f"Using stage 1 video: {distill_video_path}")
|
||||
|
||||
# Create a new generator with refinement LoRA and BSA enabled
|
||||
refine_generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/LongCat-Video-T2V-Diffusers",
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=True,
|
||||
dit_cpu_offload=True,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
enable_bsa=True,
|
||||
bsa_sparsity=0.875,
|
||||
bsa_chunk_q=[4, 4, 8],
|
||||
bsa_chunk_k=[4, 4, 8],
|
||||
lora_path="FastVideo/LongCat-Video-T2V-Refinement-LoRA",
|
||||
lora_nickname="refinement",
|
||||
)
|
||||
|
||||
refine_output_path = "outputs_video/longcat_t2v_refine_720p"
|
||||
|
||||
refine_generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
negative_prompt=NEGATIVE_PROMPT,
|
||||
output_path=refine_output_path,
|
||||
save_video=True,
|
||||
refine_from=distill_video_path,
|
||||
t_thresh=0.5,
|
||||
spatial_refine_only=False,
|
||||
num_cond_frames=0,
|
||||
height=720,
|
||||
width=1280,
|
||||
num_inference_steps=50,
|
||||
fps=30,
|
||||
guidance_scale=1.0,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
print(f"Refinement complete! Video saved to: {refine_output_path}")
|
||||
refine_generator.shutdown()
|
||||
|
||||
|
||||
def main():
|
||||
"""Run both basic and distill+refine generation pipelines."""
|
||||
print("\n" + "=" * 60)
|
||||
print("LongCat Text-to-Video Example")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
# Run basic generation
|
||||
basic_generation()
|
||||
|
||||
# Run distill+refine pipeline
|
||||
distill_refine_generation()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("All generations complete!")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,218 +0,0 @@
|
||||
"""
|
||||
LongCat Video Continuation (VC) Example Script
|
||||
|
||||
This script demonstrates LongCat VC inference using the FastVideo Python API.
|
||||
LongCat VC takes an input video and generates a continuation of it.
|
||||
|
||||
It runs both basic generation (50 steps) and distill+refine generation
|
||||
(16 steps distill + 50 steps refinement to 720p).
|
||||
|
||||
Usage:
|
||||
python examples/inference/basic/basic_longcat_vc.py
|
||||
|
||||
Prerequisites:
|
||||
- Ensure the input video exists at assets/motorcycle.mp4
|
||||
(or provide your own video)
|
||||
"""
|
||||
|
||||
import glob
|
||||
import os
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
# Common prompts and settings matching the shell script examples
|
||||
PROMPT = ("A person rides a motorcycle along a long, straight road that stretches between "
|
||||
"a body of water and a forested hillside. The rider steadily accelerates, keeping "
|
||||
"the motorcycle centered between the guardrails, while the scenery passes by on "
|
||||
"both sides. The video captures the journey from the rider's perspective, emphasizing "
|
||||
"the sense of motion and adventure.")
|
||||
|
||||
NEGATIVE_PROMPT = ("Bright tones, overexposed, static, blurred details, subtitles, style, works, "
|
||||
"paintings, images, static, overall gray, worst quality, low quality, JPEG compression "
|
||||
"residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, "
|
||||
"deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, "
|
||||
"three legs, many people in the background, walking backwards")
|
||||
|
||||
# Input video path
|
||||
VIDEO_PATH = "assets/motorcycle.mp4"
|
||||
|
||||
# Number of conditioning frames from the input video
|
||||
NUM_COND_FRAMES = 13
|
||||
|
||||
SEED = 42
|
||||
|
||||
|
||||
def basic_generation():
|
||||
"""
|
||||
Run basic LongCat VC generation (50 steps at 480p).
|
||||
|
||||
This uses the full 50-step denoising process for highest quality.
|
||||
"""
|
||||
print("=" * 60)
|
||||
print("LongCat VC: Basic Generation (50 steps, 480p)")
|
||||
print("=" * 60)
|
||||
|
||||
# Check if video exists
|
||||
if not os.path.exists(VIDEO_PATH):
|
||||
raise FileNotFoundError(f"Video not found at {VIDEO_PATH}. "
|
||||
"Please provide a valid video path.")
|
||||
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/LongCat-Video-VC-Diffusers",
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False, # set to True if GPU is out of memory
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
enable_bsa=False,
|
||||
)
|
||||
|
||||
output_path = "outputs_video/longcat_vc_basic"
|
||||
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
negative_prompt=NEGATIVE_PROMPT,
|
||||
video_path=VIDEO_PATH,
|
||||
num_cond_frames=NUM_COND_FRAMES,
|
||||
output_path=output_path,
|
||||
save_video=True,
|
||||
height=480,
|
||||
width=832,
|
||||
num_frames=93,
|
||||
num_inference_steps=50,
|
||||
fps=15,
|
||||
guidance_scale=4.0,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
print(f"\nBasic generation complete! Video saved to: {output_path}")
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
def distill_refine_generation():
|
||||
"""
|
||||
Run LongCat VC with distill+refine pipeline (16 steps + refinement to 720p).
|
||||
|
||||
This uses the distilled LoRA for fast 480p generation (16 steps),
|
||||
then refines to 720p using the refinement LoRA with BSA enabled.
|
||||
"""
|
||||
print("\n" + "=" * 60)
|
||||
print("LongCat VC: Distill + Refine Pipeline")
|
||||
print("=" * 60)
|
||||
|
||||
# Check if video exists
|
||||
if not os.path.exists(VIDEO_PATH):
|
||||
raise FileNotFoundError(f"Video not found at {VIDEO_PATH}. "
|
||||
"Please provide a valid video path.")
|
||||
|
||||
# Stage 1: Distilled generation (16 steps at 480p)
|
||||
print("\n[Stage 1] Distilled generation (16 steps, 480p)")
|
||||
print("-" * 40)
|
||||
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/LongCat-Video-VC-Diffusers",
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=True,
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
enable_bsa=False,
|
||||
lora_path="FastVideo/LongCat-Video-T2V-Distilled-LoRA",
|
||||
lora_nickname="distilled",
|
||||
)
|
||||
|
||||
distill_output_path = "outputs_video/longcat_vc_distill"
|
||||
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
negative_prompt=NEGATIVE_PROMPT,
|
||||
video_path=VIDEO_PATH,
|
||||
num_cond_frames=NUM_COND_FRAMES,
|
||||
output_path=distill_output_path,
|
||||
save_video=True,
|
||||
height=480,
|
||||
width=832,
|
||||
num_frames=93,
|
||||
num_inference_steps=16,
|
||||
fps=15,
|
||||
guidance_scale=1.0,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
print(f"Distilled generation complete! Video saved to: {distill_output_path}")
|
||||
generator.shutdown()
|
||||
|
||||
# Stage 2: Refinement (480p -> 720p)
|
||||
print("\n[Stage 2] Refinement (480p -> 720p with BSA)")
|
||||
print("-" * 40)
|
||||
|
||||
# Find the actual saved video file from stage 1
|
||||
video_files = glob.glob(os.path.join(distill_output_path, "*.mp4"))
|
||||
if not video_files:
|
||||
raise FileNotFoundError(f"No video file found in {distill_output_path}")
|
||||
# Use the most recently created video file
|
||||
distill_video_path = max(video_files, key=os.path.getmtime)
|
||||
print(f"Using stage 1 video: {distill_video_path}")
|
||||
|
||||
# Create a new generator with refinement LoRA and BSA enabled
|
||||
# Note: Refinement uses the T2V model (not VC) since it's upscaling the generated video
|
||||
refine_generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/LongCat-Video-T2V-Diffusers",
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=True,
|
||||
dit_cpu_offload=True,
|
||||
vae_cpu_offload=True,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=False,
|
||||
enable_bsa=True,
|
||||
bsa_sparsity=0.875,
|
||||
bsa_chunk_q=[4, 4, 8],
|
||||
bsa_chunk_k=[4, 4, 8],
|
||||
lora_path="FastVideo/LongCat-Video-T2V-Refinement-LoRA",
|
||||
lora_nickname="refinement",
|
||||
)
|
||||
|
||||
refine_output_path = "outputs_video/longcat_vc_refine_720p"
|
||||
|
||||
refine_generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
negative_prompt=NEGATIVE_PROMPT,
|
||||
output_path=refine_output_path,
|
||||
save_video=True,
|
||||
refine_from=distill_video_path,
|
||||
t_thresh=0.5,
|
||||
spatial_refine_only=False,
|
||||
num_cond_frames=0, # For refinement, no conditioning frames
|
||||
height=720,
|
||||
width=1280,
|
||||
num_inference_steps=50,
|
||||
fps=30,
|
||||
guidance_scale=1.0,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
print(f"Refinement complete! Video saved to: {refine_output_path}")
|
||||
refine_generator.shutdown()
|
||||
|
||||
|
||||
def main():
|
||||
"""Run both basic and distill+refine generation pipelines."""
|
||||
print("\n" + "=" * 60)
|
||||
print("LongCat Video Continuation Example")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
# Run basic generation
|
||||
basic_generation()
|
||||
|
||||
# Run distill+refine pipeline
|
||||
distill_refine_generation()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("All generations complete!")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,51 +0,0 @@
|
||||
import os
|
||||
|
||||
# Set SLA attention backend BEFORE fastvideo imports
|
||||
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "SLA_ATTN"
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
OUTPUT_PATH = "video_samples_turbodiffusion"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# TurboDiffusion: 1-4 step video generation using RCM scheduler + SLA attention
|
||||
# FastVideo will automatically use TurboDiffusionPipeline when specified
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"loayrashid/TurboWan2.1-T2V-1.3B-Diffusers",
|
||||
# FastVideo will automatically handle distributed setup
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False, # set to True if GPU is out of memory
|
||||
|
||||
# set to false if using RTX 4090
|
||||
# pin_cpu_memory=False,
|
||||
)
|
||||
|
||||
# Generate videos with the same simple API, regardless of GPU count
|
||||
# TurboDiffusion defaults: guidance_scale=1.0 and num_inference_steps=4 (from config)
|
||||
prompt = ("A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes "
|
||||
"wide with interest. The playful yet serene atmosphere is complemented by soft "
|
||||
"natural light filtering through the petals. Mid-shot, warm and cheerful tones.")
|
||||
video = generator.generate_video(
|
||||
prompt,
|
||||
output_path=OUTPUT_PATH,
|
||||
save_video=True,
|
||||
seed=42,
|
||||
)
|
||||
|
||||
# Generate another video with a different prompt, without reloading the model!
|
||||
prompt2 = ("A majestic lion strides across the golden savanna, its powerful frame "
|
||||
"glistening under the warm afternoon sun. The tall grass ripples gently in "
|
||||
"the breeze, enhancing the lion's commanding presence. The tone is vibrant, "
|
||||
"embodying the raw energy of the wild. Low angle, steady tracking shot, "
|
||||
"cinematic.")
|
||||
video2 = generator.generate_video(
|
||||
prompt2,
|
||||
output_path=OUTPUT_PATH,
|
||||
save_video=True,
|
||||
seed=42,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,45 +0,0 @@
|
||||
import os
|
||||
|
||||
# Set SLA attention backend BEFORE fastvideo imports
|
||||
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "SLA_ATTN"
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
OUTPUT_PATH = "video_samples_turbodiffusion_14B"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# TurboDiffusion 14B: 1-4 step video generation using RCM scheduler + SLA attention
|
||||
# FastVideo will automatically use TurboDiffusionPipeline when specified
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"loayrashid/TurboWan2.1-T2V-14B-Diffusers",
|
||||
# 14B model needs more GPUs
|
||||
num_gpus=2,
|
||||
)
|
||||
|
||||
prompt = ("A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes "
|
||||
"wide with interest. The playful yet serene atmosphere is complemented by soft "
|
||||
"natural light filtering through the petals. Mid-shot, warm and cheerful tones.")
|
||||
video = generator.generate_video(
|
||||
prompt,
|
||||
output_path=OUTPUT_PATH,
|
||||
save_video=True,
|
||||
seed=42,
|
||||
)
|
||||
|
||||
# Generate another video with a different prompt, without reloading the model!
|
||||
prompt2 = ("A majestic lion strides across the golden savanna, its powerful frame "
|
||||
"glistening under the warm afternoon sun. The tall grass ripples gently in "
|
||||
"the breeze, enhancing the lion's commanding presence. The tone is vibrant, "
|
||||
"embodying the raw energy of the wild. Low angle, steady tracking shot, "
|
||||
"cinematic.")
|
||||
video2 = generator.generate_video(
|
||||
prompt2,
|
||||
output_path=OUTPUT_PATH,
|
||||
save_video=True,
|
||||
seed=42,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,38 +0,0 @@
|
||||
import os
|
||||
|
||||
# Set SLA attention backend BEFORE fastvideo imports
|
||||
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "SLA_ATTN"
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
# Use local model path
|
||||
MODEL_PATH = "loayrashid/TurboWan2.2-I2V-A14B-Diffusers"
|
||||
OUTPUT_PATH = "video_samples_turbodiffusion_i2v"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# TurboDiffusion I2V: 1-4 step image-to-video generation
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
MODEL_PATH,
|
||||
num_gpus=2,
|
||||
)
|
||||
|
||||
# Example prompt and image for I2V
|
||||
prompt = (
|
||||
"Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
|
||||
)
|
||||
|
||||
# Use an example image path
|
||||
image_path = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/wan_i2v_input.JPG"
|
||||
|
||||
video = generator.generate_video(
|
||||
prompt,
|
||||
image_path=image_path,
|
||||
output_path=OUTPUT_PATH,
|
||||
save_video=True,
|
||||
seed=42,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,94 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Run Z-Image-Turbo text-to-image generation through FastVideo.
|
||||
|
||||
User story:
|
||||
"I want the official Z-Image-Turbo defaults and a deterministic PNG from
|
||||
a local or Hugging Face checkpoint."
|
||||
"""
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.api import (
|
||||
EngineConfig,
|
||||
GenerationRequest,
|
||||
GeneratorConfig,
|
||||
OutputConfig,
|
||||
ParallelismConfig,
|
||||
PipelineSelection,
|
||||
SamplingConfig,
|
||||
)
|
||||
|
||||
DEFAULT_PROMPT = (
|
||||
"Young Chinese woman in red Hanfu, intricate embroidery. Impeccable makeup, red floral forehead pattern. "
|
||||
"Elaborate high bun, golden phoenix headdress, red flowers, beads. Holds round folding fan with lady, trees, bird. "
|
||||
"Neon lightning-bolt lamp (⚡️), bright yellow glow, above extended left palm. Soft-lit outdoor night background, "
|
||||
"silhouetted tiered pagoda (西安大雁塔), blurred colorful distant lights.")
|
||||
DEFAULT_REVISION = "f332072aa78be7aecdf3ee76d5c247082da564a6"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Run Z-Image-Turbo text-to-image generation.")
|
||||
parser.add_argument("--model-path", default="Tongyi-MAI/Z-Image-Turbo")
|
||||
parser.add_argument("--revision", default=DEFAULT_REVISION)
|
||||
parser.add_argument("--output", default="outputs/zimage/zimage_turbo.png")
|
||||
parser.add_argument("--prompt", default=DEFAULT_PROMPT)
|
||||
parser.add_argument("--negative-prompt", default="")
|
||||
parser.add_argument("--height", type=int, default=1024)
|
||||
parser.add_argument("--width", type=int, default=1024)
|
||||
parser.add_argument("--steps", type=int, default=8)
|
||||
parser.add_argument("--guidance-scale", type=float, default=0.0)
|
||||
parser.add_argument("--max-sequence-length", type=int, default=512)
|
||||
parser.add_argument("--cfg-normalization", action=argparse.BooleanOptionalAction, default=False)
|
||||
parser.add_argument("--cfg-truncation", type=float, default=1.0)
|
||||
parser.add_argument("--seed", type=int, default=42)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
output = Path(args.output)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
generator = VideoGenerator.from_config(
|
||||
GeneratorConfig(
|
||||
model_path=args.model_path,
|
||||
revision=args.revision,
|
||||
engine=EngineConfig(
|
||||
num_gpus=1,
|
||||
parallelism=ParallelismConfig(tp_size=1, sp_size=1),
|
||||
use_fsdp_inference=False,
|
||||
),
|
||||
# The model registry selects the native zimage_turbo preset.
|
||||
pipeline=PipelineSelection(workload_type="t2i"),
|
||||
))
|
||||
try:
|
||||
generator.generate(
|
||||
GenerationRequest(
|
||||
prompt=args.prompt,
|
||||
negative_prompt=args.negative_prompt,
|
||||
sampling=SamplingConfig(
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
num_frames=1,
|
||||
fps=1,
|
||||
num_inference_steps=args.steps,
|
||||
guidance_scale=args.guidance_scale,
|
||||
max_sequence_length=args.max_sequence_length,
|
||||
cfg_normalization=args.cfg_normalization,
|
||||
cfg_truncation=args.cfg_truncation,
|
||||
seed=args.seed,
|
||||
),
|
||||
output=OutputConfig(
|
||||
output_path=str(output),
|
||||
save_video=True,
|
||||
return_frames=False,
|
||||
),
|
||||
))
|
||||
finally:
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,120 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Run GLM-Image image-to-image (edit) generation through FastVideo.
|
||||
|
||||
User story:
|
||||
"I have the HF `zai-org/GLM-Image` checkpoint and a condition image, and
|
||||
want a minimal edit command (text + image -> edited image), saved as a PNG."
|
||||
|
||||
GLM-Image is a single unified pipeline: passing a condition image switches it
|
||||
from text-to-image to the edit path (the condition enters the DiT via a KV-cache
|
||||
write pass), so the generator config is identical to `basic_glm_image.py` — the
|
||||
`inputs.pil_image` on the request is what selects the edit mode.
|
||||
"""
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.api import (
|
||||
EngineConfig,
|
||||
GenerationRequest,
|
||||
GeneratorConfig,
|
||||
InputConfig,
|
||||
OutputConfig,
|
||||
ParallelismConfig,
|
||||
PipelineSelection,
|
||||
SamplingConfig,
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Run GLM-Image image-to-image (edit) generation.")
|
||||
parser.add_argument(
|
||||
"--model-path",
|
||||
default="zai-org/GLM-Image",
|
||||
help="HF id or local diffusers-format GLM-Image weights directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--image",
|
||||
default="assets/images/couple.jpg",
|
||||
help="Condition image to edit.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
default="image_output/edited.png",
|
||||
help="Output PNG path.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt",
|
||||
default="Change the background to a snowy mountain landscape at golden hour.",
|
||||
help="Edit instruction.",
|
||||
)
|
||||
parser.add_argument("--height", type=int, default=1024)
|
||||
parser.add_argument("--width", type=int, default=1024)
|
||||
parser.add_argument("--steps", type=int, default=50)
|
||||
parser.add_argument("--guidance-scale", type=float, default=1.5)
|
||||
parser.add_argument("--seed", type=int, default=1024)
|
||||
parser.add_argument("--num-gpus", type=int, default=1)
|
||||
parser.add_argument("--tp-size", type=int, default=None)
|
||||
parser.add_argument("--sp-size", type=int, default=None)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
|
||||
output = Path(args.output)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
condition = Image.open(args.image).convert("RGB")
|
||||
tp_size = args.tp_size if args.tp_size is not None else (args.num_gpus if args.num_gpus > 1 else 1)
|
||||
sp_size = args.sp_size if args.sp_size is not None else (1 if args.num_gpus > 1 else args.num_gpus)
|
||||
|
||||
# GLM-Image needs trust_remote_code for its AR encoder; offload and the
|
||||
# pipeline class come from the model's registered defaults — don't override.
|
||||
# The pipeline is registered as t2i; passing inputs.pil_image below switches
|
||||
# it to the edit path.
|
||||
generator_config = GeneratorConfig(
|
||||
model_path=args.model_path,
|
||||
trust_remote_code=True,
|
||||
engine=EngineConfig(
|
||||
num_gpus=args.num_gpus,
|
||||
parallelism=ParallelismConfig(tp_size=tp_size, sp_size=sp_size),
|
||||
),
|
||||
pipeline=PipelineSelection(workload_type="t2i"),
|
||||
)
|
||||
|
||||
generator = VideoGenerator.from_config(generator_config)
|
||||
try:
|
||||
request = GenerationRequest(
|
||||
prompt=args.prompt,
|
||||
inputs=InputConfig(pil_image=condition),
|
||||
sampling=SamplingConfig(
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
num_frames=1,
|
||||
fps=1,
|
||||
num_inference_steps=args.steps,
|
||||
guidance_scale=args.guidance_scale,
|
||||
seed=args.seed,
|
||||
),
|
||||
output=OutputConfig(
|
||||
output_path=str(output.parent),
|
||||
save_video=False,
|
||||
return_frames=True,
|
||||
),
|
||||
)
|
||||
result = generator.generate(request)
|
||||
if isinstance(result, list):
|
||||
result = result[0]
|
||||
|
||||
frames = result.frames
|
||||
if frames is not None and len(frames):
|
||||
Image.fromarray(frames[0]).save(output)
|
||||
print(f"Saved image to {output}")
|
||||
finally:
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,71 +0,0 @@
|
||||
# LongCat-Video T2V 13.6B bidirectional finetune.
|
||||
|
||||
models:
|
||||
student:
|
||||
_target_: fastvideo.train.models.longcat.LongCatModel
|
||||
init_from: FastVideo/LongCat-Video-T2V-Diffusers
|
||||
trainable: true
|
||||
|
||||
method:
|
||||
_target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
|
||||
|
||||
training:
|
||||
model_path: FastVideo/LongCat-Video-T2V-Diffusers
|
||||
|
||||
distributed:
|
||||
num_gpus: 8
|
||||
sp_size: 1
|
||||
tp_size: 1
|
||||
hsdp_replicate_dim: 1
|
||||
hsdp_shard_dim: 8
|
||||
|
||||
data:
|
||||
data_path: data/LongCat-Syn
|
||||
dataloader_num_workers: 4
|
||||
train_batch_size: 1
|
||||
training_cfg_rate: 0.0
|
||||
seed: 1000
|
||||
num_latent_t: 20
|
||||
num_height: 480
|
||||
num_width: 848
|
||||
num_frames: 77
|
||||
|
||||
optimizer:
|
||||
learning_rate: 1.0e-6
|
||||
betas: [0.9, 0.999]
|
||||
weight_decay: 0.01
|
||||
lr_scheduler: constant
|
||||
lr_warmup_steps: 0
|
||||
|
||||
loop:
|
||||
max_train_steps: 4000
|
||||
gradient_accumulation_steps: 1
|
||||
|
||||
checkpoint:
|
||||
output_dir: outputs/longcat_finetune
|
||||
training_state_checkpointing_steps: 1000
|
||||
checkpoints_total_limit: 3
|
||||
|
||||
tracker:
|
||||
project_name: fastvideo
|
||||
run_name: longcat_finetune
|
||||
|
||||
model:
|
||||
enable_gradient_checkpointing_type: full
|
||||
|
||||
callbacks:
|
||||
grad_clip:
|
||||
_target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
|
||||
max_grad_norm: 1.0
|
||||
validation:
|
||||
_target_: fastvideo.train.callbacks.validation.ValidationCallback
|
||||
pipeline_target: fastvideo.pipelines.basic.longcat.longcat_pipeline.LongCatPipeline
|
||||
dataset_file: data/validation_prompts.json
|
||||
every_steps: 100
|
||||
sampling_steps: [50]
|
||||
guidance_scale: 5.0
|
||||
|
||||
pipeline:
|
||||
# Match the released LongCat scheduler config. flow_shift=0.0 collapses
|
||||
# FlowMatch training timesteps to zero in FastVideo's scheduler.
|
||||
flow_shift: 12.0
|
||||
@@ -1,89 +0,0 @@
|
||||
# TurboWan 2.1 T2V 1.3B LoRA finetune with SLA attention.
|
||||
#
|
||||
# Notes:
|
||||
# - TurboDiffusion checkpoints expect the SLA attention backend. The training
|
||||
# entrypoint now auto-selects `SLA_ATTN` for model IDs containing
|
||||
# `turbodiffusion` or `turbowan`, unless you override the environment
|
||||
# variable manually.
|
||||
# - This debug config reuses the existing crush-smol T2V parquet recipe. If you
|
||||
# preprocess data with the official TurboWan recipe, update the temporal and
|
||||
# spatial fields to match it.
|
||||
|
||||
models:
|
||||
student:
|
||||
_target_: fastvideo.train.models.wan.WanModel
|
||||
init_from: loayrashid/TurboWan2.1-T2V-1.3B-Diffusers
|
||||
trainable: true
|
||||
lora:
|
||||
enable: true
|
||||
rank: 16
|
||||
alpha: 32
|
||||
target_modules:
|
||||
- to_q
|
||||
- to_k
|
||||
- to_v
|
||||
- to_out
|
||||
|
||||
method:
|
||||
_target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
|
||||
|
||||
training:
|
||||
dit_precision: bf16
|
||||
|
||||
distributed:
|
||||
num_gpus: 1
|
||||
sp_size: 1
|
||||
tp_size: 1
|
||||
hsdp_replicate_dim: 1
|
||||
hsdp_shard_dim: 1
|
||||
|
||||
data:
|
||||
data_path: data/crush-smol_processed_t2v/combined_parquet_dataset
|
||||
dataloader_num_workers: 2
|
||||
train_batch_size: 1
|
||||
training_cfg_rate: 0.0
|
||||
seed: 42
|
||||
num_latent_t: 20
|
||||
num_height: 480
|
||||
num_width: 832
|
||||
num_frames: 77
|
||||
|
||||
optimizer:
|
||||
learning_rate: 1.0e-5
|
||||
betas: [0.9, 0.999]
|
||||
weight_decay: 0.01
|
||||
lr_scheduler: constant
|
||||
lr_warmup_steps: 0
|
||||
|
||||
loop:
|
||||
max_train_steps: 10
|
||||
gradient_accumulation_steps: 5
|
||||
|
||||
checkpoint:
|
||||
output_dir: outputs/turbo_wan_t2v_lora_sla
|
||||
training_state_checkpointing_steps: 5
|
||||
checkpoints_total_limit: 2
|
||||
resume_from_checkpoint: null
|
||||
|
||||
tracker:
|
||||
project_name: fastvideo
|
||||
run_name: turbo_wan_t2v_lora_sla
|
||||
|
||||
model:
|
||||
enable_gradient_checkpointing_type: full
|
||||
|
||||
callbacks:
|
||||
grad_clip:
|
||||
_target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
|
||||
max_grad_norm: 1.0
|
||||
|
||||
validation:
|
||||
_target_: fastvideo.train.callbacks.validation.ValidationCallback
|
||||
pipeline_target: fastvideo.pipelines.basic.turbodiffusion.turbodiffusion_pipeline.TurboDiffusionPipeline
|
||||
dataset_file: examples/training/finetune/wan_t2v_1.3B/crush_smol/validation.json
|
||||
every_steps: 5
|
||||
sampling_steps: [4]
|
||||
guidance_scale: 1.0
|
||||
|
||||
pipeline:
|
||||
flow_shift: 3
|
||||
@@ -62,28 +62,13 @@ class SamplingParam:
|
||||
prompt_attention_mask: list = field(default_factory=list)
|
||||
negative_attention_mask: list = field(default_factory=list)
|
||||
|
||||
# Camera/action control inputs (GameCraft)
|
||||
camera_states: Any | None = None # Plücker coordinates [B, T_video, 6, H, W]
|
||||
camera_trajectory: str | None = None
|
||||
action_list: list[str] | None = None
|
||||
action_speed_list: list[float] | None = None
|
||||
gt_latents: Any | None = None # Ground truth latents [B, 16, T, H, W]
|
||||
conditioning_mask: Any | None = None # Mask [B, 1, T, H, W]
|
||||
|
||||
# Camera control inputs (LingBotWorld and LingBotWorld2)
|
||||
c2ws_plucker_emb: Any | None = None # Plucker embedding: [B, C, F_lat, H_lat, W_lat]
|
||||
action_path: str | None = None # Directory containing poses.npy and intrinsics.npy
|
||||
|
||||
# Refine inputs (LongCat 480p->720p upscaling)
|
||||
# Path-based refine (load stage1 video from disk, e.g. MP4)
|
||||
refine_from: str | None = None # Path to stage1 video (480p output from distill)
|
||||
# Refine / conditioning-frame inputs
|
||||
t_thresh: float = 0.5 # Threshold for timestep scheduling in refinement
|
||||
spatial_refine_only: bool = False # If True, only spatial (no temporal doubling)
|
||||
num_cond_frames: int = 0 # Number of conditioning frames
|
||||
# In-memory refine input (for two-stage pipeline where stage1 frames are already in memory)
|
||||
# This mirrors LongCat's demo where a list of frames (e.g. np.ndarray or PIL.Image)
|
||||
# is passed directly to the refinement pipeline instead of reloading from disk.
|
||||
stage1_video: Any | None = None
|
||||
|
||||
# Text inputs
|
||||
prompt: str | list[str] | None = None
|
||||
@@ -111,11 +96,6 @@ class SamplingParam:
|
||||
guidance_scale: float = 1.0
|
||||
batch_cfg: bool = False
|
||||
guidance_scale_2: float | None = None
|
||||
# Z-Image CFG controls. ``cfg_normalization=True`` caps the guided
|
||||
# prediction norm at the positive-prediction norm; ``cfg_truncation``
|
||||
# disables CFG above the normalized-noise threshold.
|
||||
cfg_normalization: bool = False
|
||||
cfg_truncation: float | None = 1.0
|
||||
# Embedded guidance (FLUX): do not treat ``guidance_scale > 1`` as classic CFG.
|
||||
use_embedded_guidance: bool = False
|
||||
# Diffusers-style true CFG for FLUX when > 1 (requires negative prompt encoding).
|
||||
@@ -365,18 +345,6 @@ class SamplingParam:
|
||||
default=SamplingParam.guidance_scale,
|
||||
help="Classifier-free guidance scale",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cfg-normalization",
|
||||
action=StoreBoolean,
|
||||
default=SamplingParam.cfg_normalization,
|
||||
help="Cap Z-Image CFG prediction norm to the positive-prediction norm",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cfg-truncation",
|
||||
type=float,
|
||||
default=SamplingParam.cfg_truncation,
|
||||
help="Disable Z-Image CFG above this normalized-noise threshold",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--batch-cfg",
|
||||
action=StoreBoolean,
|
||||
@@ -493,24 +461,12 @@ class SamplingParam:
|
||||
default=SamplingParam.audio_path,
|
||||
help="Path to input audio for speech-driven generation (Wan S2V)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--refine-from",
|
||||
type=str,
|
||||
default=SamplingParam.refine_from,
|
||||
help="Path to stage1 video for refinement (LongCat 480p->720p)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--t-thresh",
|
||||
type=float,
|
||||
default=SamplingParam.t_thresh,
|
||||
help="Threshold for timestep scheduling in refinement (default: 0.5)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--spatial-refine-only",
|
||||
action=StoreBoolean,
|
||||
default=SamplingParam.spatial_refine_only,
|
||||
help="Only perform spatial super-resolution (no temporal doubling)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-cond-frames",
|
||||
type=int,
|
||||
|
||||
@@ -158,8 +158,6 @@ class InputConfig:
|
||||
grid_sizes: Any | None = None
|
||||
c2ws_plucker_emb: Any | None = None
|
||||
action_path: str | None = None
|
||||
refine_from: str | None = None
|
||||
stage1_video: Any | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -178,8 +176,6 @@ class SamplingConfig:
|
||||
guidance_scale: float = 1.0
|
||||
batch_cfg: bool = False
|
||||
guidance_scale_2: float | None = None
|
||||
cfg_normalization: bool = False
|
||||
cfg_truncation: float | None = 1.0
|
||||
guidance_rescale: float = 0.0
|
||||
true_cfg_scale: float | None = None
|
||||
use_embedded_guidance: bool | None = None
|
||||
|
||||
@@ -67,7 +67,7 @@ def _use_fa2(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> bool:
|
||||
# capability gate, not a runtime fallback):
|
||||
# * the backward asserts sm90+ (L40S/sm_89 dies on its arch check);
|
||||
# * GQA fails CuTeDSL JIT in pack_gqa ("ValueError: Operation creation
|
||||
# failed", observed on sm_89 with HunyuanGameCraft/LTX2).
|
||||
# failed", observed on sm_89 with LTX2).
|
||||
if q.shape[-2] != k.shape[-2]:
|
||||
return True
|
||||
return torch.is_grad_enabled() and any(t.requires_grad for t in (q, k, v))
|
||||
|
||||
@@ -1,14 +1,10 @@
|
||||
from fastvideo.configs.models.dits.cosmos import CosmosVideoConfig
|
||||
from fastvideo.configs.models.dits.cosmos2_5 import Cosmos25VideoConfig
|
||||
from fastvideo.configs.models.dits.dreamx_world import DreamXWorldARConfig, DreamXWorldConfig
|
||||
from fastvideo.configs.models.dits.flux import FluxDiTConfig
|
||||
from fastvideo.configs.models.dits.flux_2 import Flux2Config
|
||||
from fastvideo.configs.models.dits.glm_image import GlmImageDiTConfig
|
||||
from fastvideo.configs.models.dits.helios import HeliosConfig
|
||||
from fastvideo.configs.models.dits.hunyuangamecraft import HunyuanGameCraftConfig
|
||||
from fastvideo.configs.models.dits.hunyuanvideo import HunyuanVideoConfig
|
||||
from fastvideo.configs.models.dits.hunyuanvideo15 import HunyuanVideo15Config
|
||||
from fastvideo.configs.models.dits.longcat import LongCatVideoConfig
|
||||
from fastvideo.configs.models.dits.ltx2 import LTX2VideoConfig
|
||||
from fastvideo.configs.models.dits.magi_human import MagiHumanVideoConfig
|
||||
from fastvideo.configs.models.dits.minimax_h3 import MiniMaxH3Config
|
||||
@@ -22,7 +18,6 @@ from fastvideo.configs.models.dits.mmaudio import (
|
||||
)
|
||||
from fastvideo.configs.models.dits.stable_audio import StableAudioConfig
|
||||
from fastvideo.models.wan.config import WanVideoConfig
|
||||
from fastvideo.configs.models.dits.zimage import ZImageDiTConfig
|
||||
from fastvideo.configs.models.dits.hyworld import HYWorldConfig
|
||||
from fastvideo.configs.models.dits.kandinsky5 import Kandinsky5VideoConfig
|
||||
from fastvideo.configs.models.dits.kandinsky6 import Kandinsky6ArchConfig, Kandinsky6VideoAudioConfig
|
||||
@@ -32,12 +27,11 @@ from fastvideo.configs.models.dits.lingbotworld_fast import LingBotWorldFastVide
|
||||
from fastvideo.configs.models.dits.lingbot_video import LingBotVideoConfig
|
||||
|
||||
__all__ = [
|
||||
"HunyuanVideoConfig", "HunyuanVideo15Config", "HunyuanGameCraftConfig", "WanVideoConfig", "DreamXWorldConfig",
|
||||
"DreamXWorldARConfig", "CosmosVideoConfig", "Cosmos25VideoConfig", "FluxDiTConfig", "Flux2Config",
|
||||
"LongCatVideoConfig", "LTX2VideoConfig", "HYWorldConfig", "Kandinsky5VideoConfig", "Kandinsky6ArchConfig",
|
||||
"HunyuanVideoConfig", "HunyuanVideo15Config", "WanVideoConfig", "CosmosVideoConfig", "Cosmos25VideoConfig",
|
||||
"FluxDiTConfig", "Flux2Config", "LTX2VideoConfig", "HYWorldConfig", "Kandinsky5VideoConfig", "Kandinsky6ArchConfig",
|
||||
"Kandinsky6VideoAudioConfig", "Kandinsky6SRArchConfig", "Kandinsky6SRConfig", "MagiHumanVideoConfig",
|
||||
"StableAudioConfig", "GlmImageDiTConfig", "HeliosConfig", "LingBotWorld2CausalFastVideoConfig",
|
||||
"LingBotWorldFastVideoConfig", "LingBotVideoConfig", "MiniMaxH3Config", "ZImageDiTConfig", "MMAudioArchConfig",
|
||||
"MMAudioTransformerConfig", "MMAUDIO_VARIANT_ARCHITECTURES", "MMAUDIO_TRAINING_VARIANTS",
|
||||
"MMAUDIO_44K_TRAINING_VARIANTS", "get_mmaudio_transformer_config"
|
||||
"StableAudioConfig", "HeliosConfig", "LingBotWorld2CausalFastVideoConfig", "LingBotWorldFastVideoConfig",
|
||||
"LingBotVideoConfig", "MiniMaxH3Config", "MMAudioArchConfig", "MMAudioTransformerConfig",
|
||||
"MMAUDIO_VARIANT_ARCHITECTURES", "MMAUDIO_TRAINING_VARIANTS", "MMAUDIO_44K_TRAINING_VARIANTS",
|
||||
"get_mmaudio_transformer_config"
|
||||
]
|
||||
|
||||
@@ -1,71 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from fastvideo.configs.models.dits.base import DiTArchConfig, DiTConfig
|
||||
from fastvideo.models.wan.config import WanVideoArchConfig
|
||||
|
||||
|
||||
@dataclass
|
||||
class DreamXWorldArchConfig(WanVideoArchConfig):
|
||||
"""DreamX-World DiT config with camera PRoPE control fields."""
|
||||
|
||||
add_control_adapter: bool = True
|
||||
cam_method: str | None = "prope"
|
||||
attn_compress: int = 1
|
||||
cam_self_attn_layers: tuple[int, ...] | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class DreamXWorldConfig(DiTConfig):
|
||||
arch_config: DiTArchConfig = field(default_factory=DreamXWorldArchConfig)
|
||||
|
||||
prefix: str = "Wan"
|
||||
|
||||
|
||||
@dataclass
|
||||
class DreamXWorldARArchConfig(DreamXWorldArchConfig):
|
||||
"""DreamX-World-5B autoregressive causal DiT config."""
|
||||
|
||||
model_type: str = "ti2v"
|
||||
patch_size: tuple[int, int, int] = (1, 2, 2)
|
||||
text_len: int = 512
|
||||
text_dim: int = 4096
|
||||
freq_dim: int = 256
|
||||
attn_compress: int = 4
|
||||
cam_self_attn_layers: tuple[int, ...] | None = tuple(range(30))
|
||||
local_attn_size: int = 12
|
||||
sink_size: int = 3
|
||||
num_frames_per_block: int = 3
|
||||
rope_cache_policy: str = "block_relativistic"
|
||||
# The official AR checkpoint (AMAP-ML/DreamX-World ``model.safetensors``)
|
||||
# already uses FastVideo's native key names and the converter copies the
|
||||
# tensors verbatim, so every rule is an identity. The rules enumerate the
|
||||
# full state-dict surface of ``DreamXWorldARTransformer3DModel`` (norm1 /
|
||||
# norm2 / head.norm are affine-free and have no parameters).
|
||||
param_names_mapping: dict = field(
|
||||
default_factory=lambda: {
|
||||
r"^patch_embedding\.(.*)$": r"patch_embedding.\1",
|
||||
r"^text_embedding\.([02])\.(.*)$": r"text_embedding.\1.\2",
|
||||
r"^time_embedding\.([02])\.(.*)$": r"time_embedding.\1.\2",
|
||||
r"^time_projection\.1\.(.*)$": r"time_projection.1.\1",
|
||||
r"^blocks\.(\d+)\.self_attn\.(q|k|v|o)\.(.*)$": r"blocks.\1.self_attn.\2.\3",
|
||||
r"^blocks\.(\d+)\.self_attn\.norm_(q|k)\.weight$": r"blocks.\1.self_attn.norm_\2.weight",
|
||||
r"^blocks\.(\d+)\.cross_attn\.(q|k|v|o)\.(.*)$": r"blocks.\1.cross_attn.\2.\3",
|
||||
r"^blocks\.(\d+)\.cross_attn\.norm_(q|k)\.weight$": r"blocks.\1.cross_attn.norm_\2.weight",
|
||||
r"^blocks\.(\d+)\.cam_self_attn\.(q_proj|k_proj|v_proj|out_proj)\.(.*)$": r"blocks.\1.cam_self_attn.\2.\3",
|
||||
r"^blocks\.(\d+)\.cam_self_attn\.norm_(q|k)\.weight$": r"blocks.\1.cam_self_attn.norm_\2.weight",
|
||||
r"^blocks\.(\d+)\.norm3\.(.*)$": r"blocks.\1.norm3.\2",
|
||||
r"^blocks\.(\d+)\.ffn\.([02])\.(.*)$": r"blocks.\1.ffn.\2.\3",
|
||||
r"^blocks\.(\d+)\.modulation$": r"blocks.\1.modulation",
|
||||
r"^head\.head\.(.*)$": r"head.head.\1",
|
||||
r"^head\.modulation$": r"head.modulation",
|
||||
})
|
||||
reverse_param_names_mapping: dict = field(default_factory=dict)
|
||||
lora_param_names_mapping: dict = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DreamXWorldARConfig(DreamXWorldConfig):
|
||||
arch_config: DiTArchConfig = field(default_factory=DreamXWorldARArchConfig)
|
||||
|
||||
prefix: str = "Wan"
|
||||
@@ -1,61 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from fastvideo.configs.models.dits.base import DiTArchConfig, DiTConfig
|
||||
|
||||
|
||||
def is_blocks(n: str, m) -> bool:
|
||||
return "transformer_blocks" in n and str.isdigit(n.split(".")[-1])
|
||||
|
||||
|
||||
@dataclass
|
||||
class GlmImageDiTArchConfig(DiTArchConfig):
|
||||
|
||||
_fsdp_shard_conditions: list = field(default_factory=lambda: [is_blocks])
|
||||
|
||||
hidden_size: int = 4096
|
||||
num_attention_heads: int = 32
|
||||
attention_head_dim: int = 128
|
||||
in_channels: int = 16
|
||||
out_channels: int = 16
|
||||
num_layers: int = 30
|
||||
|
||||
text_embed_dim: int = 1472
|
||||
time_embed_dim: int = 512
|
||||
condition_dim: int = 256
|
||||
|
||||
prior_vq_quantizer_codebook_size: int = 16384
|
||||
|
||||
patch_size: int = 2
|
||||
|
||||
max_height: int = 2048
|
||||
max_width: int = 2048
|
||||
|
||||
qk_norm: str = "layer_norm"
|
||||
eps: float = 1e-5
|
||||
|
||||
exclude_lora_layers: list[str] = field(
|
||||
default_factory=lambda: ["image_projector", "glyph_projector", "prior_token_embedding"])
|
||||
|
||||
param_names_mapping: dict = field(
|
||||
default_factory=lambda: {
|
||||
r"^glyph_projector\.net\.0\.proj\.(.*)$": r"glyph_projector.fc_in.\1",
|
||||
r"^glyph_projector\.net\.2\.(.*)$": r"glyph_projector.fc_out.\1",
|
||||
r"^prior_projector\.net\.0\.proj\.(.*)$": r"prior_projector.fc_in.\1",
|
||||
r"^prior_projector\.net\.2\.(.*)$": r"prior_projector.fc_out.\1",
|
||||
r"^transformer_blocks\.(\d+)\.ff\.net\.0\.proj\.(.*)$": r"transformer_blocks.\1.ff.fc_in.\2",
|
||||
r"^transformer_blocks\.(\d+)\.ff\.net\.2\.(.*)$": r"transformer_blocks.\1.ff.fc_out.\2",
|
||||
})
|
||||
|
||||
reverse_param_names_mapping: dict = field(default_factory=dict)
|
||||
lora_param_names_mapping: dict = field(default_factory=dict)
|
||||
|
||||
def __post_init__(self):
|
||||
super().__post_init__()
|
||||
self.num_channels_latents = self.out_channels
|
||||
|
||||
|
||||
@dataclass
|
||||
class GlmImageDiTConfig(DiTConfig):
|
||||
arch_config: DiTArchConfig = field(default_factory=GlmImageDiTArchConfig)
|
||||
prefix: str = "GlmImage"
|
||||
@@ -1,143 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""
|
||||
Configuration for HunyuanGameCraft transformer model.
|
||||
|
||||
HunyuanGameCraft extends HunyuanVideo with:
|
||||
1. CameraNet for camera/action conditioning
|
||||
2. 33 input channels (16 latent + 16 gt_latent + 1 mask)
|
||||
3. Mask-based conditioning for autoregressive generation
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import torch
|
||||
|
||||
from fastvideo.configs.models.dits.base import DiTArchConfig, DiTConfig
|
||||
|
||||
|
||||
def is_double_block(n: str, m) -> bool:
|
||||
return "double" in n and str.isdigit(n.split(".")[-1])
|
||||
|
||||
|
||||
def is_single_block(n: str, m) -> bool:
|
||||
return "single" in n and str.isdigit(n.split(".")[-1])
|
||||
|
||||
|
||||
def is_refiner_block(n: str, m) -> bool:
|
||||
return "refiner" in n and str.isdigit(n.split(".")[-1])
|
||||
|
||||
|
||||
def is_txt_in(n: str, m) -> bool:
|
||||
return n.split(".")[-1] == "txt_in"
|
||||
|
||||
|
||||
def is_camera_net(n: str, m) -> bool:
|
||||
return "camera_net" in n
|
||||
|
||||
|
||||
@dataclass
|
||||
class HunyuanGameCraftArchConfig(DiTArchConfig):
|
||||
"""Architecture config for HunyuanGameCraft transformer."""
|
||||
|
||||
# Version field for compatibility with saved config.json
|
||||
_fastvideo_version: str = "0.1.0"
|
||||
|
||||
# Camera net flag (for config.json compatibility)
|
||||
camera_net: bool = True
|
||||
|
||||
_fsdp_shard_conditions: list = field(
|
||||
default_factory=lambda: [is_double_block, is_single_block, is_refiner_block, is_camera_net])
|
||||
|
||||
_compile_conditions: list = field(default_factory=lambda: [is_double_block, is_single_block, is_txt_in])
|
||||
|
||||
# Parameter names mapping from official checkpoint to FastVideo naming
|
||||
# GameCraft weights are already close to FastVideo format with minor adjustments
|
||||
param_names_mapping: dict = field(
|
||||
default_factory=lambda: {
|
||||
# MLP naming: fc1 -> fc_in, fc2 -> fc_out
|
||||
r"^(.*)\.img_mlp\.fc1\.(.*)$": r"\1.img_mlp.fc_in.\2",
|
||||
r"^(.*)\.img_mlp\.fc2\.(.*)$": r"\1.img_mlp.fc_out.\2",
|
||||
r"^(.*)\.txt_mlp\.fc1\.(.*)$": r"\1.txt_mlp.fc_in.\2",
|
||||
r"^(.*)\.txt_mlp\.fc2\.(.*)$": r"\1.txt_mlp.fc_out.\2",
|
||||
|
||||
# Single block MLP naming
|
||||
r"^single_blocks\.(\d+)\.mlp\.fc1\.(.*)$": r"single_blocks.\1.mlp.fc_in.\2",
|
||||
r"^single_blocks\.(\d+)\.mlp\.fc2\.(.*)$": r"single_blocks.\1.mlp.fc_out.\2",
|
||||
|
||||
# Token refiner naming
|
||||
r"^txt_in\.individual_token_refiner\.blocks\.(\d+)\.(.*)$": r"txt_in.refiner_blocks.\1.\2",
|
||||
|
||||
# Vector in naming
|
||||
r"^vector_in\.in_layer\.(.*)$": r"vector_in.fc_in.\1",
|
||||
r"^vector_in\.out_layer\.(.*)$": r"vector_in.fc_out.\1",
|
||||
|
||||
# Time embedder naming
|
||||
r"^time_in\.mlp\.0\.(.*)$": r"time_in.mlp.fc_in.\1",
|
||||
r"^time_in\.mlp\.2\.(.*)$": r"time_in.mlp.fc_out.\1",
|
||||
|
||||
# Guidance embedder naming (if present)
|
||||
r"^guidance_in\.mlp\.0\.(.*)$": r"guidance_in.mlp.fc_in.\1",
|
||||
r"^guidance_in\.mlp\.2\.(.*)$": r"guidance_in.mlp.fc_out.\1",
|
||||
|
||||
# Final layer adaLN modulation
|
||||
r"^final_layer\.adaLN_modulation\.1\.(.*)$": r"final_layer.adaLN_modulation.linear.\1",
|
||||
|
||||
# Refiner block MLP naming
|
||||
r"^txt_in\.refiner_blocks\.(\d+)\.mlp\.fc1\.(.*)$": r"txt_in.refiner_blocks.\1.mlp.fc_in.\2",
|
||||
r"^txt_in\.refiner_blocks\.(\d+)\.mlp\.fc2\.(.*)$": r"txt_in.refiner_blocks.\1.mlp.fc_out.\2",
|
||||
|
||||
# Camera net weights are already correctly named
|
||||
})
|
||||
|
||||
# Reverse mapping for saving checkpoints
|
||||
reverse_param_names_mapping: dict = field(default_factory=lambda: {})
|
||||
|
||||
# Model architecture parameters
|
||||
# patch_size can be int or tuple - if tuple, it's [T, H, W]
|
||||
patch_size: int | tuple[int, int, int] = 2
|
||||
patch_size_t: int = 1
|
||||
in_channels: int = 33 # 16 latent + 16 gt_latent + 1 mask
|
||||
out_channels: int = 16
|
||||
num_attention_heads: int = 24
|
||||
attention_head_dim: int = 128
|
||||
mlp_ratio: float = 4.0
|
||||
num_layers: int = 20 # Double stream blocks
|
||||
num_single_layers: int = 40 # Single stream blocks
|
||||
num_refiner_layers: int = 2
|
||||
rope_axes_dim: tuple[int, int, int] = (16, 56, 56)
|
||||
guidance_embeds: bool = False # GameCraft doesn't use guidance
|
||||
dtype: torch.dtype | None = None
|
||||
text_embed_dim: int = 4096 # LLaMA-3 hidden size
|
||||
pooled_projection_dim: int = 768 # CLIP pooled output dim
|
||||
rope_theta: int = 256
|
||||
qk_norm: str = "rms_norm"
|
||||
|
||||
# Camera net parameters
|
||||
camera_in_channels: int = 6 # Plücker coordinates
|
||||
camera_downscale_coef: int = 8
|
||||
camera_out_channels: int = 16
|
||||
|
||||
# Layers to exclude from LoRA
|
||||
exclude_lora_layers: list[str] = field(
|
||||
default_factory=lambda: ["img_in", "txt_in", "time_in", "vector_in", "camera_net"])
|
||||
|
||||
def __post_init__(self):
|
||||
super().__post_init__()
|
||||
self.hidden_size: int = self.attention_head_dim * self.num_attention_heads
|
||||
self.num_channels_latents: int = 16 # Output is 16 channels
|
||||
|
||||
# Convert patch_size list to tuple if needed (from JSON deserialization)
|
||||
if isinstance(self.patch_size, list):
|
||||
self.patch_size = tuple(self.patch_size)
|
||||
|
||||
# Convert rope_axes_dim list to tuple if needed
|
||||
if isinstance(self.rope_axes_dim, list):
|
||||
self.rope_axes_dim = tuple(self.rope_axes_dim)
|
||||
|
||||
|
||||
@dataclass
|
||||
class HunyuanGameCraftConfig(DiTConfig):
|
||||
"""Full config for HunyuanGameCraft model."""
|
||||
|
||||
arch_config: DiTArchConfig = field(default_factory=HunyuanGameCraftArchConfig)
|
||||
|
||||
prefix: str = "HunyuanGameCraft"
|
||||
@@ -1,133 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""
|
||||
LongCat Video DiT configuration for native FastVideo implementation.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from fastvideo.configs.models.dits.base import DiTArchConfig, DiTConfig
|
||||
from fastvideo.platforms import AttentionBackendEnum
|
||||
|
||||
|
||||
def is_longcat_blocks(n: str, m) -> bool:
|
||||
"""FSDP shard condition for LongCat transformer blocks."""
|
||||
return "blocks" in n and str.isdigit(n.split(".")[-1])
|
||||
|
||||
|
||||
@dataclass
|
||||
class LongCatVideoArchConfig(DiTArchConfig):
|
||||
"""Architecture configuration for native LongCat Video DiT."""
|
||||
|
||||
_fsdp_shard_conditions: list = field(default_factory=lambda: [is_longcat_blocks])
|
||||
|
||||
# Enable torch.compile for transformer blocks (major speedup!)
|
||||
_compile_conditions: list = field(default_factory=lambda: [is_longcat_blocks])
|
||||
|
||||
# Parameter name mapping for weight conversion
|
||||
param_names_mapping: dict = field(
|
||||
default_factory=lambda: {
|
||||
# Embedders
|
||||
r"^x_embedder\.(.*)$": r"patch_embed.\1",
|
||||
r"^t_embedder\.mlp\.0\.(.*)$": r"time_embedder.linear_1.\1",
|
||||
r"^t_embedder\.mlp\.2\.(.*)$": r"time_embedder.linear_2.\1",
|
||||
r"^y_embedder\.y_proj\.0\.(.*)$": r"caption_embedder.linear_1.\1",
|
||||
r"^y_embedder\.y_proj\.2\.(.*)$": r"caption_embedder.linear_2.\1",
|
||||
|
||||
# Transformer blocks - AdaLN modulation
|
||||
r"^blocks\.(\d+)\.adaLN_modulation\.1\.(.*)$": r"blocks.\1.adaln_linear_1.\2",
|
||||
|
||||
# Transformer blocks - Normalization
|
||||
r"^blocks\.(\d+)\.mod_norm_attn\.(.*)$": r"blocks.\1.norm_attn.\2",
|
||||
r"^blocks\.(\d+)\.mod_norm_ffn\.(.*)$": r"blocks.\1.norm_ffn.\2",
|
||||
r"^blocks\.(\d+)\.pre_crs_attn_norm\.(.*)$": r"blocks.\1.norm_cross.\2",
|
||||
|
||||
# Self-attention: QKV fused -> separate (will need splitting in converter)
|
||||
# Original has attn.qkv.weight -> need to split into to_q, to_k, to_v
|
||||
r"^blocks\.(\d+)\.attn\.qkv\.(.*)$": r"blocks.\1.self_attn.qkv_fused.\2", # Marker for splitting
|
||||
r"^blocks\.(\d+)\.attn\.proj\.(.*)$": r"blocks.\1.self_attn.to_out.\2",
|
||||
r"^blocks\.(\d+)\.attn\.q_norm\.(.*)$": r"blocks.\1.self_attn.q_norm.\2",
|
||||
r"^blocks\.(\d+)\.attn\.k_norm\.(.*)$": r"blocks.\1.self_attn.k_norm.\2",
|
||||
|
||||
# Cross-attention
|
||||
r"^blocks\.(\d+)\.cross_attn\.q_linear\.(.*)$": r"blocks.\1.cross_attn.to_q.\2",
|
||||
r"^blocks\.(\d+)\.cross_attn\.kv_linear\.(.*)$":
|
||||
r"blocks.\1.cross_attn.kv_fused.\2", # Marker for splitting
|
||||
r"^blocks\.(\d+)\.cross_attn\.proj\.(.*)$": r"blocks.\1.cross_attn.to_out.\2",
|
||||
r"^blocks\.(\d+)\.cross_attn\.q_norm\.(.*)$": r"blocks.\1.cross_attn.q_norm.\2",
|
||||
r"^blocks\.(\d+)\.cross_attn\.k_norm\.(.*)$": r"blocks.\1.cross_attn.k_norm.\2",
|
||||
|
||||
# FFN (SwiGLU)
|
||||
r"^blocks\.(\d+)\.ffn\.w1\.(.*)$": r"blocks.\1.ffn.w1.\2", # gate
|
||||
r"^blocks\.(\d+)\.ffn\.w2\.(.*)$": r"blocks.\1.ffn.w2.\2", # down
|
||||
r"^blocks\.(\d+)\.ffn\.w3\.(.*)$": r"blocks.\1.ffn.w3.\2", # up
|
||||
|
||||
# Final layer
|
||||
r"^final_layer\.adaLN_modulation\.1\.(.*)$": r"final_layer.adaln_linear.\1",
|
||||
r"^final_layer\.norm_final\.(.*)$": r"final_layer.norm.\1",
|
||||
r"^final_layer\.linear\.(.*)$": r"final_layer.proj.\1",
|
||||
})
|
||||
|
||||
# Reverse mapping for saving checkpoints: custom -> hf
|
||||
reverse_param_names_mapping: dict = field(default_factory=lambda: {})
|
||||
|
||||
# LoRA parameter name mapping
|
||||
lora_param_names_mapping: dict = field(default_factory=lambda: {})
|
||||
|
||||
# Model architecture parameters
|
||||
hidden_size: int = 4096
|
||||
depth: int = 48 # Number of transformer blocks
|
||||
num_attention_heads: int = 32
|
||||
attention_head_dim: int = 128 # hidden_size / num_attention_heads
|
||||
|
||||
in_channels: int = 16 # Latent space channels
|
||||
out_channels: int = 16
|
||||
num_channels_latents: int = 16
|
||||
|
||||
# Patch embedding
|
||||
patch_size: tuple[int, int, int] = (1, 2, 2) # [T, H, W] - no temporal compression
|
||||
|
||||
# Text/caption embedding
|
||||
caption_channels: int = 4096 # UMT5 d_model
|
||||
|
||||
# Timestep embedding
|
||||
adaln_tembed_dim: int = 512
|
||||
frequency_embedding_size: int = 256
|
||||
|
||||
# FFN
|
||||
mlp_ratio: int = 4
|
||||
|
||||
# Attention backend support
|
||||
_supported_attention_backends: tuple = field(default_factory=lambda: (
|
||||
AttentionBackendEnum.FLASH_ATTN,
|
||||
AttentionBackendEnum.TORCH_SDPA,
|
||||
))
|
||||
|
||||
# Text padding behavior
|
||||
text_tokens_zero_pad: bool = True
|
||||
|
||||
# Block Sparse Attention (BSA)
|
||||
enable_bsa: bool = False
|
||||
bsa_params: dict | None = field(default_factory=lambda: {
|
||||
"sparsity": 0.9375,
|
||||
"cdf_threshold": None,
|
||||
"chunk_3d_shape_q": [4, 4, 4],
|
||||
"chunk_3d_shape_k": [4, 4, 4],
|
||||
})
|
||||
|
||||
# LoRA exclusions
|
||||
exclude_lora_layers: list[str] = field(default_factory=lambda: [])
|
||||
|
||||
def __post_init__(self):
|
||||
super().__post_init__()
|
||||
self.out_channels = self.out_channels or self.in_channels
|
||||
# Ensure attention_head_dim matches
|
||||
self.attention_head_dim = self.hidden_size // self.num_attention_heads
|
||||
|
||||
|
||||
@dataclass
|
||||
class LongCatVideoConfig(DiTConfig):
|
||||
"""Main configuration for LongCat Video DiT."""
|
||||
|
||||
arch_config: DiTArchConfig = field(default_factory=LongCatVideoArchConfig)
|
||||
|
||||
prefix: str = "longcat"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user