Compare commits

...
5 Commits
294 changed files with 408 additions and 35081 deletions
@@ -52,8 +52,8 @@ in one commit.
script that supports a global dtype downcast flag MUST own an explicit
allowlist of fp32-keep tensors, documented at the top of the file.
2. **The `add-model-conversion` skill** should enforce two checks for any
converter that ships a `--cast-bf16`-style flag:
2. **Check two things** for any converter that ships a `--cast-bf16`-style
flag:
- Run the parity test for **every** variant of the model (base, distill,
SR, etc.), not just the headline variant. Different variants exercise
different code paths.
@@ -62,10 +62,10 @@ parity rationale:
2. **Use layer-by-layer activation hooks** when DiT parity is close-but-not-
bit-exact and the gap grows with depth. The
`fastvideo/hooks/activation_trace.py` infra exists exactly for this case
(`add-model-trace` skill). In MagiHuman's case it would have localized the
(see `docs/contributing/activation_trace.md`). In MagiHuman's case it would have localized the
first divergence point in one pass.
3. **The `add-model-port-dit` skill** should explicitly call out:
3. **DiT ports must account for:**
- SDPA input dtype must match the upstream kernel's internal cast.
- Post-attention upcast happens **before** any per-head gate, not after.
- Residual stream dtype across block boundaries is a parity invariant.
@@ -64,6 +64,5 @@ because both sides of the parity check are now correct.
`(pT pH pW C)`) at the call site and assert the layout matches the
downstream consumer's expectation.
4. The `add-model-port-dit` skill's parity gate should require an E2E hash
check for any DiT that does video token packing, not just latent
bit-exactness.
4. DiT parity for any model that does video token packing should include an
E2E hash check, not just latent bit-exactness.
-173
View File
@@ -1,173 +0,0 @@
---
name: add-model-01-prep
description: Use at the start of a FastVideo model port to gather required inputs, inspect/download HF weights, clone and install the official reference repo in the current environment, create a local_tests README skeleton, and produce a handoff before conversion or implementation.
---
# Add Model Prep
## Goal
Prepare external assets and the shared parity-test environment for a FastVideo
model port. Stop before writing conversion scripts, model components, pipeline
code, registry entries, or executable parity tests.
## Ask First
Ask once, then proceed if the HF token is already exported:
```text
Before prep: (1) official reference repo or Diffusers pipeline URL, (2) HF repo
id or local weights path and whether it has a root model_index.json, (3) target
model_family, (4) workload types, (5) which token env var is exported:
HF_TOKEN, HUGGINGFACE_HUB_TOKEN, or HF_API_KEY, (6) may I stage clone and
weights under the FastVideo repo root, and (7) may I install official reference
dependencies into the current FastVideo conda/env for parity tests?
```
Useful optional inputs: `pipeline_class`, `reference_dir`, `hf_revision`,
`official_revision`, `reuse_hints`, `download_scope`.
## Rules
- Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
escape hatches, and skip/pass semantics.
- Run from the FastVideo repo root.
- Use repo-relative defaults: `<ReferenceDir>/`,
`official_weights/<model_family>/`, `converted_weights/<model_family>/`.
- Install official reference deps into the current FastVideo environment, not a
new venv/conda env, so parity tests run both implementations with one shared
numeric stack.
- If the reference is a Diffusers class/package instead of a cloneable repo,
record import path and version instead of cloning.
- Prep may create only the local-test README and `PORT_STATUS.md` skeletons;
executable `.py` parity tests belong to `../add-model-02-parity/SKILL.md`.
## Escape Hatches
Follow `../add-model/shared/common_rules.md`. Prep-specific ask cases include
overwriting an existing clone or weight directory, installing untrusted/private
deps, choosing between incompatible official references, large downloads outside
the agreed scope, or missing gated-repo auth setup by env var name.
## Workflow
1. Verify the repo:
```bash
git rev-parse --show-toplevel
```
Expected markers: `fastvideo/`, `scripts/checkpoint_conversion/`,
`scripts/huggingface/download_hf.py`, `fastvideo/registry.py`.
2. Inspect HF or local weight layout:
```bash
python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \
"Org/Model" \
--revision "<revision>" \
--json
```
For a local path, replace `Org/Model` with `/path/to/weights`. Record
`source_layout`, `needs_conversion`, `model_index_class`, and
`components_seen`.
3. Download HF weights if needed:
```bash
python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \
"Org/Model" \
"official_weights/<model_family>" \
--revision "<revision>"
```
For selected files, repeat `--file-name`. For partial snapshots, repeat
`--allow-pattern` or `--ignore-pattern`. If the user provided a local path,
record it instead of copying large weights by default.
4. Clone the official reference repo if applicable:
```bash
python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \
"<official_repo_url>" \
"<ReferenceDir>" \
--branch "<tag-or-branch>" \
--commit "<commit-sha>" \
--update-gitignore
```
Omit `--branch`, `--commit`, or `--update-gitignore` when not needed. The
helper refuses to overwrite existing paths and prints remote/HEAD instead.
5. Keep prep assets ignored. Ensure `.gitignore` includes relevant entries:
```gitignore
/<ReferenceDir>/
/official_weights/
/converted_weights/
```
6. Follow the official repo's setup instructions in the current environment.
Inspect dependency files and README install docs before installing anything:
- `README*`, install docs, or model-card instructions.
- `requirements*.txt`, `pyproject.toml`, `setup.py`, `environment.yml`.
Use the current FastVideo conda/env. Do not create a new env even if upstream
docs recommend one; translate the needed install commands into the active env.
Prefer editable/no-deps first so the official source is importable without
changing shared pins:
```bash
uv pip install --no-deps -e ./<ReferenceDir>
```
Then install only missing official deps needed for parity imports. Stop before
installing requirements that would change FastVideo's core stack. If upstream
requires private/non-PyPI deps, record that parity needs a local stub helper
rather than pretending setup is complete.
7. Create the model-family local test skeleton and top-level port state file:
```bash
mkdir -p tests/local_tests/<model_family>
cp ".agents/skills/add-model-01-prep/templates/local_tests_readme.md" \
tests/local_tests/<model_family>/README.md
cp ".agents/skills/add-model-01-prep/templates/port_status.md" \
tests/local_tests/<model_family>/PORT_STATUS.md
```
Edit every placeholder in the README and `PORT_STATUS.md`. The README gives
later review agents enough information to reproduce the shared environment and
run/review parity work:
- official code URL or import path, local clone path, and commit/version;
- HF URL or local weight path, revision, access notes, and token env var name
only;
- commands already run and any blocked official dependency installs;
- shared-env install commands to re-run without changing core pins;
- expected local parity test paths and pytest commands;
- private-dependency stubs or known setup gaps;
- PR/review notes explaining which parity tests are required before handoff.
Do not include raw tokens, absolute cache paths that are not repo-reproducible,
or large generated outputs. If prep is blocked before imports work, still create
the README with `official_env_status=blocked` and the exact blocker.
`PORT_STATUS.md` must follow `../add-model/contracts/port_state.md`. Record open
questions and prep issues immediately, using stable IDs such as `Q001` and
`I001`. Keep resolved questions/issues in the table with a resolution instead of
deleting them.
## Handoff
End with the canonical prep handoff contract from
`../add-model/contracts/prep_handoff.md` and update the shared state files before
handoff.
## Helper Scripts
- `scripts/inspect_hf_layout.py`: classify HF/local layout.
- `scripts/download_hf_weights.py`: download HF snapshot or selected files.
- `scripts/clone_reference_repo.py`: clone reference repo safely.
@@ -1,119 +0,0 @@
#!/usr/bin/env python3
"""Clone an official reference repo without overwriting existing paths."""
from __future__ import annotations
import argparse
import subprocess
import sys
from pathlib import Path
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Clone a reference repo for FastVideo parity tests.")
parser.add_argument("repo_url", help="Official reference repository URL")
parser.add_argument("target_dir", help="Directory to clone into")
parser.add_argument("--branch", help="Branch or tag to clone")
parser.add_argument("--commit", help="Commit SHA to check out after clone")
parser.add_argument(
"--update-gitignore",
action="store_true",
help="Add the target directory to .gitignore if missing",
)
parser.add_argument(
"--gitignore",
default=".gitignore",
help="Path to gitignore file when --update-gitignore is used",
)
return parser.parse_args()
def run(command: list[str], check: bool = True) -> subprocess.CompletedProcess[str]:
return subprocess.run(
command,
check=check,
text=True,
capture_output=True,
)
def print_existing_repo_info(target: Path) -> int:
print(f"target_exists: {target}")
if not (target / ".git").exists():
print("error: target exists but is not a git repo", file=sys.stderr)
return 1
remote = run(["git", "-C", str(target), "remote", "-v"], check=False)
head = run(["git", "-C", str(target), "rev-parse", "HEAD"], check=False)
if remote.stdout:
print("remote_v:")
print(remote.stdout.rstrip())
if head.stdout:
print(f"head: {head.stdout.strip()}")
print("not_overwritten: true")
return 0
def gitignore_entry_for(target: Path) -> str:
root = Path.cwd().resolve()
resolved = target.resolve()
try:
relative = resolved.relative_to(root)
except ValueError as exc:
raise ValueError("--update-gitignore requires target_dir to be under the current directory") from exc
text = relative.as_posix().rstrip("/")
return "/" + text + "/"
def update_gitignore(path: Path, target: Path) -> bool:
entry = gitignore_entry_for(target)
existing = path.read_text().splitlines() if path.exists() else []
if entry in existing:
return False
new_text = "\n".join(existing).rstrip("\n")
if new_text:
new_text += "\n"
new_text += entry + "\n"
path.write_text(new_text)
return True
def main() -> int:
args = parse_args()
target = Path(args.target_dir)
if target.exists():
return print_existing_repo_info(target)
command = ["git", "clone", "--depth", "1"]
if args.branch:
command.extend(["--branch", args.branch])
command.extend([args.repo_url, str(target)])
try:
run(command)
if args.commit:
run(["git", "-C", str(target), "fetch", "--depth", "1", "origin", args.commit])
run(["git", "-C", str(target), "checkout", args.commit])
except subprocess.CalledProcessError as exc:
if exc.stdout:
print(exc.stdout, end="")
if exc.stderr:
print(exc.stderr, end="", file=sys.stderr)
return exc.returncode
head = run(["git", "-C", str(target), "rev-parse", "HEAD"])
print(f"cloned: {target}")
print(f"head: {head.stdout.strip()}")
if args.update_gitignore:
changed = update_gitignore(Path(args.gitignore), target)
print(f"gitignore_updated: {str(changed).lower()}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -1,103 +0,0 @@
#!/usr/bin/env python3
"""Download HF weights using the standard FastVideo token env vars."""
from __future__ import annotations
import argparse
import os
import sys
from pathlib import Path
HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY")
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Download a HF model snapshot or selected files into a local directory.")
parser.add_argument("repo_id", help="HF repo id, for example Org/Model")
parser.add_argument("local_dir", help="Destination directory")
parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)")
parser.add_argument("--revision", help="HF branch, tag, or commit")
parser.add_argument(
"--file-name",
action="append",
default=[],
help="Download one file; may be repeated. If omitted, download full snapshot.",
)
parser.add_argument(
"--allow-pattern",
action="append",
default=[],
help="Snapshot allow pattern; may be repeated. Ignored when --file-name is used.",
)
parser.add_argument(
"--ignore-pattern",
action="append",
default=[],
help="Snapshot ignore pattern; may be repeated. Ignored when --file-name is used.",
)
return parser.parse_args()
def resolve_token() -> tuple[str | None, str | None]:
for key in HF_TOKEN_ENV_KEYS:
value = os.environ.get(key)
if value:
return key, value
return None, None
def main() -> int:
args = parse_args()
token_env, token = resolve_token()
local_dir = Path(args.local_dir).expanduser()
if token_env:
print(f"token_env: {token_env}")
else:
print("token_env: none", file=sys.stderr)
try:
if local_dir.exists() and not local_dir.is_dir():
print(
f"error: destination exists and is not a directory: {local_dir}",
file=sys.stderr,
)
return 1
local_dir.mkdir(parents=True, exist_ok=True)
if args.file_name:
from huggingface_hub import hf_hub_download
for file_name in args.file_name:
path = hf_hub_download(
repo_id=args.repo_id,
filename=file_name,
repo_type=args.repo_type,
revision=args.revision,
local_dir=str(local_dir),
token=token,
)
print(f"downloaded_file: {path}")
else:
from huggingface_hub import snapshot_download
path = snapshot_download(
repo_id=args.repo_id,
repo_type=args.repo_type,
revision=args.revision,
local_dir=str(local_dir),
token=token,
allow_patterns=args.allow_pattern or None,
ignore_patterns=args.ignore_pattern or None,
)
print(f"downloaded_snapshot: {path}")
except Exception as exc: # noqa: BLE001 - CLI should print concise failures.
print(f"error: {exc}", file=sys.stderr)
return 1
print(f"local_dir: {local_dir.resolve()}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -1,260 +0,0 @@
#!/usr/bin/env python3
"""Inspect a Hugging Face repo or local weight directory layout."""
from __future__ import annotations
import argparse
import json
import os
import sys
from pathlib import Path
from typing import Any
HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY")
RAW_WEIGHT_SUFFIXES = (".safetensors", ".pt", ".pth", ".ckpt", ".bin")
KNOWN_COMPONENTS = {
"audio_vae",
"conditioner",
"feature_extractor",
"image_encoder",
"scheduler",
"text_encoder",
"text_encoder_2",
"tokenizer",
"tokenizer_2",
"transformer",
"transformer_2",
"unet",
"upsampler",
"vae",
"vocoder",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Classify a HF repo or local directory as Diffusers, raw, custom, or unknown.")
parser.add_argument("source", help="HF repo id or local weights directory")
parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)")
parser.add_argument("--revision", help="HF revision to inspect")
parser.add_argument(
"--max-local-files",
type=int,
default=20000,
help="Maximum local files to scan recursively (default: 20000)",
)
parser.add_argument(
"--sample-limit",
type=int,
default=80,
help="Number of file paths to print in human output (default: 80)",
)
parser.add_argument("--json", action="store_true", help="Emit JSON only")
return parser.parse_args()
def resolve_token() -> tuple[str | None, str | None]:
for key in HF_TOKEN_ENV_KEYS:
value = os.environ.get(key)
if value:
return key, value
return None, None
def load_local_files(root: Path, max_files: int) -> tuple[list[str], bool]:
files: list[str] = []
truncated = False
for path in root.rglob("*"):
if not path.is_file():
continue
files.append(path.relative_to(root).as_posix())
if len(files) >= max_files:
truncated = True
break
return sorted(files), truncated
def load_local_model_index(root: Path) -> tuple[dict[str, Any] | None, str | None]:
index_path = root / "model_index.json"
if not index_path.is_file():
return None, None
try:
return json.loads(index_path.read_text()), None
except Exception as exc: # noqa: BLE001 - surface malformed JSON clearly.
return None, f"failed to parse local model_index.json: {exc}"
def load_remote_files(
repo_id: str,
repo_type: str,
revision: str | None,
token: str | None,
) -> list[str]:
from huggingface_hub import list_repo_files
return sorted(list_repo_files(
repo_id,
repo_type=repo_type,
revision=revision,
token=token,
))
def load_remote_model_index(
repo_id: str,
repo_type: str,
revision: str | None,
token: str | None,
) -> tuple[dict[str, Any] | None, str | None]:
from huggingface_hub import hf_hub_download
try:
path = hf_hub_download(
repo_id=repo_id,
filename="model_index.json",
repo_type=repo_type,
revision=revision,
token=token,
)
except Exception as exc: # noqa: BLE001 - missing/inaccessible file is data.
return None, f"failed to download model_index.json: {exc}"
try:
return json.loads(Path(path).read_text()), None
except Exception as exc: # noqa: BLE001 - surface malformed JSON clearly.
return None, f"failed to parse remote model_index.json: {exc}"
def root_file_names(files: list[str]) -> set[str]:
return {name for name in files if "/" not in name}
def component_names(files: list[str], model_index: dict[str, Any] | None) -> list[str]:
components: set[str] = set()
for name in files:
parts = name.split("/", 1)
if len(parts) != 2:
continue
top, rest = parts
if top in KNOWN_COMPONENTS or rest == "config.json":
components.add(top)
if model_index:
for key, value in model_index.items():
if key.startswith("_"):
continue
if isinstance(value, list) and len(value) == 2:
components.add(key)
return sorted(components)
def classify_layout(
files: list[str],
model_index: dict[str, Any] | None,
components: list[str],
) -> tuple[str, str]:
roots = root_file_names(files)
raw_weight_files = [name for name in roots if name.endswith(RAW_WEIGHT_SUFFIXES)]
has_model_index = "model_index.json" in roots or model_index is not None
if has_model_index and components:
return "diffusers", "no"
if has_model_index:
return "custom", "unknown"
if raw_weight_files:
return "raw_official", "yes"
if any(name.endswith(RAW_WEIGHT_SUFFIXES) for name in files):
return "custom", "yes"
return "unknown", "unknown"
def build_result(args: argparse.Namespace) -> dict[str, Any]:
token_env, token = resolve_token()
source_path = Path(args.source).expanduser()
is_local = source_path.exists()
if is_local:
root = source_path.resolve()
if not root.is_dir():
raise ValueError(f"local source is not a directory: {root}")
files, truncated = load_local_files(root, args.max_local_files)
model_index, model_index_error = load_local_model_index(root)
source_kind = "local"
source = str(root)
else:
files = load_remote_files(args.source, args.repo_type, args.revision, token)
truncated = False
model_index, model_index_error = load_remote_model_index(
args.source,
args.repo_type,
args.revision,
token,
)
source_kind = "hf"
source = args.source
components = component_names(files, model_index)
source_layout, needs_conversion = classify_layout(files, model_index, components)
return {
"source": source,
"source_kind": source_kind,
"repo_type": None if is_local else args.repo_type,
"revision": args.revision,
"token_env": token_env,
"source_layout": source_layout,
"needs_conversion": needs_conversion,
"model_index_class": (model_index or {}).get("_class_name"),
"model_index_diffusers_version": (model_index or {}).get("_diffusers_version"),
"model_index_error": model_index_error,
"components_seen": components,
"file_count": len(files),
"file_scan_truncated": truncated,
"files_sample": files[:args.sample_limit],
}
def print_human(result: dict[str, Any]) -> None:
for key in (
"source",
"source_kind",
"repo_type",
"revision",
"token_env",
"source_layout",
"needs_conversion",
"model_index_class",
"model_index_diffusers_version",
"model_index_error",
"file_count",
"file_scan_truncated",
):
value = result.get(key)
if value is not None:
print(f"{key}: {value}")
components = result["components_seen"]
print("components_seen: " + (", ".join(components) if components else "none"))
print("files_sample:")
for name in result["files_sample"]:
print(f" {name}")
def main() -> int:
args = parse_args()
try:
result = build_result(args)
except Exception as exc: # noqa: BLE001 - CLI should print concise failures.
print(f"error: {exc}", file=sys.stderr)
return 1
if args.json:
print(json.dumps(result, indent=2, sort_keys=True))
else:
print_human(result)
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -1,122 +0,0 @@
# <Model Family> Local Tests
Local-only parity and smoke tests for the `<model_family>` FastVideo port. These
tests compare FastVideo against the official reference implementation and are
not expected to run in CI unless explicitly promoted later.
Port progress, open questions, issues, and handoff notes live in
`tests/local_tests/<model_family>/PORT_STATUS.md`.
## Reference Assets
| Field | Value |
|---|---|
| Model family | `<model_family>` |
| Workload types | `<T2V/I2V/V2V/T2I/or compatibility shim with rationale>` |
| Official reference | `<url or import path>` |
| Local reference dir | `<ReferenceDir or none>` |
| Official commit/version | `<sha, tag, package version, or unknown>` |
| HF weights | `<HF repo id/url or local path>` |
| HF revision | `<revision or default>` |
| Local weights dir | `<official_weights/model_family or local path>` |
| Source layout | `<diffusers/raw_official/monolithic/separate_components/mixed/custom/unknown>` |
| Needs conversion | `<yes/no/unknown>` |
Do not write token values in this file. Use only the token env var name:
`<HF_TOKEN or HUGGINGFACE_HUB_TOKEN or HF_API_KEY>`.
## Shared Environment Setup
Run from the FastVideo repo root in the same conda/env used for FastVideo.
Do not create a separate upstream environment for parity tests.
```bash
# Official reference source, if cloneable.
python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \
"<official_repo_url>" \
"<ReferenceDir>" \
--commit "<commit-sha>" \
--update-gitignore
# Editable install without changing shared core pins.
uv pip install --no-deps -e ./<ReferenceDir>
# Additional official deps installed or required for imports:
# <package list or none>
```
Do not change core dependency versions (`torch`, `diffusers`, `transformers`,
`flash-attn`, `triton`, CUDA packages) without explicit approval.
## Official Environment Status
```text
dependency_changes: <none | installed no-deps editable | installed official deps in current env | blocked on user>
official_env_status: <imports_ok | private_deps_need_stubs | blocked>
private_dep_stubs: <none or tests/local_tests/helpers/<model_family>_upstream.py>
blocked_on: <none or exact blocker>
```
## Weight Setup
```bash
python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \
"<Org/Model>" \
"official_weights/<model_family>" \
--revision "<revision>"
```
If weights are local-only, record the local path and do not copy large files into
the repository.
## Prototype And Conversion Artifacts
State-dict key/shape dumps are generated after FastVideo native prototypes exist
and are used to build the conversion mapping.
```text
official_key_dumps:
<component>: converted_weights/<model_family>/_mapping/<component>_official_keys.json
fastvideo_key_dumps:
<component>: converted_weights/<model_family>/_mapping/<component>_fastvideo_keys.json
conversion_script: scripts/checkpoint_conversion/<model_family>_to_diffusers.py
conversion_source_layout: <diffusers | separate_components | monolithic | mixed | custom>
converted_weights_dir: converted_weights/<model_family>
strict_load_status: <not_run | pass | pass_with_documented_exclusions | blocked>
```
For monolithic official checkpoints, record the component prefix split here. For
example, a single checkpoint may contain transformer, VAE/pretransform,
conditioner, and scheduler/vocoder keys that the conversion script writes into
separate FastVideo component subfolders.
## Expected Parity Tests
Planned local tests for this family:
| Component | Official files / args | Test | Concerns | Status |
|---|---|---|---|---|
| `<component>` | `<definition path; instantiation path + args>` | `tests/local_tests/<bucket>/test_<model_family>_<component>_parity.py` | `<prototype or setup concerns>` | `<planned/scaffold_skip/debug_red/non_skip_pass/blocked>` |
| `pipeline` | `<official pipeline call>` | `tests/local_tests/pipelines/test_<model_family>_pipeline_parity.py` | `<pipeline concerns>` | `<planned/scaffold_skip/debug_red/non_skip_pass/blocked>` |
Include reused components in this table. Reuse is accepted only after the
FastVideo component definition and official instantiation arguments have both
been checked and the component parity test passes non-skip.
Run the relevant tests with:
```bash
pytest tests/local_tests/<bucket>/test_<model_family>_<component>_parity.py -v -s
pytest tests/local_tests/pipelines/test_<model_family>_pipeline_parity.py -v -s
```
## Review Notes
- Required before handoff: non-skip PASS for each required component parity
test, including reused components that own weights or numerical behavior.
- Pipeline parity may start as a scaffold, but final handoff requires non-skip
PASS or an explicit blocker accepted through the escape-hatch process.
- User decisions and pause points are tracked as `E###` rows in
`PORT_STATUS.md`; do not rely on chat history for escape-hatch context.
- Review agents should verify this README's setup commands still match the PR,
then run the listed parity tests or report the exact blocker.
@@ -1,69 +0,0 @@
# <Model Family> Port Status
## Summary
- model_family: `<model_family>`
- workload_types: `<T2V/I2V/V2V/T2I/or compatibility shim with rationale>`
- official_ref: `<url or import path>`
- official_ref_dir: `<ReferenceDir or none>`
- hf_weights_path: `<HF repo id/url or local path>`
- local_weights_dir: `<official_weights/model_family or local path>`
- source_layout: `<diffusers/raw_official/monolithic/separate_components/mixed/custom/unknown>`
- local_tests_readme: `tests/local_tests/<model_family>/README.md`
## Current Phase
- phase: `prep`
- status: `in_progress`
- owner: `prep`
- last_updated: `<YYYY-MM-DD>`
## Component Matrix
| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues |
|---|---|---|---|---|---|---|---|---|---|
| `<component>` | `<dit/vae/encoder/generic>` | `<unknown/reuse/port>` | `<path + symbols>` | `<path + args>` | `<target files>` | `<not_started/in_progress/pass/blocked>` | `<not_started/pass/blocked>` | `<not_started/scaffold_skip/debug_red/non_skip_pass/blocked>` | `<none or IDs>` |
## Conversion State
- conversion_script: `scripts/checkpoint_conversion/<model_family>_to_diffusers.py`
- converted_weights_dir: `converted_weights/<model_family>`
- source_layout: `<diffusers/separate_components/monolithic/mixed/custom/unknown>`
- strict_load_status: `not_run`
- passthrough_components: `<none or list>`
- retry_history: `<none>`
## Parity Commands
| Scope | Command | Last Result | Notes |
|---|---|---|---|
| component | `pytest tests/local_tests/<bucket>/test_<model_family>_<component>_parity.py -v -s` | `not_run` | `<notes>` |
| pipeline | `pytest tests/local_tests/pipelines/test_<model_family>_pipeline_parity.py -v -s` | `not_run` | `<notes>` |
## Open Questions
| ID | Question | Owner | Needed By Phase | Status | Resolution |
|---|---|---|---|---|---|
| Q001 | `<question>` | `<owner>` | `<phase>` | `<open/resolved>` | `<resolution or blank>` |
## Issues And Blockers
| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution |
|---|---|---|---|---|---|---|---|---|
| I001 | `<phase>` | `<component or all>` | `<low/medium/high/blocker>` | `<issue>` | `<logs/paths/commands>` | `<owner>` | `<open/resolved>` | `<resolution or blank>` |
## Escape Hatches
| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution |
|---|---|---|---|---|---|---|
| E001 | `<phase>` | `<scope/dependency/auth/cost/destructive/ambiguity/blocker>` | `<one precise question>` | `<safe recommended option>` | `<open/resolved>` | `<resolution or blank>` |
## Decisions
| Date | Decision | Rationale | Impact |
|---|---|---|---|
| `<YYYY-MM-DD>` | `<decision>` | `<why>` | `<affected components/phases>` |
## Handoff Notes
- `<short notes for the next agent>`
-234
View File
@@ -1,234 +0,0 @@
---
name: add-model-02-parity
description: Use during /add-model after reference/architecture study to scaffold and later activate local FastVideo component parity tests. Emphasizes early test creation, official-reference loading, standardized FastVideo loading, and non-skip handoff gates.
---
# Add Model Parity
## Goal
Create parity tests as early as possible in a FastVideo port. The first pass can
land before conversion or component implementation as an executable scaffold;
handoff is blocked until the same tests become non-skip PASS with real weights.
## When To Run
Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
escape hatches, and skip/pass semantics.
Run immediately after `/add-model` Phase 1 has identified:
- official component classes and call signatures;
- FastVideo target component buckets/classes/configs;
- local reference clone or import path from `add-model-01-prep`;
- local raw or Diffusers weight path;
- `official_env_status=imports_ok`, or private deps that will be stubbed
locally in tests;
- `local_tests_readme` documenting setup and planned review/test commands;
- expected component inputs and output tensors.
Do not wait for all FastVideo components to be implemented. Write the tests
first, then let component-porting subagents make them pass.
## Outputs
- One component parity test per required component, including reused components:
`tests/local_tests/<bucket>/test_<family>_<component>_parity.py`.
- Optional helper for upstream private deps:
`tests/local_tests/helpers/<family>_upstream.py`.
- Pipeline parity is owned later by `../add-model-09-pipeline/SKILL.md` after all
component parity tests pass non-skip.
- A parity status block for the `/add-model` parity verification phase.
## Early Scaffold Rules
- A scaffold may skip while the FastVideo class, converted weights, or official
import is missing.
- A scaffold must already encode the real official load path, FastVideo load
path, deterministic inputs, expected output extraction, and tolerance target.
- Each parity test must declare its coverage scope in the file docstring or a
module constant: `production_loader`, `implementation_subcomponent`, or `both`.
Implementation/subcomponent parity may bypass production loaders deliberately,
but final handoff still needs production-loader coverage somewhere before the
pipeline depends on that component.
- Official reference imports must run in the current FastVideo environment; do
not create or assume a separate upstream venv/conda env.
- A scaffold is not evidence of correctness. It becomes evidence only after a
local non-skip PASS.
- Prefer env-var path overrides with repo-relative defaults.
- Keep tests local-only under `tests/local_tests/`; package/CI quality tests are
added later.
- Update shared state files as described in
`../add-model/shared/common_rules.md` whenever adding or activating parity
tests.
## Component Template
Copy `templates/component_parity_test.py` and fill every `TODO` marker. The
template is distilled from:
- `tests/local_tests/transformers/test_ltx2.py`
- `tests/local_tests/transformers/test_gamecraft_parity.py`
- `tests/local_tests/encoders/test_ltx2_gemma_parity.py`
- `tests/local_tests/vaes/test_oobleck_vae_parity.py`
- `tests/local_tests/sd35/test_sd35_component_parity.py`
The template supports three states:
| State | Meaning |
|---|---|
| Scaffold skip | Test is committed early, but official import, FastVideo class, or weights are not available yet. |
| Debug red | Both sides load and the test fails numerically. This is useful: porting can chase the first drift. |
| Non-skip pass | Required before `/add-model` handoff. |
## Subagent Dispatch Pattern
After Phase 1, dispatch one parity subagent per component before or alongside
component implementation:
```text
Create a local parity test scaffold for <family> <component>.
Use the prep handoff:
- official_ref_dir/import: <...>
- local_weights_dir: <...>
- source_layout: <...>
- needs_conversion: <yes/no>
- official_env_status: <imports_ok | private_deps_need_stubs>
- local_tests_readme: tests/local_tests/<model_family>/README.md
- port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
- official_definition_files: <paths + classes/functions>
- official_instantiation_files: <paths + factory/pipeline/config call sites + args>
- concerns_or_unknowns: <known ambiguous inputs, outputs, deps, or args>
The complete per-component packet must match
`../add-model/contracts/component_context.md`.
Read the official component call path and the planned FastVideo component API.
Add tests/local_tests/<bucket>/test_<family>_<component>_parity.py based on
add-model-02-parity/templates/component_parity_test.py.
The scaffold must load the official model with real weights when available,
load the FastVideo model through the standardized config/class/loader path when
available, create deterministic inputs, compare concrete outputs, and skip only
when a dependency is genuinely missing. Do not make an unconditional skip or a
shape-only test.
```
## FastVideo Load Patterns
Pick the narrowest load path that matches the component:
| Component | Preferred FastVideo load path |
|---|---|
| DiT / transformer | Bucket config + model class, or `TransformerLoader` when testing converted Diffusers component dirs. |
| VAE | VAE class `from_pretrained(...)` when implemented, or bucket config + class for local converted dirs. |
| Text/image encoder | Bucket config + model class; pass HF subpaths from `local_weights_dir` or converted component dirs. |
| Scheduler/conditioner | Native class/config plus exact official kwargs. |
For early scaffolds, an import of the planned FastVideo class may be inside a
helper that calls `pytest.skip` if the class does not exist yet. Replace that
skip with a real import once the component PR adds the class.
Direct class/config construction is allowed for implementation or subcomponent
parity, such as connector-only encoder checks or official monolithic-checkpoint
mapping tests. Label that scope explicitly and add separate production-loader
coverage when converted component dirs are available.
## Official Load Patterns
- Clone/reference repo path: add its source dir to `sys.path` before imports.
- HF/Diffusers reference: import only inside the test, not production code.
- Private deps: add a helper under `tests/local_tests/helpers/` to install
stubs before importing upstream modules; do not rely on an external upstream
environment.
- Gated HF repos: resolve `HF_TOKEN`, `HUGGINGFACE_HUB_TOKEN`, or `HF_API_KEY`
under the token rules in `../add-model/shared/common_rules.md`.
## Non-Skip Activation Checklist
Before `/add-model` handoff, each scaffolded test must be activated:
```text
[ ] Official side imports and loads real weights.
[ ] FastVideo side imports and loads the converted or original weights.
[ ] Test executes at least one real forward call on both sides.
[ ] Test compares output tensors, not only shapes or state-dict keys.
[ ] Local pytest output contains PASSED, not SKIPPED or XFAIL.
[ ] Tolerance is justified for the component scope and kernel alignment.
```
## Component Parity Details
Reference imports:
- Import from `official_ref_dir` or the recorded package/import path.
- If upstream has private deps, add a helper under
`tests/local_tests/helpers/<family>_upstream.py` that installs minimal stubs
before importing upstream modules.
- Common stubs: identity compile/op-registration decorators, CP world size set to
1, identity scatter/gather, and test-friendly custom-op kernels.
- Stub decorators that register `torch.ops.<ns>.<op>` must preserve the
`torch.library` registration side-effect. Identity decorators alone are not
enough.
- Delete stub helpers and every `install_stubs()` call as soon as the real deps
become required installs. No-op shims are dead code.
Kernel and wrapper pitfalls:
- If parity routes flash-attn GQA through SDPA, expand KV heads manually on the
SDPA side with `repeat_interleave` along the head axis.
- If upstream VAE `decode()` denormalizes internally but FastVideo/Diffusers
expects pre-denormalized latents, apply `z = z * std + mean` only on the
FastVideo side in the parity test.
- Per-channel VAE `latents_mean` / `latents_std` must be reshaped explicitly,
e.g. `.view(1, z_dim, 1, 1, 1)` for 5D video latents.
Tolerance guide:
| Scope | Start `atol` / `rtol` | Notes |
|---|---|---|
| Single block, same kernel | `1e-4` / `1e-4` | Tight default. |
| Full DiT, aligned kernels | `1e-2` / `1e-2` | Cross-layer accumulation. |
| Full DiT, cross-kernel bf16 | `0.1` / `0.1` | Also require abs-mean drift below 5% and per-modality diagnostics. |
| VAE decode fp32 | `5e-2` / `5e-2` | After normalization alignment. |
| Encoder wrapper around same HF class | `1e-3` / `1e-3` | Should be near-zero. |
Element-wise `assert_close` alone is not enough for deep full-DiT parity. Also
log global abs-mean drift and per-modality summaries.
When a non-skip component parity run is numerically red after weight/input
checks, invoke `../add-model-08-trace/SKILL.md` before adding bespoke forward
hooks. Use `docs/contributing/activation_trace.md` to keep
`FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and `FASTVIDEO_TRACE_STEPS`
identical across FastVideo and upstream traces.
Useful local commands:
```bash
pytest tests/local_tests/<bucket>/test_<family>_*parity*.py -v -s
pytest tests/local_tests -k "<family> and parity" -v -s
```
## Escape Hatches
Follow `../add-model/shared/common_rules.md`. Parity-specific ask cases include
private dependency approval, choosing between incompatible official references,
accepting a shape-only substitute, or loosening required tolerances.
## Pipeline Parity
Pipeline parity is later than component parity because it needs stages, presets,
registry wiring, converted weights, and green component parity. Record official
pipeline call notes in `local_tests_readme`, but do not treat pipeline parity as
owned by this skill.
Use `../add-model-09-pipeline/SKILL.md` and its
`templates/pipeline_parity_test.py` for pipeline parity scaffolding and
debugging. Compare denoised latents or decoded media, not just successful
generation.
## Handoff Status Block
Return `../add-model/contracts/parity_status.md` to `/add-model` and update the
shared state files before handoff.
@@ -1,188 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""Component parity scaffold for <FAMILY> <COMPONENT>.
This file is intended to be created early in a port. It may skip until the
official reference, FastVideo class, and real weights are available, but it must
never become an unconditional skip or shape-only test.
Fill every TODO before considering this test active.
"""
from __future__ import annotations
import importlib
import os
from pathlib import Path
import sys
import pytest
import torch
from torch.testing import assert_close
os.environ.setdefault("MASTER_ADDR", "localhost")
os.environ.setdefault("MASTER_PORT", "29519")
os.environ.setdefault("DISABLE_SP", "1")
os.environ.setdefault("FASTVIDEO_ATTENTION_BACKEND", "TORCH_SDPA")
REPO_ROOT = Path(__file__).resolve().parents[3]
FAMILY = "<family>" # TODO: snake_case family name.
COMPONENT = "<component>" # TODO: transformer | vae | encoder | conditioner | ...
PARITY_SCOPE = "implementation_subcomponent" # TODO: production_loader | implementation_subcomponent | both
OFFICIAL_MODULE = "<official.module>" # TODO: e.g. "ltx_core.model.transformer".
OFFICIAL_CLASS = "<OfficialClass>" # TODO: official class/factory name.
FASTVIDEO_CONFIG_MODULE = "fastvideo.configs.models.<bucket>" # TODO.
FASTVIDEO_CONFIG_CLASS = "<FastVideoConfig>" # TODO.
FASTVIDEO_MODEL_MODULE = "fastvideo.models.<bucket>.<module>" # TODO.
FASTVIDEO_MODEL_CLASS = "<FastVideoModel>" # TODO.
OFFICIAL_REF_DIR = Path(os.getenv("<FAMILY_UPPER>_OFFICIAL_REF_DIR", REPO_ROOT / "<ReferenceDir>"))
LOCAL_WEIGHTS_DIR = Path(os.getenv("<FAMILY_UPPER>_LOCAL_WEIGHTS_DIR", REPO_ROOT / "official_weights" / FAMILY))
CONVERTED_WEIGHTS_DIR = Path(os.getenv("<FAMILY_UPPER>_CONVERTED_WEIGHTS_DIR",
REPO_ROOT / "converted_weights" / FAMILY))
def _resolve_hf_token() -> str | None:
for key in ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY"):
value = os.environ.get(key)
if value:
return value
return None
def _add_official_to_path() -> None:
"""Add the official source path before importing upstream modules."""
# TODO: adjust for the official repo layout. Common examples:
# OFFICIAL_REF_DIR / "src"
# OFFICIAL_REF_DIR / "packages" / "<pkg>" / "src"
# OFFICIAL_REF_DIR
official_src = OFFICIAL_REF_DIR / "src"
if not official_src.exists():
official_src = OFFICIAL_REF_DIR
if official_src.exists() and str(official_src) not in sys.path:
sys.path.insert(0, str(official_src))
def _import_or_skip(module_name: str, attr_name: str | None = None):
if "<" in module_name or (attr_name is not None and "<" in attr_name):
pytest.skip(f"Template import placeholder not filled: {module_name}.{attr_name}")
try:
module = importlib.import_module(module_name)
except Exception as exc: # noqa: BLE001 - local parity should skip missing refs.
pytest.skip(f"Cannot import {module_name}: {exc}")
if attr_name is None:
return module
try:
return getattr(module, attr_name)
except AttributeError:
pytest.skip(f"{module_name} has no attribute {attr_name}")
def _load_official_model(device: torch.device, dtype: torch.dtype) -> torch.nn.Module:
"""Load the official component with real weights."""
_add_official_to_path()
if not OFFICIAL_REF_DIR.exists():
pytest.skip(f"Official reference missing: {OFFICIAL_REF_DIR}")
if not LOCAL_WEIGHTS_DIR.exists():
pytest.skip(f"Local weights missing: {LOCAL_WEIGHTS_DIR}")
# TODO: import official class/factory and load real weights strictly.
# Examples in-tree:
# - LTX2: SingleGPUModelBuilder(...).build(device=device, dtype=dtype)
# - GameCraft: torch.load(...)["module"] -> official_model.load_state_dict(...)
# - Oobleck: create_model_from_config(config) + ckpt state_dict
OfficialClass = _import_or_skip(OFFICIAL_MODULE, OFFICIAL_CLASS)
model = OfficialClass() # TODO: pass official config kwargs.
state_dict = {} # TODO: load official state dict from LOCAL_WEIGHTS_DIR.
missing, unexpected = model.load_state_dict(state_dict, strict=True)
assert not missing and not unexpected, (f"official load mismatch missing={missing[:5]} unexpected={unexpected[:5]}")
return model.to(device=device, dtype=dtype).eval()
def _load_fastvideo_model(device: torch.device, dtype: torch.dtype) -> torch.nn.Module:
"""Load the FastVideo component with the same tensor content."""
if not CONVERTED_WEIGHTS_DIR.exists() and not LOCAL_WEIGHTS_DIR.exists():
pytest.skip(f"No FastVideo loadable weights: {CONVERTED_WEIGHTS_DIR} or {LOCAL_WEIGHTS_DIR}")
# TODO: replace with the bucket-specific FastVideo config/class/loader.
# DiT examples:
# from fastvideo.configs.models.dits import <Config>
# from fastvideo.models.dits.<module> import <Model>
# VAE examples:
# from fastvideo.models.vaes.<module> import <VAE>
# model = <VAE>.from_pretrained(...)
FastVideoConfig = _import_or_skip(FASTVIDEO_CONFIG_MODULE, FASTVIDEO_CONFIG_CLASS)
FastVideoModel = _import_or_skip(FASTVIDEO_MODEL_MODULE, FASTVIDEO_MODEL_CLASS)
config = FastVideoConfig()
model = FastVideoModel(config=config)
state_dict = {} # TODO: load converted or directly mapped state dict.
missing, unexpected = model.load_state_dict(state_dict, strict=True)
assert not missing and not unexpected, (
f"FastVideo load mismatch missing={missing[:5]} unexpected={unexpected[:5]}")
return model.to(device=device, dtype=dtype).eval()
def _make_inputs(device: torch.device, dtype: torch.dtype) -> dict[str, torch.Tensor]:
"""Create deterministic inputs matching the official component call."""
torch.manual_seed(0)
# TODO: replace with component-specific tensors and metadata.
return {
"hidden_states": torch.randn(1, 4, 16, device=device, dtype=dtype),
"timestep": torch.tensor([10], device=device),
}
def _run_official(model: torch.nn.Module, inputs: dict[str, torch.Tensor]) -> torch.Tensor:
"""Run official component and return the tensor to compare."""
with torch.inference_mode():
output = model(**inputs) # TODO: adapt official call signature.
if isinstance(output, dict):
sample = output.get("sample")
output = sample if sample is not None else output.get("x")
elif hasattr(output, "sample"):
output = output.sample
elif isinstance(output, tuple):
output = output[0]
assert torch.is_tensor(output), f"official output is not tensor: {type(output)}"
return output.detach().float().cpu()
def _run_fastvideo(model: torch.nn.Module, inputs: dict[str, torch.Tensor]) -> torch.Tensor:
"""Run FastVideo component and return the tensor to compare."""
with torch.inference_mode():
output = model(**inputs) # TODO: adapt FastVideo call signature.
if isinstance(output, dict):
sample = output.get("sample")
output = sample if sample is not None else output.get("x")
elif hasattr(output, "sample"):
output = output.sample
elif isinstance(output, tuple):
output = output[0]
assert torch.is_tensor(output), f"FastVideo output is not tensor: {type(output)}"
return output.detach().float().cpu()
@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required for this parity test.")
def test_component_parity():
"""Compare official and FastVideo outputs on identical inputs."""
device = torch.device("cuda:0")
dtype = torch.bfloat16
official = _load_official_model(device, dtype)
fastvideo = _load_fastvideo_model(device, dtype)
inputs = _make_inputs(device, dtype)
official_out = _run_official(official, inputs)
fastvideo_out = _run_fastvideo(fastvideo, inputs)
assert official_out.shape == fastvideo_out.shape
diff = (official_out - fastvideo_out).abs()
print(f"official abs_mean={official_out.abs().mean().item():.6f} "
f"fastvideo abs_mean={fastvideo_out.abs().mean().item():.6f} "
f"diff_max={diff.max().item():.6f} diff_mean={diff.mean().item():.6f}")
# TODO: pick tolerance by scope:
# - single block / same kernel: 1e-4
# - full DiT aligned kernels: 1e-2
# - full DiT cross-kernel bf16: 1e-1 + abs_mean drift check
# - VAE decode fp32: 5e-2 after normalization alignment
assert_close(fastvideo_out, official_out, atol=1e-4, rtol=1e-4)
@@ -1,114 +0,0 @@
---
name: add-model-03-port-dit
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native DiT/transformer component.
---
# Add Model Port DiT
## Goal
Prototype or parity-debug one diffusion transformer in FastVideo-native code.
This skill is for one component only; do not work on the VAE, encoders,
pipeline, or unrelated conversion code unless the current component cannot load
without a minimal fix there.
## Inputs
Follow `../add-model/shared/component_skill_common.md` and require the complete
packet from `../add-model/contracts/component_context.md`.
DiT-specific packet fields:
- `component`: transformer or DiT name.
- `parity_test`: `tests/local_tests/<bucket>/test_<family>_<component>_parity.py`.
- `weights`: converted transformer dir or local official path.
- `target_files`: `fastvideo/models/dits/<family>.py` and
`fastvideo/configs/models/dits/<family>.py`.
## Modes
Use the common prototype and parity-debug modes from
`../add-model/shared/component_skill_common.md`.
DiT-specific prototype concerns include ambiguous official flags, shape
mismatches, missing FastVideo layer equivalents, and dedicated output heads.
## Reuse Proof
Apply the shared reuse proof. DiT-specific comparison must include attention
algorithm, positional embeddings, RoPE/patching, timestep/guidance embeddings,
scaling constants, dtype casts, state-dict names, and every output head.
## Existing FastVideo Patterns
- Base class: `fastvideo/models/dits/base.py::BaseDiT`.
- Config bases: `DiTConfig` and `DiTArchConfig` in
`fastvideo/configs/models/dits/base.py`.
- Use the matching DiT config bucket. Wrong bucket inheritance can typecheck but
fail during pipeline wiring.
- Config export: add the config to
`fastvideo/configs/models/dits/__init__.py`.
- Registry discovery: set `EntryClass = <ClassName>` in the model file.
- Loader path: `TransformerLoader` reads `transformer/config.json`, calls
`dit_config.update_model_arch(config)`, resolves `_class_name` through
`ModelRegistry`, and constructs the class with `config` and `hf_config`.
- Reference examples: `stable_audio.py`, `wanvideo.py`, `sd3.py`, `longcat.py`,
and `ltx2.py`.
- Layer guidance: `fastvideo/layers/AGENTS.md`.
## Implementation Rules
- Use FastVideo-native layers by default: `ReplicatedLinear` for DiT hot-path
linears, `DistributedAttention` for standard full-sequence attention, and
`LocalAttention` for local/window attention or simple single-GPU parity paths.
- Raw SDPA is acceptable for cross-modality flat streams when no FastVideo
distributed equivalent exists; document the SP gap in the module docstring.
- Mirror official tensor contracts exactly: latent packing, patch ordering,
timestep embedding scale, RoPE/positional embedding, guidance embedding,
cross-attention context order, output head order, and dtype casts.
- Preserve all output heads that the official DiT emits. Do not silently drop
audio, depth, pose, mask, or auxiliary heads.
- Put architecture fields on `DiTArchConfig`; keep inference steps, CFG scales,
FPS, flow shift, and sampling defaults out of the arch config.
- Define `_fsdp_shard_conditions`, `_compile_conditions`,
`param_names_mapping`, and `reverse_param_names_mapping` where needed.
- Follow the production import boundary in
`../add-model/shared/common_rules.md`.
## Prototype Checks
Follow the shared prototype success criteria. A useful one-off check is:
```bash
python - <<'PY'
# Import the target config/class, instantiate with random weights, and print
# state_dict names/shapes for the conversion mapping.
PY
```
## Parity-Debug Loop
Run the shared parity-debug loop. The component test command is:
```bash
pytest <parity_test> -v -s
```
For numerical drift, use `../add-model-08-trace/SKILL.md` before writing bespoke
hooks. Start with FastVideo's activation trace (`fastvideo/hooks/activation_trace.py`;
`docs/contributing/activation_trace.md`) and a block-level regex such as
`FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$"`. Only fall back to custom
per-block hooks if the needed boundary or statistic is not exposed by
`FASTVIDEO_TRACE_STATS`.
## Escape Hatches
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
in `../add-model/shared/component_skill_common.md`. DiT-specific ask cases include
dropping an output head/modality, accepting an unsupported kernel/private op, or
choosing between incompatible official transformer definitions.
## Handoff
Return `../add-model/contracts/component_skill_handoff.md` following the common
handoff rules in `../add-model/shared/component_skill_common.md`.
@@ -1,100 +0,0 @@
---
name: add-model-04-port-vae
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native VAE component.
---
# Add Model Port VAE
## Goal
Prototype or parity-debug one VAE or autoencoder in FastVideo-native code. This
skill covers video, image, and audio VAEs.
## Inputs
Follow `../add-model/shared/component_skill_common.md` and require the complete
packet from `../add-model/contracts/component_context.md`.
VAE-specific packet fields:
- `component`: VAE or autoencoder name.
- `parity_test`: `tests/local_tests/vaes/test_<family>_<component>_parity.py`.
- `weights`: converted VAE dir, HF subfolder, or local official path.
- `target_files`: `fastvideo/models/vaes/<arch_or_family>.py` and
`fastvideo/configs/models/vaes/<arch_or_family>.py`.
## Modes
Use the common prototype and parity-debug modes from
`../add-model/shared/component_skill_common.md`.
VAE-specific prototype concerns include latent normalization, stochastic
posterior behavior, tiling incompatibility, temporal/spatial/audio layout, and
decode output containers.
## Reuse Proof
Apply the shared reuse proof. VAE-specific comparison must include latent layout,
temporal/spatial/audio compression, scaling factor, mean/std normalization,
posterior behavior, encode/decode output objects, tiling flags, and cropping.
## Existing FastVideo Patterns
- Shared tiling wrapper: `fastvideo/models/vaes/common.py::ParallelTiledVAE`.
- Config bases: `VAEConfig` and `VAEArchConfig` in
`fastvideo/configs/models/vaes/base.py`.
- Use the matching VAE config bucket. Wrong bucket inheritance can typecheck but
fail during pipeline wiring.
- Config export: add the config to
`fastvideo/configs/models/vaes/__init__.py`.
- Registry discovery: set `EntryClass = <ClassName>` in the model file.
- Loader path: VAE loaders resolve `_class_name` through `ModelRegistry` and
load converted component weights from the VAE subdir.
- Reference examples: `oobleck.py`, `autoencoder_kl.py`, `wanvae.py`,
`ltx2vae.py`, and `gamecraftvae.py`.
- Layer guidance: `fastvideo/layers/AGENTS.md`.
## Implementation Rules
- Name reusable VAE architectures by architecture (`oobleck.py`,
`autoencoder_kl.py`); name family-specific VAEs by family.
- Match official encode/decode contracts exactly: input layout, latent layout,
temporal/spatial/audio compression, scaling factor, mean/std normalization,
posterior sampling behavior, decode output object, and frame/sample cropping.
- Compare deterministic outputs in parity: decode outputs, encode mean/mode, or
round-trip tensors. Do not compare stochastic samples unless the RNG path is
explicitly controlled.
- Use FastVideo tiling only when it preserves official numerics for the tested
shape; disable it in config for audio or unsupported dimensions.
- Put architecture constants on `VAEArchConfig`; put `load_encoder`,
`load_decoder`, tiling, dtype, and pretrained path fields on `VAEConfig`.
- Follow the production import boundary in
`../add-model/shared/common_rules.md`.
## Prototype Checks
Follow the shared prototype success criteria.
## Parity-Debug Loop
Run the shared parity-debug loop. The component test command is:
```bash
pytest <parity_test> -v -s
```
For numerical drift, check normalization, latent scaling, posterior mode vs
sample, channel order, and temporal/spatial/audio cropping before changing
layers.
## Escape Hatches
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
in `../add-model/shared/component_skill_common.md`. VAE-specific ask cases include
dropping an encode/decode path, accepting an unsupported private op, or choosing
between incompatible official VAE definitions.
## Handoff
Return `../add-model/contracts/component_skill_handoff.md` following the common
handoff rules in `../add-model/shared/component_skill_common.md`.
@@ -1,119 +0,0 @@
---
name: add-model-05-port-encoder
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native text, image, audio, or compound encoder component.
---
# Add Model Port Encoder
## Goal
Prototype or parity-debug one encoder or encoder-like conditioner in
FastVideo-native code. Use this for text encoders, image encoders, audio
encoders, and compound conditioners that fit the encoder config/loader bucket.
## Inputs
Follow `../add-model/shared/component_skill_common.md` and require the complete
packet from `../add-model/contracts/component_context.md`.
Encoder-specific packet fields:
- `component`: encoder or encoder-like conditioner name.
- `parity_test`: `tests/local_tests/encoders/test_<family>_<component>_parity.py`.
- `weights`: converted encoder dir, HF subfolder, or external HF id.
- `target_files`: `fastvideo/models/encoders/<arch_or_family>.py` and
`fastvideo/configs/models/encoders/<arch_or_family>.py`.
## Modes
Use the common prototype and parity-debug modes from
`../add-model/shared/component_skill_common.md`.
Encoder-specific prototype concerns include tokenizer kwargs, hidden-state
extraction, output packing, connector order, and external/passthrough weight
needs.
## Reuse Proof
Apply the shared reuse proof. Encoder-specific comparison must include tokenizer
contracts, hidden-state extraction, masks, positional IDs, output packing,
connector/projection ordering, passthrough paths, and returned dataclass shape.
## Existing FastVideo Patterns
- Base classes: `TextEncoder` and `ImageEncoder` in
`fastvideo/models/encoders/base.py`.
- Output type: `BaseEncoderOutput`.
- Config bases: `TextEncoderConfig`, `ImageEncoderConfig`,
`TextEncoderArchConfig`, and `ImageEncoderArchConfig` in
`fastvideo/configs/models/encoders/base.py`.
- Use the matching encoder config bucket. Wrong bucket inheritance can typecheck
but fail during pipeline wiring.
- Config export: add the config to
`fastvideo/configs/models/encoders/__init__.py`.
- Registry discovery: set `EntryClass = <ClassName>` or a list of class names in
the model file.
- Reference examples: native `t5.py`, `clip.py`, `siglip.py`, `llama.py`,
`qwen2_5.py`, `gemma.py`, and compound `stable_audio_conditioner.py`.
- Layer guidance: `fastvideo/layers/AGENTS.md`.
## Implementation Rules
- Reuse tokenizers and pure data utilities when needed, but do not add runtime
third-party model-class imports as a placeholder for a component that owns
weights or numerical behavior.
- For LLM-style encoders, follow existing tensor-parallel patterns such as
`QKVParallelLinear`, `MergedColumnParallelLinear`, `RowParallelLinear`,
`VocabParallelEmbedding`, and `RMSNorm` when matching native examples.
- Match official hidden-state extraction exactly: layer index, pooled output,
attention mask dtype, padding side, truncation, special tokens, final norm,
output_hidden_states, and returned tuple/dataclass shape.
- For connector or conditioner modules, preserve sub-conditioner order and the
exact packing of cross-attention tokens, masks, and global conditioning.
- Put tokenizer kwargs and architecture constants on the arch config when they
affect numerical behavior.
- If an external HF encoder is explicitly accepted as a lazy wrapper, keep it
isolated, document why it is not a native port, and still require parity for
the wrapper's output contract.
Hybrid external-HF encoder checklist:
- Put external model folders in passthrough subfolders such as
`text_encoder/<external_name>/`, or record a root `model_index.json` path field
that the loader resolves to a local directory.
- Keep external model parameters out of the FastVideo-owned state-dict surface
when the external model is loaded lazily from its own HF files.
- Convert and strict-check only the FastVideo-owned connector/projection weights;
document external model weights as passthrough.
- Add parity for the wrapper's final output contract and, when useful, a narrower
connector-only parity test that labels its scope as
`implementation_subcomponent`.
- Verify the production loader resolves the same external path used by the
pipeline, not just the direct class used in the parity test.
## Prototype Checks
Follow the shared prototype success criteria.
## Parity-Debug Loop
Run the shared parity-debug loop. The component test command is:
```bash
pytest <parity_test> -v -s
```
For numerical drift, check tokenization, masks, hidden-state selection,
positional IDs, dtype/autocast, and output packing before changing layers.
## Escape Hatches
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
in `../add-model/shared/component_skill_common.md`. Encoder-specific ask cases
include accepting private model-code execution, choosing between incompatible
tokenizer/encoder references, or dropping a required conditioning stream.
## Handoff
Return `../add-model/contracts/component_skill_handoff.md` following the common
handoff rules in `../add-model/shared/component_skill_common.md`.
@@ -1,111 +0,0 @@
---
name: add-model-06-port-generic
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one non-DiT, non-VAE, non-encoder FastVideo component.
---
# Add Model Port Generic
## Goal
Prototype or parity-debug one scheduler, conditioner, upsampler, vocoder,
adapter, preprocessor, or unknown component in FastVideo-native code.
## Inputs
Follow `../add-model/shared/component_skill_common.md` and require the complete
packet from `../add-model/contracts/component_context.md`.
Generic-component packet fields:
- `component`: component name.
- `component_type`: scheduler, conditioner, upsampler, vocoder, adapter,
preprocessor, or unknown.
- `parity_test`: `tests/local_tests/<bucket>/test_<family>_<component>_parity.py`.
- `weights`: converted component dir, HF subfolder, or none.
- `target_files`: matching `fastvideo/models/` and `fastvideo/configs/models/`
bucket files when applicable.
## Modes
Use the common prototype and parity-debug modes from
`../add-model/shared/component_skill_common.md`.
Generic-component prototype concerns include stateless/stateful ambiguity,
missing loader buckets, source prefixes, mutable scheduler state, and output
container shape.
## Reuse Proof
Apply the shared reuse proof. Generic-component comparison must include mutable
state, scaling constants, scheduler/conditioner semantics, output containers, and
whether the component owns state or is stateless.
## Existing FastVideo Patterns
- Schedulers live under `fastvideo/models/schedulers/` and expose `EntryClass`.
- Upsamplers use `fastvideo/models/upsamplers/` plus configs under
`fastvideo/configs/models/upsamplers/`; see `hunyuan15.py`.
- Vocoders and audio-specific modules can live under `fastvideo/models/audio/`
with configs under `fastvideo/configs/models/audio/`; see `ltx2_audio_vae.py`.
- Compound conditioners may fit the encoder bucket when the pipeline loader uses
`ConditionerLoader`; see `stable_audio_conditioner.py`.
- Registry discovery uses `EntryClass`; config bucket exports are required when
pipeline configs import them by bucket.
- Use the narrowest matching config bucket. Wrong bucket inheritance can typecheck
but fail during pipeline wiring.
- Layer guidance: `fastvideo/layers/AGENTS.md`.
## Bucket Decision
- If the component is a transformer/DiT, stop and use `add-model-03-port-dit`.
- If the component is a VAE/autoencoder, stop and use `add-model-04-port-vae`.
- If the component is a text/image/audio encoder or encoder-like conditioner,
stop and use `add-model-05-port-encoder` unless the loader requires a different
bucket.
- Otherwise choose the narrowest existing bucket. Add a new bucket only when no
existing loader/config shape can represent the component without misleading
names or unsafe runtime behavior.
## Implementation Rules
- Match official behavior, not just shapes: constructor args, default values,
runtime flags, RNG use, dtype/autocast, scaling constants, masks, and output
containers all matter.
- Keep the implementation minimal and native. Do not keep a runtime import of
the official implementation as the production component.
- For schedulers, compare timesteps, sigmas/noise levels, step outputs, shift
handling, prediction type, and any mutable internal state.
- For upsamplers, compare resize mode, align_corners, residual branches,
causal padding, normalization, and exact target-shape behavior.
- For vocoders/audio components, compare waveform shape, sample-rate contract,
channel order, hop length, normalization, and dtype.
- If private upstream deps are required only for tests, keep stubs under
`tests/local_tests/helpers/` and do not import them from production code.
## Prototype Checks
Follow the shared prototype success criteria.
## Parity-Debug Loop
Run the shared parity-debug loop. The component test command is:
```bash
pytest <parity_test> -v -s
```
For numerical drift, add targeted intermediate comparisons in the test to
identify the first divergent operation.
## Escape Hatches
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
in `../add-model/shared/component_skill_common.md`. Generic-component ask cases
include creating a new loader bucket, accepting an unsupported private op,
choosing between incompatible official definitions, or dropping a required
component.
## Handoff
Return `../add-model/contracts/component_skill_handoff.md` following the common
handoff rules in `../add-model/shared/component_skill_common.md`.
@@ -1,186 +0,0 @@
---
name: add-model-07-conversion
description: Use during /add-model Phase 5 to write and verify a FastVideo checkpoint conversion script after native component prototypes expose FastVideo state-dict keys/shapes.
---
# Add Model Conversion
## Goal
Convert official weights into a FastVideo-loadable component layout after Phase 4
native prototypes exist. The conversion script owns parameter mapping, component
splitting, passthrough assets, config emission, and strict-load verification.
## Inputs
Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
escape hatches, production boundaries, and skip/pass semantics.
Require the initial request from
`../add-model/contracts/conversion_request.md`.
If the FastVideo key/shape dump is missing, return to `/add-model` Phase 4. Do
not write a final mapping against an unimplemented component.
For Phase 6 retry requests from component skills, also require the retry shape
from `../add-model/contracts/conversion_request.md`.
## Output
- `scripts/checkpoint_conversion/<family>_to_diffusers.py`.
- `converted_weights/<family>/` with `model_index.json` and per-component
subfolders.
- Updated `tests/local_tests/<model_family>/README.md` with conversion command,
source layout, output path, and strict-load status.
- Updated `tests/local_tests/<model_family>/PORT_STATUS.md` with conversion
state, retry history, open questions, and issues/blockers.
## Reference Scripts
- `scripts/checkpoint_conversion/convert_ltx2_weights.py`: component prefix
splitting, metadata config extraction, passthrough Gemma/tokenizer assets, and
optional component-only output.
- `scripts/checkpoint_conversion/stable_audio_to_diffusers.py`: monolithic
`model.safetensors` split into transformer/VAE/conditioner, plus copied
passthrough subfolders. Use this shape for single-checkpoint official repos.
- `scripts/checkpoint_conversion/convert_gamecraft_full.py`: separate official
sources for transformer, VAE, text encoders, tokenizers, scheduler, and root
`model_index.json`.
- `scripts/checkpoint_conversion/longcat_to_fastvideo.py`: fused QKV/KV split,
renamed native transformer weights, and copied existing Diffusers components.
- `scripts/checkpoint_conversion/pt_to_safetensors.py`: simple `.pt` extraction
helper for nested checkpoint dictionaries.
## Source Layout Decision
Choose exactly one primary layout:
| Layout | Conversion behavior |
|---|---|
| `diffusers` | Usually no tensor remap; verify configs/classes and copy or update `_class_name` only when needed. |
| `raw_official` | Convert a raw official checkpoint file or directory. Choose explicit component ownership before writing output. |
| `separate_components` | Convert/copy each component from its own file or directory. |
| `monolithic` | Load one model checkpoint and split state dict by authoritative prefixes into component buckets. |
| `mixed` | Convert some components and copy passthrough components such as tokenizers, text encoders, schedulers, or already-Diffusers VAE dirs. |
| `custom` | Document why none of the above fits before writing conversion code. |
Monolithic checkpoints need explicit prefix ownership. For example, Stable Audio
uses one `model.safetensors` with DiT, pretransform/VAE, and conditioner keys;
the converter splits those keys into FastVideo component subfolders and writes
per-component configs.
## Script Shape
Start from `templates/family_to_diffusers.py` or the closest reference script.
Keep the script explicit and reviewable:
- `COMPONENT_SPECS` or `COMPONENT_PREFIXES` declares component ownership.
- `PARAM_NAME_MAP` declares key renames.
- `SKIP_PATTERNS` declares intentionally dropped training-only keys.
- tensor split/fuse helpers are named by operation, e.g. `split_qkv`.
- `build_component_configs(...)` writes loader-compatible config files. Most
model components use `config.json`; schedulers use `scheduler_config.json`.
- `build_model_index(...)` writes a root `model_index.json` matching the target
FastVideo pipeline and component classes.
- verification reports missing, unexpected, skipped, unchanged, renamed, and
shape-mismatched keys.
`model_index.json` library tokens must match FastVideo loaders:
- standard native DiT/VAE/audio/vocoder/upsampler components loaded by existing
Diffusers-style loaders usually use `"diffusers"` with a FastVideo
`_class_name` in the component `config.json`;
- text encoders, tokenizers, image encoders, processors, and feature extractors
usually use `"transformers"`;
- `conditioner` currently expects `"fastvideo"`;
- use fully qualified `"fastvideo.<module>"` only when intentionally relying on
the custom fastvideo-library escape path;
- do not write bare `"fastvideo"` for transformer, VAE, or other loaders that
expect `"diffusers"` unless the loader explicitly expects it.
## Mapping Rules
- Use Phase 4 key/shape dumps to derive mappings. Do not guess from official key
names alone.
- Preserve each component's official file paths, parity test path, and prototype
concerns in comments or structured constants near the mapping that uses them.
- Every official inference parameter should be mapped, copied through, or listed
as intentionally skipped with a reason.
- Every FastVideo prototype parameter should receive a tensor or be listed as an
intentional external/passthrough parameter.
- Shape matches are necessary but not sufficient; check semantic pairing for
Q/K/V, gate/up/down, norm scale/bias, LoRA/base, and modality-specific heads.
- If official and FastVideo fuse or split tensors differently, convert tensors in
the script rather than changing production code to match checkpoint quirks.
## Verification
Run conversion locally, then verify before returning to Phase 6. For retry
requests, update the mapping, rerun conversion, and refresh only the implicated
converted component when safe; otherwise rerun the full conversion.
```bash
python scripts/checkpoint_conversion/<family>_to_diffusers.py \
--src <official_weights> \
--revision <hf_revision> \
--dst converted_weights/<model_family>
```
Omit `--revision` for local sources or when prep recorded `default` / `none`.
Minimum output layout:
```text
converted_weights/<family>/
model_index.json
transformer/config.json
transformer/*.safetensors
vae/config.json
vae/*.safetensors
scheduler/scheduler_config.json as needed
text_encoder/... as needed
```
Required checks:
- `model_index.json` exists and lists every required component.
- Each converted component has the config filename its loader expects and
safetensors weights when it owns weights. Scheduler dirs require
`scheduler_config.json`; most other native model dirs use `config.json`.
- Weight filenames may vary by loader: transformer and VAE loaders glob all
`*.safetensors`; text encoders may load `*.safetensors`, `*.bin`, and
sometimes `*.pt`; `conditioner` currently expects
`diffusion_pytorch_model.safetensors`. Use the loader's actual accepted layout
rather than assuming one global filename.
- Passthrough components are copied or referenced deliberately.
- Each emitted component config validates through the same path production
loaders use. Instantiate the relevant config and call `update_model_arch(...)`
or `update_model_config(...)` with the emitted JSON so unknown keys fail during
conversion, not at pipeline load time.
- Record production loader strictness for every stateful component. If the loader
intentionally uses non-strict loading, add explicit missing/unexpected-key
assertions in the parity test and document exactly which keys are allowed.
- Each new FastVideo component strict-loads converted weights where its production
loader is strict. If strict loading is impossible, record the exact allowed
missing/unexpected keys and why they are not inference weights.
- Retry fixes include the original component parity evidence and the new
strict-load result in `local_tests_readme` so the component subagent can resume
without rediscovering context.
- `local_tests_readme` records the command, output directory, and strict-load
result.
Do not chase numerical parity in this skill except to identify a conversion
mapping bug. Long parity-debug loops belong to `/add-model` Phase 6.
## Escape Hatches
Follow `../add-model/shared/common_rules.md`. Conversion-specific ask cases
include selecting between incompatible official checkpoints, publishing/uploading
weights, overwriting an existing converted repo not created by this run,
accepting non-strict missing inference weights, or dropping a component/output
from scope.
## Handoff
Return `../add-model/contracts/conversion_handoff.md` and update the shared state
files before handoff.
@@ -1,293 +0,0 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
"""Convert <model_family> official weights to a FastVideo Diffusers-style tree.
This template supports both separate component sources and a monolithic pipeline
checkpoint that must be split by component prefix. Replace every TODO before
using it for a real port.
"""
from __future__ import annotations
import argparse
import json
import os
import re
import shutil
from collections import OrderedDict
from pathlib import Path
from typing import Any
import torch
from safetensors import safe_open
from safetensors.torch import load_file, save_file
try:
from huggingface_hub import snapshot_download
except ImportError: # pragma: no cover - optional local conversion dependency
snapshot_download = None
# TODO: fill with authoritative component prefixes for monolithic checkpoints.
# Example: {"model.model.": "transformer", "pretransform.model.": "vae"}
COMPONENT_PREFIXES: dict[str, str] = {}
# TODO: fill with component-specific source paths for separate-component repos.
# Example: {"transformer": "transformer/model.safetensors", "vae": "vae/"}
SEPARATE_COMPONENT_PATHS: dict[str, str] = {}
# TODO: copy passthrough dirs that are already loadable by FastVideo/Diffusers.
PASSTHROUGH_SUBFOLDERS: tuple[str, ...] = ("tokenizer", "scheduler")
# TODO: add regex renames derived from Phase 4 key/shape dumps.
PARAM_NAME_MAP: dict[str, str] = {}
# TODO: include training-only or dynamically-computed keys that must not load.
SKIP_PATTERNS: tuple[str, ...] = ()
def _hf_token() -> str | None:
return (os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACE_HUB_TOKEN") or os.environ.get("HF_API_KEY"))
def resolve_src(src: str, revision: str | None) -> Path:
if os.path.exists(src):
return Path(src)
if snapshot_download is None:
raise RuntimeError("huggingface_hub is required when --src is a repo id")
return Path(snapshot_download(repo_id=src, revision=revision, token=_hf_token()))
def load_checkpoint(path: Path) -> dict[str, torch.Tensor]:
if path.is_dir():
weights: dict[str, torch.Tensor] = {}
for shard in sorted(path.glob("*.safetensors")):
weights.update(load_file(str(shard)))
if weights:
return weights
raise FileNotFoundError(f"No safetensors found in {path}")
if path.suffix == ".safetensors":
return load_file(str(path))
checkpoint = torch.load(path, map_location="cpu", weights_only=True)
if isinstance(checkpoint, dict):
for key in ("state_dict", "model_state_dict", "model", "module", "ema"):
if key in checkpoint and isinstance(checkpoint[key], dict):
return checkpoint[key]
return checkpoint
raise TypeError(f"Unsupported checkpoint type: {type(checkpoint)!r}")
def should_skip_key(key: str) -> bool:
return any(re.search(pattern, key) for pattern in SKIP_PATTERNS)
def apply_mapping(key: str) -> str | None:
if should_skip_key(key):
return None
for pattern, replacement in PARAM_NAME_MAP.items():
if re.match(pattern, key):
return re.sub(pattern, replacement, key)
return key
def split_monolithic(state: dict[str, torch.Tensor], ) -> dict[str, OrderedDict[str, torch.Tensor]]:
components: dict[str, OrderedDict[str, torch.Tensor]] = {
name: OrderedDict()
for name in set(COMPONENT_PREFIXES.values())
}
intentionally_skipped: list[str] = []
unowned: list[str] = []
for key, value in state.items():
if should_skip_key(key):
intentionally_skipped.append(key)
continue
for prefix, component in COMPONENT_PREFIXES.items():
if key.startswith(prefix):
mapped = apply_mapping(key[len(prefix):])
if mapped is not None:
components[component][mapped] = value
break
else:
unowned.append(key)
if unowned:
sample = ", ".join(unowned[:10])
raise ValueError(f"Unowned monolithic keys: {len(unowned)}. "
f"Add COMPONENT_PREFIXES or SKIP_PATTERNS entries. Sample: {sample}")
if intentionally_skipped:
print(f"Intentionally skipped {len(intentionally_skipped)} keys")
return {name: weights for name, weights in components.items() if weights}
def load_separate_components(src_dir: Path) -> dict[str, OrderedDict[str, torch.Tensor]]:
components: dict[str, OrderedDict[str, torch.Tensor]] = {}
for component, rel_path in SEPARATE_COMPONENT_PATHS.items():
state = load_checkpoint(src_dir / rel_path)
converted: OrderedDict[str, torch.Tensor] = OrderedDict()
for key, value in state.items():
mapped = apply_mapping(key)
if mapped is not None:
converted[mapped] = value
components[component] = converted
return components
def build_component_configs(_src_dir: Path) -> dict[str, dict[str, Any]]:
# TODO: emit config content accepted by FastVideo loaders. Most components use
# config.json; schedulers use scheduler_config.json.
return {
"transformer": {
"_class_name": "<FastVideoTransformerClass>"
},
"vae": {
"_class_name": "<FastVideoVAEClass>"
},
}
def config_filename(component: str) -> str:
if component == "scheduler":
return "scheduler_config.json"
return "config.json"
def source_label(src: str) -> str:
if os.path.exists(src):
return Path(src).name
return src
def build_model_index(
src: str,
revision: str | None,
available_components: set[str],
) -> dict[str, Any]:
# TODO: match the target pipeline and every required component.
index: dict[str, Any] = {
"_class_name": "<FastVideoPipelineClass>",
"_diffusers_version": "0.30.0",
"_fastvideo_converted_from": source_label(src),
# Existing transformer/VAE loaders expect "diffusers" even when
# _class_name names a FastVideo-native class registered in FastVideo.
"transformer": ["diffusers", "<FastVideoTransformerClass>"],
"vae": ["diffusers", "<FastVideoVAEClass>"],
}
if revision:
index["_fastvideo_converted_revision"] = revision
return {key: value for key, value in index.items() if key.startswith("_") or key in available_components}
def validate_component_configs(configs: dict[str, dict[str, Any]]) -> None:
# TODO: instantiate each FastVideo config and call update_model_arch(...) or
# update_model_config(...) with this JSON so unknown emitted keys fail here.
placeholder_configs = [name for name, config in configs.items() if "<" in json.dumps(config)]
if placeholder_configs:
raise ValueError(f"Replace config placeholders for: {placeholder_configs}")
def verify_conversion(
dst_dir: Path,
components: dict[str, OrderedDict[str, torch.Tensor]],
) -> None:
del dst_dir, components
# TODO: load each emitted stateful component through its production loader and
# assert strict load, or document exact allowed missing/unexpected keys.
raise NotImplementedError("Implement production config validation and strict-load checks")
def write_component(
dst_dir: Path,
name: str,
state: dict[str, torch.Tensor],
config: dict[str, Any] | None,
) -> None:
component_dir = dst_dir / name
if component_dir.exists() and any(component_dir.iterdir()):
shutil.rmtree(component_dir)
component_dir.mkdir(parents=True, exist_ok=True)
save_file(dict(state), str(component_dir / "diffusion_pytorch_model.safetensors"))
if config is not None:
config_path = component_dir / config_filename(name)
with config_path.open("w", encoding="utf-8") as f:
json.dump(config, f, indent=2)
f.write("\n")
print(f"Wrote {name}: {len(state)} tensors")
def copy_passthrough(src_dir: Path, dst_dir: Path) -> list[str]:
copied: list[str] = []
for subfolder in PASSTHROUGH_SUBFOLDERS:
src = src_dir / subfolder
if not src.is_dir():
continue
dst = dst_dir / subfolder
if dst.exists():
shutil.rmtree(dst)
shutil.copytree(src, dst)
copied.append(subfolder)
print(f"Copied {subfolder}/")
return copied
def default_monolithic_checkpoint(src_path: Path) -> Path:
if src_path.is_file():
return src_path
return src_path / "model.safetensors"
def convert(
src: str,
dst: str,
layout: str,
revision: str | None,
) -> None:
src_path = resolve_src(src, revision)
dst_dir = Path(dst)
dst_dir.mkdir(parents=True, exist_ok=True)
model_index_path = dst_dir / "model_index.json"
if layout in {"monolithic", "raw_official"}:
# TODO: replace model.safetensors with the official monolithic file name.
components = split_monolithic(load_checkpoint(default_monolithic_checkpoint(src_path)))
elif layout in {"separate_components", "mixed"}:
if not src_path.is_dir():
raise ValueError(f"{layout} layout requires a source directory: {src_path}")
components = load_separate_components(src_path)
else:
raise ValueError(f"Unsupported template layout: {layout}")
copied = (copy_passthrough(src_path, dst_dir) if src_path.is_dir() else [])
configs = build_component_configs(src_path if src_path.is_dir() else src_path.parent)
validate_component_configs(configs)
for name, state in components.items():
write_component(dst_dir, name, state, configs.get(name))
available = set(components) | set(copied)
with model_index_path.open("w", encoding="utf-8") as f:
json.dump(build_model_index(src, revision, available), f, indent=2)
f.write("\n")
print(f"Wrote {dst_dir / 'model_index.json'}")
verify_conversion(dst_dir, components)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--src", required=True, help="HF repo id, local dir, or checkpoint path")
parser.add_argument("--revision", help="HF branch, tag, or commit for repo sources")
parser.add_argument(
"--dst",
required=True,
help="Output converted_weights/<model_family> directory",
)
parser.add_argument(
"--layout",
choices=("raw_official", "monolithic", "separate_components", "mixed"),
required=True,
help="Official source layout",
)
args = parser.parse_args()
convert(args.src, args.dst, args.layout, args.revision)
if __name__ == "__main__":
main()
-214
View File
@@ -1,214 +0,0 @@
---
name: add-model-08-trace
description: Use during /add-model Phase 6 when component parity has failed and root cause requires layer-by-layer divergence analysis. Uses FastVideo activation trace first, falling back to custom hooks only for boundaries or stats the utility cannot observe.
---
# Add-Model Trace
## Manual Invocation
Load this skill when `/add-model` Phase 6 component parity has failed and the
root cause requires layer-by-layer divergence analysis. This skill is not
auto-fired. The calling subagent (DiT, VAE, encoder, or generic port skill)
loads it when its standard parity-debug loop hits a wall and cannot isolate
the divergence from end-to-end tensor comparisons alone.
Do not load this skill for first-pass parity failures. Try weight-diff and
end-to-end tensor comparison first. Load this skill only when those do not
isolate the cause.
## Goal
Find the first numerical divergence point between FastVideo's port and the
official reference, layer by layer, by instrumenting both sides at matching
tensor boundaries. The investigation must leave zero source residue in
production code when it closes.
## When To Run
After a component parity test FAILS at a bf16-noise-realistic tolerance AND
the calling subagent's first-pass debug (weight-diff, end-to-end tensor
compare) does not isolate the cause.
Required inputs before starting:
- A working FastVideo loader for the component under investigation.
- A working official loader, typically via
`tests/local_tests/helpers/<family>_upstream.py::load_upstream_<component>`.
- Shared deterministic test inputs (same tensors on both sides).
- The component parity test file path and its current failure output.
## Primary Path: FastVideo Activation Trace
Use FastVideo's first-class activation trace before writing custom hooks:
`fastvideo/hooks/activation_trace.py`, documented in
`docs/contributing/activation_trace.md`.
Pipeline runs attach trace to the transformer during pipeline initialization.
Component-only parity harnesses may call `attach_activation_trace(model)` from
local test/debug code; do not add trace calls to production model code.
Prefix the failing parity command with a tight layer regex:
```bash
FASTVIDEO_TRACE_ACTIVATIONS=1 \
FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$" \
FASTVIDEO_TRACE_STATS="abs_mean,sum,max,shape" \
FASTVIDEO_TRACE_STEPS="0" \
FASTVIDEO_TRACE_OUTPUT="/tmp/opencode/fv_trace.jsonl" \
pytest tests/local_tests -k "parity" -v -s
```
Match the layer regex to the actual `model.named_modules()` names. Empty or
broad regexes are expensive; prefer block-level names first, then narrow to
submodules after the first divergent block is known.
## Trace Compare Contract
One JSONL file per side. FastVideo output should use `FASTVIDEO_TRACE_OUTPUT`;
the upstream harness should emit the same JSONL shape:
```json
{"module":"block.layers.0","tensor":"out","step":0,"abs_mean":0.0123,"sum":1.0,"max":0.5,"shape":[1,16,32]}
```
Compare rows by `(module, step, tensor)`. The first row whose `shape`,
`abs_mean`, or `max` diverges beyond the component tolerance is the first broken
boundary. Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
`FASTVIDEO_TRACE_STEPS` identical between sides; if row order differs, sort or
normalize before diffing.
## Drill-Down Loop
**Initial run:** trace every top-level block (`^block\.layers\.[0-9]+$` or the
family's equivalent). Identify the first block index where `abs_mean` or `max`
drifts beyond tolerance while earlier blocks match.
**Drill run:** tighten `FASTVIDEO_TRACE_LAYERS` to submodules inside the first
divergent block: attention output, MLP projections, norm outputs, modality
adapters, or other named boundaries exposed by `named_modules()`.
**Iterate:** if the first divergent operation is a free function or tensor op not
visible as an `nn.Module`, use the fallback instrumentation hierarchy below.
The loop ends when the first divergent submodule or operation is identified with
a file:line citation in the official source.
## Fallback Instrumentation Hierarchy
Use these only when activation trace cannot observe the needed boundary or
statistic.
### (1) Custom forward hooks
`module.register_forward_hook(...)` and `register_forward_pre_hook(...)`.
Always within `try/finally` with `handle.remove()`. Zero source residue.
### (2) Runtime monkey-patch
`module.attr = wrapped_func` or `cls.method = wrapped_method`, restored via
`try/finally` (save original first). Use for free functions and non-Module sites
such as activation functions (`swiglu`, `apply_rotary_emb`).
### (3) Source edits in FastVideo's own code
Only when (1) and (2) are insufficient. Track all edits within a single named
`git stash` boundary OR a temporary branch. Run `git diff` before closing the
investigation to confirm cleanup.
### (4) Source edits in official repo source
Allowed only when hook and monkey-patch approaches cannot capture the site.
For git-tracked or editable official clones, use `git diff` in the clone path to
verify cleanup. For non-editable site-packages, back up the target file before
editing and restore it before handoff.
## Hypothesis Toggles
Use env-var-gated monkey-patches to A/B test suspect implementations without
source edits. Pattern: `<FAMILY>_DEBUG_PATCH_<HYPOTHESIS>=1`.
Example from the magi-human investigation:
```
MAGI_DEBUG_PATCH_LINEAR=1
```
This patched `PackedExpertLinear.forward` to mirror upstream's
`_BF16ComputeLinear` explicit-cast pattern, isolating a dtype-cast difference
as the root cause.
Document all toggles in the script docstring. Each toggle must:
- save the original before patching;
- restore the original in a `try/finally` block;
- print a `[debug] Patched <ClassName>.<method>` line to stdout when active.
## Cleanup Gate
The calling agent MUST report `[cleanup-gate] PASS` on all five items before
handoff. Do not hand off with any item unresolved.
1. `git diff` in the FastVideo repo: empty. No stray prints, hooks, or
monkey-patches in production code.
2. `git diff` in the official-repo clone (if used): empty. For non-editable
site-packages installs: `diff original.py original.py.trace-backup` is
empty OR `pip install --force-reinstall <pkg>` succeeded and the installed
file matches the original.
3. `git stash list`: only the named investigation stash (or empty). No
unnamed stashes left from this session.
4. No new untracked files outside `/tmp/opencode/` (logs) and the existing
debug script directory (`tests/local_tests/transformers/` or equivalent).
5. `mypy` clean on any production files touched during the investigation.
## Escape Hatches
Escalate to the calling bucket skill when:
- A forward hook on an official module raises because of a custom `forward`
signature or varlen handler args that the hook closure cannot satisfy. The
bucket skill has component-specific knowledge to work around this.
- The first divergent layer is `block[0]`, meaning the divergence is in the
adapter, modality dispatcher, coordinate embedding, or packing step before
any block runs. Check those sites first; the bug is not in attention or MLP.
- Per-block drift is never zero anywhere across all blocks. This usually means
the inputs are not bit-identical between sides. Verify with a state-dict
compare (weight-diff script) AND confirm the input tensors are the same
object or have identical values before the forward call.
## Handoff
Return to the calling subagent with:
- FastVideo trace JSONL path and upstream trace JSONL path.
- Trace settings used: `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
`FASTVIDEO_TRACE_STEPS`.
- The first divergent `(module, step, tensor)` row and observed drift.
- The upstream file:line citation where the divergence originates.
- Fallback hook/patch verdict if activation trace could not observe the boundary.
- Hypothesis verdict if an A/B toggle was used, for example `PATCH_LINEAR=1`.
- Cleanup-gate status: `[cleanup-gate] PASS` or a list of unresolved items.
The calling agent uses this to scope the production fix in the FastVideo
component file.
## References
- `docs/contributing/activation_trace.md` for canonical activation-trace env vars,
JSONL output, cost model, and troubleshooting.
- `fastvideo/hooks/activation_trace.py` for the implementation and
`attach_activation_trace(model)` entry point.
- `templates/block_trace_debug.py` in this skill directory: fallback custom-hook
template when activation trace cannot observe the needed boundary or stat.
- `tests/local_tests/transformers/_debug_magi_human_block_parity.py` in the
FastVideo3 repo: historical worked example for custom hook/patch debugging.
- `add-model/SKILL.md` Phase 6: the calling context for this skill.
- `add-model-03-port-dit/SKILL.md`, `add-model-04-port-vae/SKILL.md`,
`add-model-05-port-encoder/SKILL.md`, `add-model-06-port-generic/SKILL.md`:
bucket-specific debug language and component-specific escape-hatch knowledge.
## Changelog
| Date | Change |
|---|---|
| 2026-05-01 | Initial skill extracted from `_debug_magi_human_block_parity.py` pattern. |
@@ -1,334 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""Per-block divergence debugger template for FastVideo model ports.
Run directly (not a pytest test):
python tests/local_tests/transformers/_debug_<family>_<component>_parity.py
Generalizes: tests/local_tests/transformers/_debug_magi_human_block_parity.py
Fill FAMILY, COMPONENT, and the two loader functions. Run once for the initial
drift table, then set <FAMILY>_DEBUG_DRILL_LAYER=NN to drill into submodules.
Add <FAMILY>_DEBUG_PATCH_<HYPOTHESIS>=1 to A/B test a suspect implementation.
CLEANUP: all hooks removed in try/finally; monkey-patches restored in
try/finally; source edits tracked in a named git stash. Zero source residue.
See add-model-08-trace/SKILL.md for the full cleanup gate checklist.
"""
from __future__ import annotations
import gc
import os
import sys
from pathlib import Path
from typing import Any
import torch
FAMILY: str = "<family>" # e.g. "magi_human", "ltx2", "wan"
COMPONENT: str = "<component>" # e.g. "dit", "vae", "encoder"
DRILL_LAYER_ENV: str = "<FAMILY>_DEBUG_DRILL_LAYER"
HYPOTHESIS_ENV: str = "<FAMILY>_DEBUG_PATCH_<HYPOTHESIS>"
REL_THRESHOLD: float = 0.005 # 0.5% abs_mean drift flags a block as divergent
LOG_DIR: Path = Path("/tmp/opencode")
REPO_ROOT = Path(__file__).resolve().parents[3]
sys.path.insert(0, str(REPO_ROOT))
def load_official(device: torch.device) -> torch.nn.Module:
"""Load the official upstream model. TODO: implement for your family.
Example (magi-human):
from tests.local_tests.helpers.magi_human_upstream import install_stubs, load_upstream_dit
install_stubs()
return load_upstream_dit(base_shard_dir, device=device, dtype=None)
"""
raise NotImplementedError(f"Fill load_official() for {FAMILY}/{COMPONENT}.")
def load_fastvideo(device: torch.device) -> torch.nn.Module:
"""Load the FastVideo-native model. TODO: implement for your family.
Example (magi-human):
from fastvideo.configs.models.dits.magi_human import MagiHumanVideoConfig
from fastvideo.models.dits.magi_human import MagiHumanDiT
from safetensors.torch import load_file; import glob
fv = MagiHumanDiT(MagiHumanVideoConfig())
state = {}
for shard in sorted(glob.glob(str(transformer_dir / "*.safetensors"))): state.update(load_file(shard))
fv.load_state_dict(state, strict=False); return fv.to(device).eval()
"""
raise NotImplementedError(f"Fill load_fastvideo() for {FAMILY}/{COMPONENT}.")
def build_inputs(device: torch.device) -> dict[str, Any]:
"""Return deterministic inputs shared by both sides. TODO: replace.
Both sides must receive the SAME tensors (clone before each forward call).
Non-identical inputs cause non-zero drift everywhere.
"""
torch.manual_seed(0)
return {"x": torch.randn(64, 1024, dtype=torch.bfloat16, device=device)}
def _stat(name: str, t: torch.Tensor) -> dict:
f = t.detach().float()
return {
"name": name,
"shape": tuple(t.shape),
"abs_mean": f.abs().mean().item(),
"sum": f.sum().item(),
"min": f.min().item(),
"max": f.max().item(),
}
def _attach_block_hooks(
model: torch.nn.Module,
label: str,
log: list[dict],
tensors: dict[str, torch.Tensor] | None = None,
drill_layer: int | None = None,
) -> list[Any]:
"""Return hook handles. Caller MUST remove them in try/finally."""
handles: list[Any] = []
def _hook(name: str):
def fn(_module, _inputs, outputs):
t = outputs[0] if isinstance(outputs, tuple) else outputs
if not torch.is_tensor(t):
return
log.append({"side": label, **_stat(name, t)})
if tensors is not None:
tensors[name] = t.detach().float().cpu()
return fn
def _pre_hook(name: str):
# Pre-hooks observe a free function's output by intercepting the next
# module's input (useful when the activation is not an nn.Module).
def fn(_module, inputs):
t = inputs[0] if isinstance(inputs, tuple) else inputs
if not torch.is_tensor(t):
return
key = f"{name}<in>"
log.append({"side": label, **_stat(key, t)})
if tensors is not None:
tensors[key] = t.detach().float().cpu()
return fn
# TODO: adapt attribute paths to your model. Remove adapter block if absent.
if hasattr(model, "adapter"):
handles.append(model.adapter.register_forward_hook(_hook("adapter")))
# TODO: adapt model.block.layers to your block container.
# Alternatives: model.transformer.layers, model.blocks, model.layers
block_layers = model.block.layers # type: ignore[attr-defined]
for i, layer in enumerate(block_layers):
handles.append(layer.register_forward_hook(_hook(f"block[{i:02d}]")))
if drill_layer is not None and i == drill_layer:
tag = f"L{i:02d}"
# TODO: adapt submodule names to your layer's attributes.
# magi-human uses: attention, mlp.pre_norm, mlp.up_gate_proj,
# mlp.down_proj (pre+post), mlp, attn_post_norm, mlp_post_norm.
if hasattr(layer, "attention"):
handles.append(layer.attention.register_forward_hook(_hook(f"{tag}.attention")))
if hasattr(layer, "mlp"):
mlp = layer.mlp
if hasattr(mlp, "pre_norm"):
handles.append(mlp.pre_norm.register_forward_hook(_hook(f"{tag}.mlp.pre_norm")))
if hasattr(mlp, "up_gate_proj"):
handles.append(mlp.up_gate_proj.register_forward_hook(_hook(f"{tag}.mlp.up_gate_proj")))
if hasattr(mlp, "down_proj"):
handles.append(mlp.down_proj.register_forward_pre_hook(_pre_hook(f"{tag}.mlp.down_proj")))
handles.append(mlp.down_proj.register_forward_hook(_hook(f"{tag}.mlp.down_proj")))
handles.append(mlp.register_forward_hook(_hook(f"{tag}.mlp")))
if hasattr(layer, "attn_post_norm"):
handles.append(layer.attn_post_norm.register_forward_hook(_hook(f"{tag}.attn_post_norm")))
if hasattr(layer, "mlp_post_norm"):
handles.append(layer.mlp_post_norm.register_forward_hook(_hook(f"{tag}.mlp_post_norm")))
return handles
def _apply_hypothesis_patch() -> bool:
"""Apply an optional monkey-patch gated by HYPOTHESIS_ENV. TODO: implement.
Pattern: save original on the class, patch, restore in _restore_hypothesis_patch().
"""
if os.getenv(HYPOTHESIS_ENV) != "1":
return False
# TODO: import FastVideo class, save original, apply patch.
print(f"[debug] Hypothesis patch {HYPOTHESIS_ENV}=1 applied.")
return True
def _restore_hypothesis_patch() -> None:
if os.getenv(HYPOTHESIS_ENV) != "1":
return
# TODO: restore original, e.g.: _mod.TargetClass.method = _mod._ORIGINAL_METHOD
def _write_log(entries: list[dict], path: Path) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "w") as f:
for e in entries:
f.write(f"{e['name']} {e['shape']} "
f"{e['abs_mean']:.8f} {e['sum']:.4f} "
f"{e['min']:.6f} {e['max']:.6f}\n")
def _sort_key(name: str, drill_layer: int) -> tuple:
if name == "adapter":
return (0, "")
if name.startswith(f"L{drill_layer:02d}."):
sub_order = {
"attention": 0,
"attn_post_norm": 1,
"mlp.pre_norm": 2,
"mlp.up_gate_proj": 3,
"mlp.down_proj<in>": 4,
"mlp.down_proj": 5,
"mlp": 6,
"mlp_post_norm": 7,
}.get(name.split(".", 1)[1], 9)
return (1, f"block[{drill_layer:02d}]", sub_order)
if name.startswith("block["):
return (1, name, 99)
return (2, name, 0)
def _print_table(by_name: dict[str, dict], drill_layer: int) -> int | None:
hdr = (f"{'name':<18} {'up_shape':<22} {'up_absmean':>12} {'fv_absmean':>12} "
f"{'absmean_diff':>14} {'rel%':>8} {'up_sum':>14} {'fv_sum':>14} {'sum_diff':>12}")
print(f"\n{hdr}\n{'-' * len(hdr)}")
first_div: int | None = None
for name in sorted(by_name.keys(), key=lambda n: _sort_key(n, drill_layer)):
d = by_name[name]
up, fv = d.get("up"), d.get("fv")
if up is None or fv is None:
continue
am_diff = abs(up["abs_mean"] - fv["abs_mean"])
am_rel = am_diff / max(up["abs_mean"], 1e-9)
sum_diff = abs(up["sum"] - fv["sum"])
flag = ""
if name.startswith("block[") and am_rel > REL_THRESHOLD:
flag = " <<< DIVERGE"
if first_div is None:
first_div = int(name[len("block["):-1])
print(f"{name:<18} {str(up['shape']):<22} {up['abs_mean']:>12.6f} "
f"{fv['abs_mean']:>12.6f} {am_diff:>14.6f} {am_rel * 100:>7.3f}% "
f"{up['sum']:>14.4f} {fv['sum']:>14.4f} {sum_diff:>12.4f}{flag}")
return first_div
def _print_elementwise(up_t: dict[str, torch.Tensor], fv_t: dict[str, torch.Tensor], drill_layer: int) -> None:
common = set(up_t.keys()) & set(fv_t.keys())
if not common:
return
hdr = f"{'name':<30} {'shape':<22} {'diff_max':>12} {'diff_mean':>12} {'diff_rel%':>10}"
print(f"\nElement-wise diffs for drilled L{drill_layer:02d} submodules:\n{hdr}\n{'-' * len(hdr)}")
for name in sorted(common):
a, b = up_t[name], fv_t[name]
if a.shape != b.shape:
continue
diff = (a - b).abs()
rel = (diff.mean().item() / max(a.abs().mean().item(), 1e-9)) * 100
print(f"{name:<30} {str(tuple(a.shape)):<22} "
f"{diff.max().item():>12.6f} {diff.mean().item():>12.6f} {rel:>9.4f}%")
def main() -> None:
if not torch.cuda.is_available():
print("Need CUDA. Skipping.")
return
# TODO: add precondition checks (official clone present, weights available).
drill_layer = int(os.getenv(DRILL_LAYER_ENV, "0"))
device = torch.device("cuda:0")
patched = _apply_hypothesis_patch()
try:
inputs = build_inputs(device)
print("Loading official model...")
official = load_official(device)
up_log: list[dict] = []
up_t: dict[str, torch.Tensor] = {}
up_handles = _attach_block_hooks(official, "up", up_log, up_t, drill_layer)
print("Running official forward (with hooks)...")
try:
with torch.inference_mode():
# TODO: adapt forward call signature to your component.
ref_out = official(**{k: v.clone() for k, v in inputs.items()})
if isinstance(ref_out, dict):
sample = ref_out.get("sample")
ref_out = sample if sample is not None else ref_out.get("x")
elif hasattr(ref_out, "sample"):
ref_out = ref_out.sample
elif isinstance(ref_out, tuple):
ref_out = ref_out[0]
assert torch.is_tensor(ref_out), f"official output is not tensor: {type(ref_out)}"
ref_out = ref_out.detach().float().cpu()
finally:
for h in up_handles:
h.remove()
del official
gc.collect()
torch.cuda.empty_cache()
print("Loading FastVideo model...")
fv = load_fastvideo(device)
fv_log: list[dict] = []
fv_t: dict[str, torch.Tensor] = {}
fv_handles = _attach_block_hooks(fv, "fv", fv_log, fv_t, drill_layer)
print("Running FastVideo forward (with hooks)...")
try:
with torch.inference_mode():
# TODO: adapt forward call signature to your component.
fv_out = fv(**{k: v.clone() for k, v in inputs.items()})
if isinstance(fv_out, dict):
sample = fv_out.get("sample")
fv_out = sample if sample is not None else fv_out.get("x")
elif hasattr(fv_out, "sample"):
fv_out = fv_out.sample
elif isinstance(fv_out, tuple):
fv_out = fv_out[0]
assert torch.is_tensor(fv_out), f"FastVideo output is not tensor: {type(fv_out)}"
fv_out = fv_out.detach().float().cpu()
finally:
for h in fv_handles:
h.remove()
finally:
_restore_hypothesis_patch()
LOG_DIR.mkdir(parents=True, exist_ok=True)
up_path = LOG_DIR / f"{FAMILY}_{COMPONENT}_up_layers.log"
fv_path = LOG_DIR / f"{FAMILY}_{COMPONENT}_fv_layers.log"
_write_log(up_log, up_path)
_write_log(fv_log, fv_path)
print(f"\nLogs: {up_path} {fv_path}\nDiff: diff {up_path} {fv_path}")
by_name: dict[str, dict] = {}
for entry in up_log + fv_log:
by_name.setdefault(entry["name"], {})[entry["side"]] = entry
first_div = _print_table(by_name, drill_layer)
print()
if first_div is not None:
print(f"First block exceeding {REL_THRESHOLD * 100:.2f}% drift: block[{first_div:02d}]")
print(f"Re-run with {DRILL_LAYER_ENV}={first_div} to drill submodules.")
else:
print(f"No block exceeded {REL_THRESHOLD * 100:.2f}% -- divergence is amortized or pre-block.")
diff = (ref_out - fv_out).abs()
print(f"\nFinal ref_abs={ref_out.abs().mean():.6f} fv_abs={fv_out.abs().mean():.6f} "
f"diff_max={diff.max():.6f} diff_mean={diff.mean():.6f}")
_print_elementwise(up_t, fv_t, drill_layer)
if patched:
print(f"\n[debug] Hypothesis {HYPOTHESIS_ENV}=1 was active this run.")
if __name__ == "__main__":
main()
@@ -1,255 +0,0 @@
---
name: add-model-09-pipeline
description: Use during /add-model Phase 7 after all required component parity tests pass to define FastVideo pipeline wiring, configs, presets, registry entries, examples, smoke tests, and pipeline parity tests.
---
# Add Model Pipeline
## Goal
Implement and verify the end-to-end FastVideo pipeline after the native
components and converted weights have passed non-skip component parity. This
skill owns pipeline class/stage wiring, pipeline configs, presets, registry
entries, examples, smoke tests, and pipeline parity-debug.
FastVideo has one pipeline architecture: stage-based composition through
`ComposedPipelineBase`. Add or specialize stages only when existing stages cannot
represent the official behavior safely.
## Hard Gate
Do not start pipeline work until every required component, including reused
components, has a non-skip local parity PASS.
If any component row is missing, skipped, red, or blocked, return to `/add-model`
Phase 6. Pipeline parity cannot distinguish stage wiring mistakes from broken
component numerics when component parity is still unresolved.
## Inputs
Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
escape hatches, production boundaries, and skip/pass semantics.
Require a complete packet matching
`../add-model/contracts/pipeline_context.md`.
The packet must include:
- official pipeline files and official call/default sources;
- workload types, input/output modalities, and output contract;
- converted or source `model_index.json` path;
- component parity rows, all `non_skip_pass`;
- target FastVideo pipeline/config/preset/registry/example/test paths;
- `local_tests_readme` and `port_state_file` paths.
## Outputs
- Pipeline package under `fastvideo/pipelines/basic/<family>/`.
- Pipeline config under `fastvideo/configs/pipelines/<family>.py` or a documented
family-local config file when that matches existing project style.
- Presets under `fastvideo/pipelines/basic/<family>/presets.py`.
- Registry updates in `fastvideo/registry.py`.
- Basic example under `examples/inference/basic/basic_<family>*.py`.
- Local smoke and parity tests under `tests/local_tests/pipelines/`.
- Updated `tests/local_tests/<model_family>/README.md`.
- Updated `tests/local_tests/<model_family>/PORT_STATUS.md`.
- Handoff matching `../add-model/contracts/pipeline_handoff.md`.
## Mode: Pipeline Definition
Use this mode first.
1. Read the official pipeline call path before editing FastVideo code.
2. Compare official defaults against the planned FastVideo config and presets:
steps, CFG scales, secondary CFG, flow shift, schedulers, sigmas, seed/RNG,
resolution, frames, FPS, duration, VAE scaling, decode slicing, negative
prompt defaults, and output heads.
3. Create or update the pipeline class with `_required_config_modules` matching
the emitted `model_index.json` and `ComposedPipelineBase.load_modules`.
Runtime pipeline resolution is exact: `model_index.json["_class_name"]` must
match a registered `EntryClass.__name__`, or a wrapper/alias class in
`EntryClass`. Registry detectors do not select the executable pipeline class.
4. Add new public generation kwargs to `fastvideo/api/sampling_param.py` before
examples or presets use them. `SamplingParam.update()` ignores unknown keys
except for logging, and preset defaults apply only to declared fields. Add CLI
args when the option should be available from command-line entrypoints.
5. Put loader-time changes in `load_modules()` or earlier, not
`initialize_pipeline()`. `ComposedPipelineBase.__init__` loads modules before
`post_init()` calls `initialize_pipeline()`, so process-global flags, loader
path rewrites, dtype overrides, and tokenizer path changes needed for loading
cannot be introduced there.
6. Use `self.get_module("transformer_2", None)` and similar optional accessors
for truly optional modules. Do not hard-require optional modules by accident.
7. Avoid mutating class-level `_required_config_modules` in custom code. If a
pipeline needs dynamic modules, copy the list to an instance-owned value or
pass `required_config_modules` explicitly so one pipeline instance cannot leak
module requirements into another.
8. Create the stage chain in official execution order. Prefer existing shared
stages for standard text encoding, timestep preparation, latent preparation,
denoising, and decoding.
9. Add model-specific stages only for family-specific behavior that does not fit
the shared stage contracts.
10. Add pipeline config classes for wiring and runtime defaults. Do not duplicate
component architecture fields unless a loader requires them in the subconfig.
Family-local config files such as
`fastvideo/pipelines/basic/<family>/pipeline_configs.py` are valid only when
`fastvideo/registry.py` imports and registers the classes explicitly.
11. Add `InferencePreset` objects with `model_family`, `name`, `version`,
`defaults`, optional validation-only `stage_schemas`, and an `ALL_PRESETS`
tuple. `stage_schemas` validates user-facing `stage_overrides` names; it does
not drive `create_pipeline_stages()` execution.
12. Register config classes and presets in `fastvideo/registry.py`: add
`register_configs(...)`, import the family's `ALL_PRESETS`, and append it to
`_register_presets()`. Detectors should cover HF paths and `_class_name`
strings for config/preset lookup, but not as a replacement for exact pipeline
class-name resolution.
13. Add a basic example with a user-story docstring and normal file-path inputs
for image, audio, or video references. Keep orchestration glue in the
pipeline or a helper, not in the example.
14. Add a separate smoke test
`tests/local_tests/pipelines/test_<family>_pipeline_smoke.py` that proves
imports, `EntryClass`, registry, presets, config defaults, and at least one
real load/generate path when weights are local. Older local tests sometimes
colocate smoke checks in parity files; new ports should use the separate file
convention.
15. Add or update pipeline parity test scaffolding with
`templates/pipeline_parity_test.py`.
16. Update `local_tests_readme` and `port_state_file` with commands, statuses,
default sources, decisions, and blockers.
Production import boundaries are defined in
`../add-model/shared/common_rules.md`.
## Mode: Pipeline Parity Debug
Run after pipeline definition and after smoke can execute far enough to load the
pipeline. Loop until pipeline parity is a non-skip PASS or a precise blocker is
returned.
Mandatory order:
```bash
pytest tests/local_tests/pipelines/test_<family>_pipeline_smoke.py -v -s
DISABLE_SP=1 pytest tests/local_tests/pipelines/test_<family>_pipeline_parity.py -v -s
python examples/inference/basic/basic_<family>.py
```
Pipeline parity must compare real outputs, not only successful generation:
- denoised latents when decode parity is expensive or nondeterministic;
- decoded videos/images when visual output should be deterministic enough;
- decoded waveform or audio features for audio pipelines;
- separate video and audio targets for joint AV pipelines unless a validated
joint metric exists.
Debug pipeline drift in this order:
1. Confirm both sides use the same component weights and component parity PASS
results are still valid.
2. Align official and FastVideo call arguments, presets, and default values.
3. Align scheduler timesteps, sigmas/noise levels, prediction type, flow shift,
guidance math, and secondary-guidance branches.
4. Align RNG: initial latents/noise, generator device, seed, per-step noise, VAE
sampling, and any official `+1 frame` or crop/slice behavior.
5. Align conditioning: prompt templates, negative prompts, masks, image/audio
preprocessing, modality packing, text truncation, and dtype/autocast.
6. Align decode: latent scaling, per-channel mean/std, tiling flags, output
channel order, sample rate, FPS, and final slicing.
7. Add targeted stage-level diagnostics to identify the first divergent stage.
If stage diagnostics show the first bad stage is transformer/denoising or a
mid-DiT block, enable activation trace before adding ad hoc pipeline prints; see
`docs/contributing/activation_trace.md` and `../add-model-08-trace/SKILL.md`.
Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
`FASTVIDEO_TRACE_STEPS` identical across reruns so pipeline parity traces diff
one-to-one.
If the first divergence belongs to component implementation, strict loading, or
conversion mapping, stop pipeline edits and return `next_step=return_to_phase_6`
with the exact failing evidence. Do not patch conversion from this skill.
## Stage And Variant Rules
- Canonical video T2V order: `InputValidationStage`, `TextEncodingStage`,
`ConditioningStage`, `TimestepPreparationStage`, `LatentPreparationStage`,
`DenoisingStage`, `DecodingStage`.
- Canonical video I2V delta adds image loading/encoding and image VAE encoding in
the official order, commonly: `TextEncodingStage`, `ImageEncodingStage`,
`ConditioningStage`, `TimestepPreparationStage`, `LatentPreparationStage`,
`ImageVAEEncodingStage`, `DenoisingStage`, `DecodingStage`.
- Treat `ConditioningStage` as default-present for Wan-style pipelines, but still
follow the reference if another family truly skips or replaces it.
- T2V video pipelines usually use validation, text encoding, conditioning,
timestep preparation, latent preparation, denoising, and decoding.
- I2V adds image loading/encoding and image-latent preparation according to the
official pipeline, not by assuming CLIP or Wan-specific branches.
- Pick image, audio, and video encoders from the reference. Do not assume CLIP or
any other common encoder unless the reference uses it.
- Cross-attention class names are not prescribed; match the family style and
preserve the official tensor contract.
- `WorkloadType` currently has no `T2A`, `A2A`, or `AV` values. Until that enum
is extended, audio-only pipelines may register with `WorkloadType.T2V` and
preset `workload_type="t2v"` as a compatibility shim, but must document the
rationale in code and `PORT_STATUS.md`.
- Audio-only pipelines should not force real video semantics into presets. Use
minimal video-shaped placeholders such as small `height`/`width` and
`num_frames=1` only when shared `VideoGenerator`/validation paths require them,
and document that the real output is audio.
- Record modality-specific shape knobs and output contract in the pipeline
handoff: video uses `height`, `width`, `num_frames`, and `fps`; audio uses
`audio_seconds` and `sampling_rate`; joint AV records both plus whether output
is muxed or paired files.
- Use sibling pipeline classes/configs when required modules, HF repo layout,
stage chains, or inputs differ materially.
- Use one kwargs-driven pipeline class only when variants share weights,
modules, stage chain, and safe call semantics.
- Split later if components diverge, workload tags require separate discovery,
signatures become unsafe, or stage branches become substantial.
- If the DiT branches on `added_kv_proj_dim`, document the T2V/I2V split.
- If the reference uses `transformer_2`, `boundary_ratio`, `guidance_scale_2`, or
DMD step lists, keep those on config, presets, or stages deliberately.
- Support every official output head in scope. If a head is out of scope, record
explicit user approval in `PORT_STATUS.md`.
Pipeline verification order:
```bash
pytest tests/local_tests/pipelines/test_<family>_pipeline_smoke.py -v -s
DISABLE_SP=1 pytest -v -s tests/local_tests/pipelines/test_<family>_pipeline_parity.py
python examples/inference/basic/basic_<family>.py
```
Smoke tests prove loadability only. They are not a substitute for numerical
component or pipeline parity.
## Escape Hatches
Follow `../add-model/shared/common_rules.md`. Pipeline-specific ask cases include
dropping a public mode, modality, or output head; adding a new workload enum;
changing official defaults for user-facing behavior; accepting a known pipeline
parity blocker; running GPU-heavy quality work outside the agreed scope; or
publishing/uploading generated references or converted weights.
## Handoff
Return `../add-model/contracts/pipeline_handoff.md` and update the shared state
files before handoff.
Do not hand back a green pipeline if smoke or parity skipped locally. A skip is a
setup gap, not a pass.
## References
- `fastvideo/pipelines/composed_pipeline_base.py` for module loading and stage
execution.
- `fastvideo/pipelines/basic/wan/` for standard video T2V/I2V/DMD variants.
- `fastvideo/pipelines/basic/stable_audio/` for audio-specific stage composition.
- `fastvideo/configs/pipelines/stable_audio.py` and
`fastvideo/pipelines/basic/stable_audio/presets.py` for config/preset shape.
- `fastvideo/registry.py` for `register_configs(...)` and preset registration.
- `tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py` for latent
parity structure.
- `tests/local_tests/pipelines/test_stable_audio_pipeline_parity.py` for audio
parity structure.
- `tests/local_tests/pipelines/test_stable_audio_pipeline_smoke.py` for no-GPU
import/registry/preset preflight shape.
@@ -1,147 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""Pipeline parity scaffold for TODO_MODEL_FAMILY.
Copy this file to
`tests/local_tests/pipelines/test_<family>_pipeline_parity.py` and replace every
TODO before treating it as an executable scaffold.
The filled test should compare denoised latents, decoded media, audio waveform,
or another concrete output from the official pipeline against FastVideo. A
successful generation without tensor/media comparison is not parity.
"""
from __future__ import annotations
import os
import sys
from pathlib import Path
from typing import Any
import pytest
import torch
from torch.testing import assert_close
_REPO_ROOT = Path(__file__).resolve().parents[3]
_MODEL_FAMILY = "TODO_MODEL_FAMILY"
_OFFICIAL_REF_ENV = "TODO_OFFICIAL_REF_PATH"
_OFFICIAL_REF_DEFAULT = _REPO_ROOT / "TODO_OFFICIAL_REF_DIR"
_FASTVIDEO_MODEL_ENV = "TODO_FASTVIDEO_MODEL_PATH"
_FASTVIDEO_MODEL_DEFAULT = _REPO_ROOT / "converted_weights" / _MODEL_FAMILY
def _path_from_env(env_name: str, default: Path) -> Path:
return Path(os.getenv(env_name, str(default))).expanduser()
def _add_official_to_path() -> Path:
official_path = _path_from_env(_OFFICIAL_REF_ENV, _OFFICIAL_REF_DEFAULT)
if not official_path.exists():
pytest.skip(f"Official reference not found at {official_path}")
if str(official_path) not in sys.path:
sys.path.insert(0, str(official_path))
return official_path
def _log_tensor_stats(label: str, tensor: torch.Tensor) -> None:
value = tensor.detach().float()
print(f"[{_MODEL_FAMILY} PIPELINE] {label}: shape={tuple(tensor.shape)} "
f"dtype={tensor.dtype} device={tensor.device} "
f"min={value.min().item():.6f} max={value.max().item():.6f} "
f"mean={value.mean().item():.6f} std={value.std().item():.6f}")
def _extract_tensor(output: Any, key: str) -> torch.Tensor:
if isinstance(output, dict):
value = output.get(key)
else:
value = getattr(output, key, None)
if value is None:
raise AssertionError(f"Pipeline output did not contain {key!r}")
if not torch.is_tensor(value):
try:
import numpy as np
value = torch.from_numpy(np.asarray(value))
except Exception as exc: # pragma: no cover - scaffold guard
raise AssertionError(f"Could not convert {key!r} to tensor") from exc
return value.detach().float().cpu()
def _run_official_pipeline(
official_path: Path,
params: dict[str, Any],
device: torch.device,
) -> Any:
del official_path, params, device
pytest.skip("TODO: import the official pipeline/factory, load official weights, "
"run with params, and return the comparison target.")
def _run_fastvideo_pipeline(model_path: Path, params: dict[str, Any]) -> Any:
from fastvideo import VideoGenerator
generator = VideoGenerator.from_pretrained(
str(model_path),
num_gpus=1,
use_fsdp_inference=False,
dit_cpu_offload=False,
vae_cpu_offload=False,
text_encoder_cpu_offload=False,
)
try:
return generator.generate_video(
prompt=params["prompt"],
negative_prompt=params.get("negative_prompt"),
output_path=f"outputs_{_MODEL_FAMILY}/pipeline_parity",
save_video=False,
height=params.get("height"),
width=params.get("width"),
num_frames=params.get("num_frames"),
fps=params.get("fps"),
num_inference_steps=params["num_inference_steps"],
guidance_scale=params.get("guidance_scale"),
seed=params["seed"],
)
finally:
generator.shutdown()
@pytest.mark.skipif(
not torch.cuda.is_available(),
reason="TODO_MODEL_FAMILY pipeline parity requires CUDA.",
)
def test_todo_model_family_pipeline_official_parity() -> None:
official_path = _add_official_to_path()
fastvideo_model_path = _path_from_env(
_FASTVIDEO_MODEL_ENV,
_FASTVIDEO_MODEL_DEFAULT,
)
if not fastvideo_model_path.exists():
pytest.skip(f"FastVideo model path not found at {fastvideo_model_path}")
device = torch.device("cuda:0")
params = {
"prompt": "TODO: stable parity prompt",
"negative_prompt": "",
"height": 64,
"width": 64,
"num_frames": 9,
"fps": 8,
"num_inference_steps": 4,
"guidance_scale": 1.0,
"seed": 0,
}
official_output = _run_official_pipeline(official_path, params, device)
fastvideo_output = _run_fastvideo_pipeline(fastvideo_model_path, params)
comparison_key = "TODO_COMPARISON_KEY"
official_tensor = _extract_tensor(official_output, comparison_key)
fastvideo_tensor = _extract_tensor(fastvideo_output, comparison_key)
_log_tensor_stats("official", official_tensor)
_log_tensor_stats("fastvideo", fastvideo_tensor)
assert official_tensor.shape == fastvideo_tensor.shape
diff = (official_tensor - fastvideo_tensor).abs()
print(f"diff max={diff.max().item():.6f} "
f"mean={diff.mean().item():.6f} median={diff.median().item():.6f}")
assert_close(fastvideo_tensor, official_tensor, atol=1e-2, rtol=1e-2)
@@ -1,141 +0,0 @@
---
name: add-model-10-pr-review
description: Review rubric for FastVideo PRs that add or modify model families, variants, first-class components, checkpoint conversion, pipelines, parity coverage, or generated-media quality baselines. Use when reviewing a PR whose diff touches fastvideo/models/, fastvideo/pipelines/basic/, fastvideo/registry.py, scripts/checkpoint_conversion/, fastvideo/tests/ssim/, or related model-port surfaces. Pairs with review-pr-link as a project-scoped review pass; produces findings, not fixes.
---
# Add-Model PR Review
Use this skill when a reviewed PR appears to add, port, or substantially modify
a FastVideo model family, model variant, first-class model component,
checkpoint conversion, model pipeline, or local parity coverage.
This is a review skill, not an implementation workflow. Do not run `/add-model`
or start writing missing port code during review. Use the add-model skill stack
as a rubric for findings.
## Trigger Paths
Trigger this skill if `git diff --name-only <base>...HEAD` includes any of:
- `fastvideo/models/dits/`, `fastvideo/configs/models/dits/`
- `fastvideo/models/vaes/`, `fastvideo/configs/models/vaes/`
- `fastvideo/models/encoders/`, `fastvideo/configs/models/encoders/`
- `fastvideo/models/schedulers/`, `fastvideo/configs/models/schedulers/`
- `fastvideo/models/upsamplers/`, `fastvideo/configs/models/upsamplers/`
- `fastvideo/models/audio/`, `fastvideo/configs/models/audio/`
- `fastvideo/pipelines/basic/`, `fastvideo/configs/pipelines/`
- `fastvideo/registry.py`, `fastvideo/api/sampling_param.py`
- `scripts/checkpoint_conversion/`
- `examples/inference/basic/`
- `tests/local_tests/`, especially component or pipeline parity tests
- `fastvideo/tests/ssim/` or other quality-regression tests for generated media
Also trigger when the PR title/body claims a new model, model variant, VAE,
encoder, scheduler, conditioner, pipeline, conversion script, or generated-media
quality baseline even if the path list is incomplete.
## Review Inputs
Read these add-model references as review checklists:
- `../add-model/SKILL.md`: phase gates and final handoff requirements.
- `../add-model/shared/common_rules.md`: token/auth safety, production import
boundaries, state files, and skip/pass semantics.
- `../add-model/contracts/final_handoff.md`: final evidence expected from a
complete port.
- `../add-model/contracts/component_context.md` and
`../add-model/contracts/component_skill_handoff.md`: component evidence and
parity-debug expectations.
- `../add-model/contracts/conversion_request.md` and
`../add-model/contracts/conversion_handoff.md`: conversion evidence,
strict-load status, config validation, and retry context.
- `../add-model/contracts/pipeline_context.md` and
`../add-model/contracts/pipeline_handoff.md`: pipeline class/stage/config/
preset/registry/example evidence.
Then read only the satellite skill(s) that match touched areas:
- DiT/transformer changes: `../add-model-03-port-dit/SKILL.md`.
- VAE changes: `../add-model-04-port-vae/SKILL.md`.
- Encoder/conditioner changes: `../add-model-05-port-encoder/SKILL.md`.
- Scheduler/upsampler/vocoder/other components:
`../add-model-06-port-generic/SKILL.md`.
- Component parity tests: `../add-model-02-parity/SKILL.md`.
- Checkpoint conversion: `../add-model-07-conversion/SKILL.md`.
- Pipeline/config/presets/registry/examples:
`../add-model-09-pipeline/SKILL.md`.
- Prep/state docs: `../add-model-01-prep/SKILL.md`.
## Required Review Lanes
For a full model-family or model-variant PR, cover all lanes. For a
component-only PR, cover the component, conversion/parity as applicable, and the
documented downstream consumer.
1. Scope and source-of-truth lane:
Verify the PR clearly identifies the official reference, weights/revision,
supported variants, modalities, output heads, and any approved scope cuts.
2. Component lane:
Verify each required component is FastVideo-native or has a documented and
accepted lazy-wrapper exception. Check bucket/config inheritance, `EntryClass`,
state-dict surface, reused-component evidence, and output heads.
3. Conversion lane:
Verify mappings are derived from prototype key/shape dumps, source layout is
supported, skipped keys are intentional, emitted configs validate through
production paths, component strict-load status is recorded, `model_index.json`
library tokens match loaders, and revisions are pinned when converting from
HF.
4. Component parity lane:
Verify local parity tests exist for every required component, including reused
components. Scaffolds may skip in CI, but the PR must provide local non-skip
PASS evidence or an explicit accepted blocker.
5. Pipeline lane:
Verify stage order, required modules, `_class_name` / `EntryClass.__name__`
resolution, config defaults, presets, `SamplingParam` fields, registry
registration, examples, smoke tests, and pipeline parity.
6. Quality and evidence lane:
Verify media quality regression is added or explicitly deferred, examples run,
generated outputs are non-corrupt, `tests/local_tests/<family>/README.md` and
`PORT_STATUS.md` are current, and final blockers are surfaced in the review.
## Findings To Prioritize
Prioritize review findings in this order:
- Missing or skipped required component parity without accepted blocker.
- Pipeline parity/smoke/example missing or skipped for a pipeline PR.
- Conversion emits unloadable or unvalidated configs/weights.
- Wrong `model_index.json` `_class_name`, component library token, or registry
class resolution.
- Runtime diffusers/transformers model-class imports for components that own
weights or numerical behavior.
- Dropped modalities, output heads, variants, or conditioning streams without
explicit approval.
- Reused FastVideo component lacks exact definition/instantiation proof or
non-skip parity.
- Public generation kwargs/preset defaults missing from `SamplingParam`.
- Tests only check shapes, importability, or successful generation without
numerical/media comparison.
- Tokens, credentials, reference clones, staged weights, or generated bulk assets
committed to the PR.
## Output Format
Write normal code-review findings first, ordered by severity. Include file and
line references from the PR diff when possible.
Use this phrasing for missing add-model evidence:
```text
This PR does not satisfy the add-model <component|conversion|pipeline|final>
gate because <specific required evidence> is missing. The risk is <runtime load,
numerical parity, dropped output, registry resolution, etc.>.
```
Keep the summary short. Mention which lanes were reviewed and which could not be
verified because assets, GPU time, or external credentials were unavailable.
-438
View File
@@ -1,438 +0,0 @@
---
name: add-model
description: Manual /add-model workflow for implementing a FastVideo model or first-class component port after add-model-01-prep has staged reference code and weights. Organizes the port into numbered phases with conversion rules, component policies, parity gates, and handoff checks.
---
# Add Model
## Manual Invocation
This skill is for explicit `/add-model` use only. Do not auto-start it from a
casual model-port mention. The setup-only workflow is
`../add-model-01-prep/SKILL.md`.
## Goal
Port a new FastVideo model family, model variant, or first-class reusable
component so it can be loaded through FastVideo's native model, config, stage,
registry, preset, and test infrastructure.
FastVideo has one pipeline architecture: stage-based composition via
`ComposedPipelineBase`. Vary the stages and modules, not the architecture.
## Scope Shapes
Use this skill for either shape:
| Shape | Required output |
|---|---|
| Full model family or variant | Native components, conversion if needed, pipeline config/class, presets, registry, smoke test, local parity tests, example, quality regression. |
| First-class component contribution | Native component class/config, bucket export, component parity test, and a documented downstream pipeline that will consume it. Skip pipeline/preset/registry rows only when the contribution is intentionally component-only. |
If upstream ships many variants, lock scope before coding. "Base model" means
checkpoint variant, not a modality subset. If the base checkpoint produces
audio, pose, depth, masks, or other output heads, either support those outputs
or get explicit user agreement to drop them.
## Required Input
Start from an `add-model-01-prep` handoff, or equivalent fields matching
`contracts/prep_handoff.md`.
Before Phase 0, read the shared rules and all relevant schemas:
- `shared/common_rules.md`
- `contracts/prep_handoff.md`
- `contracts/port_state.md`
- `contracts/escape_hatch.md`
- `contracts/component_context.md`
- `contracts/parity_status.md`
- `contracts/conversion_request.md`
- `contracts/conversion_handoff.md`
- `contracts/component_skill_handoff.md`
- `contracts/pipeline_context.md`
- `contracts/pipeline_handoff.md`
- `contracts/final_handoff.md`
## Hard Rules
- Follow `shared/common_rules.md` for token/auth safety, state files, escape
hatches, production import boundaries, and skip/pass semantics.
- If the prep handoff is missing or ambiguous, stop and run
`../add-model-01-prep/SKILL.md`.
- If a needed component is not ported, do not ship the pipeline that needs it.
- Wan is grandfathered for missing local parity; do not copy its missing-test
precedent for new work.
## Escape Hatches
Follow `shared/common_rules.md` and `contracts/escape_hatch.md`. The main
orchestrator should ask only when no phase skill can safely continue under the
shared rules.
## Files Map
| Area | Paths |
|---|---|
| DiT | `fastvideo/models/dits/<family>.py`, `fastvideo/configs/models/dits/<family>.py`, bucket `__init__.py`. |
| VAE | `fastvideo/models/vaes/<arch_or_family>.py`, `fastvideo/configs/models/vaes/<arch_or_family>.py`, bucket `__init__.py`. Name by shared arch when reusable (`oobleck.py`, `autoencoder_kl.py`), otherwise by family (`wanvae.py`). |
| Encoder / conditioner / scheduler / upsampler | Native class/config in the matching `fastvideo/models/<bucket>/` and `fastvideo/configs/models/<bucket>/` bucket. |
| Lazy loader wrapper | Optional `fastvideo/models/<bucket>/<family>_loader.py` or similar thin `nn.Module` wrapper when a component is fetched from an external HF repo and should be hidden from host-pipeline state-dict matching. |
| Conversion | `scripts/checkpoint_conversion/<family>_to_diffusers.py` only when `needs_conversion=yes`. |
| Pipeline | `fastvideo/pipelines/basic/<family>/<family>_pipeline.py` plus sibling files for variants whose components or required modules differ. |
| Pipeline config | `fastvideo/configs/pipelines/<family>.py` or `fastvideo/pipelines/basic/<family>/pipeline_configs.py`. |
| Stages | `fastvideo/pipelines/basic/<family>/stages/` only for model-specific stage subclasses. |
| Presets / registry | `fastvideo/pipelines/basic/<family>/presets.py`, `fastvideo/registry.py`. |
| Tests | Component parity under `tests/local_tests/<bucket>/`; pipeline smoke/parity under `tests/local_tests/pipelines/`; CI-backed quality tests under `fastvideo/tests/`. |
| Example | `examples/inference/basic/basic_<family>*.py`, one per public mode/variant. |
## Phase 0: Scope And Handoff Gate
1. Validate every required handoff field.
2. Resolve `needs_conversion=unknown` before component work:
```bash
python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \
"<hf-or-local-path>" \
--json
```
3. List first-PR scope across both axes:
- Variant axis: base, distill, SR/refine, causal, DMD, I2V, V2V, etc.
- Modality axis: video, image, audio, pose, depth, masks, text, etc.
4. For component-only work, explicitly name the downstream full-pipeline PR or
planned consumer.
5. Confirm `official_env_status` is `imports_ok` or
`private_deps_need_stubs`. If it is `blocked`, return to
`../add-model-01-prep/SKILL.md` before parity scaffolding.
6. Confirm `local_tests_readme` exists and records official setup, HF weights,
dependency changes, and planned parity commands for reviewers.
7. Confirm `port_state_file` exists, follows `contracts/port_state.md`, and has
rows for open questions/issues found during prep.
8. If there are multiple official implementations, choose the one whose
architecture matches the published weights. A blessed library port can be a
better parity reference than a highly configurable research repo; document
the choice in tests.
## Phase 1: Reference And Architecture Study
Read the official pipeline call path before writing code.
Record:
- Required modules from `model_index.json` or equivalent: transformer, VAE,
text encoders, tokenizers, scheduler, image encoders, audio VAE, vocoder,
conditioners, upsamplers.
- Input/output modalities and every dedicated DiT output head.
- Text/image/audio encoding flow, latent shape, dtype, scaling, packing,
scheduler/timestep math, guidance math, VAE normalization, and decode flow.
- Whether the official code relies on private deps, custom ops, or special
kernels that parity tests must stub.
Arch config rule:
- `ArchConfig` fields must match the emitted per-component config, especially
`transformer/config.json`, one-to-one.
- Pipeline knobs do not belong on the DiT arch config: inference steps, CFG
scales, flow shift, FPS, VAE stride, text target length, data-proxy knobs,
eval defaults, and sampling defaults go on `PipelineConfig`, presets, or
stages.
- If the HF repo is raw or has empty configs, synthesize
`transformer/config.json` from the official Python model-config class, not
from data/eval config classes.
## Phase 2: Early Parity Scaffolding
Create component parity tests before or alongside implementation. Use
`../add-model-02-parity/SKILL.md` and its `templates/component_parity_test.py`.
The official reference must import in the current FastVideo environment, or the
prep handoff must identify private deps that will be stubbed locally for tests.
Use `local_tests_readme` as the reviewer-facing source for setup commands and
update its planned test table as parity scaffolds are added.
This phase is early by design:
- Official loading can be implemented from the reference study.
- FastVideo loading can target planned standardized class/config/loader paths.
- Tests may initially skip because the FastVideo class or converted weights do
not exist yet.
- The scaffold must still contain real official loading, deterministic inputs,
output extraction, and concrete tensor comparisons. No unconditional skips,
no shape-only tests.
Use subagents here: dispatch one parity-test subagent per required component,
including components that may be reused. Their output becomes the red/skip
target that porting or reuse-verification subagents make pass later.
## Phase 3: Reuse Gate And Component Dispatch
Build a component inventory before implementation:
| Field | Meaning |
|---|---|
| Component | transformer, VAE, text encoder, image encoder, scheduler, conditioner, upsampler, vocoder, etc. |
| Official definition | Repo-relative source file, class/function name, and relevant line/range if known. |
| Official instantiation | Repo-relative pipeline/config/factory call site plus constructor args and runtime flags. |
| FastVideo target | Existing class to reuse or new bucket/file/config to add. |
| Parity test | Required local test path, including reused components. |
| Status | `reuse_pending`, `reuse_proven`, `port_pending`, `non_skip_pass`, or `blocked`. |
Reuse is allowed only from the checked-out FastVideo tree. Do not wait for or
depend on an open PR adding a native class; add the native port directly in this
PR if the current tree cannot be reused.
Reuse decision:
1. Record exact official definition and instantiation evidence for every
component.
2. If an existing FastVideo class and config match both definition and
instantiation, pass that reused target to the bucket-specific skill in
`mode=prototype` and require reuse evidence plus key/shape dumps.
3. If either definition or instantiation differs, port the component directly as
FastVideo-native code through the bucket-specific skill.
4. Reused components still require non-skip component parity against the exact
official instantiation used by the target pipeline.
Porting subagent dispatch:
- Dispatch one subagent per component after Phase 2 parity scaffolds exist.
- Use `../add-model-03-port-dit/SKILL.md` for DiTs/transformers.
- Use `../add-model-04-port-vae/SKILL.md` for VAEs.
- Use `../add-model-05-port-encoder/SKILL.md` for text, image, audio, or compound
encoders/conditioners that fit the encoder config bucket.
- Use `../add-model-06-port-generic/SKILL.md` for schedulers, upsamplers,
vocoders, adapters, preprocessors, or unknown components.
- Each subagent owns one component only and must loop on that component's local
parity test until it produces a non-skip PASS or returns a precise blocker.
Every component subagent must receive a complete packet matching
`contracts/component_context.md`. If any required path is unknown, pass `unknown`
plus the exact search already performed. Do not silently omit ambiguous official
files or prototype concerns.
Bucket, layer, and attention rules live in the bucket-specific skills and
`fastvideo/layers/AGENTS.md`.
## Phase 4: Native Component Prototype
Conversion needs a FastVideo state-dict surface. Use the Phase 3
bucket-specific skill in `mode=prototype` for every required component, including
reused components.
Prototype success criteria:
- the FastVideo-native or reused class/config can import and instantiate with the
exact official architecture args;
- official and FastVideo key/shape dumps exist for every stateful component;
- `local_tests_readme` and `port_state_file` record prototype status and concerns;
- the returned handoff matches `contracts/component_skill_handoff.md`.
Do not chase numerical parity in Phase 4. Prototype mode ends when conversion has
the key/shape surface it needs, or when the component skill returns a precise
blocker or escape hatch.
## Phase 5: Param Mapping And Weight Conversion
Use `../add-model-07-conversion/SKILL.md` after Phase 4 prototypes exist.
Send a request matching `contracts/conversion_request.md`; consume the returned
`contracts/conversion_handoff.md` update before Phase 6.
Use the prep handoff's `needs_conversion` value:
- `no`: verify the source already has the component layout FastVideo loaders can
consume, then record any passthrough components.
- `yes`: write `scripts/checkpoint_conversion/<family>_to_diffusers.py` and
output `converted_weights/<family>/`.
- `unknown`: return to Phase 0.
The conversion skill owns source-layout handling, mapping derivation, config and
`model_index.json` emission, passthrough assets, strict-load verification, and
Phase 6 retry requests. Component skills must not patch conversion scripts or
converted weights ad hoc.
## Phase 6: Component Parity Debug
This is the expected expensive loop. Dispatch one subagent per required
component, including reused components, using the bucket-specific skill in
`mode=parity-debug`.
Each subagent gets:
- the complete component context packet from Phase 3/4;
- updated conversion mapping notes and strict-load result from Phase 5;
- any prototype concerns or unknowns that were not resolved before conversion.
The bucket-specific skills own parity-debug tactics. If a failure belongs to
conversion, route it through `../add-model-07-conversion/SKILL.md` with a retry
request matching `contracts/conversion_request.md`, then resume the component
skill with the updated conversion handoff.
When a component failure narrows to layer-by-layer numerical drift, load
`../add-model-08-trace/SKILL.md` before writing custom hooks. It uses
`fastvideo/hooks/activation_trace.py`; canonical env vars and JSONL format are
documented in `docs/contributing/activation_trace.md`.
Phase 6 ends only when every required component handoff reports
`parity_status=non_skip_pass`, or when a precise blocker or escape hatch is
recorded in `port_state_file`.
## Phase 7: Pipeline, Stages, And Variants
Do not start Phase 7 until every required component, reused or ported, has a
non-skip local parity PASS from Phase 6. If any component parity test is still
`scaffold_skip`, `debug_red`, `blocked`, or missing, resume Phase 6 first.
Use `../add-model-09-pipeline/SKILL.md` for pipeline definition and parity-debug.
Send a complete packet matching `contracts/pipeline_context.md`; consume the
returned `contracts/pipeline_handoff.md` before moving to quality regression or
final handoff.
The pipeline skill owns:
- pipeline class, stage chain, and optional model-specific stages;
- pipeline config, presets, registry updates, and examples;
- official args/defaults/presets comparison before setting FastVideo defaults;
- pipeline smoke and parity tests;
- continuous pipeline parity-debug until non-skip PASS or precise blocker;
- updates to `local_tests_readme` and `port_state_file`.
The pipeline handoff must explicitly cover stage order, variants, modality and
output-head handling, config/preset/registry/example status, smoke/parity tests,
and any return-to-Phase-6 evidence.
## Phase 8: PipelineConfig, Presets, Registry, Examples
This phase is implemented through `../add-model-09-pipeline/SKILL.md` after the
Phase 7 component-parity gate passes. Accept the pipeline handoff only if it
covers configs, presets, registry detection/exact class resolution, examples,
new `SamplingParam` fields for public kwargs/defaults, and local smoke/parity
status. Detailed rules live in `../add-model-09-pipeline/SKILL.md`.
## Phase 9: Parity Activation And Local Verification
Local parity is author-run, not CI-enforced. CI may only run package-level
quality tests later. Before handoff, Phase 2 scaffolds must be activated into
non-skip PASS results.
Order is mandatory:
1. Run conversion if needed.
2. Run component parity for every required component, including reused ones.
3. Run pipeline smoke.
4. Run pipeline parity.
5. Run the basic example.
If pipeline smoke or parity points back to component implementation,
strict-load, or conversion mapping, return to Phase 6 or Phase 5 rather than
patching around the issue in the pipeline.
Skip policy:
- Follow `shared/common_rules.md`: a committed local test may skip for absent
clones/weights, but a local skip is not a verified pass.
Use the commands and tolerance guidance from `../add-model-02-parity/SKILL.md` for
component checks and from `../add-model-09-pipeline/SKILL.md` for pipeline smoke,
pipeline parity, and examples. Record exact commands, status, and blockers in
`local_tests_readme` and `port_state_file`.
## Phase 10: Quality Regression
Video outputs:
- Add `fastvideo/tests/ssim/test_<family>_similarity.py` when output video
quality must be preserved.
- Seed references through `seed-ssim-references` after the test exists.
Audio outputs:
- SSIM does not apply. Use an audio-specific regression metric such as
mel-spectrogram L1, multi-resolution STFT, CLAP cosine, or a project-approved
learned metric.
- Document the metric and hardware/runtime assumptions in the test.
Joint AV outputs:
- Keep video and audio regression checks separate unless there is a validated
joint metric.
## Phase 11: Post-Parity Review And Handoff
After parity is green, run a hot-path review before handoff:
- Hoist constant tensor allocations out of sampler/denoising loops.
- Replace per-step `randn_like` churn with preallocated buffers plus
`.normal_()` when safe.
- Move `torch.backends.*` flag changes to one-shot setup/load paths.
- Delete `batch.extra` writes that nothing reads.
- Derive magic constants from configs when possible.
Pre-handoff checklist:
```text
[ ] Prep handoff is complete and committed nowhere with token values.
[ ] Conversion was run if needed and output loads with real weights.
[ ] Every required component, reused or newly ported, has a non-skip local parity PASS.
[ ] `local_tests_readme` lists every component parity test, command, status, and blocker if any.
[ ] `port_state_file` has every open question/issue either resolved or listed as an explicit blocker.
[ ] Any `next_step=ask_user` has a matching `escape_hatch` block and `E###` row.
[ ] Pipeline smoke has a non-skip local PASS.
[ ] Pipeline parity has a non-skip local PASS against the official reference.
[ ] Basic example runs and writes a non-corrupt output.
[ ] Video SSIM or audio-specific quality regression is added or explicitly deferred.
[ ] Runtime production code has no diffusers/transformers model-class imports.
[ ] Production comments are WHY-focused; examples have user-story docstrings.
[ ] Post-parity hot-path pass is complete.
```
Ask before deleting any reference clone or staged weights created by
`add-model-01-prep`. Leave `.gitignore` entries so future parity assets stay
untracked. Never commit the clone, weights, `.env`, credentials, or anything
matching `*secret*`.
## References
- `../add-model-01-prep/SKILL.md` for user-input collection, HF inspection,
weight staging, reference cloning, and setup handoff.
- `contracts/` for canonical handoff schemas used by prep, parity, conversion,
component porting, escape hatches, and final handoff.
- `../add-model-02-parity/SKILL.md` for early component parity scaffolds and
activation templates.
- `../add-model-07-conversion/SKILL.md` for Phase 5 mapping, conversion scripts,
monolithic checkpoint splitting, and strict-load checks.
- `../add-model-03-port-dit/SKILL.md`, `../add-model-04-port-vae/SKILL.md`,
`../add-model-05-port-encoder/SKILL.md`, and
`../add-model-06-port-generic/SKILL.md` for component subagent implementation
and parity-debug loops.
- `../add-model-09-pipeline/SKILL.md` for pipeline definition, config/preset/
registry/example wiring, smoke tests, and pipeline parity-debug.
- `fastvideo/layers/AGENTS.md` for native layer selection and state-dict surface
guidance.
- `docs/contributing/coding_agents.md` for narrative context.
- `docs/design/overview.md` for pipeline/config/registry architecture.
- `fastvideo/pipelines/basic/wan/` for standard T2V/I2V/DMD/Causal variants.
- `fastvideo/pipelines/basic/ltx2/` for non-standard stages and audio/video
patterns.
- `tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py` for pipeline
parity shape.
- `tests/local_tests/transformers/test_ltx2.py`,
`tests/local_tests/vaes/test_ltx2_vae.py`, and
`tests/local_tests/encoders/test_ltx2_gemma_parity.py` for component parity.
- `scripts/checkpoint_conversion/convert_ltx2_weights.py` for modern conversion
script shape.
- `scripts/checkpoint_conversion/wan_to_diffusers.py` for legacy regex mapping
reference only.
## Changelog
| Date | Change |
|---|---|
| 2026-04-24 | Initial FastVideo add-model workflow. |
| 2026-04-30 | Split external setup into `add-model-01-prep`. |
| 2026-04-30 | Rewrote as manual `/add-model` phase workflow and incorporated prior review decisions. |
| 2026-04-30 | Extracted early parity scaffolding into `add-model-02-parity` and moved it before conversion/component implementation. |
| 2026-04-30 | Added component reuse proof gate, bucket-specific porting skills, and parity PASS requirement for reused components. |
| 2026-04-30 | Split prototype, conversion, and parity-debug phases; added conversion skill for monolithic and separate checkpoint layouts. |
| 2026-04-30 | Extracted handoff schemas into `contracts/` for shared use across skills. |
| 2026-04-30 | Added pipeline skill contract and Phase 7 component-parity gate. |
| 2026-04-30 | Added escape-hatch contract for user decisions and `ask_user` handoffs. |
@@ -1,29 +0,0 @@
# Add Model Contracts
Canonical handoff schemas for the `/add-model` workflow. When a skill needs to
send or receive structured context, use these files instead of inventing a local
schema.
| Contract | Use |
|---|---|
| `prep_handoff.md` | `add-model-01-prep` output and `/add-model` Phase 0 input. |
| `port_state.md` | Per-port `PORT_STATUS.md` file tracking progress, open questions, and issues. |
| `escape_hatch.md` | Shared pause-and-ask schema for user decisions the workflow cannot safely choose. |
| `component_context.md` | Per-component packet passed to parity, prototype, conversion, and parity-debug subagents. |
| `parity_status.md` | `add-model-02-parity` scaffold/activation status returned to `/add-model`. |
| `conversion_request.md` | Phase 5 conversion input and Phase 6 conversion retry request. |
| `conversion_handoff.md` | `add-model-07-conversion` output back to `/add-model` and component subagents. |
| `component_skill_handoff.md` | Component porting skill output in prototype or parity-debug mode. |
| `pipeline_context.md` | Phase 7 packet passed to `add-model-09-pipeline` after component parity is green. |
| `pipeline_handoff.md` | `add-model-09-pipeline` output back to `/add-model` after pipeline definition or parity-debug. |
| `final_handoff.md` | Final `/add-model` pre-handoff checklist summary. |
Rules:
- Do not omit required fields. Use `unknown` plus the search already performed
when the value is not known yet.
- Do not include raw token values. Use env var names only.
- Keep model-specific mapping details in conversion scripts and the local tests
README/status notes, not in generic skill docs.
- Use `next_step=ask_user` only with an `escape_hatch` block matching
`escape_hatch.md`.
@@ -1,55 +0,0 @@
# Component Context Contract
Canonical per-component packet passed from `/add-model` to parity, prototype,
conversion, and parity-debug subagents.
```text
component_context:
model_family: <snake_case>
component: <name>
component_type: <dit|vae|encoder|scheduler|conditioner|upsampler|vocoder|generic>
mode: parity-scaffold | prototype | parity-debug
official_ref_dir: <path or import path>
official_definition_files:
- path: <repo-relative or absolute path in official repo>
symbols: <class/function names>
notes: <layer graph, output contract, state-dict owner>
official_instantiation_files:
- path: <repo-relative or absolute path in official repo>
symbols: <factory/pipeline/config names>
args: <constructor args, config values, runtime flags>
official_weight_source: <checkpoint file, subfolder, prefix, or passthrough source>
fastvideo_target_files:
- fastvideo/models/<bucket>/<file>.py
- fastvideo/configs/models/<bucket>/<file>.py
local_tests_readme: tests/local_tests/<model_family>/README.md
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
parity_test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py
prototype_key_dumps:
official: converted_weights/<family>/_mapping/<component>_official_keys.json | planned | unknown
fastvideo: converted_weights/<family>/_mapping/<component>_fastvideo_keys.json | planned | unknown
conversion:
script: scripts/checkpoint_conversion/<family>_to_diffusers.py | not_created | not_needed | unknown
converted_component_dir: converted_weights/<family>/<component> | not_created | not_needed | unknown
model_index_library: <diffusers|transformers|fastvideo|fastvideo.*|unknown|none>
config_file: <config.json|scheduler_config.json|none|unknown>
mapping_notes: <key prefixes, split/fuse concerns, skipped keys, not_created, not_needed, or unknown>
production_loader_strictness: <strict|non_strict_with_allowed_keys|stateless|unknown>
strict_load: <not_run | pass | pass_with_documented_exclusions | blocked>
concerns_or_unknowns:
- <prototype mismatch, ambiguous arg, missing op, dtype concern, output head, etc.>
```
Rules:
- If any required path is unknown, pass `unknown` plus the exact search already
performed.
- Do not silently omit ambiguous official files, instantiation args, or prototype
concerns.
- For reused components, still fill every field and set `fastvideo_target_files`
to the reused class/config.
- In `mode=parity-scaffold`, prototype and conversion fields may be `planned`,
`not_created`, `not_needed`, or `unknown`; do not invent paths or statuses that
do not exist yet.
- Update `port_state_file` when concerns, issues, conversion status, or parity
status change.
@@ -1,41 +0,0 @@
# Component Skill Handoff Contract
Returned by `add-model-03-port-dit`, `add-model-04-port-vae`,
`add-model-05-port-encoder`, and `add-model-06-port-generic`.
```text
component: <name>
mode: prototype | parity-debug
files_changed: <model/config/export/test/readme paths>
official_files_used: <definition files, instantiation files>
prototype_key_dumps: <official path, fastvideo path, or none>
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
concerns_or_unknowns: <remaining or newly discovered concerns>
parity_test: <path>
parity_status: scaffold_skip | debug_red | non_skip_pass | blocked
production_loader_strictness: strict | non_strict_with_allowed_keys | stateless
strict_load: pass | pass_with_documented_exclusions | blocked | not_run
pytest_output: <command + short result>
blocker: <none or exact missing dependency/weights/numeric mismatch>
conversion_retry_request: <none or failing keys/shapes/prefixes/evidence for add-model-07-conversion>
readme_updated: yes | no
next_step: phase_5_conversion | phase_5_conversion_retry | phase_6_continue | ask_user | blocked
escape_hatch: <none or block matching contracts/escape_hatch.md>
```
Rules:
- In `mode=prototype`, parity may be `scaffold_skip` or `blocked`; key dumps are
the required artifact. Successful prototype handoff should use
`next_step=phase_5_conversion`.
- In `mode=parity-debug`, final success requires `parity_status=non_skip_pass`.
- If conversion is implicated, return `conversion_retry_request` and do not edit
conversion scripts or converted weights directly.
- If production loading is non-strict, list allowed missing/unexpected keys in
the parity test or handoff and mark `strict_load=pass_with_documented_exclusions`.
- Update `port_state_file` before returning: component row, open questions,
issues/blockers, decisions, and handoff notes.
- Return an `escape_hatch` only for user decisions, not for normal component
implementation or parity-debug failures.
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
`PORT_STATUS.md` row.
@@ -1,41 +0,0 @@
# Conversion Handoff Contract
Returned by `../add-model-07-conversion/SKILL.md` to `/add-model` and component
parity-debug subagents.
```text
conversion_script: scripts/checkpoint_conversion/<family>_to_diffusers.py
source_layout: <diffusers|raw_official|separate_components|monolithic|mixed|custom>
converted_weights_dir: converted_weights/<model_family>
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
components_written: <list>
passthrough_components: <list>
strict_load: pass | pass_with_documented_exclusions | blocked
component_context_updates:
- component: <name>
converted_component_dir: <path>
model_index_library: <diffusers|transformers|fastvideo|fastvideo.*>
config_file: <path or none>
config_validation: pass | blocked | not_applicable
mapping_notes: <prefixes, split/fuse ops, skipped keys>
production_loader_strictness: strict | non_strict_with_allowed_keys | stateless
strict_load: pass | pass_with_documented_exclusions | blocked | not_run
retry_resolved: <yes | no | not_a_retry>
concerns_or_unknowns: <remaining list>
blocked_on: <none or exact blocker>
next_step: phase_6_component_parity_debug | ask_user
escape_hatch: <none or block matching contracts/escape_hatch.md>
```
Rules:
- Include strict-load evidence for every stateful converted component.
- If a component intentionally loads non-strictly, list the exact missing or
unexpected keys and why they are safe.
- Include the actual `model_index.json` library token, config filename, and config
validation result for every emitted component.
- Preserve retry evidence so the requesting component subagent can resume with
updated context.
- Keep `port_state_file` synchronized with `component_context_updates`.
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
`PORT_STATUS.md` row.
@@ -1,54 +0,0 @@
# Conversion Request Contract
Consumed by `../add-model-07-conversion/SKILL.md` in Phase 5 and during Phase 6
conversion retries.
Initial conversion request:
```text
model_family: <snake_case>
source_layout: diffusers | raw_official | monolithic | separate_components | mixed | custom
official_weights: <HF repo, local dir, or checkpoint file>
hf_revision: <revision | default | none>
converted_weights_dir: converted_weights/<model_family>
local_tests_readme: tests/local_tests/<model_family>/README.md
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
components:
- name: <transformer|vae|text_encoder|conditioner|scheduler|...>
component_type: <dit|vae|encoder|scheduler|conditioner|upsampler|vocoder|generic>
official_definition_files: <paths + symbols>
official_instantiation_files: <paths + call sites + args>
official_weight_source: <checkpoint file, prefix, subfolder, or passthrough source>
official_keys: <path to official key/shape dump>
fastvideo_keys: <path to FastVideo prototype key/shape dump>
fastvideo_class: <class name>
model_index_library: <diffusers|transformers|fastvideo|fastvideo.*>
config_filename: <config.json|scheduler_config.json|none>
production_loader_strictness: <strict|non_strict_with_allowed_keys|stateless>
source_prefix_or_path: <prefix or path>
parity_test: <component parity test path>
prototype_concerns_or_unknowns: <short list>
```
Retry request from a component skill:
```text
conversion_retry_request:
component: <name>
parity_test: <path>
failing_keys: <official and FastVideo keys, if known>
expected_actual_shapes: <expected vs actual shapes, if known>
source_prefix_or_path: <prefix/path implicated by the failure>
evidence: <strict-load error, first divergent tensor, parity log excerpt>
suspected_fix: <rename | split | fuse | skip | component bucket | config | unknown>
```
Rules:
- Phase 5 conversion requires Phase 4 official/FastVideo key dumps.
- Component skills must use the retry request instead of editing conversion
scripts or converted weights directly.
- Conversion must update `port_state_file` with conversion status, retry history,
strict-load status, new issues, and resolved issues.
- Conversion must validate emitted config keys through the production config
update path and record the config filename expected by each loader.
@@ -1,51 +0,0 @@
# Escape Hatch Contract
Canonical pause-and-ask schema for `/add-model` skills. Use this when the next
action requires user input instead of autonomous debugging.
```text
escape_hatch:
needs_user_input: yes | no
decision_type: scope | dependency | auth | cost | destructive | ambiguity | blocker
question: <one precise question>
recommended_option: <safe recommended choice>
options:
- <option + consequence>
safe_default: <what the agent will do after approval, or none>
blocked_until_answered: yes | no
state_snapshot:
phase: <phase or skill mode>
files_changed:
- <paths>
command_or_test: <last relevant command, or not_run>
evidence: <short logs, paths, error text, or blocker ID>
```
Use `needs_user_input=no` when the handoff is green or the next step is already
specified by the workflow.
Ask the user only for decisions the workflow cannot safely choose:
- product or PR scope changes, including dropping a modality, output head, or
variant;
- core dependency changes, version pin changes, or installing untrusted/private
dependencies;
- auth setup for gated repos, using env var names only and never token values;
- large downloads, publishing weights, SSIM/reference uploads, or GPU-heavy work
where cost/runtime approval is needed;
- destructive file/git operations, overwriting existing clones/weights, or
deleting staged assets;
- ambiguous official sources of truth with incompatible behavior;
- accepting a known blocker, loosening parity/quality tolerances, or shipping
without required non-skip parity.
Do not ask for normal recoverable failures:
- missing imports, missing local paths, skipped tests, failing parity, conversion
mapping errors, strict-load failures, format/lint failures, or implementation
bugs covered by the skill workflow.
Before returning `next_step=ask_user`, update
`tests/local_tests/<model_family>/PORT_STATUS.md` with the blocker/question ID,
include the exact evidence, and provide one recommended option plus at most three
alternatives.
@@ -1,38 +0,0 @@
# Final Handoff Contract
Completed by `/add-model` before handing work back to the user or opening a PR.
```text
final_handoff:
prep_handoff_complete: yes | no
conversion_status: not_needed | pass | blocked
components:
- name: <component>
reuse_or_port: reused | ported
parity_test: <path>
parity_status: non_skip_pass | blocked
concerns_or_unknowns: <none or list>
pipeline_smoke: pass | blocked | not_run
pipeline_parity: pass | blocked | not_run
example_status: pass | blocked | not_run
quality_regression: added | deferred_with_reason | not_applicable
local_tests_readme: tests/local_tests/<model_family>/README.md
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
token_values_committed: no
runtime_third_party_model_imports: none | listed_with_rationale
blockers: <none or list>
escape_hatch: <none or block matching contracts/escape_hatch.md>
```
Required before handoff:
- Every required component, reused or ported, has non-skip local parity PASS.
- Pipeline smoke and pipeline parity are non-skip PASS, or a blocker is explicit.
- Basic example runs and writes a non-corrupt output.
- `local_tests_readme` lists every component parity command/status/blocker.
- `port_state_file` has no unresolved blocker that is omitted from the final
response or PR notes.
- No raw HF token values, credentials, `.env`, reference clone, or staged weight
blobs are committed.
- If final handoff is blocked on user input, include an `escape_hatch` block and
matching `PORT_STATUS.md` row.
@@ -1,44 +0,0 @@
# Parity Status Contract
Returned by `../add-model-02-parity/SKILL.md` to `/add-model` and later updated by
component parity-debug subagents.
```text
component_parity:
- component: <name>
test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py
status: scaffold_skip | debug_red | non_skip_pass | blocked
missing: <none | fastvideo_class | converted_weights | official_import | ...>
coverage_scope: production_loader | implementation_subcomponent | both
official_definition_files: <paths>
official_instantiation_files: <paths>
concerns_or_unknowns: <short list>
pipeline_parity:
test: <path or not-created>
status: not_started | scaffold_skip | debug_red | non_skip_pass | blocked
local_tests_readme: tests/local_tests/<model_family>/README.md
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
notes: <short list>
escape_hatch: <none or block matching contracts/escape_hatch.md>
```
Status meanings:
- `scaffold_skip`: test is present but skips for a specific missing dependency,
FastVideo class, or weights.
- `debug_red`: both sides load and the test fails numerically.
- `non_skip_pass`: required before final handoff for every required component,
including reused components.
- `blocked`: a precise missing dependency, weight, official call path, or
component/conversion regression prevents local activation.
Coverage meanings:
- `production_loader`: FastVideo side loads through the same loader/path used by
a pipeline.
- `implementation_subcomponent`: FastVideo side constructs classes or remaps
tensors directly to isolate implementation behavior.
- `both`: the test covers both production loading and implementation behavior.
Use `escape_hatch` only when blocked status requires a user decision. Normal
skips or red parity should be debugged by the workflow without asking.
@@ -1,82 +0,0 @@
# Pipeline Context Contract
Canonical packet passed from `/add-model` to `add-model-09-pipeline` for pipeline
definition and pipeline parity-debug work.
```text
pipeline_context:
model_family: <snake_case>
mode: pipeline-definition | pipeline-parity-debug
workload_types:
- <T2V|I2V|V2V|T2I|compatibility-shim-with-rationale>
modalities:
inputs: <text/image/video/audio/pose/depth/mask/etc.>
outputs: <video/image/audio/joint-av/latents/etc.>
official_ref_dir: <path or import path>
official_pipeline_files:
- path: <repo-relative or absolute path in official repo>
symbols: <pipeline/factory/sample functions>
notes: <stage order, mutable state, output contract>
official_call:
command_or_api: <official CLI, Python call, or package entrypoint>
args_and_defaults: <height, width, frames, fps, duration, steps, CFG, scheduler, seeds, etc.>
preset_source: <model card, config file, official script, or unknown>
scheduler_and_rng: <timestep/sigma/noise/generator behavior>
output_contract: <decoded media, denoised latents, waveform, dict keys, etc.>
model_index:
class_name: <FastVideo pipeline class name to emit in model_index.json>
entry_class_names: <registered EntryClass.__name__ values that must include class_name>
required_modules: <text_encoder, tokenizer, vae, transformer, scheduler, etc.>
passthrough_modules: <tokenizer, scheduler, processor, external HF dirs, or none>
sampling_param:
new_fields: <none or list of public kwargs/preset defaults to add to SamplingParam>
cli_fields: <none or list of fields that need CLI args>
placeholder_fields: <none or video-shaped compatibility placeholders with rationale>
components:
- name: <component>
component_type: <dit|vae|encoder|scheduler|conditioner|upsampler|vocoder|generic>
parity_test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py
parity_status: non_skip_pass
fastvideo_target_files: <model/config/export files>
converted_component_dir: converted_weights/<family>/<component>
conversion:
converted_weights_dir: converted_weights/<family>
source_layout: <diffusers|raw_official|monolithic|separate_components|mixed|custom>
model_index_path: converted_weights/<family>/model_index.json
fastvideo_targets:
pipeline_files:
- fastvideo/pipelines/basic/<family>/<family>_pipeline.py
stage_files:
- fastvideo/pipelines/basic/<family>/stages/<stage>.py
pipeline_config_files:
- fastvideo/configs/pipelines/<family>.py
preset_file: fastvideo/pipelines/basic/<family>/presets.py
registry_file: fastvideo/registry.py
example_files:
- examples/inference/basic/basic_<family>.py
smoke_test: tests/local_tests/pipelines/test_<family>_pipeline_smoke.py
parity_test: tests/local_tests/pipelines/test_<family>_pipeline_parity.py
local_tests_readme: tests/local_tests/<model_family>/README.md
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
concerns_or_unknowns:
- <pipeline branch, unsupported workload, output head, preset ambiguity, etc.>
```
Rules:
- Start only after every required component, reused or ported, has
`parity_status=non_skip_pass`. If any row is missing or skipped, return to
`/add-model` Phase 6.
- Record official call arguments and default sources before writing FastVideo
presets. Do not invent inference defaults from memory.
- `model_index.class_name` must match a registered pipeline `EntryClass.__name__`;
registry detectors are not sufficient for executable pipeline resolution.
- Every public generation kwarg or preset default must be represented in
`SamplingParam`, or documented as an intentional internal-only field.
- `T2A`, `A2A`, and `AV` may be used only after `WorkloadType` supports them;
otherwise record the compatibility shim and rationale explicitly.
- Keep token values out of the packet. Use only token environment variable names.
- If a target path is unknown, use `unknown` plus the exact search already
performed.
- Update `local_tests_readme` and `port_state_file` whenever pipeline smoke,
parity, presets, registry, examples, or blockers change.
@@ -1,71 +0,0 @@
# Pipeline Handoff Contract
Returned by `add-model-09-pipeline` to `/add-model` after pipeline definition or
pipeline parity-debug work.
```text
pipeline_handoff:
model_family: <snake_case>
mode: pipeline-definition | pipeline-parity-debug
files_changed:
- <pipeline/config/preset/registry/stage/example/test/readme/status paths>
official_files_used:
- <definition/call/default source paths>
required_config_modules:
emitted: <list from pipeline class>
model_index: <list from converted or source model_index.json>
status: match | mismatch | blocked
pipeline_class_resolution:
model_index_class_name: <_class_name>
entry_class_names: <registered EntryClass.__name__ values>
status: exact_match | alias_added | blocked
sampling_param:
fields_added: <none or list>
cli_fields_added: <none or list>
unknown_kwargs_checked: yes | no | blocked
stage_chain:
- <stage names in execution order>
pipeline_config:
file: <path>
classes: <class names>
official_defaults_checked: yes | no | blocked
presets:
file: <path>
names: <preset names>
status: pass | blocked | not_run
registry:
status: pass | blocked | not_run
detectors: <HF paths and model_index _class_name strings covered>
smoke_test:
path: tests/local_tests/pipelines/test_<family>_pipeline_smoke.py
status: non_skip_pass | blocked | not_run
pytest_output: <command + short result>
pipeline_parity:
path: tests/local_tests/pipelines/test_<family>_pipeline_parity.py
status: scaffold_skip | debug_red | non_skip_pass | blocked
pytest_output: <command + short result>
comparison_target: <latents|decoded video|audio|joint outputs>
example:
path: examples/inference/basic/basic_<family>.py
status: pass | blocked | not_run
output: <path or none>
readme_updated: yes | no
port_state_updated: yes | no
blockers: <none or exact blocker list>
next_step: phase_10_quality_regression | return_to_phase_6 | ask_user
escape_hatch: <none or block matching contracts/escape_hatch.md>
```
Rules:
- `pipeline-definition` may return with parity still `scaffold_skip` only if the
exact missing dependency, weight, or call-path blocker is recorded.
- Final `/add-model` handoff requires `smoke_test.status=non_skip_pass` and
`pipeline_parity.status=non_skip_pass`, unless the user explicitly accepts a
documented blocker.
- If parity failure traces to a component, conversion, or strict-load issue,
return `next_step=return_to_phase_6` and include the exact failing evidence.
- Keep `local_tests_readme` and `port_state_file` synchronized with this
handoff before returning.
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
`PORT_STATUS.md` row.
@@ -1,89 +0,0 @@
# Port State Contract
Canonical per-port state file created during prep and updated by every
`/add-model` phase.
Path:
```text
tests/local_tests/<model_family>/PORT_STATUS.md
```
Purpose:
- Single source of truth for resumable port progress.
- Tracks component status, conversion status, parity status, open questions,
blockers, escape hatches, and issue history.
- Lets review agents run the same setup/tests without reconstructing handoffs
from conversation history.
Required sections:
```text
# <Model Family> Port Status
## Summary
- model_family:
- workload_types:
- official_ref:
- official_ref_dir:
- hf_weights_path:
- local_weights_dir:
- source_layout:
- local_tests_readme:
## Current Phase
- phase:
- status: not_started | in_progress | blocked | complete
- owner: orchestrator | prep | parity | conversion | component:<name> | pipeline
- last_updated:
## Component Matrix
| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues |
|---|---|---|---|---|---|---|---|---|---|
## Conversion State
- conversion_script:
- converted_weights_dir:
- source_layout:
- strict_load_status:
- passthrough_components:
- retry_history:
## Parity Commands
| Scope | Command | Last Result | Notes |
|---|---|---|---|
## Open Questions
| ID | Question | Owner | Needed By Phase | Status | Resolution |
|---|---|---|---|---|---|
## Issues And Blockers
| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution |
|---|---|---|---|---|---|---|---|---|
## Escape Hatches
| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution |
|---|---|---|---|---|---|---|
## Decisions
| Date | Decision | Rationale | Impact |
|---|---|---|---|
## Handoff Notes
- <short notes for the next agent>
```
Rules:
- Update this file whenever a phase starts, blocks, resolves an issue, or hands
off to another skill.
- Record open questions and issues immediately. Do not leave blockers only in
chat history or subagent responses.
- Use stable IDs: `Q001`, `Q002`, `I001`, `I002`, etc.
- Use stable escape-hatch IDs: `E001`, `E002`, etc. Link them from handoff
`escape_hatch.state_snapshot.evidence` when returning `next_step=ask_user`.
- Do not include raw token values, machine-local cache internals, or large output
dumps. Use repo-relative paths when possible.
- If a question or issue is resolved, keep the row and fill `Resolution` instead
of deleting it.
@@ -1,42 +0,0 @@
# Prep Handoff Contract
Produced by `../add-model-01-prep/SKILL.md` and consumed by `/add-model` Phase 0.
```text
model_family: <snake_case>
workload_types: <T2V/I2V/V2V/T2I/or compatibility shim with rationale>
official_ref: <url or import path>
official_ref_dir: <ReferenceDir or none>
official_ref_commit: <sha or unknown>
hf_weights_path: <HF id or local path>
hf_revision: <revision or default>
local_weights_dir: official_weights/<model_family> or <local path>
source_layout: diffusers | raw_official | monolithic | separate_components | mixed | custom | unknown
model_index_class: <_class_name or none>
components_seen: <components>
needs_conversion: yes | no | unknown
hf_token_env: <env var name only>
dependency_changes: none | installed no-deps editable | installed official deps in current env | blocked on user
official_env_status: imports_ok | private_deps_need_stubs | blocked
local_tests_readme: tests/local_tests/<model_family>/README.md
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
gitignore_entries_added: <list>
next_step: add-model | ask_user
open_questions: <short list>
escape_hatch: <none or block matching contracts/escape_hatch.md>
```
Validation:
- `official_env_status` must be `imports_ok` or `private_deps_need_stubs` before
component parity scaffolding.
- `local_tests_readme` must exist and describe official setup, HF weights,
dependency changes, planned parity commands, and review notes.
- `port_state_file` must exist and follow `contracts/port_state.md`.
- Prep does not go directly to conversion; `/add-model` must run component
prototype/key-dump Phase 4 before Phase 5 conversion.
- `T2A`, `A2A`, and `AV` may be used only after `WorkloadType` supports them;
otherwise record the compatibility shim and rationale explicitly.
- Never include HF token values.
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
`PORT_STATUS.md` row.
@@ -1,96 +0,0 @@
# Shared Add-Model Rules
These rules apply to every `add-model` related skill: prep, parity, conversion,
component porting, pipeline, and the main `/add-model` orchestrator.
## Token And Auth Safety
- Never accept, print, echo, log, hard-code, or commit raw HF token values.
- Refer only to token environment variable names: `HF_TOKEN`,
`HUGGINGFACE_HUB_TOKEN`, or `HF_API_KEY`.
- Scripts may read those environment variables but must not print their values.
- Ask for auth setup only by env var name. Do not ask the user to paste a token.
- Read scope is needed for gated repos during conversion/load. Write scope is
needed for publishing converted weights or seeding generated references.
## Shared State Files
- `tests/local_tests/<model_family>/README.md` is the reviewer-facing setup and
verification log. Keep it current with setup commands, dependency blockers,
parity commands, conversion commands, and pass/blocker status.
- `tests/local_tests/<model_family>/PORT_STATUS.md` is the per-port state file.
It must follow `../contracts/port_state.md` and keep stable `Q###`, `I###`, and
`E###` IDs.
- Keep resolved questions/issues in `PORT_STATUS.md` with the resolution instead
of deleting them.
- Before returning a handoff, update both state files when the skill changed
setup, tests, conversion, parity status, blockers, or decisions.
- Do not include raw tokens, non-reproducible absolute cache paths, large
generated outputs, `.env`, credentials, or anything matching `*secret*`.
## Escape Hatches
Continue autonomously for recoverable setup, implementation, conversion,
strict-load, smoke, parity-debug, lint, or test failures. Stop and ask the user
only when the next action requires a product, cost, safety, auth, dependency, or
scope decision the workflow cannot safely choose.
Use `../contracts/escape_hatch.md` whenever returning `next_step=ask_user`.
Ask for user input only for:
- scope changes, such as dropping a modality, output head, variant, component, or
public mode;
- core dependency changes, untrusted/private dependency installs, or version pin
changes;
- auth setup for gated repos, using env var names only;
- large downloads, publishing weights, SSIM/reference uploads, or GPU-heavy work
where cost/runtime approval is needed;
- destructive file/git operations, overwriting existing clones/weights, or
deleting staged assets;
- incompatible official sources of truth where no reference can be chosen from
published weights and docs;
- accepting a blocker, loosening tolerances, using shape-only substitutes, or
shipping without required non-skip parity.
Do not ask for normal recoverable failures: missing imports, missing local paths,
skipped tests, failing parity, conversion mapping bugs, strict-load errors,
format/lint failures, smoke failures, registry import issues, example failures,
or implementation bugs covered by the phase workflow.
Before asking:
- update `PORT_STATUS.md` with an `E###` escape-hatch row plus any linked `Q###`
or `I###` row;
- include exact evidence: command, path, short error text, parity or strict-load
excerpt, or blocker ID;
- provide one recommended option and at most three alternatives;
- set the relevant handoff `next_step=ask_user` and include the `escape_hatch`
block.
Skill-specific escape-hatch sections may add extra examples, but they must not
weaken these shared rules.
## Production Boundary
- No runtime `from diffusers import <model class>` or
`from transformers import <model class>` in `fastvideo/` production code.
- Components that own weights or numerical behavior must be FastVideo-native
unless the user explicitly accepts a documented lazy-wrapper exception.
- Allowed third-party runtime exceptions are tokenizers and pure data utilities
when they match existing project patterns.
- Tests may import diffusers/transformers as parity references.
- Production comments explain why, not what or provenance. Avoid narrative
comments like `vendored from`, `matches upstream`, `REVIEW`, or session-history
commentary.
## Verification Semantics
- A committed local test may skip when clones, weights, or private deps are absent
so CI and other contributors are not blocked.
- On the porter's machine, a skip is not a pass. Fix the missing import, weights,
or path before claiming verification.
- New ports require local non-skip parity for required components and pipeline
parity when a pipeline is in scope.
- Smoke tests prove loadability only. They are not a substitute for numerical
component or pipeline parity.
@@ -1,117 +0,0 @@
# Component Skill Common Instructions
These instructions apply to `add-model-03-port-dit`, `add-model-04-port-vae`,
`add-model-05-port-encoder`, and `add-model-06-port-generic`. Bucket-specific skills
add target paths, implementation patterns, drift checks, and scope questions.
## Required Context
Require the complete packet from `../contracts/component_context.md`.
Do not start if the official definition files, official instantiation files, or
parity test path are missing. Ask the `/add-model` orchestrator for the complete
component context packet instead of rediscovering broad scope silently.
If the parity scaffold is missing, create it first with
`../../add-model-02-parity/templates/component_parity_test.py`.
## Prototype Mode
Prototype mode runs before conversion:
- implement or prove reuse for the minimal native component, config, export, and
`EntryClass` surface needed by the relevant loader;
- instantiate with random weights using the exact official architecture args, or
instantiate/document stateless components with no weights;
- dump official and FastVideo `state_dict()` names/shapes for every stateful
component so conversion can derive mappings from real surfaces;
- return concerns discovered during prototype work, such as ambiguous official
flags, shape mismatches, private ops, missing loader buckets, passthrough
weights, or output heads;
- update `local_tests_readme` with prototype status and key-dump paths;
- update `port_state_file` with prototype status, open questions, issues, and
handoff notes;
- do not chase numerical parity and do not block on converted weights.
Prototype mode succeeds when the component imports, instantiates with official
args, and required key/shape dumps exist. Converted weights and parity PASS are
not required yet.
## Parity-Debug Mode
Parity-debug mode runs after conversion:
- strict-load converted weights through the same path the pipeline will use, or
document that the component is stateless or an approved passthrough;
- use `conversion_context` and `concerns_or_unknowns` to decide whether a failure
belongs to mapping, loading, implementation, tokenization, normalization,
scheduler semantics, or the parity test;
- run only the component parity test first with `pytest <parity_test> -v -s`;
- if it skips, fix the missing official import, FastVideo class, tokenizer,
converted weights, or path;
- if it fails numerically, add targeted intermediate comparisons to identify the
first divergent operation or tensor;
- update component implementation only when the failure is a component
layer/config/forward/contract bug;
- update `local_tests_readme` with the command, result, and blocker or PASS;
- update `port_state_file` with parity status, resolved/new issues, open
questions, and handoff notes;
- keep iterating until the test is a non-skip PASS or return a precise blocker.
## Conversion Boundary
Component skills must not patch conversion scripts or converted weights ad hoc.
If the first drift or strict-load failure points to wrong keys, missing tensors,
shape mismatches, component prefixes, split/fuse logic, skipped-key policy, or
config emission, return a conversion retry request for
`../../add-model-07-conversion/SKILL.md` matching
`../contracts/conversion_request.md`.
Resume parity-debug only after conversion returns an updated handoff.
## Reuse Proof
When `fastvideo_target_files` point to existing FastVideo code instead of a new
port:
- compare the official definition against the FastVideo target: graph/operation
structure, parameter or state shapes, normalization, activation, positional or
temporal behavior, scaling constants, dtype behavior, state-dict names, output
containers, and output tensors;
- compare the official instantiation against the FastVideo config and loader
args: constructor args, config values, defaults, variant flags, optional
submodules, checkpoint metadata, tokenizer/media paths, and loader path;
- treat a matching class instantiated with different args as not reusable;
- record reuse evidence in `local_tests_readme` and keep the reused component in
`prototype_key_dumps` when it owns state so conversion and parity-debug use the
same surface;
- still run parity-debug to a non-skip PASS. If mismatch is found, return the
concern so `/add-model` can switch the component to a native port.
## Handoff
Return `../contracts/component_skill_handoff.md`.
Mode-specific expectations:
- In `mode=prototype`, `parity_status` may be `scaffold_skip` or `blocked`; key
dumps are the required artifact and successful prototype handoff should use
`next_step=phase_5_conversion`.
- In `mode=parity-debug`, final success requires
`parity_status=non_skip_pass`.
- If conversion is implicated, return `conversion_retry_request` and leave
conversion edits to `add-model-07-conversion`.
- If production loading is non-strict, list allowed missing/unexpected keys in
the parity test or handoff and mark
`strict_load=pass_with_documented_exclusions`.
## Escape Hatches
Follow `common_rules.md`. Do not ask for normal prototype or parity-debug
failures such as missing imports, tokenizer/path issues, red parity,
strict-load failures, key mismatches, shape mismatches, or implementation bugs.
Return conversion retry requests or precise blockers as directed by the workflow.
Ask only when component work requires a scope or safety decision, such as
dropping a required stream/output/path, changing core dependencies, accepting
private model code or unsupported private ops, choosing between incompatible
official definitions, creating a new loader bucket, or loosening required parity.
@@ -1,338 +0,0 @@
---
name: decompose-pipeline-pr
description: Decompose an oversized FastVideo pipeline PR into a stack of independently-reviewable PRs. Tiers the diff by blast radius (invisible / dead code / cross-cutting infra / activation), produces a branch graph and worktree bootstrap, drafts the AGENTS.md manifest, flags missing tests on cross-cutting infra changes, and extracts lessons from the PR body.
---
# Decompose Pipeline PR
## Purpose
When a PR adds a new pipeline (or first-class component port) and crosses
~3,000 LOC, single-shot review converges to rubber-stamping. This skill
decomposes such a PR into a stack of independently-reviewable PRs without
disturbing `main`.
It is the inverse of `add-model`: where `add-model` walks adding a new
pipeline as a fresh PR, this skill walks decomposing an existing oversized
pipeline PR.
**Worked example:** PR #1280 (daVinci-MagiHuman, 9,812 LOC, 56 files) →
2 prerequisite PRs off main + 8-PR stack:
- #1293 `will/activation-trace` (prerequisite)
- #1294 `will/loader-infra` (prerequisite)
- #1295 (1/8) housekeeping
- #1296 (2/8) t5gemma encoder
- #1297 (3/8) DiT
- #1298 (4/8) pipeline stages
- #1299 (5/8) pipeline orchestrator
- #1300 (6/8) provenance (AGENTS.md, JOURNAL.md, lessons)
- #1301 (7/8) conversion scripts
- #1302 (8/8) registry activation
## Prerequisites
- Open PR number on `hao-ai-lab/FastVideo` (or any FastVideo fork)
- `gh` CLI authenticated against the target remote
- Local git worktree support (`git worktree`)
- Git config `user.name` / `user.email` set
- Pre-commit installed (`pre-commit install --hook-type pre-commit --hook-type commit-msg`)
- The target PR's branch fetched locally as `origin/<feature-branch>`
## Inputs
| Parameter | Required | Description |
|-----------|----------|-------------|
| PR number or URL | Yes | E.g. `1280` or `https://github.com/hao-ai-lab/FastVideo/pull/1280` |
| Max desired PR size | No | Defaults to ~2,500 LOC of code per stack PR (excluding generated/journal files) |
| Output directory | No | Defaults to `.agents/tmp/decompose-<pr-number>/` (gitignored) |
## Steps
### 1. Verify ground truth (do not trust `gh pr diff --name-only`)
`gh pr diff <N> --name-only` has been observed to emit phantom file entries.
Always cross-check against the authoritative `git diff`:
```bash
mkdir -p .agents/tmp/decompose-<N>
git fetch origin pull/<N>/head:<feature-branch>
git diff origin/main..origin/<feature-branch> --name-status \
> .agents/tmp/decompose-<N>/files.txt
git diff origin/main..origin/<feature-branch> --stat
```
Use the `--name-status` output as the authoritative file list. If it
disagrees with `gh pr diff --name-only`, trust the git diff.
### 2. Tier the diff by blast radius
Classify every changed file into one of four tiers:
| Tier | Description | Examples |
|---|---|---|
| **Tier 0 — Invisible** | Lint/style/CI configs that don't affect runtime | `.gitignore`, `pyproject.toml` (codespell only), agent documentation |
| **Tier 1 — Dead code** | New files in their own dirs; aggregator one-liners | `fastvideo/models/dits/<new>/`, `fastvideo/pipelines/basic/<new>/`, `examples/inference/basic/basic_<new>*.py`, `tests/local_tests/<new>/`, `__init__.py` exports |
| **Tier 2 — Cross-cutting infra** | Modifications to files used by every pipeline | See protected-paths list below |
| **Tier 3 — Activation switch** | `register_configs(...)` calls + the example scripts that demo them | `fastvideo/registry.py` |
**FastVideo Tier 2 protected paths:**
```
fastvideo/utils.py
fastvideo/pipelines/composed_pipeline_base.py
fastvideo/models/loader/component_loader.py
fastvideo/configs/models/dits/__init__.py
fastvideo/configs/models/encoders/__init__.py
fastvideo/configs/models/vaes/__init__.py
fastvideo/envs.py
fastvideo/fastvideo_args.py
fastvideo/distributed/**
fastvideo/layers/**
fastvideo/attention/**
fastvideo/registry.py # treat as Tier 3 if change is the activation
```
Tier 3 detection (mechanical):
```bash
git diff origin/main..origin/<feature-branch> -- fastvideo/registry.py | \
grep -E "^\+.*register_configs\("
```
If `registry.py` only contains `register_configs` additions, treat it as
Tier 3. If it modifies existing behavior, treat it as Tier 2 (rare).
### 3. Identify reusable Tier-1 components
Within Tier 1, look for sub-trees that are **not** model-specific and could
land separately:
- Encoders matching a known multi-model base (T5/T5-Gemma/Llama/Gemma/CLIP variants)
- New stage classes that subclass shared bases without referencing the new model
- Hook/profiler/debug infra under `fastvideo/hooks/`
- New helpers that have no model-specific dependencies
These get split into their own PRs (e.g. PR 4 `t5gemma-encoder` in the
MagiHuman example).
### 4. Hunt for missing test coverage on Tier 2 changes
For every Tier-2 file modified, check whether the original PR added unit
tests for the new behavior:
```bash
for f in <list-of-tier-2-files>; do
echo "=== Tests for $f ==="
git diff origin/main..origin/<feature-branch> -- \
"$(echo $f | sed 's|fastvideo/|fastvideo/tests/|; s|\.py|*|')"
done
```
If a Tier-2 PR has no accompanying tests, **emit a "must-add tests" list**
with a sketch of the case grid. Tier-2 PRs do not ship without those tests.
The MagiHuman example required this for PR-B (`utils.py`): the original PR
shipped no `test_utils_loader.py`, so the decomposition added 9 unit-test
cases covering the umbrella-detector boundary, the optional-component-dirs
relaxation, and regression coverage on every existing 2-segment HF id.
### 5. Build the dependency DAG and topo-sort
Edges:
- Tier 2 infra → Tier 1 code that imports it
- Reusable Tier 1 components → model-specific Tier 1 code that uses them
(encoder before DiT before pipeline)
- Tier 1 → Tier 3 (activation always last)
- Tier 0 has no dependents (lands first as a freebie)
Topo-sort produces the stack ordering. Pull Tier-2 PRs **out of the stack**
when they have no model-specific dependency — they should land off main
with their own focused review, not buried in a model port.
Render as a tree (markdown):
```
main
├─ <prereq-A>
│ └─ <prereq-B>
│ ├─ <stack-01-housekeeping>
│ │ └─ <stack-02-encoder>
│ │ └─ <stack-03-dit>
│ │ └─ ...
│ │ └─ <stack-N-activate>
│ └─ (parallel) <skill-pr> off main
```
### 6. Detect mis-shelved docs and debug scratch
Two categories to flag:
- **Mis-shelved docs**: Markdown files under `tests/local_tests/` are
journals, not tests. Flag for relocation to the package dir as
`JOURNAL.md`.
- **Debug scratch**: files starting with `_debug_`, `_scratch_`, or
`_explore_`. Flag for drop (do not carry into any output PR).
For MagiHuman: `tests/local_tests/magi-human.md` → relocate. Two
`_debug_magi_human_*.py` files → drop.
### 7. Author the AGENTS.md manifest skeleton
For the new pipeline package, generate a 6-section `AGENTS.md` scaffold
with the file table pre-populated from the diff:
1. **Manifest** — file table by role
2. **Parity invariants** — load-bearing rules with one-paragraph each + lesson refs
3. **Cross-refs** — "If you change X, re-run Y" matrix
4. **Run book** — single pytest command + prereqs (HF tokens, GPU, wall-time)
5. **Open questions** — known issues (e.g. tolerance carve-outs)
6. **Provenance** — PR table with branch names and source SHA
The provenance section is filled incrementally during stack execution and
finalized in the activation PR.
### 8. Extract lessons from the PR body
Scan the PR body for sections titled "Key implementation work", "Bug hunt",
"Lessons", or sentences with patterns like "took N waves to localize",
"silent regression", "investigation revealed". Each becomes a candidate
`.agents/lessons/<YYYY-MM-DD>_<slug>.md` draft.
Lessons MUST follow the existing template in
`.agents/lessons/README.md`:
- YAML frontmatter: `date`, `experiment`, `category`, `severity`
- Sections: What Happened, Root Cause, Fix / Workaround, Prevention
- Filename: `<YYYY-MM-DD>_<short-slug>.md`
Lessons co-locate with the code they concern: a conversion-script lesson
lands in the same PR as the conversion script, not in the docs PR.
### 9. Emit the commit-footer convention
Every commit in the stack ends with:
```
<Feature>-Stack: N/M
```
E.g. `Magi-Stack: 5/8`. Use the package directory name as the feature key.
After all PRs squash-merge, `git log --grep='^<Feature>-Stack:'` reconstructs
the lineage even if PR numbers later get renumbered.
### 10. Produce the worktree bootstrap
Generate a runnable bash script:
```bash
#!/bin/bash
set -euo pipefail
REPO=/home/<user>/FastVideo
WORKTREE=/home/<user>/FastVideoMagi # NB: directory name must be a valid
# Python identifier (no hyphens) so
# mypy doesn't choke
SOURCE_PR=<N>
SOURCE_BRANCH=will/<feature>
SOURCE_SHA=$(git -C "$REPO" rev-parse "origin/$SOURCE_BRANCH")
git -C "$REPO" fetch origin main:main
git -C "$REPO" fetch "origin/$SOURCE_BRANCH"
git -C "$REPO" worktree add "$WORKTREE" origin/main
# Capture baseline for provenance. Everything under .agents/tmp is transient
# and ignored by git.
OUTPUT_DIR="$REPO/.agents/tmp/decompose-$SOURCE_PR"
mkdir -p "$OUTPUT_DIR"
cat > "$OUTPUT_DIR/<feature>-baseline-${SOURCE_SHA:0:8}.txt" <<EOF
Source PR: <repo>#$SOURCE_PR
Source SHA: $SOURCE_SHA
Authoritative file count: $(git -C "$REPO" diff origin/main..origin/$SOURCE_BRANCH --name-only | wc -l)
Date captured: $(date -u +%Y-%m-%dT%H:%M:%SZ)
EOF
```
### 11. Author preserve via `git checkout`, not `cherry-pick`
For each stack PR:
```bash
git -C "$WORKTREE" switch -c <new-branch> <base-branch>
git -C "$WORKTREE" checkout origin/<source-branch> -- <file1> <file2> ...
git -C "$WORKTREE" commit -m "[<scope>]: <subject>
<body>
<Feature>-Stack: N/M"
git -C "$WORKTREE" push -u origin <new-branch>
gh pr create --base <base-branch> --head <new-branch> --title "..." --body "$(cat <<EOF ... EOF)"
```
Notes:
- `git checkout origin/<source> -- <files>` extracts only the named files,
preserving the diff. The original PR's author is **not** preserved on the
new commit (it's authored by whoever runs the script). Reference the
original PR + source SHA in every commit body and PR description for
authorship attribution.
- **Never use `git cherry-pick`** for this workflow — cherry-pick applies
whole commits, which mixes concerns across PR boundaries.
## Outputs
The skill produces all transient planning artifacts under
`.agents/tmp/decompose-<pr>/`:
1. A markdown decomposition plan (`plan.md`)
2. A proposed branch graph
3. A worktree-bootstrap script (`bootstrap.sh`)
4. Per-PR file allocation lists (under `stack/`)
5. AGENTS.md scaffolds for any new pipeline packages
6. Draft lesson files (placed alongside the PR that owns the code they concern)
7. A finalized provenance table for the package AGENTS.md
## Anti-Patterns
The skill should warn against:
- **"Just rebase the megaPR into smaller commits."** Doesn't help review;
reviewer still sees one PR.
- **Co-locating tests under the new package.** FastVideo's convention is
by-kind under `fastvideo/tests/` and `tests/local_tests/<family>/`. Don't
invent a new layout per pipeline.
- **Splitting Tier 2 changes into "one file per PR."** Tier 2 PRs are
about semantic units (e.g., "loader umbrella + optional component dirs"
together because they jointly define the new diffusers-format contract),
not file-count.
- **Landing the activation switch first** ("just register, the code can
be empty"). The skill enforces activation-last so every intermediate
state is dead code, not broken code.
- **Trusting `gh pr diff --name-only`.** Cross-check against
`git diff origin/main..origin/<feature-branch> --name-status` —
`gh`'s output has been observed to include phantom entries.
- **Worktree dir names with hyphens.** mypy interprets them as invalid
Python package names and refuses to run. Use CamelCase or underscores.
- **Skipping the lesson-extraction step.** PR bodies contain the most
expensive learnings of the original implementation. Losing them to a
squash-merge is the silent decay of institutional knowledge.
## Example Usage
```
User: split PR 1280
Agent: [invokes decompose-pipeline-pr]
→ produces .agents/tmp/decompose-1280/plan.md with:
- tiered file table (56 files: 3 tier-0, 35 tier-1, 9 tier-2,
9 tier-3)
- branch graph (PR-A + PR-B + 8-PR stack)
- worktree bootstrap script
- per-PR file lists
- AGENTS.md scaffold for fastvideo/pipelines/basic/magi_human/
- 3 draft lessons extracted from the PR body
→ asks user to confirm before opening branches
```
## References
- The MagiHuman decomposition (worked example):
`fastvideo/pipelines/basic/magi_human/AGENTS.md` (after PR #1302 merges)
- Existing skill: `.agents/skills/add-model/SKILL.md` (the inverse — adding
a new pipeline as a fresh PR)
- Lesson template: `.agents/lessons/README.md`
- Skill template: `.agents/skills/SKILL_TEMPLATE.md`
+2 -2
View File
@@ -105,8 +105,8 @@ Detect artefact type by inspecting the file's imports / helper call:
- **pixel** (`.mp4`) — file imports
`run_text_to_video_similarity_test` / `run_image_to_video_similarity_test`
from `fastvideo.tests.ssim.inference_similarity_utils`, OR uses the
legacy custom-inline helper pattern (see `test_gamecraft`,
`test_longcat`, etc.). Default to pixel when both heuristics fail.
legacy custom-inline helper pattern (see `test_gen3c`). Default to pixel
when both heuristics fail.
Record `ARTEFACT_TYPE ∈ {pixel, latent}` for use in step 4. Steps 2, 3, 5,
and 6 are artefact-type-agnostic — `_iter_reference_files`,
-74
View File
@@ -205,40 +205,6 @@ pull_request_rules:
label:
add: ["scope: model"]
# ============================================================
# Pre-commit failure help comment
# ============================================================
- name: comment on pre-commit failure
conditions:
- check-failure~=pre-commit
- -closed
actions:
comment:
message: |
## Pre-commit checks failed
Hi @{{author}}, the pre-commit checks have failed. To fix them locally:
```bash
# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install
# Run all checks and auto-fix what's possible
pre-commit run --all-files
```
Common fixes:
- **yapf**: `yapf -i <file>` (formatting)
- **ruff**: `ruff check --fix <file>` (linting)
- **codespell**: `codespell --write-changes <file>` (spelling)
After fixing, commit and push the changes. The checks will re-run automatically.
For future commits, `pre-commit` will run automatically on changed files before each commit.
# ============================================================
# Merge conflict detection
# ============================================================
@@ -251,16 +217,6 @@ pull_request_rules:
actions:
label:
add: [needs-rebase]
comment:
message: |
This PR has merge conflicts with the base branch. Please rebase:
```bash
git fetch origin main
git rebase origin/main
# Resolve any conflicts, then:
git push --force-with-lease
```
- name: remove conflict label when resolved
conditions:
@@ -290,35 +246,5 @@ pull_request_rules:
merge:
method: squash
# ============================================================
# PR title format help
# ============================================================
- name: comment on invalid PR title format
conditions:
- -closed
- -draft
- "-title~=(?i)^\\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\\]"
actions:
comment:
message: |
## ⚠️ PR title format required
Your PR title must start with a type tag in brackets. Examples:
- `[feat] Add new model support`
- `[bugfix] Fix VAE tiling corruption`
- `[refactor] Restructure training pipeline`
- `[perf] Optimize attention kernel`
- `[ci] Update test infrastructure`
- `[infra] Add activation trace hooks`
- `[docs] Add inference guide`
- `[misc] Clean up configs`
- `[new-model] Port Flux2 to FastVideo`
- `[skill] Add add-model agent skill`
Valid tags: `feat`, `feature`, `bugfix`, `fix`, `refactor`, `perf`, `ci`, `infra`, `doc`, `docs`, `misc`, `chore`, `kernel`, `new-model`, `skill`, `skills`
Please update your PR title and the merge protection check will pass automatically.
merge_protections_settings:
reporting_method: check-runs
+28 -42
View File
@@ -12,6 +12,7 @@ from __future__ import annotations
import argparse
import fnmatch
import re
from collections.abc import Iterable
from dataclasses import dataclass, field
from pathlib import Path
from typing import TextIO
@@ -130,11 +131,6 @@ class FamilyCoverage:
FAMILY_COVERAGE = (
FamilyCoverage(
re.compile(r"(^|[/_.-])dreamx(_world)?([/_.-]|$)"),
("test_dreamx.py", ),
("test_dreamx_world_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])flux[_-]?2([/_.-]|$)"),
("test_flux2_klein.py", ),
@@ -145,21 +141,11 @@ FAMILY_COVERAGE = (
("test_flux.py", ),
("test_flux_t2i_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])(hunyuan)?gamecraft([/_.-]|$)"),
("test_gamecraft.py", ),
("test_gamecraft_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])gen3c([/_.-]|$)"),
("test_gen3c.py", ),
("test_gen3c_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])glm[_-]?image([/_.-]|$)"),
("test_glm_image.py", ),
("test_glm_image_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])hunyuan(video)?15([a-z0-9_-]*)([/_.-]|$)"),
(),
@@ -175,11 +161,6 @@ FAMILY_COVERAGE = (
("test_lingbot.py", ),
("test_lingbot_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])longcat([/_.-]|$)"),
("test_longcat.py", ),
("test_longcat_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])ltx[_-]?2([/_.-]|$)"),
("test_ltx2.py", ),
@@ -210,11 +191,6 @@ FAMILY_COVERAGE = (
("test_stable_audio.py", ),
("test_stable_audio_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])turbo(diffusion)?([/_.-]|$)"),
(),
("test_turbodiffusion_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])wan(video|vae)?([/_.-]|$)"),
("test_wan_t2v.py", "test_wan_vae.py", "test_wan_causal.py", "test_wan_denoising.py"),
@@ -225,11 +201,6 @@ FAMILY_COVERAGE = (
"test_wan_ti2v_similarity.py",
),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])z[_-]?image([/_.-]|$)"),
("test_zimage.py", ),
("test_zimage_similarity.py", ),
),
)
@@ -325,16 +296,22 @@ def _select_output_coverage(plan: MergePlan, path: str) -> None:
plan.add_ssim(SSIM_SMOKE_TESTS, reason=f"shared output SSIM smoke coverage: {path}")
def classify_paths(paths: list[str]) -> MergePlan:
def _normalize_path(raw_path: str) -> str:
path = raw_path.strip()
while path.startswith("./"):
path = path[2:]
return path
def classify_paths(paths: list[str], removed_paths: Iterable[str] = ()) -> MergePlan:
"""Plan merge lanes for ``paths``.
``removed_paths`` lists changed paths that no longer exist at the PR head
(deleted files and rename sources); removed golden/SSIM tests are not run.
"""
plan = MergePlan()
normalized_paths: list[str] = []
for raw_path in paths:
path = raw_path.strip()
while path.startswith("./"):
path = path[2:]
if path:
normalized_paths.append(path)
normalized_paths = sorted(set(normalized_paths))
normalized_paths = sorted({path for path in map(_normalize_path, paths) if path})
removed = {path for path in map(_normalize_path, removed_paths) if path}
if not normalized_paths:
plan.require_all("changed-file list was empty; failing closed")
return plan
@@ -374,7 +351,10 @@ def classify_paths(paths: list[str]) -> MergePlan:
if path.startswith("fastvideo/tests/golden_gate/"):
name = Path(path).name
if name.startswith("test_") and name.endswith(".py"):
plan.add_golden((name, ), reason=f"changed golden test: {path}")
if path in removed:
plan.reasons.append(f"removed golden test has nothing to run: {path}")
else:
plan.add_golden((name, ), reason=f"changed golden test: {path}")
elif name in {"AGENTS.md", "README.md"}:
plan.reasons.append(f"golden documentation only: {path}")
else:
@@ -385,7 +365,10 @@ def classify_paths(paths: list[str]) -> MergePlan:
if path.startswith("fastvideo/tests/ssim/"):
name = Path(path).name
if name.startswith("test_") and name.endswith(".py"):
plan.add_ssim((name, ), reason=f"changed SSIM test: {path}")
if path in removed:
plan.reasons.append(f"removed SSIM test has nothing to run: {path}")
else:
plan.add_ssim((name, ), reason=f"changed SSIM test: {path}")
elif path.endswith((".py", ".json", ".pt", ".png", ".mp4")):
plan.ssim_all = True
plan.add_lanes("ssim", reason=f"shared SSIM harness/reference: {path}")
@@ -565,6 +548,7 @@ def _write_summary(output: TextIO, plan: MergePlan) -> None:
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--paths-file", type=Path, required=True)
parser.add_argument("--removed-paths-file", type=Path)
parser.add_argument("--github-output", type=Path)
parser.add_argument("--summary-file", type=Path)
return parser.parse_args()
@@ -573,7 +557,9 @@ def parse_args() -> argparse.Namespace:
def main() -> int:
args = parse_args()
paths = args.paths_file.read_text(encoding="utf-8").splitlines()
plan = classify_paths(paths)
removed_paths = (args.removed_paths_file.read_text(encoding="utf-8").splitlines()
if args.removed_paths_file else [])
plan = classify_paths(paths, removed_paths)
print(f"MERGE_TEST_PLAN={plan.encoded_lanes()}")
print(f"MERGE_GOLDEN_TESTS={plan.encoded_golden_tests()}")
print(f"MERGE_SSIM_TESTS={plan.encoded_ssim_tests()}")
@@ -180,6 +180,8 @@ jobs:
set -euo pipefail
changed_json="$RUNNER_TEMP/merge-changed-files.json"
changed_paths="$RUNNER_TEMP/merge-changed-paths.txt"
removed_paths="$RUNNER_TEMP/merge-removed-paths.txt"
: > "$removed_paths"
if gh api --paginate --slurp \
"repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/files?per_page=100" \
> "$changed_json"; then
@@ -187,6 +189,10 @@ jobs:
if [ "$observed" = "$EXPECTED_CHANGED_FILES" ]; then
jq -r '.[][] | .filename, (.previous_filename // empty)' "$changed_json" \
| sort -u > "$changed_paths"
# Paths absent from the PR head, so the planner never selects a deleted test.
jq -r '.[][] | if .status == "removed" then .filename
elif .status == "renamed" then (.previous_filename // empty) else empty end' \
"$changed_json" | sort -u > "$removed_paths"
else
echo "::warning::Changed-file API returned $observed of $EXPECTED_CHANGED_FILES paths; selecting all merge lanes."
echo '__FASTVIDEO_CI_PLAN_ALL__' > "$changed_paths"
@@ -202,6 +208,7 @@ jobs:
run: |
python3 .github/scripts/plan_merge_ci.py \
--paths-file "$RUNNER_TEMP/merge-changed-paths.txt" \
--removed-paths-file "$RUNNER_TEMP/merge-removed-paths.txt" \
--github-output "$GITHUB_OUTPUT" \
--summary-file "$GITHUB_STEP_SUMMARY"
-51
View File
@@ -1,51 +0,0 @@
name: Close Stale Issues and PRs
on:
schedule:
# Daily at 1:30 AM UTC
- cron: '30 1 * * *'
jobs:
stale:
if: github.repository == 'hao-ai-lab/FastVideo'
permissions:
issues: write
pull-requests: write
actions: write
runs-on: ubuntu-latest
steps:
- uses: actions/stale@997185467fa4f803885201cee163a9f38240193d # v10.1.1
with:
operations-per-run: 500
exempt-draft-pr: true
exempt-issue-labels: 'keep-open,pinned,security,Bug,RFC'
exempt-pr-labels: 'keep-open,pinned'
labels-to-add-when-unstale: 'unstale'
labels-to-remove-when-stale: 'unstale'
days-before-issue-stale: 90
days-before-issue-close: 30
stale-issue-label: 'stale'
stale-issue-message: >
This issue has been automatically marked as stale because it has not
had any activity within 90 days. It will be automatically closed if
no further activity occurs within 30 days. Leave a comment if you
feel this issue should remain open. Thank you!
close-issue-message: >
This issue has been automatically closed due to inactivity. Please
feel free to reopen if you feel it is still relevant. Thank you!
days-before-pr-stale: 60
days-before-pr-close: 14
stale-pr-label: 'stale'
stale-pr-message: >
This pull request has been automatically marked as stale because it
has not had any activity within 60 days. It will be automatically
closed if no further activity occurs within 14 days. Leave a comment
if you feel this pull request should remain open. Thank you!
close-pr-message: >
This pull request has been automatically closed due to inactivity.
Please feel free to reopen if you intend to continue working on it.
Thank you!
-56
View File
@@ -1,56 +0,0 @@
name: Welcome First-Time Contributors
on:
issues:
types: [opened]
pull_request_target:
types: [opened]
permissions:
issues: write
pull-requests: write
jobs:
welcome:
if: github.repository == 'hao-ai-lab/FastVideo'
runs-on: ubuntu-latest
steps:
- uses: actions/first-interaction@34f15e814fe48ac9312ccf29db4e74fa767cbab7 # v1.3.0
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
issue-message: |
Welcome to FastVideo! Thanks for opening your first issue.
To help us investigate, please include:
- **FastVideo version**: `pip show fastvideo`
- **GPU**: `nvidia-smi` output (GPU model, driver, CUDA version)
- **Python version**: `python --version`
- **OS**: e.g., Ubuntu 22.04
If this is a bug, a minimal reproduction script helps us fix it faster.
Useful links:
- [Documentation](https://hao-ai-lab.github.io/FastVideo)
- [Contributing Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
- [Slack](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ)
pr-message: |
Welcome to FastVideo! Thanks for your first pull request.
**How our CI works:**
PRs run a three-tier CI system:
1. **Pre-commit** — formatting (yapf), linting (ruff), type checking (mypy). Runs immediately on every PR.
2. **Fastcheck** — six core GPU lanes run automatically via Buildkite (~10-15 min).
3. **Merge gate** — a reviewer adds `ready`; changed paths select only the relevant integration, training, golden, or SSIM coverage.
**Before your PR is reviewed:**
- [ ] `pre-commit run --all-files` passes locally
- [ ] You've added or updated tests for your changes
- [ ] The PR description explains what and why
If pre-commit fails, a bot comment will explain how to fix it. Fastcheck and merge-gate results appear in the Checks section below.
**Useful links:**
- [Contributing Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
- [Development Roadmap](https://github.com/hao-ai-lab/FastVideo/issues/899)
- [Slack](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ)
-1
View File
@@ -36,7 +36,6 @@ env
*.log
weights/
logs/
/Z-Image/
official_weights/
converted_weights/
@@ -160,8 +160,8 @@ class FastH3Backend:
Each clip length is a torch.compile shape, and the fullgraph regional
route treats exceeding dynamo's recompile limit as a crash rather than
a fallback. FastVideo's own imports lower it (`layers/lora/linear.py`
to 16, longcat's `bsa_interface.py` to 32), so this raises it again
and turns overflow back into a recompile.
to 16), so this raises it again and turns overflow back into a
recompile.
One pinned clip length keeps every process far under even the lowered
limit; a varied `warmup_lengths` is where this becomes the seatbelt it
+1 -127
View File
@@ -1,5 +1,5 @@
{
"version": 15,
"version": 16,
"recipes": [
{
"id": "fastwan21-t2v",
@@ -161,53 +161,6 @@
"Native MLX FastMetal T2V on 36 GB+ unified memory. Same --fast, --fast-spatial, and --refine flags as the 1.3B script."
]
},
{
"id": "turbodiffusion-wan21-1-3b-t2v",
"family": "turbodiffusion",
"stage": "inference",
"task": "Text to video",
"label": "TurboWan2.1 1.3B",
"summary": "A TurboDiffusion-accelerated Wan2.1 1.3B text-to-video run from its maintained single-GPU example.",
"model": "loayrashid/TurboWan2.1-T2V-1.3B-Diffusers",
"source": "examples/inference/basic/basic_turbodiffusion.py",
"command": "python examples/inference/basic/basic_turbodiffusion.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under video_samples_turbodiffusion/",
"related": ["turbodiffusion-wan21-14b-t2v", "turbowan22-i2v"]
},
{
"id": "turbodiffusion-wan21-14b-t2v",
"family": "turbodiffusion",
"stage": "inference",
"task": "Text to video",
"label": "TurboWan2.1 14B",
"summary": "TurboDiffusion acceleration applied to the 14B Wan2.1 text-to-video checkpoint; the checked-in source is configured for two GPUs.",
"model": "loayrashid/TurboWan2.1-T2V-14B-Diffusers",
"source": "examples/inference/basic/basic_turbodiffusion_14b.py",
"command": "python examples/inference/basic/basic_turbodiffusion_14b.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 2, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under video_samples_turbodiffusion_14B/",
"related": ["turbodiffusion-wan21-1-3b-t2v", "turbowan22-i2v"]
},
{
"id": "turbowan22-i2v",
"family": "turbodiffusion",
"stage": "inference",
"task": "Image to video",
"label": "TurboWan2.2 A14B",
"summary": "A one-to-four-step image-to-video path using TurboDiffusion and the SLA attention backend from its maintained example.",
"model": "loayrashid/TurboWan2.2-I2V-A14B-Diffusers",
"source": "examples/inference/basic/basic_turbodiffusion_i2v.py",
"command": "python examples/inference/basic/basic_turbodiffusion_i2v.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 2, "evidence": "source-configured"},
"evidence": "Source-backed",
"related": ["turbodiffusion-wan21-14b-t2v"]
},
{
"id": "ltx2-distilled-t2v",
"family": "ltx2",
@@ -476,53 +429,6 @@
"expected_artifact": "PNG images under outputs/flux_dev/samples/",
"limitations": ["FLUX.1 is loadable by ID but registers no model_family in fastvideo/registry.py; it is grouped under FLUX for documentation only."]
},
{
"id": "glm-image-t2i",
"family": "glm_image",
"stage": "inference",
"task": "Text to image",
"label": "GLM-Image",
"summary": "GLM-Image text-to-image generation from its maintained example.",
"model": "zai-org/GLM-Image",
"source": "examples/inference/basic/basic_glm_image.py",
"command": "python examples/inference/basic/basic_glm_image.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG image at image_output/landscape.png",
"related": ["glm-image-edit"]
},
{
"id": "glm-image-edit",
"family": "glm_image",
"stage": "inference",
"task": "Image editing",
"label": "GLM-Image editing",
"summary": "Edit an input image with an instruction prompt using GLM-Image, from its maintained editing example.",
"model": "zai-org/GLM-Image",
"source": "examples/inference/basic/edit_glm_image.py",
"command": "python examples/inference/basic/edit_glm_image.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG image at image_output/edited.png (input: assets/images/couple.jpg)",
"related": ["glm-image-t2i"]
},
{
"id": "zimage-turbo-t2i",
"family": "zimage",
"stage": "inference",
"task": "Text to image",
"label": "Z-Image Turbo",
"summary": "Z-Image Turbo text-to-image on a single GPU from its maintained example.",
"model": "Tongyi-MAI/Z-Image-Turbo",
"source": "examples/inference/basic/basic_zimage.py",
"command": "python examples/inference/basic/basic_zimage.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG image at outputs/zimage/zimage_turbo.png"
},
{
"id": "sd35-medium-t2i",
"family": "sd35",
@@ -914,38 +820,6 @@
],
"limitations": ["Supply a compatible FastH3 adapter. The script infers dense or VSA attention from the adapter payload unless you override it."]
},
{
"id": "longcat-t2v",
"family": "longcat",
"stage": "inference",
"task": "Text to video",
"label": "LongCat Video T2V",
"summary": "LongCat Video text-to-video at 480p (50 steps), with distilled and 720p refinement passes included in the same maintained script.",
"model": "FastVideo/LongCat-Video-T2V-Diffusers",
"source": "examples/inference/basic/basic_longcat_t2v.py",
"command": "python examples/inference/basic/basic_longcat_t2v.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under outputs_video/longcat_t2v_basic/, longcat_t2v_distill/, and longcat_t2v_refine_720p/",
"related": ["longcat-i2v"]
},
{
"id": "longcat-i2v",
"family": "longcat",
"stage": "inference",
"task": "Image to video",
"label": "LongCat Video I2V",
"summary": "LongCat Video image-to-video with optional distilled and refinement passes, from its maintained example.",
"model": "FastVideo/LongCat-Video-I2V-Diffusers",
"source": "examples/inference/basic/basic_longcat_i2v.py",
"command": "python examples/inference/basic/basic_longcat_i2v.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under outputs_video/longcat_i2v_basic/ and longcat_i2v_distill/",
"related": ["longcat-t2v"]
},
{
"id": "stable-audio-open-t2a",
"family": "stable_audio",
+4 -7
View File
@@ -9,18 +9,15 @@ plain typographic tile instead — that tile is a UI placeholder, not a logo.
| --- | --- | --- |
| `wan-ai.webp` | [Official Wan-AI Hugging Face organization avatar](https://huggingface.co/Wan-AI) | Wan family card and page header |
| `ltx.webp` | [Official Lightricks Hugging Face organization avatar](https://huggingface.co/Lightricks) | LTX family card and page header |
| `tencent-hunyuan.webp` | [Official Tencent Hunyuan Hugging Face organization avatar](https://huggingface.co/Tencent-Hunyuan) | Hunyuan and GameCraft cards, Hunyuan page header |
| `tencent-hunyuan.webp` | [Official Tencent Hunyuan Hugging Face organization avatar](https://huggingface.co/Tencent-Hunyuan) | Hunyuan family card and page header |
| `nvidia.webp` | [Official NVIDIA Hugging Face organization avatar](https://huggingface.co/nvidia) | Cosmos and GEN3C cards, Cosmos page header |
| `kandinsky.webp` | [Official Kandinsky Lab Hugging Face organization avatar](https://huggingface.co/kandinskylab) | Kandinsky 5 and Kandinsky 6 cards and page headers |
| `black-forest-labs.webp` | [Official Black Forest Labs Hugging Face organization avatar](https://huggingface.co/black-forest-labs) | FLUX family card and page header |
| `minimax.webp` | [Official MiniMax Hugging Face organization avatar](https://huggingface.co/MiniMaxAI) | MiniMax H3 family card and page header |
| `tongyi.webp` | [Official Tongyi MAI Hugging Face organization avatar](https://huggingface.co/Tongyi-MAI) | Z-Image family card and page header |
| `zai.webp` | [Official Z.ai Hugging Face organization avatar](https://huggingface.co/zai-org) | GLM-Image family card and page header |
| `stabilityai.webp` | [Official Stability AI Hugging Face organization avatar](https://huggingface.co/stabilityai) | Stable Diffusion and Stable Audio cards and page headers |
| `meituan-longcat.webp` | [Official Meituan LongCat Hugging Face organization avatar](https://huggingface.co/meituan-longcat) | LongCat family card and page header |
| `fastvideo.webp` | [Official FastVideo Hugging Face organization avatar](https://huggingface.co/FastVideo) | Matrix Game and MMAudio cards (converted weights published by this org), DreamX card |
| `fastvideo.webp` | [Official FastVideo Hugging Face organization avatar](https://huggingface.co/FastVideo) | Matrix Game and MMAudio cards (converted weights published by this org) |
Typographic tiles (no vendored image): TurboDiffusion ("Turbo"), HY-World
("HY"), LingBot ("LB"), MMAudio page header ("MMA"). These publishers have no
Typographic tiles (no vendored image): HY-World ("HY"), LingBot ("LB"),
MMAudio page header ("MMA"). These publishers have no
single official mark appropriate for reuse in the catalog; add a licensed
asset here if one becomes available.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.1 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.4 KiB

+14
View File
@@ -0,0 +1,14 @@
# Contact and Maintainers
## Contact
- **Contact email:** wlsaidhi@gmail.com
- **Bug reports and feature requests:** open a
[GitHub issue](https://github.com/hao-ai-lab/FastVideo/issues).
- **Documentation:** see the [Developer Guide](contributing/overview.md).
## Maintainers
FastVideo is an open-source project of the
[hao-ai-lab](https://github.com/hao-ai-lab) GitHub organization. The source code
lives in [hao-ai-lab/FastVideo](https://github.com/hao-ai-lab/FastVideo).
-4
View File
@@ -247,9 +247,6 @@ upstream model, walks its `named_modules()`, and attaches hooks externally.
This is the same pattern used in
`tests/local_tests/transformers/_debug_magi_human_block_parity.py`.
The `add-model-08-trace` skill at `.agents/skills/add-model-08-trace/`
provides a script template for this purpose.
## Future extensions (design only, not yet implemented)
### Extension 1: FX/Dynamo backend graph rewrite
@@ -319,7 +316,6 @@ granularity isn't enough.
- Env vars: `fastvideo/envs.py` (`FASTVIDEO_TRACE_ACTIVATIONS` and friends)
- Pipeline integration: `fastvideo/pipelines/composed_pipeline_base.py`
- Tests: `fastvideo/tests/hooks/test_activation_trace.py`
- Companion skill (for ad-hoc port investigations): `.agents/skills/add-model-08-trace/`
## Changelog
+5 -2
View File
@@ -461,8 +461,11 @@ The reusable implementation lives in
| Workflow | Trigger | Purpose |
|---|---|---|
| `community-issue-labeler.yml` | Issue opened or edited | Adds scope/platform labels from issue keywords. |
| `community-welcome.yml` | First contribution | Posts a welcome comment for first-time contributors. |
| `community-stale.yml` | Daily schedule | Marks and closes stale issues and PRs, with exemption labels. |
None of these automations post comments or messages on issues or pull
requests; they only set labels. Issues and pull requests are never marked
stale or closed automatically. Keep it that way: new automation must report
through checks, commit statuses, or labels instead of PR/issue comments.
## File Reference
+3
View File
@@ -405,6 +405,9 @@ Action:
causal masks, and broadcast shapes).
- **Scheduler / sigma schedule mismatch**: even small differences in schedules
or timestep shapes can cause noticeable drift.
- **Skipped parity tests**: local parity tests skip when the reference clone,
weights, or a GPU are missing. A skip is not a pass; run them where the
reference and weights are available before claiming parity.
## Diffusers vs manual conversion
+25 -17
View File
@@ -304,36 +304,44 @@ longer exists also fails the test, so the fixing pull request deletes its entry.
| `FASTVIDEO_TEST_WAN22_5B_ALLOW_LOW_MEMORY` | bool | `0` | test | Run the MLX Wan2.2 5B real-weights parity test on hosts with little memory. Deprecated names: `FASTVIDEO_WAN22_5B_ALLOW_LOW_MEMORY`. |
| `FASTVIDEO_TEST_WAN22_5B_ROOT` | str | unset | test | Local Wan2.2 5B checkpoint for the MLX real-weights parity test. Deprecated names: `FASTVIDEO_WAN22_5B_ROOT`. |
| `FASTVIDEO_TEST_GRADNORM_UPDATE` | bool | `0` | test | Gradient-norm regression tests update their references. Deprecated names: `FASTVIDEO_GRADNORM_UPDATE`. |
| `FASTVIDEO_TEST_DREAMX_WORLD_SSIM_MODEL_PATH` | str | `FastVideo/DreamX-World-5B-Cam-Diffusers` | test | Model for the DreamX-World camera SSIM test. Deprecated names: `DREAMX_WORLD_SSIM_MODEL_PATH`. |
| `FASTVIDEO_TEST_DREAMX_WORLD_AR_SSIM_MODEL_PATH` | str | `FastVideo/DreamX-World-5B-Diffusers` | test | Model for the DreamX-World autoregressive SSIM test. Deprecated names: `DREAMX_WORLD_AR_SSIM_MODEL_PATH`. |
| `FASTVIDEO_TEST_FLUX_T2I_MODEL_DIR` | str | `black-forest-labs/FLUX.1-dev` | test | Model for the Flux text-to-image SSIM test. Deprecated names: `FLUX_T2I_MODEL_DIR`. |
| `FASTVIDEO_TEST_FLUX_TRANSFORMER_PATH` | str | unset | test | Local Flux transformer for the Flux transformer test. Deprecated names: `FLUX_TRANSFORMER_PATH`. |
| `FASTVIDEO_TEST_GAMECRAFT_MODEL_PATH` | str | `FastVideo/HunyuanGameCraft-Diffusers` | test | Model for the HunyuanGameCraft SSIM test. Deprecated names: `GAMECRAFT_MODEL_PATH`. |
| `FASTVIDEO_TEST_GEN3C_MODEL_PATH` | str | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | test | Model for the GEN3C SSIM test. Deprecated names: `GEN3C_MODEL_PATH`. |
| `FASTVIDEO_TEST_GEN3C_IMAGE_PATH` | str | unset | test | Input image for the GEN3C SSIM test. Deprecated names: `GEN3C_TEST_IMAGE_PATH`. |
| `FASTVIDEO_TEST_GLM_IMAGE_LOCAL_WEIGHTS_DIR` | str | unset | test | Local official GLM-Image weights for the GLM-Image SSIM test. Deprecated names: `GLM_IMAGE_LOCAL_WEIGHTS_DIR`. |
| `FASTVIDEO_TEST_GLM_IMAGE_MODEL_DIR` | str | unset | test | Model for the GLM-Image SSIM test. Deprecated names: `GLM_IMAGE_MODEL_DIR`. |
| `FASTVIDEO_TEST_KANDINSKY5_E2E_NUM_GPUS` | int | `1` | test | GPUs for the Kandinsky5 nightly end-to-end overfit test. Deprecated names: `KANDINSKY5_E2E_NUM_GPUS`. |
| `FASTVIDEO_TEST_KANDINSKY5_E2E_WRITE_REFERENCE` | bool | `0` | test | The Kandinsky5 nightly end-to-end test writes a missing reference video. Deprecated names: `KANDINSKY5_E2E_WRITE_REFERENCE`. |
| `FASTVIDEO_TEST_LONGCAT_MODEL_ROOT` | str | unset | test | Local LongCat-Video checkpoint for the golden-gate test. Deprecated names: `LONGCAT_MODEL_ROOT`. |
| `FASTVIDEO_TEST_MINIMAX_H3_GATE_GOLDEN_DIR` | str | unset | test | Local directory of MiniMax-H3 golden-gate tensors. Deprecated names: `MINIMAX_H3_GATE_GOLDEN_DIR`. |
| `FASTVIDEO_TEST_MINIMAX_H3_GATE_LAYER` | int | `0` | test | Transformer layer that the MiniMax-H3 golden-gate test checks. Deprecated names: `MINIMAX_H3_GATE_LAYER`. |
| `FASTVIDEO_TEST_MINIMAX_H3_MODEL_ROOT` | str | unset | test | Local MiniMax-H3 checkpoint for the golden-gate test. Deprecated names: `MINIMAX_H3_MODEL_ROOT`. |
| `FASTVIDEO_TEST_SD35_MODEL_DIR` | str | `stabilityai/stable-diffusion-3.5-medium` | test | Model for the Stable Diffusion 3.5 SSIM test. Deprecated names: `SD35_MODEL_DIR`. |
| `FASTVIDEO_TEST_TAEH3_REFERENCE_DIR` | str | unset | test | Upstream taehv checkout for the MLX TAEH3 parity test. Deprecated names: `TAEH3_REFERENCE_DIR`. |
| `FASTVIDEO_TEST_WAN_ANIMATE_MODEL_DIR` | str | unset | test | Local Wan2.2-Animate-14B checkpoint for the Wan-Animate weight tests. Deprecated names: `WAN_ANIMATE_MODEL_PATH`. |
| `FASTVIDEO_TEST_ZIMAGE_MODEL_DIR` | str | `Tongyi-MAI/Z-Image-Turbo` | test | Model for the Z-Image SSIM test. Deprecated names: `ZIMAGE_MODEL_DIR`. |
| `FASTVIDEO_TEST_ZIMAGE_MODEL_REVISION` | str | `f332072aa78be7aecdf3ee76d5c247082da564a6` | test | Hugging Face revision of the Z-Image model for its SSIM test. Deprecated names: `ZIMAGE_MODEL_REVISION`. |
Variables that FastVideo no longer reads; setting one logs a warning:
| Deprecated variable | Reason |
| ----------------------------------------- | ---------------- |
| `FASTVIDEO_TARGET_DEVICE` | no code reads it |
| `FASTVIDEO_USE_PRECOMPILED` | no code reads it |
| `FASTVIDEO_RINGBUFFER_WARNING_INTERVAL` | no code reads it |
| `FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S` | no code reads it |
| `FASTVIDEO_SERVER_DEV_MODE` | no code reads it |
| `FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE` | no code reads it |
| `FASTVIDEO_TRACE_FUNCTION` | no code reads it |
| Deprecated variable | Reason |
| ------------------------------------------------ | ---------------------------- |
| `FASTVIDEO_TARGET_DEVICE` | no code reads it |
| `FASTVIDEO_USE_PRECOMPILED` | no code reads it |
| `FASTVIDEO_RINGBUFFER_WARNING_INTERVAL` | no code reads it |
| `FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S` | no code reads it |
| `FASTVIDEO_SERVER_DEV_MODE` | no code reads it |
| `FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE` | no code reads it |
| `FASTVIDEO_TRACE_FUNCTION` | no code reads it |
| `FASTVIDEO_TEST_DREAMX_WORLD_SSIM_MODEL_PATH` | DreamX World was removed |
| `DREAMX_WORLD_SSIM_MODEL_PATH` | DreamX World was removed |
| `FASTVIDEO_TEST_DREAMX_WORLD_AR_SSIM_MODEL_PATH` | DreamX World was removed |
| `DREAMX_WORLD_AR_SSIM_MODEL_PATH` | DreamX World was removed |
| `FASTVIDEO_TEST_GAMECRAFT_MODEL_PATH` | HunyuanGameCraft was removed |
| `GAMECRAFT_MODEL_PATH` | HunyuanGameCraft was removed |
| `FASTVIDEO_TEST_GLM_IMAGE_LOCAL_WEIGHTS_DIR` | GLM-Image was removed |
| `GLM_IMAGE_LOCAL_WEIGHTS_DIR` | GLM-Image was removed |
| `FASTVIDEO_TEST_GLM_IMAGE_MODEL_DIR` | GLM-Image was removed |
| `GLM_IMAGE_MODEL_DIR` | GLM-Image was removed |
| `FASTVIDEO_TEST_LONGCAT_MODEL_ROOT` | LongCat was removed |
| `LONGCAT_MODEL_ROOT` | LongCat was removed |
| `FASTVIDEO_TEST_ZIMAGE_MODEL_DIR` | Z-Image was removed |
| `ZIMAGE_MODEL_DIR` | Z-Image was removed |
| `FASTVIDEO_TEST_ZIMAGE_MODEL_REVISION` | Z-Image was removed |
| `ZIMAGE_MODEL_REVISION` | Z-Image was removed |
<!-- END GENERATED ENV TABLE -->
+1 -1
View File
@@ -21,7 +21,7 @@ Common examples:
[docs] Add LoRA finetuning guide
[new-model] Port HunyuanVideo 1.5 to FastVideo
[infra] Add activation trace hooks for pipeline debugging
[skill] Add add-model agent skill
[skill] Add reseed-ssim-references agent skill
```
Accepted tags are `feat`, `feature`, `bugfix`, `fix`, `refactor`, `perf`,
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Cosmos recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="cosmos" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="cosmos" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# FLUX recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="flux" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="flux" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
-140
View File
@@ -1,140 +0,0 @@
---
hide:
- toc
---
# GLM-Image recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="glm_image" data-recipes="../../assets/cookbook-recipes.json?v=15">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
<span class="cookbook-family-header__logo">
<img class="off-glb" src="../../assets/logos/zai.webp" alt="Z.ai" width="112" height="112">
</span>
<div>
<p class="cookbook-eyebrow">Maintained family · Inference</p>
<h2>GLM-Image inference recipes</h2>
<p>GLM-Image from Z.ai supports both text-to-image generation and instruction-based image editing, each with a maintained example.</p>
</div>
</div>
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
<span class="cookbook-lifecycle__stage cookbook-lifecycle__stage--active">Inference <small>live</small></span>
<span class="cookbook-lifecycle__stage">Distillation <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Fine-tuning <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Training <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Evaluation <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Optimization <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Deployment <small>planned</small></span>
</div>
</header>
<nav class="cookbook-jumpnav" aria-label="Recipe page sections">
<a href="#recipe-builder">Builder</a>
<a href="#cookbook-setup">Setup</a>
<a href="#cookbook-troubleshooting">Troubleshooting</a>
<a href="#cookbook-evidence">Evidence</a>
</nav>
<section class="cookbook-builder" id="recipe-builder" aria-labelledby="builder-heading">
<div class="cookbook-builder__intro">
<h2 id="builder-heading">Pick a recipe and runtime</h2>
<p>Start with the result you want, then choose one of the runtimes FastVideo actually maintains for it.</p>
</div>
<div class="cookbook-builder__layout">
<div class="cookbook-controls">
<div class="cookbook-selection-row">
<div class="cookbook-selection-row__label">
<strong>Recipe</strong>
<span>Task and checkpoint</span>
</div>
<div class="cookbook-option-grid cookbook-option-grid--models" data-cookbook-model-options role="group" aria-label="Recipe">
<button type="button" disabled>Loading GLM-Image recipes...</button>
</div>
</div>
<div class="cookbook-selection-row">
<div class="cookbook-selection-row__label">
<strong>Runtime</strong>
<span>Maintained paths only</span>
</div>
<div class="cookbook-option-grid cookbook-option-grid--hardware" data-cookbook-hardware-options role="group" aria-label="Runtime">
<button type="button" disabled>Loading runtimes...</button>
</div>
</div>
<p class="cookbook-selection-description" data-cookbook-description>Loading recipe details...</p>
<p class="cookbook-hardware-note">Exact device and memory details appear only when a recorded run supports them.</p>
<div class="cookbook-hardware-state" data-cookbook-hardware-state role="status" aria-live="polite">
Reading recipe evidence...
</div>
</div>
<article class="cookbook-result">
<div class="cookbook-result__header">
<h3 data-cookbook-label>Loading...</h3>
<div class="cookbook-result__badges">
<span class="cookbook-badge">Maintained</span>
<span class="cookbook-badge" data-cookbook-evidence>Source-backed</span>
<span class="cookbook-badge cookbook-badge--neutral" data-cookbook-hardware-badge>Source config</span>
</div>
</div>
<dl class="cookbook-result__facts">
<div><dt>Model</dt><dd data-cookbook-model>Loading...</dd></div>
<div><dt>Workload</dt><dd data-cookbook-task>Loading...</dd></div>
<div><dt>Source configuration</dt><dd data-cookbook-gpus>Loading...</dd></div>
<div><dt>Expected output</dt><dd data-cookbook-artifact>Loading...</dd></div>
</dl>
<div class="cookbook-command">
<div class="cookbook-command__bar">
<span>Terminal</span>
</div>
<pre><code class="language-bash" data-cookbook-command>Loading...</code></pre>
</div>
<div class="cookbook-result__footer">
<a data-cookbook-source href="../../inference/examples/basic/">Open example source</a>
<a data-cookbook-model-link href="https://huggingface.co/zai-org">View model card</a>
</div>
<p class="cookbook-picker__status" role="status" aria-live="polite" data-cookbook-status></p>
</article>
</div>
<noscript>
<div class="cookbook-noscript">
JavaScript is needed for the guided selector. You can still browse the
<a href="../../inference/examples/examples_inference_index/">maintained inference examples</a>.
</div>
</noscript>
</section>
</div>
<details class="cookbook-collapsible" id="cookbook-setup">
<summary>Setup</summary>
<div class="cookbook-collapsible__body">
<p>The generated commands expect a local clone:</p>
<pre><code>git clone https://github.com/hao-ai-lab/FastVideo.git
cd FastVideo</code></pre>
<p>Use <a href="../../inference/configuration/">Configuration</a> for supported Python and CLI settings, <a href="../../inference/optimizations/">Optimizations</a> for attention and memory tradeoffs, and the <a href="../../inference/support_matrix/">support matrix</a> for the supported model and optimization surface.</p>
</div>
</details>
<details class="cookbook-collapsible" id="cookbook-troubleshooting">
<summary>Troubleshooting</summary>
<div class="cookbook-collapsible__body">
<ul>
<li>The editing example reads <code>assets/images/couple.jpg</code> from the repository root, so run it from a repo checkout rather than an arbitrary working directory.</li>
<li>Output paths default under <code>image_output/</code>; pass <code>--output</code> to change them.</li>
</ul>
</div>
</details>
<details class="cookbook-collapsible" id="cookbook-evidence">
<summary>Evidence status</summary>
<div class="cookbook-collapsible__body">
<p>All recipes on this page are <strong>Source-backed</strong>: their commands, model IDs, and flags were validated against the checked-in FastVideo sources listed above (static validation). No runtime GPU validation is recorded for these recipes, so GPU model fit, memory use, throughput, and runtime duration are <strong>Unknown</strong> and deliberately not claimed. Runtime buttons show only the GPU counts configured in checked-in sources.</p>
</div>
</details>
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Hunyuan recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="hunyuan" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="hunyuan" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
-115
View File
@@ -163,50 +163,6 @@ hide:
</ul>
</span>
</a>
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./longcat/" aria-label="Open LongCat recipes">
<span class="cookbook-family-tile__visual" data-evervault>
<span class="cookbook-evervault" aria-hidden="true">
<span class="cookbook-evervault__gradient"></span>
<span class="cookbook-evervault__noise" data-cookbook-pattern></span>
</span>
<span class="cookbook-family-tile__logo-wrap">
<img class="off-glb" src="../assets/logos/meituan-longcat.webp" alt="" width="132" height="132" loading="lazy">
</span>
</span>
<span class="cookbook-family-tile__footer">
<span class="cookbook-family-tile__footer-top">
<span><strong>LongCat</strong><small>T2V, I2V, optional refine</small></span>
<span class="cookbook-count">2 recipes</span>
</span>
<ul class="cookbook-mode-row">
<li>T2V</li>
<li>I2V</li>
</ul>
</span>
</a>
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./turbodiffusion/" aria-label="Open TurboDiffusion recipes">
<span class="cookbook-family-tile__visual" data-evervault>
<span class="cookbook-evervault" aria-hidden="true">
<span class="cookbook-evervault__gradient"></span>
<span class="cookbook-evervault__noise" data-cookbook-pattern></span>
</span>
<span class="cookbook-family-tile__logo-wrap">
<span class="cookbook-family-tile__monogram" aria-hidden="true">Turbo</span>
</span>
</span>
<span class="cookbook-family-tile__footer">
<span class="cookbook-family-tile__footer-top">
<span><strong>TurboDiffusion</strong><small>Accelerated Wan profiles</small></span>
<span class="cookbook-count">3 recipes</span>
</span>
<ul class="cookbook-mode-row">
<li>T2V</li>
<li>I2V</li>
</ul>
</span>
</a>
</div>
</section>
@@ -236,49 +192,6 @@ hide:
</span>
</a>
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./glm-image/" aria-label="Open GLM-Image recipes">
<span class="cookbook-family-tile__visual" data-evervault>
<span class="cookbook-evervault" aria-hidden="true">
<span class="cookbook-evervault__gradient"></span>
<span class="cookbook-evervault__noise" data-cookbook-pattern></span>
</span>
<span class="cookbook-family-tile__logo-wrap">
<img class="off-glb" src="../assets/logos/zai.webp" alt="" width="132" height="132" loading="lazy">
</span>
</span>
<span class="cookbook-family-tile__footer">
<span class="cookbook-family-tile__footer-top">
<span><strong>GLM-Image</strong><small>Generate and edit</small></span>
<span class="cookbook-count">2 recipes</span>
</span>
<ul class="cookbook-mode-row">
<li>T2I</li>
<li>Edit</li>
</ul>
</span>
</a>
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./z-image/" aria-label="Open Z-Image recipes">
<span class="cookbook-family-tile__visual" data-evervault>
<span class="cookbook-evervault" aria-hidden="true">
<span class="cookbook-evervault__gradient"></span>
<span class="cookbook-evervault__noise" data-cookbook-pattern></span>
</span>
<span class="cookbook-family-tile__logo-wrap">
<img class="off-glb" src="../assets/logos/tongyi.webp" alt="" width="132" height="132" loading="lazy">
</span>
</span>
<span class="cookbook-family-tile__footer">
<span class="cookbook-family-tile__footer-top">
<span><strong>Z-Image</strong><small>Turbo text to image</small></span>
<span class="cookbook-count">1 recipe</span>
</span>
<ul class="cookbook-mode-row">
<li>T2I</li>
</ul>
</span>
</a>
<a class="cookbook-family-tile cookbook-family-tile--ready" href="./stable-diffusion/" aria-label="Open Stable Diffusion recipes">
<span class="cookbook-family-tile__visual" data-evervault>
<span class="cookbook-evervault" aria-hidden="true">
@@ -407,20 +320,6 @@ hide:
<p>These families already have runnable examples. The cookbook page is not ready, so the cards are not links.</p>
</div>
<div class="cookbook-family-grid">
<article class="cookbook-family-tile cookbook-family-tile--coming" aria-label="GameCraft cookbook page planned; runnable examples exist">
<span class="cookbook-family-tile__visual">
<span class="cookbook-family-tile__logo-wrap">
<img class="off-glb" src="../assets/logos/tencent-hunyuan.webp" alt="" width="132" height="132" loading="lazy">
</span>
</span>
<span class="cookbook-family-tile__footer">
<span class="cookbook-family-tile__footer-top">
<span><strong>GameCraft</strong><small>Game world generation</small></span>
<span class="cookbook-count">Page planned</span>
</span>
</span>
</article>
<article class="cookbook-family-tile cookbook-family-tile--coming" aria-label="GEN3C cookbook page planned; runnable examples exist">
<span class="cookbook-family-tile__visual">
<span class="cookbook-family-tile__logo-wrap">
@@ -449,20 +348,6 @@ hide:
</span>
</article>
<article class="cookbook-family-tile cookbook-family-tile--coming" aria-label="DreamX cookbook page planned; runnable examples exist">
<span class="cookbook-family-tile__visual">
<span class="cookbook-family-tile__logo-wrap">
<img class="off-glb" src="../assets/logos/fastvideo.webp" alt="" width="132" height="132" loading="lazy">
</span>
</span>
<span class="cookbook-family-tile__footer">
<span class="cookbook-family-tile__footer-top">
<span><strong>DreamX</strong><small>World generation</small></span>
<span class="cookbook-count">Page planned</span>
</span>
</span>
</article>
<article class="cookbook-family-tile cookbook-family-tile--coming" aria-label="LingBot cookbook page planned; runnable examples exist">
<span class="cookbook-family-tile__visual">
<span class="cookbook-family-tile__logo-wrap">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Kandinsky 5 recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky5" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky5" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Kandinsky 6 recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky6" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky6" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
-140
View File
@@ -1,140 +0,0 @@
---
hide:
- toc
---
# LongCat recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="longcat" data-recipes="../../assets/cookbook-recipes.json?v=15">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
<span class="cookbook-family-header__logo">
<img class="off-glb" src="../../assets/logos/meituan-longcat.webp" alt="Meituan LongCat" width="112" height="112">
</span>
<div>
<p class="cookbook-eyebrow">Maintained family · Inference</p>
<h2>LongCat inference recipes</h2>
<p>LongCat Video from Meituan covers text-to-video and image-to-video, and its maintained examples chain optional distilled and 720p refinement passes.</p>
</div>
</div>
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
<span class="cookbook-lifecycle__stage cookbook-lifecycle__stage--active">Inference <small>live</small></span>
<span class="cookbook-lifecycle__stage">Distillation <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Fine-tuning <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Training <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Evaluation <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Optimization <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Deployment <small>planned</small></span>
</div>
</header>
<nav class="cookbook-jumpnav" aria-label="Recipe page sections">
<a href="#recipe-builder">Builder</a>
<a href="#cookbook-setup">Setup</a>
<a href="#cookbook-troubleshooting">Troubleshooting</a>
<a href="#cookbook-evidence">Evidence</a>
</nav>
<section class="cookbook-builder" id="recipe-builder" aria-labelledby="builder-heading">
<div class="cookbook-builder__intro">
<h2 id="builder-heading">Pick a recipe and runtime</h2>
<p>Start with the result you want, then choose one of the runtimes FastVideo actually maintains for it.</p>
</div>
<div class="cookbook-builder__layout">
<div class="cookbook-controls">
<div class="cookbook-selection-row">
<div class="cookbook-selection-row__label">
<strong>Recipe</strong>
<span>Task and checkpoint</span>
</div>
<div class="cookbook-option-grid cookbook-option-grid--models" data-cookbook-model-options role="group" aria-label="Recipe">
<button type="button" disabled>Loading LongCat recipes...</button>
</div>
</div>
<div class="cookbook-selection-row">
<div class="cookbook-selection-row__label">
<strong>Runtime</strong>
<span>Maintained paths only</span>
</div>
<div class="cookbook-option-grid cookbook-option-grid--hardware" data-cookbook-hardware-options role="group" aria-label="Runtime">
<button type="button" disabled>Loading runtimes...</button>
</div>
</div>
<p class="cookbook-selection-description" data-cookbook-description>Loading recipe details...</p>
<p class="cookbook-hardware-note">Exact device and memory details appear only when a recorded run supports them.</p>
<div class="cookbook-hardware-state" data-cookbook-hardware-state role="status" aria-live="polite">
Reading recipe evidence...
</div>
</div>
<article class="cookbook-result">
<div class="cookbook-result__header">
<h3 data-cookbook-label>Loading...</h3>
<div class="cookbook-result__badges">
<span class="cookbook-badge">Maintained</span>
<span class="cookbook-badge" data-cookbook-evidence>Source-backed</span>
<span class="cookbook-badge cookbook-badge--neutral" data-cookbook-hardware-badge>Source config</span>
</div>
</div>
<dl class="cookbook-result__facts">
<div><dt>Model</dt><dd data-cookbook-model>Loading...</dd></div>
<div><dt>Workload</dt><dd data-cookbook-task>Loading...</dd></div>
<div><dt>Source configuration</dt><dd data-cookbook-gpus>Loading...</dd></div>
<div><dt>Expected output</dt><dd data-cookbook-artifact>Loading...</dd></div>
</dl>
<div class="cookbook-command">
<div class="cookbook-command__bar">
<span>Terminal</span>
</div>
<pre><code class="language-bash" data-cookbook-command>Loading...</code></pre>
</div>
<div class="cookbook-result__footer">
<a data-cookbook-source href="../../inference/examples/basic/">Open example source</a>
<a data-cookbook-model-link href="https://huggingface.co/meituan-longcat">View model card</a>
</div>
<p class="cookbook-picker__status" role="status" aria-live="polite" data-cookbook-status></p>
</article>
</div>
<noscript>
<div class="cookbook-noscript">
JavaScript is needed for the guided selector. You can still browse the
<a href="../../inference/examples/examples_inference_index/">maintained inference examples</a>.
</div>
</noscript>
</section>
</div>
<details class="cookbook-collapsible" id="cookbook-setup">
<summary>Setup</summary>
<div class="cookbook-collapsible__body">
<p>The generated commands expect a local clone:</p>
<pre><code>git clone https://github.com/hao-ai-lab/FastVideo.git
cd FastVideo</code></pre>
<p>Use <a href="../../inference/configuration/">Configuration</a> for supported Python and CLI settings, <a href="../../inference/optimizations/">Optimizations</a> for attention and memory tradeoffs, and the <a href="../../inference/support_matrix/">support matrix</a> for the supported model and optimization surface.</p>
</div>
</details>
<details class="cookbook-collapsible" id="cookbook-troubleshooting">
<summary>Troubleshooting</summary>
<div class="cookbook-collapsible__body">
<ul>
<li>Each LongCat script runs multiple passes (basic, distilled, refine); total runtime scales accordingly, and every pass prints its own output directory.</li>
<li>Out of memory: the sources already enable VAE and text-encoder CPU offload; further options are covered in <a href="../../inference/offloading/">Offloading</a>.</li>
</ul>
</div>
</details>
<details class="cookbook-collapsible" id="cookbook-evidence">
<summary>Evidence status</summary>
<div class="cookbook-collapsible__body">
<p>All recipes on this page are <strong>Source-backed</strong>: their commands, model IDs, and flags were validated against the checked-in FastVideo sources listed above (static validation). No runtime GPU validation is recorded for these recipes, so GPU model fit, memory use, throughput, and runtime duration are <strong>Unknown</strong> and deliberately not claimed. Runtime buttons show only the GPU counts configured in checked-in sources.</p>
</div>
</details>
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# LTX recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="ltx2" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="ltx2" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Matrix Game recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="matrixgame" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="matrixgame" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -18,7 +18,7 @@ eight-forward T2VA command](../getting_started/installation/mlx.md#pruned-eight-
It reads `fastvideo_inference.json` for the trained schedule. The command
uses native 832x480 resolution and all requested frames.
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="minimax_h3" data-default-recipe="fasth3-preview-cuda" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="minimax_h3" data-default-recipe="fasth3-preview-cuda" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# MMAudio recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="mmaudio" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="mmaudio" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Stable Audio recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="stable_audio" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="stable_audio" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Stable Diffusion recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="sd35" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="sd35" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
-140
View File
@@ -1,140 +0,0 @@
---
hide:
- toc
---
# TurboDiffusion recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="turbodiffusion" data-recipes="../../assets/cookbook-recipes.json?v=15">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
<span class="cookbook-family-header__logo">
<span class="cookbook-family-tile__monogram" aria-hidden="true">Turbo</span>
</span>
<div>
<p class="cookbook-eyebrow">Maintained family · Inference</p>
<h2>TurboDiffusion inference recipes</h2>
<p>TurboDiffusion profiles accelerate Wan checkpoints with step-distilled sampling and the SLA attention backend. These recipes follow the registry's `turbodiffusion` model family.</p>
</div>
</div>
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
<span class="cookbook-lifecycle__stage cookbook-lifecycle__stage--active">Inference <small>live</small></span>
<span class="cookbook-lifecycle__stage">Distillation <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Fine-tuning <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Training <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Evaluation <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Optimization <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Deployment <small>planned</small></span>
</div>
</header>
<nav class="cookbook-jumpnav" aria-label="Recipe page sections">
<a href="#recipe-builder">Builder</a>
<a href="#cookbook-setup">Setup</a>
<a href="#cookbook-troubleshooting">Troubleshooting</a>
<a href="#cookbook-evidence">Evidence</a>
</nav>
<section class="cookbook-builder" id="recipe-builder" aria-labelledby="builder-heading">
<div class="cookbook-builder__intro">
<h2 id="builder-heading">Pick a recipe and runtime</h2>
<p>Start with the result you want, then choose one of the runtimes FastVideo actually maintains for it.</p>
</div>
<div class="cookbook-builder__layout">
<div class="cookbook-controls">
<div class="cookbook-selection-row">
<div class="cookbook-selection-row__label">
<strong>Recipe</strong>
<span>Task and checkpoint</span>
</div>
<div class="cookbook-option-grid cookbook-option-grid--models" data-cookbook-model-options role="group" aria-label="Recipe">
<button type="button" disabled>Loading TurboDiffusion recipes...</button>
</div>
</div>
<div class="cookbook-selection-row">
<div class="cookbook-selection-row__label">
<strong>Runtime</strong>
<span>Maintained paths only</span>
</div>
<div class="cookbook-option-grid cookbook-option-grid--hardware" data-cookbook-hardware-options role="group" aria-label="Runtime">
<button type="button" disabled>Loading runtimes...</button>
</div>
</div>
<p class="cookbook-selection-description" data-cookbook-description>Loading recipe details...</p>
<p class="cookbook-hardware-note">Exact device and memory details appear only when a recorded run supports them.</p>
<div class="cookbook-hardware-state" data-cookbook-hardware-state role="status" aria-live="polite">
Reading recipe evidence...
</div>
</div>
<article class="cookbook-result">
<div class="cookbook-result__header">
<h3 data-cookbook-label>Loading...</h3>
<div class="cookbook-result__badges">
<span class="cookbook-badge">Maintained</span>
<span class="cookbook-badge" data-cookbook-evidence>Source-backed</span>
<span class="cookbook-badge cookbook-badge--neutral" data-cookbook-hardware-badge>Source config</span>
</div>
</div>
<dl class="cookbook-result__facts">
<div><dt>Model</dt><dd data-cookbook-model>Loading...</dd></div>
<div><dt>Workload</dt><dd data-cookbook-task>Loading...</dd></div>
<div><dt>Source configuration</dt><dd data-cookbook-gpus>Loading...</dd></div>
<div><dt>Expected output</dt><dd data-cookbook-artifact>Loading...</dd></div>
</dl>
<div class="cookbook-command">
<div class="cookbook-command__bar">
<span>Terminal</span>
</div>
<pre><code class="language-bash" data-cookbook-command>Loading...</code></pre>
</div>
<div class="cookbook-result__footer">
<a data-cookbook-source href="../../inference/examples/basic/">Open example source</a>
<a data-cookbook-model-link href="https://huggingface.co/loayrashid">View model card</a>
</div>
<p class="cookbook-picker__status" role="status" aria-live="polite" data-cookbook-status></p>
</article>
</div>
<noscript>
<div class="cookbook-noscript">
JavaScript is needed for the guided selector. You can still browse the
<a href="../../inference/examples/examples_inference_index/">maintained inference examples</a>.
</div>
</noscript>
</section>
</div>
<details class="cookbook-collapsible" id="cookbook-setup">
<summary>Setup</summary>
<div class="cookbook-collapsible__body">
<p>The generated commands expect a local clone:</p>
<pre><code>git clone https://github.com/hao-ai-lab/FastVideo.git
cd FastVideo</code></pre>
<p>Use <a href="../../inference/configuration/">Configuration</a> for supported Python and CLI settings, <a href="../../inference/optimizations/">Optimizations</a> for attention and memory tradeoffs, and the <a href="../../inference/support_matrix/">support matrix</a> for the supported model and optimization surface.</p>
</div>
</details>
<details class="cookbook-collapsible" id="cookbook-troubleshooting">
<summary>Troubleshooting</summary>
<div class="cookbook-collapsible__body">
<ul>
<li>TurboDiffusion paths load community-published <code>loayrashid/TurboWan*</code> checkpoints; availability is governed by those repos.</li>
<li>The SLA attention backend used by the I2V recipe is selected inside the example source; do not combine it with another <code>FASTVIDEO_ATTENTION_BACKEND</code> override in the same shell.</li>
</ul>
</div>
</details>
<details class="cookbook-collapsible" id="cookbook-evidence">
<summary>Evidence status</summary>
<div class="cookbook-collapsible__body">
<p>All recipes on this page are <strong>Source-backed</strong>: their commands, model IDs, and flags were validated against the checked-in FastVideo sources listed above (static validation). No runtime GPU validation is recorded for these recipes, so GPU model fit, memory use, throughput, and runtime duration are <strong>Unknown</strong> and deliberately not claimed. Runtime buttons show only the GPU counts configured in checked-in sources.</p>
</div>
</details>
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Wan recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="wan" data-recipes="../../assets/cookbook-recipes.json?v=15">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="wan" data-recipes="../../assets/cookbook-recipes.json?v=16">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
-140
View File
@@ -1,140 +0,0 @@
---
hide:
- toc
---
# Z-Image recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="zimage" data-recipes="../../assets/cookbook-recipes.json?v=15">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
<span class="cookbook-family-header__logo">
<img class="off-glb" src="../../assets/logos/tongyi.webp" alt="Tongyi MAI" width="112" height="112">
</span>
<div>
<p class="cookbook-eyebrow">Maintained family · Inference</p>
<h2>Z-Image inference recipes</h2>
<p>Z-Image Turbo from Tongyi MAI is a fast text-to-image model. The maintained example runs it on a single GPU.</p>
</div>
</div>
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
<span class="cookbook-lifecycle__stage cookbook-lifecycle__stage--active">Inference <small>live</small></span>
<span class="cookbook-lifecycle__stage">Distillation <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Fine-tuning <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Training <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Evaluation <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Optimization <small>planned</small></span>
<span class="cookbook-lifecycle__stage">Deployment <small>planned</small></span>
</div>
</header>
<nav class="cookbook-jumpnav" aria-label="Recipe page sections">
<a href="#recipe-builder">Builder</a>
<a href="#cookbook-setup">Setup</a>
<a href="#cookbook-troubleshooting">Troubleshooting</a>
<a href="#cookbook-evidence">Evidence</a>
</nav>
<section class="cookbook-builder" id="recipe-builder" aria-labelledby="builder-heading">
<div class="cookbook-builder__intro">
<h2 id="builder-heading">Pick a recipe and runtime</h2>
<p>Start with the result you want, then choose one of the runtimes FastVideo actually maintains for it.</p>
</div>
<div class="cookbook-builder__layout">
<div class="cookbook-controls">
<div class="cookbook-selection-row">
<div class="cookbook-selection-row__label">
<strong>Recipe</strong>
<span>Task and checkpoint</span>
</div>
<div class="cookbook-option-grid cookbook-option-grid--models" data-cookbook-model-options role="group" aria-label="Recipe">
<button type="button" disabled>Loading Z-Image recipes...</button>
</div>
</div>
<div class="cookbook-selection-row">
<div class="cookbook-selection-row__label">
<strong>Runtime</strong>
<span>Maintained paths only</span>
</div>
<div class="cookbook-option-grid cookbook-option-grid--hardware" data-cookbook-hardware-options role="group" aria-label="Runtime">
<button type="button" disabled>Loading runtimes...</button>
</div>
</div>
<p class="cookbook-selection-description" data-cookbook-description>Loading recipe details...</p>
<p class="cookbook-hardware-note">Exact device and memory details appear only when a recorded run supports them.</p>
<div class="cookbook-hardware-state" data-cookbook-hardware-state role="status" aria-live="polite">
Reading recipe evidence...
</div>
</div>
<article class="cookbook-result">
<div class="cookbook-result__header">
<h3 data-cookbook-label>Loading...</h3>
<div class="cookbook-result__badges">
<span class="cookbook-badge">Maintained</span>
<span class="cookbook-badge" data-cookbook-evidence>Source-backed</span>
<span class="cookbook-badge cookbook-badge--neutral" data-cookbook-hardware-badge>Source config</span>
</div>
</div>
<dl class="cookbook-result__facts">
<div><dt>Model</dt><dd data-cookbook-model>Loading...</dd></div>
<div><dt>Workload</dt><dd data-cookbook-task>Loading...</dd></div>
<div><dt>Source configuration</dt><dd data-cookbook-gpus>Loading...</dd></div>
<div><dt>Expected output</dt><dd data-cookbook-artifact>Loading...</dd></div>
</dl>
<div class="cookbook-command">
<div class="cookbook-command__bar">
<span>Terminal</span>
</div>
<pre><code class="language-bash" data-cookbook-command>Loading...</code></pre>
</div>
<div class="cookbook-result__footer">
<a data-cookbook-source href="../../inference/examples/basic/">Open example source</a>
<a data-cookbook-model-link href="https://huggingface.co/Tongyi-MAI">View model card</a>
</div>
<p class="cookbook-picker__status" role="status" aria-live="polite" data-cookbook-status></p>
</article>
</div>
<noscript>
<div class="cookbook-noscript">
JavaScript is needed for the guided selector. You can still browse the
<a href="../../inference/examples/examples_inference_index/">maintained inference examples</a>.
</div>
</noscript>
</section>
</div>
<details class="cookbook-collapsible" id="cookbook-setup">
<summary>Setup</summary>
<div class="cookbook-collapsible__body">
<p>The generated commands expect a local clone:</p>
<pre><code>git clone https://github.com/hao-ai-lab/FastVideo.git
cd FastVideo</code></pre>
<p>Use <a href="../../inference/configuration/">Configuration</a> for supported Python and CLI settings, <a href="../../inference/optimizations/">Optimizations</a> for attention and memory tradeoffs, and the <a href="../../inference/support_matrix/">support matrix</a> for the supported model and optimization surface.</p>
</div>
</details>
<details class="cookbook-collapsible" id="cookbook-troubleshooting">
<summary>Troubleshooting</summary>
<div class="cookbook-collapsible__body">
<ul>
<li>Output defaults to <code>outputs/zimage/zimage_turbo.png</code>; pass <code>--output</code> to redirect.</li>
<li>Gated or missing checkpoints: run <code>huggingface-cli login</code> and confirm you accepted the model's license on Hugging Face.</li>
</ul>
</div>
</details>
<details class="cookbook-collapsible" id="cookbook-evidence">
<summary>Evidence status</summary>
<div class="cookbook-collapsible__body">
<p>All recipes on this page are <strong>Source-backed</strong>: their commands, model IDs, and flags were validated against the checked-in FastVideo sources listed above (static validation). No runtime GPU validation is recorded for these recipes, so GPU model fit, memory use, throughput, and runtime duration are <strong>Unknown</strong> and deliberately not claimed. Runtime buttons show only the GPU counts configured in checked-in sources.</p>
</div>
</details>
@@ -219,9 +219,6 @@ surfaces:
sources:
- fastvideo.configs.pipelines.gen3c.Gen3CConfig
- fastvideo.configs.pipelines.gen3c.Gen3CInferenceConfig
color_correction_strength:
sources:
- fastvideo.configs.pipelines.dreamx_world.DreamXWorld5BARPipelineConfig
default_camera_rotation:
sources:
- fastvideo.configs.pipelines.gen3c.Gen3CConfig
@@ -296,11 +293,6 @@ surfaces:
sources:
- fastvideo.configs.pipelines.lingbotworld.LingBotWorldI2V480PConfig
- fastvideo.configs.pipelines.lingbotworld.Wan2_2_I2V_A14B_Config
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionI2VConfig
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionI2V_A14B_Config
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2VConfig
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_14B_Config
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_1_3B_Config
- fastvideo.configs.pipelines.wan.FastWan2_1_T2V_480P_Config
- fastvideo.configs.pipelines.wan.FastWan2_2_TI2V_5B_Config
- fastvideo.configs.pipelines.matrixgame2.MatrixGame2BaseI2V480PConfig
@@ -319,11 +311,6 @@ surfaces:
sources:
- fastvideo.configs.pipelines.lingbotworld.LingBotWorldI2V480PConfig
- fastvideo.configs.pipelines.lingbotworld.Wan2_2_I2V_A14B_Config
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionI2VConfig
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionI2V_A14B_Config
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2VConfig
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_14B_Config
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_1_3B_Config
- fastvideo.configs.pipelines.wan.FastWan2_1_T2V_480P_Config
- fastvideo.configs.pipelines.wan.FastWan2_2_TI2V_5B_Config
- fastvideo.configs.pipelines.matrixgame2.MatrixGame2BaseI2V480PConfig
@@ -338,48 +325,21 @@ surfaces:
- fastvideo.configs.pipelines.wan.WanI2V720PConfig
- fastvideo.configs.pipelines.wan.WanT2V480PConfig
- fastvideo.configs.pipelines.wan.WanT2V720PConfig
bsa_cdf_threshold:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
bsa_chunk_k:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
bsa_chunk_q:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
bsa_params:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
bsa_sparsity:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
enable_bsa:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
enable_kv_cache:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
enhance_hf:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
offload_kv_cache:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
t_thresh:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
use_distill:
sources: [fastvideo.configs.pipelines.longcat.LongCatT2V480PConfig, fastvideo.configs.pipelines.longcat.LongCatT2V704PConfig]
scheduler_arch:
sources:
- fastvideo.configs.pipelines.sd35.SD35Config
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
text_encoder_archs:
sources:
- fastvideo.configs.pipelines.sd35.SD35Config
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
tokenizer_archs:
sources:
- fastvideo.configs.pipelines.sd35.SD35Config
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
transformer_arch:
sources:
- fastvideo.configs.pipelines.sd35.SD35Config
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
vae_arch:
sources:
- fastvideo.configs.pipelines.sd35.SD35Config
- fastvideo.configs.pipelines.zimage.ZImagePipelineConfig
expand_timesteps:
sources:
- fastvideo.configs.pipelines.wan.FastWan2_2_TI2V_5B_Config
@@ -494,8 +454,6 @@ surfaces:
image_conditioning: "MagiHuman preset variant marker for reference-image conditioning."
pdd_step_indices: "MiniMax-H3 PDD fine-grid partition; resolve_checkpoint_settings sets it from fastvideo_inference.json."
ref_audio_offset: "MagiHuman internal data-proxy audio alignment offset."
scheduler_sigma_min: "Z-Image scheduler parity invariant; not part of the public typed inference API."
scheduler_use_reference_discrete_timesteps: "Z-Image scheduler parity invariant; not part of the public typed inference API."
sr_local_attn_layers: "MagiHuman SR internal sparse-attention layer selection."
text_offset: "MagiHuman internal data-proxy text alignment offset."
vae_stride: "MagiHuman internal VAE/data-proxy stride setting."
@@ -527,8 +485,6 @@ surfaces:
pose: request.inputs.pose
c2ws_plucker_emb: request.inputs.c2ws_plucker_emb
action_path: request.inputs.action_path
refine_from: request.inputs.refine_from
stage1_video: request.inputs.stage1_video
prompt: request.prompt
negative_prompt: request.negative_prompt
prompt_path: request.inputs.prompt_path
@@ -548,8 +504,6 @@ surfaces:
guidance_scale: request.sampling.guidance_scale
batch_cfg: request.sampling.batch_cfg
guidance_scale_2: request.sampling.guidance_scale_2
cfg_normalization: request.sampling.cfg_normalization
cfg_truncation: request.sampling.cfg_truncation
guidance_rescale: request.sampling.guidance_rescale
use_embedded_guidance: request.sampling.use_embedded_guidance
true_cfg_scale: request.sampling.true_cfg_scale
@@ -573,19 +527,12 @@ surfaces:
return_continuation_state: request.output.return_state
preset_owned:
t_thresh: request.stage_overrides.refine.t_thresh
spatial_refine_only: request.stage_overrides.refine.spatial_refine_only
num_cond_frames: request.stage_overrides.refine.num_cond_frames
trajectory_type: request.extensions.gen3c.trajectory_type
movement_distance: request.extensions.gen3c.movement_distance
camera_rotation: request.extensions.gen3c.camera_rotation
prompt_attention_mask: request.extensions.hyworld.prompt_attention_mask
negative_attention_mask: request.extensions.hyworld.negative_attention_mask
camera_states: request.extensions.hunyuangamecraft.camera_states
camera_trajectory: request.extensions.hunyuangamecraft.camera_trajectory
action_list: request.extensions.hunyuangamecraft.action_list
action_speed_list: request.extensions.hunyuangamecraft.action_speed_list
gt_latents: request.extensions.hunyuangamecraft.gt_latents
conditioning_mask: request.extensions.hunyuangamecraft.conditioning_mask
ltx2_cfg_scale_video: request.extensions.ltx2.cfg_scale_video
ltx2_cfg_scale_audio: request.extensions.ltx2.cfg_scale_audio
ltx2_modality_scale_video: request.extensions.ltx2.modality_scale_video
-4
View File
@@ -38,18 +38,14 @@ COOKBOOK_SOURCE_ROOTS = (
# (e.g. black-forest-labs/FLUX.1-dev) and are grouped for documentation only.
COOKBOOK_FAMILIES = {
"wan",
"turbodiffusion",
"ltx2",
"hunyuan",
"cosmos",
"kandinsky5",
"kandinsky6",
"flux",
"glm_image",
"zimage",
"sd35",
"minimax_h3",
"longcat",
"stable_audio",
"mmaudio",
"matrixgame",
+1 -3
View File
@@ -232,8 +232,7 @@ Dataclass carrying all pipeline state between stages. Key field groups:
- **Scheduler**: `timesteps`, `num_inference_steps`, `guidance_scale`,
`sigmas`.
- **Task-specific**: `mouse_cond`/`keyboard_cond` (Matrix-Game 2.0), `pose`
(HYWorld), `camera_states` (GameCraft), `c2ws_plucker_emb`
(LingBotWorld).
(HYWorld), `c2ws_plucker_emb` (LingBotWorld).
- **Output**: `output: Tensor | None`.
- **Logging**: `logging_info: PipelineLoggingInfo`.
@@ -256,7 +255,6 @@ Standard stages (typical execution order):
| `DecodingStage` | `stages/decoding.py` | Decodes latents to video via VAE |
Specialized variants: `CausalDenoisingStage`, `LTX2DenoisingStage`,
`LongCatDenoisingStage`, `GameCraftDenoisingStage`,
`HYWorldDenoisingStage`, `MatrixGame2CausalDenoisingStage`,
`SRDenoisingStage`, `LTX2AudioDecodingStage`, `SD35ConditioningStage`,
`LTX2TextEncodingStage`, `LTX2LatentPreparationStage`.
+2 -2
View File
@@ -168,8 +168,8 @@ and logs a warning if the flag is set.
Deferral is opt-in per pipeline. Releasing a component and loading it again is
only safe when nothing outside the loader has changed it, and two common habits
break that without raising: mutating a component after load, as LongCat does
when it enables block-sparse attention, and reading a component's attributes
break that without raising: mutating a component after load, such as enabling
a sparse-attention mode in `initialize_pipeline`, and reading a component's attributes
while stages are built, as the shared denoising stage does to pick an attention
backend. A pipeline therefore lists the components it has checked in
`_lazy_module_names`, which is empty in the base class. MiniMax-H3 opts in. On
+21 -43
View File
@@ -31,14 +31,10 @@ column links a runnable script in `examples/inference/basic/` where one exists.
| cosmos | `nvidia/Cosmos-Predict2-2B-Video2World` | T2V | — |
| cosmos25 | `KyleShao/Cosmos-Predict2.5-2B-Diffusers` | T2V | [basic_cosmos2_5_t2w.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_cosmos2_5_t2w.py) |
| cosmos25 | `nvidia/Cosmos-Predict2.5-14B` | T2V | [basic_cosmos2_5_t2w.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_cosmos2_5_t2w.py) |
| dreamx_world | `FastVideo/DreamX-World-5B-Cam-Diffusers` | I2V | [basic_dreamx_world.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_dreamx_world.py) |
| dreamx_world | `FastVideo/DreamX-World-5B-Diffusers` | I2V | [basic_dreamx_world.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_dreamx_world.py) |
| flux | `black-forest-labs/FLUX.1-dev` | T2I | [basic_flux_dev.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_flux_dev.py) |
| flux2 | `black-forest-labs/FLUX.2-klein-4B`<br>`black-forest-labs/FLUX.2-klein-9B` | T2I | [basic_flux2_klein.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_flux2_klein.py) |
| flux2 | `black-forest-labs/FLUX.2-dev` | T2I | [basic_flux2.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_flux2.py) |
| gamecraft | `FastVideo/HunyuanGameCraft-Diffusers` | I2V | [basic_gamecraft.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_gamecraft.py) |
| gen3c | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | T2V | [basic_gen3c.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_gen3c.py) |
| glm_image | `zai-org/GLM-Image` | T2I | [basic_glm_image.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_glm_image.py) |
| hunyuan | `hunyuanvideo-community/HunyuanVideo` | T2V | — |
| hunyuan | `FastVideo/FastHunyuan-diffusers` | T2V | — |
| hunyuan15 | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | T2V | [basic_hy15.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_hy15.py) |
@@ -62,9 +58,6 @@ column links a runnable script in `examples/inference/basic/` where one exists.
| lingbotworld | `FastVideo/LingBot-World-Base-Cam-Diffusers` | I2V | [basic_lingbotworld_base_cam.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_lingbotworld_base_cam.py) |
| lingbotworld2 | `robbyant/lingbot-world-v2-14b-causal-fast` | I2V | [basic_lingbotworld2_causal_fast.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_lingbotworld2_causal_fast.py) |
| lingbotworld_fast | `FastVideo/LingBot-World-Fast-Diffusers` | I2V | [basic_lingbotworld_fast.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_lingbotworld_fast.py) |
| longcat | `FastVideo/LongCat-Video-T2V-Diffusers` | T2V | [basic_longcat_t2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_longcat_t2v.py) |
| longcat | `FastVideo/LongCat-Video-I2V-Diffusers` | I2V | [basic_longcat_i2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_longcat_i2v.py) |
| longcat | `FastVideo/LongCat-Video-VC-Diffusers` | — | [basic_longcat_vc.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_longcat_vc.py) |
| ltx2 | `FastVideo/LTX2-Distilled-Diffusers`<br>`FastVideo/LTX2.3-Distilled-Diffusers`<br>`FastVideo/LTX-2.3-Distilled-Diffusers` | T2V | [basic_ltx2_distilled.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_distilled.py) |
| ltx2 | `FastVideo/LTX-2.5-Distilled-Diffusers` | T2V, I2V | [T2AV](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_5_t2av.py)<br>[I2AV](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_5_i2av.py) |
| ltx2 | `FastVideo/LTX-2.5-Dev-Diffusers` | T2V, I2V | [T2AV](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_5_t2av.py)<br>[I2AV](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_ltx2_5_i2av.py) |
@@ -77,9 +70,6 @@ column links a runnable script in `examples/inference/basic/` where one exists.
| sd35 | `stabilityai/stable-diffusion-3.5-medium` | T2I | [basic_sd35_t2i.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_sd35_t2i.py) |
| stable_audio | `FastVideo/stable-audio-open-1.0-Diffusers` | T2V | [basic_stable_audio.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_stable_audio.py) |
| stable_audio | `FastVideo/stable-audio-open-small-Diffusers` | T2V | [basic_stable_audio_small.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_stable_audio_small.py) |
| turbodiffusion | `loayrashid/TurboWan2.1-T2V-1.3B-Diffusers` | T2V | [basic_turbodiffusion.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_turbodiffusion.py) |
| turbodiffusion | `loayrashid/TurboWan2.1-T2V-14B-Diffusers` | T2V | [basic_turbodiffusion_14b.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_turbodiffusion_14b.py) |
| turbodiffusion | `loayrashid/TurboWan2.2-I2V-A14B-Diffusers` | I2V | [basic_turbodiffusion_i2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_turbodiffusion_i2v.py) |
| wan | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | T2V | [basic.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic.py) |
| wan | `Wan-AI/Wan2.1-T2V-14B-Diffusers`<br>`FastVideo/Wan2.1-VSA-T2V-14B-720P-Diffusers` | T2V | — |
| wan | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | I2V | — |
@@ -97,7 +87,6 @@ column links a runnable script in `examples/inference/basic/` where one exists.
| wan | `wlsaidhi/SFWan2.1-T2V-1.3B-Diffusers` | T2V | [basic_self_forcing_causal.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_self_forcing_causal.py) |
| wan | `rand0nmr/SFWan2.2-T2V-A14B-Diffusers` | T2V | [basic_self_forcing_causal_wan2_2_t2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_self_forcing_causal_wan2_2_t2v.py) |
| wan | `FastVideo/SFWan2.2-I2V-A14B-Preview-Diffusers` | I2V | [basic_self_forcing_causal_wan2_2_i2v.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_self_forcing_causal_wan2_2_i2v.py) |
| zimage | `Tongyi-MAI/Z-Image-Turbo` | T2I | [basic_zimage.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_zimage.py) |
**Note (stable_audio)**: the Stable Audio Open pipelines generate audio
(`StableAudioT2AConfig` / `StableAudioOpenSmallConfig`); they are registered
@@ -184,33 +173,27 @@ optimizations: absence means **untested**, not incompatible.
}
</style>
| Model Name | HuggingFace Model ID | Resolutions | TeaCache | Sliding Tile Attn (Legacy Branch) | Sage Attn | VSA | BSA |
|------------|---------------------|-------------|----------|-------------------|-----------|-----|-----|
| FastWan2.1 T2V 1.3B | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ✅ | ⭕ |
| FastWan2.2 TI2V 5B Full Attn | `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | 720P | ⭕ | ⭕ | ⭕ | ✅ | ⭕ |
| Wan2.2 TI2V 5B | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | 720P | ⭕ | ⭕ | ✅ | ⭕ | ⭕ |
| DreamX-World 5B Cam | `FastVideo/DreamX-World-5B-Cam-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| DreamX-World 5B AR | `FastVideo/DreamX-World-5B-Diffusers` | 704px1280p | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| Lucy Edit Dev 5B*** | `decart-ai/Lucy-Edit-Dev` | 480P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| Wan2.2 S2V 14B**** | `FastVideo/Wan2.2-S2V-14B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| Wan2.2 T2V A14B | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 480P<br>720P | ❌ | ❌ | ✅ | ⭕ | ⭕ |
| Wan2.2 I2V A14B | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | 480P<br>720P | ❌ | ❌ | ✅ | ⭕ | ⭕ |
| Wan2.2 Animate 14B | `Wan-AI/Wan2.2-Animate-14B-Diffusers` | 720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| HunyuanVideo | `hunyuanvideo-community/HunyuanVideo` | 720px1280p<br>544px960p | ❌ | ✅ | ✅ | ⭕ | ⭕ |
| FastHunyuan | `FastVideo/FastHunyuan-diffusers` | 720px1280p<br>544px960p | ❌ | ✅ | ✅ | ⭕ | ⭕ |
| Wan2.1 T2V 1.3B | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | 480P | ✅ | ✅ | ✅ | ⭕ | ⭕ |
| Wan2.1 T2V 14B | `Wan-AI/Wan2.1-T2V-14B-Diffusers` | 480P, 720P | ✅ | ✅ | ✅ | ⭕ | ⭕ |
| Wan2.1 I2V 480P | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | 480P | ✅ | ✅ | ✅ | ⭕ | ⭕ |
| Wan2.1 I2V 720P | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers` | 720P | ✅ | ✅ | ✅ | ⭕ | ⭕ |
| TurboWan2.1 T2V 1.3B | `loayrashid/TurboWan2.1-T2V-1.3B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| TurboWan2.1 T2V 14B | `loayrashid/TurboWan2.1-T2V-14B-Diffusers` | 480P, 720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| TurboWan2.2 I2V A14B | `loayrashid/TurboWan2.2-I2V-A14B-Diffusers` | 480P<br>720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| LongCat T2V 13.6B | `FastVideo/LongCat-Video-T2V-Diffusers` | 480P<br>720P | ❌ | ❌ | ❌ | ⭕ | ✅ |
| Matrix Game 2.0 Base Distilled | `FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| Matrix Game 2.0 GTA Distilled | `FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| Matrix Game 2.0 TempleRun Distilled | `FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| Matrix Game 3.0 Base Distilled | `FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers` | 720x1280 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
| GEN3C Cosmos 7B | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | 704px1280p | ❌ | ❌ | ❌ | ⭕ | ⭕ |
| Model Name | HuggingFace Model ID | Resolutions | TeaCache | Sliding Tile Attn (Legacy Branch) | Sage Attn | VSA |
|------------|---------------------|-------------|----------|-------------------|-----------|-----|
| FastWan2.1 T2V 1.3B | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ✅ |
| FastWan2.2 TI2V 5B Full Attn | `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | 720P | ⭕ | ⭕ | ⭕ | ✅ |
| Wan2.2 TI2V 5B | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | 720P | ⭕ | ⭕ | ✅ | ⭕ |
| Lucy Edit Dev 5B*** | `decart-ai/Lucy-Edit-Dev` | 480P | ⭕ | ⭕ | ⭕ | ⭕ |
| Wan2.2 S2V 14B**** | `FastVideo/Wan2.2-S2V-14B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ⭕ |
| Wan2.2 T2V A14B | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 480P<br>720P | ❌ | ❌ | ✅ | ⭕ |
| Wan2.2 I2V A14B | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | 480P<br>720P | ❌ | ❌ | ✅ | ⭕ |
| Wan2.2 Animate 14B | `Wan-AI/Wan2.2-Animate-14B-Diffusers` | 720P | ⭕ | ⭕ | ⭕ | ⭕ |
| HunyuanVideo | `hunyuanvideo-community/HunyuanVideo` | 720px1280p<br>544px960p | ❌ | ✅ | ✅ | ⭕ |
| FastHunyuan | `FastVideo/FastHunyuan-diffusers` | 720px1280p<br>544px960p | ❌ | ✅ | ✅ | ⭕ |
| Wan2.1 T2V 1.3B | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | 480P | ✅ | ✅ | ✅ | ⭕ |
| Wan2.1 T2V 14B | `Wan-AI/Wan2.1-T2V-14B-Diffusers` | 480P, 720P | ✅ | ✅ | ✅ | ⭕ |
| Wan2.1 I2V 480P | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | 480P | ✅ | ✅ | ✅ | ⭕ |
| Wan2.1 I2V 720P | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers` | 720P | ✅ | ✅ | ✅ | ⭕ |
| Matrix Game 2.0 Base Distilled | `FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ |
| Matrix Game 2.0 GTA Distilled | `FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ |
| Matrix Game 2.0 TempleRun Distilled | `FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ |
| Matrix Game 3.0 Base Distilled | `FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers` | 720x1280 | ⭕ | ⭕ | ⭕ | ⭕ |
| GEN3C Cosmos 7B | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | 704px1280p | ❌ | ❌ | ❌ | ⭕ |
## Apple Silicon native runtime
@@ -284,11 +267,6 @@ listed under [Special requirements](#special-requirements).
https://github.com/hao-ai-lab/FastVideo/tree/sta_do_not_delete
- STA currently requires Hopper GPUs (H100s).
### TurboWan2.1 (TurboDiffusion)
- Uses TurboDiffusionPipeline with RCM scheduler for 1-4 step generation
- Requires SLA attention backend: `export FASTVIDEO_ATTENTION_BACKEND=SLA_ATTN`
- Uses `guidance_scale=1.0` (no classifier-free guidance)
### Matrix Game 2.0
- Image-to-video game world models with keyboard/mouse control input
- Three variants available: Base (universal), GTA, and TempleRun
@@ -1,61 +0,0 @@
import os
from fastvideo import VideoGenerator
OUTPUT_PATH = os.getenv("DREAMX_WORLD_OUTPUT_PATH", "video_samples_dreamx_world")
def _env_int(name: str, default: int) -> int:
return int(os.getenv(name, str(default)))
def _env_float(name: str, default: float) -> float:
return float(os.getenv(name, str(default)))
def main():
model_name = os.getenv("DREAMX_WORLD_MODEL_DIR", "FastVideo/DreamX-World-5B-Cam-Diffusers")
generator = VideoGenerator.from_pretrained(
model_name,
num_gpus=1,
use_fsdp_inference=False,
dit_cpu_offload=False,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
override_pipeline_cls_name="DreamXWorldPipeline",
)
prompt = os.getenv(
"DREAMX_WORLD_PROMPT",
"A cinematic first-person drive through a futuristic coastal city at "
"sunrise, reflective glass towers, clean streets, soft volumetric light.",
)
image_path = os.getenv(
"DREAMX_WORLD_IMAGE_PATH",
"https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/wan_i2v_input.JPG",
)
kwargs = {
"output_path": OUTPUT_PATH,
"save_video": os.getenv("DREAMX_WORLD_SAVE_VIDEO", "1") != "0",
"height": _env_int("DREAMX_WORLD_HEIGHT", 480),
"width": _env_int("DREAMX_WORLD_WIDTH", 832),
"num_frames": _env_int("DREAMX_WORLD_NUM_FRAMES", 161),
"num_inference_steps": _env_int("DREAMX_WORLD_STEPS", 30),
"guidance_scale": _env_float("DREAMX_WORLD_GUIDANCE", 5.0),
"action_list": os.getenv("DREAMX_WORLD_ACTIONS", "w,d,w").split(","),
"action_speed_list":
[float(value) for value in os.getenv("DREAMX_WORLD_ACTION_SPEEDS", "4.0,2.0,4.0").split(",")],
}
if image_path:
kwargs["image_path"] = image_path
try:
generator.generate_video(prompt, **kwargs)
finally:
generator.shutdown()
if __name__ == "__main__":
main()
-118
View File
@@ -1,118 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""
Basic inference script for HunyuanGameCraft video generation.
HunyuanGameCraft generates game-like videos with camera/action control.
It takes an optional image input and generates video with camera motion
based on simple action commands (forward, left, right, backward, rotations).
Available actions:
- forward (w): Move camera forward
- backward (s): Move camera backward
- left (a): Move camera left (strafe)
- right (d): Move camera right (strafe)
- left_rot: Rotate camera left (pan)
- right_rot: Rotate camera right (pan)
- up_rot: Rotate camera up (tilt)
- down_rot: Rotate camera down (tilt)
T2V vs I2V:
- Default: I2V (uses a default reference image). Set GAMECRAFT_I2V_IMAGE to a
URL or path to use a different image.
- T2V only (no reference image): run with GAMECRAFT_I2V_IMAGE= (empty).
"""
import os
import torch
from fastvideo import VideoGenerator
from fastvideo.models.camera import create_camera_trajectory
# Model configuration (use GAMECRAFT_MODEL_PATH for local weights)
MODEL_PATH = os.environ.get("GAMECRAFT_MODEL_PATH", "FastVideo/HunyuanGameCraft-Diffusers")
# Default prompts for demo
DEFAULT_PROMPTS = {
"village":
"A charming medieval village with cobblestone streets, thatched-roof houses, and vibrant flower gardens under a bright blue sky.",
"temple":
"A majestic ancient temple stands under a clear blue sky, its grandeur highlighted by towering Doric columns and intricate architectural details.",
"forest":
"A lush green forest with tall trees, dappled sunlight filtering through the leaves, and a winding dirt path.",
"beach": "A tropical beach with crystal clear turquoise water, white sand, and palm trees swaying in the breeze.",
}
# I2V: default reference image (URL). Can override with a local path.
DEFAULT_I2V_IMAGE_URL = ("https://huggingface.co/datasets/huggingface/documentation-images/"
"resolve/main/diffusers/astronaut.jpg")
DEFAULT_I2V_PROMPT = ("An astronaut hatching from an egg, on the surface of the moon, "
"the darkness and depth of space realised in the background.")
OUTPUT_PATH = "video_samples_gamecraft"
def main():
# Initialize generator
# FastVideo will automatically download weights from HuggingFace
generator = VideoGenerator.from_pretrained(
MODEL_PATH,
num_gpus=1,
use_fsdp_inference=True,
dit_cpu_offload=True,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=True,
)
# Video parameters
height = 704
width = 1280
num_frames = 33
action = "forward"
action_speed = 0.2
# Create camera trajectory (Plücker coordinates)
camera_states = create_camera_trajectory(
action=action,
height=height,
width=width,
num_frames=num_frames,
action_speed=action_speed,
dtype=torch.bfloat16,
)
print(f"Camera states shape: {camera_states.shape}")
# I2V vs T2V: unset GAMECRAFT_I2V_IMAGE -> I2V (default image). Set to "" -> T2V.
env_image = os.environ.get("GAMECRAFT_I2V_IMAGE")
if env_image is None:
image_path = DEFAULT_I2V_IMAGE_URL # default: I2V
elif env_image.strip() == "":
image_path = None # T2V
else:
image_path = env_image.strip() # I2V with given URL/path
is_i2v = image_path is not None
prompt = DEFAULT_I2V_PROMPT if is_i2v else DEFAULT_PROMPTS["temple"]
print(f"Mode: {'I2V' if is_i2v else 'T2V'}, prompt: {prompt[:60]}...")
gen_kw = dict(
prompt=prompt,
negative_prompt="",
camera_states=camera_states,
height=height,
width=width,
num_frames=num_frames,
num_inference_steps=50,
guidance_scale=6.0,
seed=42,
fps=24,
output_path=OUTPUT_PATH,
save_video=True,
)
if is_i2v:
gen_kw["image_path"] = image_path
generator.generate_video(**gen_kw)
if __name__ == "__main__":
main()
-107
View File
@@ -1,107 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""Run GLM-Image text-to-image generation through FastVideo.
User story:
"I have the HF `zai-org/GLM-Image` checkpoint and want a minimal
text-to-image generation command, saved as a PNG."
"""
import argparse
from pathlib import Path
from PIL import Image
from fastvideo import VideoGenerator
from fastvideo.api import (
EngineConfig,
GenerationRequest,
GeneratorConfig,
OutputConfig,
ParallelismConfig,
PipelineSelection,
SamplingConfig,
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run GLM-Image text-to-image generation.")
parser.add_argument(
"--model-path",
default="zai-org/GLM-Image",
help="HF id or local diffusers-format GLM-Image weights directory.",
)
parser.add_argument(
"--output",
default="image_output/landscape.png",
help="Output PNG path.",
)
parser.add_argument(
"--prompt",
default=("A beautiful landscape photography with rolling hills, "
"a winding river, and a vibrant sunset in the background. "
"Warm golden light, photorealistic style."),
help="Text prompt.",
)
parser.add_argument("--height", type=int, default=1024)
parser.add_argument("--width", type=int, default=1024)
parser.add_argument("--steps", type=int, default=50)
parser.add_argument("--guidance-scale", type=float, default=1.5)
parser.add_argument("--seed", type=int, default=1024)
parser.add_argument("--num-gpus", type=int, default=1)
parser.add_argument("--tp-size", type=int, default=None)
parser.add_argument("--sp-size", type=int, default=None)
return parser.parse_args()
def main() -> None:
args = parse_args()
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
tp_size = args.tp_size if args.tp_size is not None else (args.num_gpus if args.num_gpus > 1 else 1)
sp_size = args.sp_size if args.sp_size is not None else (1 if args.num_gpus > 1 else args.num_gpus)
# GLM-Image needs trust_remote_code for its AR encoder; offload and the
# pipeline class come from the model's registered defaults — don't override.
generator_config = GeneratorConfig(
model_path=args.model_path,
trust_remote_code=True,
engine=EngineConfig(
num_gpus=args.num_gpus,
parallelism=ParallelismConfig(tp_size=tp_size, sp_size=sp_size),
),
pipeline=PipelineSelection(workload_type="t2i"),
)
generator = VideoGenerator.from_config(generator_config)
try:
request = GenerationRequest(
prompt=args.prompt,
sampling=SamplingConfig(
height=args.height,
width=args.width,
num_frames=1,
fps=1,
num_inference_steps=args.steps,
guidance_scale=args.guidance_scale,
seed=args.seed,
),
output=OutputConfig(
output_path=str(output.parent),
save_video=False,
return_frames=True,
),
)
result = generator.generate(request)
if isinstance(result, list):
result = result[0]
frames = result.frames
if frames is not None and len(frames):
Image.fromarray(frames[0]).save(output)
print(f"Saved image to {output}")
finally:
generator.shutdown()
if __name__ == "__main__":
main()
@@ -1,204 +0,0 @@
"""
LongCat Image-to-Video (I2V) Example Script
This script demonstrates LongCat I2V inference using the FastVideo Python API.
LongCat I2V takes an input image and generates a video from it.
It runs both basic generation (50 steps) and distill+refine generation
(16 steps distill + 50 steps refinement to 720p with BSA).
Usage:
python examples/inference/basic/basic_longcat_i2v.py
Note:
Refinement uses 768x768 dimensions where latent (48x48) is divisible by 8,
compatible with BSA chunks [4, 4, 8].
"""
import glob
import os
from fastvideo import VideoGenerator
# Common prompts and settings matching the shell script examples
PROMPT = ("A woman sits at a wooden table by the window in a cozy café. She reaches out "
"with her right hand, picks up the white coffee cup from the saucer, and gently "
"brings it to her lips to take a sip. After drinking, she places the cup back on "
"the table and looks out the window, enjoying the peaceful atmosphere.")
NEGATIVE_PROMPT = ("Bright tones, overexposed, static, blurred details, subtitles, style, works, "
"paintings, images, static, overall gray, worst quality, low quality, JPEG compression "
"residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, "
"deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, "
"three legs, many people in the background, walking backwards")
# Input image path
IMAGE_PATH = "assets/girl.png"
SEED = 42
def basic_generation():
"""
Run basic LongCat I2V generation (50 steps at 480p).
This uses the full 50-step denoising process for highest quality.
"""
print("=" * 60)
print("LongCat I2V: Basic Generation (50 steps, 480p)")
print("=" * 60)
generator = VideoGenerator.from_pretrained(
"FastVideo/LongCat-Video-I2V-Diffusers",
num_gpus=1,
use_fsdp_inference=False, # set to True if GPU is out of memory
dit_cpu_offload=False,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
enable_bsa=False,
)
output_path = "outputs_video/longcat_i2v_basic"
generator.generate_video(
prompt=PROMPT,
negative_prompt=NEGATIVE_PROMPT,
image_path=IMAGE_PATH,
output_path=output_path,
save_video=True,
height=480,
width=480, # Square
num_frames=93,
num_inference_steps=50,
fps=15,
guidance_scale=4.0,
seed=SEED,
)
print(f"\nBasic generation complete! Video saved to: {output_path}")
generator.shutdown()
def distill_refine_generation():
"""
Run LongCat I2V with distill+refine pipeline (16 steps + refinement to 768p).
This uses the distilled LoRA for fast 480p generation (16 steps),
then refines to 768p using the refinement LoRA with BSA enabled.
"""
print("\n" + "=" * 60)
print("LongCat I2V: Distill + Refine Pipeline")
print("=" * 60)
# Stage 1: Distilled generation (16 steps at 480p)
print("\n[Stage 1] Distilled generation (16 steps, 480p)")
print("-" * 40)
generator = VideoGenerator.from_pretrained(
"FastVideo/LongCat-Video-I2V-Diffusers",
num_gpus=1,
use_fsdp_inference=True,
dit_cpu_offload=False,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
enable_bsa=False,
lora_path="FastVideo/LongCat-Video-T2V-Distilled-LoRA",
lora_nickname="distilled",
)
distill_output_path = "outputs_video/longcat_i2v_distill"
generator.generate_video(
prompt=PROMPT,
negative_prompt=NEGATIVE_PROMPT,
image_path=IMAGE_PATH,
output_path=distill_output_path,
save_video=True,
height=480,
width=480, # Square
num_frames=93,
num_inference_steps=16,
fps=15,
guidance_scale=1.0,
seed=SEED,
)
print(f"Distilled generation complete! Video saved to: {distill_output_path}")
generator.shutdown()
# Stage 2: Refinement (480p -> 768p)
print("\n[Stage 2] Refinement (480p -> 768p with BSA)")
print("-" * 40)
# Find the actual saved video file from stage 1
video_files = glob.glob(os.path.join(distill_output_path, "*.mp4"))
if not video_files:
raise FileNotFoundError(f"No video file found in {distill_output_path}")
# Use the most recently created video file
distill_video_path = max(video_files, key=os.path.getmtime)
print(f"Using stage 1 video: {distill_video_path}")
# Create a new generator with refinement LoRA and BSA enabled
# Note: Refinement uses the T2V model (not I2V) since it's upscaling the generated video
# For BSA [4, 4, 8]: latent must be divisible by 8
# 768x768: latent 48x48, 48%8=0 ✓
refine_generator = VideoGenerator.from_pretrained(
"FastVideo/LongCat-Video-T2V-Diffusers",
num_gpus=1,
use_fsdp_inference=True,
dit_cpu_offload=True,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
enable_bsa=True,
bsa_sparsity=0.875,
bsa_chunk_q=[4, 4, 4],
bsa_chunk_k=[4, 4, 4],
lora_path="FastVideo/LongCat-Video-T2V-Refinement-LoRA",
lora_nickname="refinement",
)
refine_output_path = "outputs_video/longcat_i2v_refine_720p"
refine_generator.generate_video(
prompt=PROMPT,
negative_prompt=NEGATIVE_PROMPT,
output_path=refine_output_path,
save_video=True,
refine_from=distill_video_path,
t_thresh=0.5,
spatial_refine_only=False,
num_cond_frames=0,
height=720,
width=720,
num_inference_steps=50,
fps=30,
guidance_scale=1.0,
seed=SEED,
)
print(f"Refinement complete! Video saved to: {refine_output_path}")
refine_generator.shutdown()
def main():
"""Run both basic and distill+refine generation pipelines."""
print("\n" + "=" * 60)
print("LongCat Image-to-Video Example")
print("=" * 60 + "\n")
# Run basic generation
basic_generation()
# Run distill+refine pipeline
distill_refine_generation()
print("\n" + "=" * 60)
print("All generations complete!")
print("=" * 60)
if __name__ == "__main__":
main()
@@ -1,192 +0,0 @@
"""
LongCat Text-to-Video (T2V) Example Script
This script demonstrates LongCat T2V inference using the FastVideo Python API.
It runs both basic generation (50 steps) and distill+refine generation
(16 steps distill + 50 steps refinement to 720p).
Usage:
python examples/inference/basic/basic_longcat_t2v.py
"""
import glob
import os
from fastvideo import VideoGenerator
# Common prompts and settings matching the shell script examples
PROMPT = ("In a realistic photography style, a white boy around seven or eight years old "
"sits on a park bench, wearing a light blue T-shirt, denim shorts, and white sneakers. "
"He holds an ice cream cone with vanilla and chocolate flavors, and beside him is a "
"medium-sized golden Labrador. Smiling, the boy offers the ice cream to the dog, "
"who eagerly licks it with its tongue. The sun is shining brightly, and the background "
"features a green lawn and several tall trees, creating a warm and loving scene.")
NEGATIVE_PROMPT = ("Bright tones, overexposed, static, blurred details, subtitles, style, works, "
"paintings, images, static, overall gray, worst quality, low quality, JPEG compression "
"residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, "
"deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, "
"three legs, many people in the background, walking backwards")
SEED = 42
def basic_generation():
"""
Run basic LongCat T2V generation (50 steps at 480p).
This uses the full 50-step denoising process for highest quality.
"""
print("=" * 60)
print("LongCat T2V: Basic Generation (50 steps, 480p)")
print("=" * 60)
generator = VideoGenerator.from_pretrained(
"FastVideo/LongCat-Video-T2V-Diffusers",
num_gpus=1,
use_fsdp_inference=False, # set to True if GPU is out of memory
dit_cpu_offload=False,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
enable_bsa=False,
)
output_path = "outputs_video/longcat_t2v_basic"
generator.generate_video(
prompt=PROMPT,
negative_prompt=NEGATIVE_PROMPT,
output_path=output_path,
save_video=True,
height=480,
width=832,
num_frames=93,
num_inference_steps=50,
fps=15,
guidance_scale=4.0,
seed=SEED,
)
print(f"\nBasic generation complete! Video saved to: {output_path}")
generator.shutdown()
def distill_refine_generation():
"""
Run LongCat T2V with distill+refine pipeline (16 steps + refinement to 720p).
This uses the distilled LoRA for fast 480p generation (16 steps),
then refines to 720p using the refinement LoRA with BSA enabled.
"""
print("\n" + "=" * 60)
print("LongCat T2V: Distill + Refine Pipeline")
print("=" * 60)
# Stage 1: Distilled generation (16 steps at 480p)
print("\n[Stage 1] Distilled generation (16 steps, 480p)")
print("-" * 40)
generator = VideoGenerator.from_pretrained(
"FastVideo/LongCat-Video-T2V-Diffusers",
num_gpus=1,
use_fsdp_inference=True,
dit_cpu_offload=False,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
enable_bsa=False,
lora_path="FastVideo/LongCat-Video-T2V-Distilled-LoRA",
lora_nickname="distilled",
)
distill_output_path = "outputs_video/longcat_t2v_distill"
generator.generate_video(
prompt=PROMPT,
negative_prompt=NEGATIVE_PROMPT,
output_path=distill_output_path,
save_video=True,
height=480,
width=832,
num_frames=93,
num_inference_steps=16,
fps=15,
guidance_scale=1.0,
seed=SEED,
)
print(f"Distilled generation complete! Video saved to: {distill_output_path}")
generator.shutdown()
# Stage 2: Refinement (480p -> 720p)
print("\n[Stage 2] Refinement (480p -> 720p with BSA)")
print("-" * 40)
# Find the actual saved video file from stage 1
video_files = glob.glob(os.path.join(distill_output_path, "*.mp4"))
if not video_files:
raise FileNotFoundError(f"No video file found in {distill_output_path}")
# Use the most recently created video file
distill_video_path = max(video_files, key=os.path.getmtime)
print(f"Using stage 1 video: {distill_video_path}")
# Create a new generator with refinement LoRA and BSA enabled
refine_generator = VideoGenerator.from_pretrained(
"FastVideo/LongCat-Video-T2V-Diffusers",
num_gpus=1,
use_fsdp_inference=True,
dit_cpu_offload=True,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
enable_bsa=True,
bsa_sparsity=0.875,
bsa_chunk_q=[4, 4, 8],
bsa_chunk_k=[4, 4, 8],
lora_path="FastVideo/LongCat-Video-T2V-Refinement-LoRA",
lora_nickname="refinement",
)
refine_output_path = "outputs_video/longcat_t2v_refine_720p"
refine_generator.generate_video(
prompt=PROMPT,
negative_prompt=NEGATIVE_PROMPT,
output_path=refine_output_path,
save_video=True,
refine_from=distill_video_path,
t_thresh=0.5,
spatial_refine_only=False,
num_cond_frames=0,
height=720,
width=1280,
num_inference_steps=50,
fps=30,
guidance_scale=1.0,
seed=SEED,
)
print(f"Refinement complete! Video saved to: {refine_output_path}")
refine_generator.shutdown()
def main():
"""Run both basic and distill+refine generation pipelines."""
print("\n" + "=" * 60)
print("LongCat Text-to-Video Example")
print("=" * 60 + "\n")
# Run basic generation
basic_generation()
# Run distill+refine pipeline
distill_refine_generation()
print("\n" + "=" * 60)
print("All generations complete!")
print("=" * 60)
if __name__ == "__main__":
main()
@@ -1,218 +0,0 @@
"""
LongCat Video Continuation (VC) Example Script
This script demonstrates LongCat VC inference using the FastVideo Python API.
LongCat VC takes an input video and generates a continuation of it.
It runs both basic generation (50 steps) and distill+refine generation
(16 steps distill + 50 steps refinement to 720p).
Usage:
python examples/inference/basic/basic_longcat_vc.py
Prerequisites:
- Ensure the input video exists at assets/motorcycle.mp4
(or provide your own video)
"""
import glob
import os
from fastvideo import VideoGenerator
# Common prompts and settings matching the shell script examples
PROMPT = ("A person rides a motorcycle along a long, straight road that stretches between "
"a body of water and a forested hillside. The rider steadily accelerates, keeping "
"the motorcycle centered between the guardrails, while the scenery passes by on "
"both sides. The video captures the journey from the rider's perspective, emphasizing "
"the sense of motion and adventure.")
NEGATIVE_PROMPT = ("Bright tones, overexposed, static, blurred details, subtitles, style, works, "
"paintings, images, static, overall gray, worst quality, low quality, JPEG compression "
"residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, "
"deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, "
"three legs, many people in the background, walking backwards")
# Input video path
VIDEO_PATH = "assets/motorcycle.mp4"
# Number of conditioning frames from the input video
NUM_COND_FRAMES = 13
SEED = 42
def basic_generation():
"""
Run basic LongCat VC generation (50 steps at 480p).
This uses the full 50-step denoising process for highest quality.
"""
print("=" * 60)
print("LongCat VC: Basic Generation (50 steps, 480p)")
print("=" * 60)
# Check if video exists
if not os.path.exists(VIDEO_PATH):
raise FileNotFoundError(f"Video not found at {VIDEO_PATH}. "
"Please provide a valid video path.")
generator = VideoGenerator.from_pretrained(
"FastVideo/LongCat-Video-VC-Diffusers",
num_gpus=1,
use_fsdp_inference=False, # set to True if GPU is out of memory
dit_cpu_offload=False,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
enable_bsa=False,
)
output_path = "outputs_video/longcat_vc_basic"
generator.generate_video(
prompt=PROMPT,
negative_prompt=NEGATIVE_PROMPT,
video_path=VIDEO_PATH,
num_cond_frames=NUM_COND_FRAMES,
output_path=output_path,
save_video=True,
height=480,
width=832,
num_frames=93,
num_inference_steps=50,
fps=15,
guidance_scale=4.0,
seed=SEED,
)
print(f"\nBasic generation complete! Video saved to: {output_path}")
generator.shutdown()
def distill_refine_generation():
"""
Run LongCat VC with distill+refine pipeline (16 steps + refinement to 720p).
This uses the distilled LoRA for fast 480p generation (16 steps),
then refines to 720p using the refinement LoRA with BSA enabled.
"""
print("\n" + "=" * 60)
print("LongCat VC: Distill + Refine Pipeline")
print("=" * 60)
# Check if video exists
if not os.path.exists(VIDEO_PATH):
raise FileNotFoundError(f"Video not found at {VIDEO_PATH}. "
"Please provide a valid video path.")
# Stage 1: Distilled generation (16 steps at 480p)
print("\n[Stage 1] Distilled generation (16 steps, 480p)")
print("-" * 40)
generator = VideoGenerator.from_pretrained(
"FastVideo/LongCat-Video-VC-Diffusers",
num_gpus=1,
use_fsdp_inference=True,
dit_cpu_offload=False,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
enable_bsa=False,
lora_path="FastVideo/LongCat-Video-T2V-Distilled-LoRA",
lora_nickname="distilled",
)
distill_output_path = "outputs_video/longcat_vc_distill"
generator.generate_video(
prompt=PROMPT,
negative_prompt=NEGATIVE_PROMPT,
video_path=VIDEO_PATH,
num_cond_frames=NUM_COND_FRAMES,
output_path=distill_output_path,
save_video=True,
height=480,
width=832,
num_frames=93,
num_inference_steps=16,
fps=15,
guidance_scale=1.0,
seed=SEED,
)
print(f"Distilled generation complete! Video saved to: {distill_output_path}")
generator.shutdown()
# Stage 2: Refinement (480p -> 720p)
print("\n[Stage 2] Refinement (480p -> 720p with BSA)")
print("-" * 40)
# Find the actual saved video file from stage 1
video_files = glob.glob(os.path.join(distill_output_path, "*.mp4"))
if not video_files:
raise FileNotFoundError(f"No video file found in {distill_output_path}")
# Use the most recently created video file
distill_video_path = max(video_files, key=os.path.getmtime)
print(f"Using stage 1 video: {distill_video_path}")
# Create a new generator with refinement LoRA and BSA enabled
# Note: Refinement uses the T2V model (not VC) since it's upscaling the generated video
refine_generator = VideoGenerator.from_pretrained(
"FastVideo/LongCat-Video-T2V-Diffusers",
num_gpus=1,
use_fsdp_inference=True,
dit_cpu_offload=True,
vae_cpu_offload=True,
text_encoder_cpu_offload=True,
pin_cpu_memory=False,
enable_bsa=True,
bsa_sparsity=0.875,
bsa_chunk_q=[4, 4, 8],
bsa_chunk_k=[4, 4, 8],
lora_path="FastVideo/LongCat-Video-T2V-Refinement-LoRA",
lora_nickname="refinement",
)
refine_output_path = "outputs_video/longcat_vc_refine_720p"
refine_generator.generate_video(
prompt=PROMPT,
negative_prompt=NEGATIVE_PROMPT,
output_path=refine_output_path,
save_video=True,
refine_from=distill_video_path,
t_thresh=0.5,
spatial_refine_only=False,
num_cond_frames=0, # For refinement, no conditioning frames
height=720,
width=1280,
num_inference_steps=50,
fps=30,
guidance_scale=1.0,
seed=SEED,
)
print(f"Refinement complete! Video saved to: {refine_output_path}")
refine_generator.shutdown()
def main():
"""Run both basic and distill+refine generation pipelines."""
print("\n" + "=" * 60)
print("LongCat Video Continuation Example")
print("=" * 60 + "\n")
# Run basic generation
basic_generation()
# Run distill+refine pipeline
distill_refine_generation()
print("\n" + "=" * 60)
print("All generations complete!")
print("=" * 60)
if __name__ == "__main__":
main()
@@ -1,51 +0,0 @@
import os
# Set SLA attention backend BEFORE fastvideo imports
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "SLA_ATTN"
from fastvideo import VideoGenerator
OUTPUT_PATH = "video_samples_turbodiffusion"
def main() -> None:
# TurboDiffusion: 1-4 step video generation using RCM scheduler + SLA attention
# FastVideo will automatically use TurboDiffusionPipeline when specified
generator = VideoGenerator.from_pretrained(
"loayrashid/TurboWan2.1-T2V-1.3B-Diffusers",
# FastVideo will automatically handle distributed setup
num_gpus=1,
use_fsdp_inference=False, # set to True if GPU is out of memory
# set to false if using RTX 4090
# pin_cpu_memory=False,
)
# Generate videos with the same simple API, regardless of GPU count
# TurboDiffusion defaults: guidance_scale=1.0 and num_inference_steps=4 (from config)
prompt = ("A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes "
"wide with interest. The playful yet serene atmosphere is complemented by soft "
"natural light filtering through the petals. Mid-shot, warm and cheerful tones.")
video = generator.generate_video(
prompt,
output_path=OUTPUT_PATH,
save_video=True,
seed=42,
)
# Generate another video with a different prompt, without reloading the model!
prompt2 = ("A majestic lion strides across the golden savanna, its powerful frame "
"glistening under the warm afternoon sun. The tall grass ripples gently in "
"the breeze, enhancing the lion's commanding presence. The tone is vibrant, "
"embodying the raw energy of the wild. Low angle, steady tracking shot, "
"cinematic.")
video2 = generator.generate_video(
prompt2,
output_path=OUTPUT_PATH,
save_video=True,
seed=42,
)
if __name__ == "__main__":
main()
@@ -1,45 +0,0 @@
import os
# Set SLA attention backend BEFORE fastvideo imports
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "SLA_ATTN"
from fastvideo import VideoGenerator
OUTPUT_PATH = "video_samples_turbodiffusion_14B"
def main() -> None:
# TurboDiffusion 14B: 1-4 step video generation using RCM scheduler + SLA attention
# FastVideo will automatically use TurboDiffusionPipeline when specified
generator = VideoGenerator.from_pretrained(
"loayrashid/TurboWan2.1-T2V-14B-Diffusers",
# 14B model needs more GPUs
num_gpus=2,
)
prompt = ("A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes "
"wide with interest. The playful yet serene atmosphere is complemented by soft "
"natural light filtering through the petals. Mid-shot, warm and cheerful tones.")
video = generator.generate_video(
prompt,
output_path=OUTPUT_PATH,
save_video=True,
seed=42,
)
# Generate another video with a different prompt, without reloading the model!
prompt2 = ("A majestic lion strides across the golden savanna, its powerful frame "
"glistening under the warm afternoon sun. The tall grass ripples gently in "
"the breeze, enhancing the lion's commanding presence. The tone is vibrant, "
"embodying the raw energy of the wild. Low angle, steady tracking shot, "
"cinematic.")
video2 = generator.generate_video(
prompt2,
output_path=OUTPUT_PATH,
save_video=True,
seed=42,
)
if __name__ == "__main__":
main()
@@ -1,38 +0,0 @@
import os
# Set SLA attention backend BEFORE fastvideo imports
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "SLA_ATTN"
from fastvideo import VideoGenerator
# Use local model path
MODEL_PATH = "loayrashid/TurboWan2.2-I2V-A14B-Diffusers"
OUTPUT_PATH = "video_samples_turbodiffusion_i2v"
def main() -> None:
# TurboDiffusion I2V: 1-4 step image-to-video generation
generator = VideoGenerator.from_pretrained(
MODEL_PATH,
num_gpus=2,
)
# Example prompt and image for I2V
prompt = (
"Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
)
# Use an example image path
image_path = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/wan_i2v_input.JPG"
video = generator.generate_video(
prompt,
image_path=image_path,
output_path=OUTPUT_PATH,
save_video=True,
seed=42,
)
if __name__ == "__main__":
main()
-94
View File
@@ -1,94 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""Run Z-Image-Turbo text-to-image generation through FastVideo.
User story:
"I want the official Z-Image-Turbo defaults and a deterministic PNG from
a local or Hugging Face checkpoint."
"""
import argparse
from pathlib import Path
from fastvideo import VideoGenerator
from fastvideo.api import (
EngineConfig,
GenerationRequest,
GeneratorConfig,
OutputConfig,
ParallelismConfig,
PipelineSelection,
SamplingConfig,
)
DEFAULT_PROMPT = (
"Young Chinese woman in red Hanfu, intricate embroidery. Impeccable makeup, red floral forehead pattern. "
"Elaborate high bun, golden phoenix headdress, red flowers, beads. Holds round folding fan with lady, trees, bird. "
"Neon lightning-bolt lamp (⚡️), bright yellow glow, above extended left palm. Soft-lit outdoor night background, "
"silhouetted tiered pagoda (西安大雁塔), blurred colorful distant lights.")
DEFAULT_REVISION = "f332072aa78be7aecdf3ee76d5c247082da564a6"
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run Z-Image-Turbo text-to-image generation.")
parser.add_argument("--model-path", default="Tongyi-MAI/Z-Image-Turbo")
parser.add_argument("--revision", default=DEFAULT_REVISION)
parser.add_argument("--output", default="outputs/zimage/zimage_turbo.png")
parser.add_argument("--prompt", default=DEFAULT_PROMPT)
parser.add_argument("--negative-prompt", default="")
parser.add_argument("--height", type=int, default=1024)
parser.add_argument("--width", type=int, default=1024)
parser.add_argument("--steps", type=int, default=8)
parser.add_argument("--guidance-scale", type=float, default=0.0)
parser.add_argument("--max-sequence-length", type=int, default=512)
parser.add_argument("--cfg-normalization", action=argparse.BooleanOptionalAction, default=False)
parser.add_argument("--cfg-truncation", type=float, default=1.0)
parser.add_argument("--seed", type=int, default=42)
return parser.parse_args()
def main() -> None:
args = parse_args()
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
generator = VideoGenerator.from_config(
GeneratorConfig(
model_path=args.model_path,
revision=args.revision,
engine=EngineConfig(
num_gpus=1,
parallelism=ParallelismConfig(tp_size=1, sp_size=1),
use_fsdp_inference=False,
),
# The model registry selects the native zimage_turbo preset.
pipeline=PipelineSelection(workload_type="t2i"),
))
try:
generator.generate(
GenerationRequest(
prompt=args.prompt,
negative_prompt=args.negative_prompt,
sampling=SamplingConfig(
height=args.height,
width=args.width,
num_frames=1,
fps=1,
num_inference_steps=args.steps,
guidance_scale=args.guidance_scale,
max_sequence_length=args.max_sequence_length,
cfg_normalization=args.cfg_normalization,
cfg_truncation=args.cfg_truncation,
seed=args.seed,
),
output=OutputConfig(
output_path=str(output),
save_video=True,
return_frames=False,
),
))
finally:
generator.shutdown()
if __name__ == "__main__":
main()
-120
View File
@@ -1,120 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""Run GLM-Image image-to-image (edit) generation through FastVideo.
User story:
"I have the HF `zai-org/GLM-Image` checkpoint and a condition image, and
want a minimal edit command (text + image -> edited image), saved as a PNG."
GLM-Image is a single unified pipeline: passing a condition image switches it
from text-to-image to the edit path (the condition enters the DiT via a KV-cache
write pass), so the generator config is identical to `basic_glm_image.py` — the
`inputs.pil_image` on the request is what selects the edit mode.
"""
import argparse
from pathlib import Path
from PIL import Image
from fastvideo import VideoGenerator
from fastvideo.api import (
EngineConfig,
GenerationRequest,
GeneratorConfig,
InputConfig,
OutputConfig,
ParallelismConfig,
PipelineSelection,
SamplingConfig,
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run GLM-Image image-to-image (edit) generation.")
parser.add_argument(
"--model-path",
default="zai-org/GLM-Image",
help="HF id or local diffusers-format GLM-Image weights directory.",
)
parser.add_argument(
"--image",
default="assets/images/couple.jpg",
help="Condition image to edit.",
)
parser.add_argument(
"--output",
default="image_output/edited.png",
help="Output PNG path.",
)
parser.add_argument(
"--prompt",
default="Change the background to a snowy mountain landscape at golden hour.",
help="Edit instruction.",
)
parser.add_argument("--height", type=int, default=1024)
parser.add_argument("--width", type=int, default=1024)
parser.add_argument("--steps", type=int, default=50)
parser.add_argument("--guidance-scale", type=float, default=1.5)
parser.add_argument("--seed", type=int, default=1024)
parser.add_argument("--num-gpus", type=int, default=1)
parser.add_argument("--tp-size", type=int, default=None)
parser.add_argument("--sp-size", type=int, default=None)
return parser.parse_args()
def main() -> None:
args = parse_args()
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
condition = Image.open(args.image).convert("RGB")
tp_size = args.tp_size if args.tp_size is not None else (args.num_gpus if args.num_gpus > 1 else 1)
sp_size = args.sp_size if args.sp_size is not None else (1 if args.num_gpus > 1 else args.num_gpus)
# GLM-Image needs trust_remote_code for its AR encoder; offload and the
# pipeline class come from the model's registered defaults — don't override.
# The pipeline is registered as t2i; passing inputs.pil_image below switches
# it to the edit path.
generator_config = GeneratorConfig(
model_path=args.model_path,
trust_remote_code=True,
engine=EngineConfig(
num_gpus=args.num_gpus,
parallelism=ParallelismConfig(tp_size=tp_size, sp_size=sp_size),
),
pipeline=PipelineSelection(workload_type="t2i"),
)
generator = VideoGenerator.from_config(generator_config)
try:
request = GenerationRequest(
prompt=args.prompt,
inputs=InputConfig(pil_image=condition),
sampling=SamplingConfig(
height=args.height,
width=args.width,
num_frames=1,
fps=1,
num_inference_steps=args.steps,
guidance_scale=args.guidance_scale,
seed=args.seed,
),
output=OutputConfig(
output_path=str(output.parent),
save_video=False,
return_frames=True,
),
)
result = generator.generate(request)
if isinstance(result, list):
result = result[0]
frames = result.frames
if frames is not None and len(frames):
Image.fromarray(frames[0]).save(output)
print(f"Saved image to {output}")
finally:
generator.shutdown()
if __name__ == "__main__":
main()
@@ -1,71 +0,0 @@
# LongCat-Video T2V 13.6B bidirectional finetune.
models:
student:
_target_: fastvideo.train.models.longcat.LongCatModel
init_from: FastVideo/LongCat-Video-T2V-Diffusers
trainable: true
method:
_target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
training:
model_path: FastVideo/LongCat-Video-T2V-Diffusers
distributed:
num_gpus: 8
sp_size: 1
tp_size: 1
hsdp_replicate_dim: 1
hsdp_shard_dim: 8
data:
data_path: data/LongCat-Syn
dataloader_num_workers: 4
train_batch_size: 1
training_cfg_rate: 0.0
seed: 1000
num_latent_t: 20
num_height: 480
num_width: 848
num_frames: 77
optimizer:
learning_rate: 1.0e-6
betas: [0.9, 0.999]
weight_decay: 0.01
lr_scheduler: constant
lr_warmup_steps: 0
loop:
max_train_steps: 4000
gradient_accumulation_steps: 1
checkpoint:
output_dir: outputs/longcat_finetune
training_state_checkpointing_steps: 1000
checkpoints_total_limit: 3
tracker:
project_name: fastvideo
run_name: longcat_finetune
model:
enable_gradient_checkpointing_type: full
callbacks:
grad_clip:
_target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
max_grad_norm: 1.0
validation:
_target_: fastvideo.train.callbacks.validation.ValidationCallback
pipeline_target: fastvideo.pipelines.basic.longcat.longcat_pipeline.LongCatPipeline
dataset_file: data/validation_prompts.json
every_steps: 100
sampling_steps: [50]
guidance_scale: 5.0
pipeline:
# Match the released LongCat scheduler config. flow_shift=0.0 collapses
# FlowMatch training timesteps to zero in FastVideo's scheduler.
flow_shift: 12.0
@@ -1,89 +0,0 @@
# TurboWan 2.1 T2V 1.3B LoRA finetune with SLA attention.
#
# Notes:
# - TurboDiffusion checkpoints expect the SLA attention backend. The training
# entrypoint now auto-selects `SLA_ATTN` for model IDs containing
# `turbodiffusion` or `turbowan`, unless you override the environment
# variable manually.
# - This debug config reuses the existing crush-smol T2V parquet recipe. If you
# preprocess data with the official TurboWan recipe, update the temporal and
# spatial fields to match it.
models:
student:
_target_: fastvideo.train.models.wan.WanModel
init_from: loayrashid/TurboWan2.1-T2V-1.3B-Diffusers
trainable: true
lora:
enable: true
rank: 16
alpha: 32
target_modules:
- to_q
- to_k
- to_v
- to_out
method:
_target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
training:
dit_precision: bf16
distributed:
num_gpus: 1
sp_size: 1
tp_size: 1
hsdp_replicate_dim: 1
hsdp_shard_dim: 1
data:
data_path: data/crush-smol_processed_t2v/combined_parquet_dataset
dataloader_num_workers: 2
train_batch_size: 1
training_cfg_rate: 0.0
seed: 42
num_latent_t: 20
num_height: 480
num_width: 832
num_frames: 77
optimizer:
learning_rate: 1.0e-5
betas: [0.9, 0.999]
weight_decay: 0.01
lr_scheduler: constant
lr_warmup_steps: 0
loop:
max_train_steps: 10
gradient_accumulation_steps: 5
checkpoint:
output_dir: outputs/turbo_wan_t2v_lora_sla
training_state_checkpointing_steps: 5
checkpoints_total_limit: 2
resume_from_checkpoint: null
tracker:
project_name: fastvideo
run_name: turbo_wan_t2v_lora_sla
model:
enable_gradient_checkpointing_type: full
callbacks:
grad_clip:
_target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
max_grad_norm: 1.0
validation:
_target_: fastvideo.train.callbacks.validation.ValidationCallback
pipeline_target: fastvideo.pipelines.basic.turbodiffusion.turbodiffusion_pipeline.TurboDiffusionPipeline
dataset_file: examples/training/finetune/wan_t2v_1.3B/crush_smol/validation.json
every_steps: 5
sampling_steps: [4]
guidance_scale: 1.0
pipeline:
flow_shift: 3
+1 -45
View File
@@ -62,28 +62,13 @@ class SamplingParam:
prompt_attention_mask: list = field(default_factory=list)
negative_attention_mask: list = field(default_factory=list)
# Camera/action control inputs (GameCraft)
camera_states: Any | None = None # Plücker coordinates [B, T_video, 6, H, W]
camera_trajectory: str | None = None
action_list: list[str] | None = None
action_speed_list: list[float] | None = None
gt_latents: Any | None = None # Ground truth latents [B, 16, T, H, W]
conditioning_mask: Any | None = None # Mask [B, 1, T, H, W]
# Camera control inputs (LingBotWorld and LingBotWorld2)
c2ws_plucker_emb: Any | None = None # Plucker embedding: [B, C, F_lat, H_lat, W_lat]
action_path: str | None = None # Directory containing poses.npy and intrinsics.npy
# Refine inputs (LongCat 480p->720p upscaling)
# Path-based refine (load stage1 video from disk, e.g. MP4)
refine_from: str | None = None # Path to stage1 video (480p output from distill)
# Refine / conditioning-frame inputs
t_thresh: float = 0.5 # Threshold for timestep scheduling in refinement
spatial_refine_only: bool = False # If True, only spatial (no temporal doubling)
num_cond_frames: int = 0 # Number of conditioning frames
# In-memory refine input (for two-stage pipeline where stage1 frames are already in memory)
# This mirrors LongCat's demo where a list of frames (e.g. np.ndarray or PIL.Image)
# is passed directly to the refinement pipeline instead of reloading from disk.
stage1_video: Any | None = None
# Text inputs
prompt: str | list[str] | None = None
@@ -111,11 +96,6 @@ class SamplingParam:
guidance_scale: float = 1.0
batch_cfg: bool = False
guidance_scale_2: float | None = None
# Z-Image CFG controls. ``cfg_normalization=True`` caps the guided
# prediction norm at the positive-prediction norm; ``cfg_truncation``
# disables CFG above the normalized-noise threshold.
cfg_normalization: bool = False
cfg_truncation: float | None = 1.0
# Embedded guidance (FLUX): do not treat ``guidance_scale > 1`` as classic CFG.
use_embedded_guidance: bool = False
# Diffusers-style true CFG for FLUX when > 1 (requires negative prompt encoding).
@@ -365,18 +345,6 @@ class SamplingParam:
default=SamplingParam.guidance_scale,
help="Classifier-free guidance scale",
)
parser.add_argument(
"--cfg-normalization",
action=StoreBoolean,
default=SamplingParam.cfg_normalization,
help="Cap Z-Image CFG prediction norm to the positive-prediction norm",
)
parser.add_argument(
"--cfg-truncation",
type=float,
default=SamplingParam.cfg_truncation,
help="Disable Z-Image CFG above this normalized-noise threshold",
)
parser.add_argument(
"--batch-cfg",
action=StoreBoolean,
@@ -493,24 +461,12 @@ class SamplingParam:
default=SamplingParam.audio_path,
help="Path to input audio for speech-driven generation (Wan S2V)",
)
parser.add_argument(
"--refine-from",
type=str,
default=SamplingParam.refine_from,
help="Path to stage1 video for refinement (LongCat 480p->720p)",
)
parser.add_argument(
"--t-thresh",
type=float,
default=SamplingParam.t_thresh,
help="Threshold for timestep scheduling in refinement (default: 0.5)",
)
parser.add_argument(
"--spatial-refine-only",
action=StoreBoolean,
default=SamplingParam.spatial_refine_only,
help="Only perform spatial super-resolution (no temporal doubling)",
)
parser.add_argument(
"--num-cond-frames",
type=int,
-4
View File
@@ -158,8 +158,6 @@ class InputConfig:
grid_sizes: Any | None = None
c2ws_plucker_emb: Any | None = None
action_path: str | None = None
refine_from: str | None = None
stage1_video: Any | None = None
@dataclass
@@ -178,8 +176,6 @@ class SamplingConfig:
guidance_scale: float = 1.0
batch_cfg: bool = False
guidance_scale_2: float | None = None
cfg_normalization: bool = False
cfg_truncation: float | None = 1.0
guidance_rescale: float = 0.0
true_cfg_scale: float | None = None
use_embedded_guidance: bool | None = None
+1 -1
View File
@@ -67,7 +67,7 @@ def _use_fa2(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> bool:
# capability gate, not a runtime fallback):
# * the backward asserts sm90+ (L40S/sm_89 dies on its arch check);
# * GQA fails CuTeDSL JIT in pack_gqa ("ValueError: Operation creation
# failed", observed on sm_89 with HunyuanGameCraft/LTX2).
# failed", observed on sm_89 with LTX2).
if q.shape[-2] != k.shape[-2]:
return True
return torch.is_grad_enabled() and any(t.requires_grad for t in (q, k, v))
+6 -12
View File
@@ -1,14 +1,10 @@
from fastvideo.configs.models.dits.cosmos import CosmosVideoConfig
from fastvideo.configs.models.dits.cosmos2_5 import Cosmos25VideoConfig
from fastvideo.configs.models.dits.dreamx_world import DreamXWorldARConfig, DreamXWorldConfig
from fastvideo.configs.models.dits.flux import FluxDiTConfig
from fastvideo.configs.models.dits.flux_2 import Flux2Config
from fastvideo.configs.models.dits.glm_image import GlmImageDiTConfig
from fastvideo.configs.models.dits.helios import HeliosConfig
from fastvideo.configs.models.dits.hunyuangamecraft import HunyuanGameCraftConfig
from fastvideo.configs.models.dits.hunyuanvideo import HunyuanVideoConfig
from fastvideo.configs.models.dits.hunyuanvideo15 import HunyuanVideo15Config
from fastvideo.configs.models.dits.longcat import LongCatVideoConfig
from fastvideo.configs.models.dits.ltx2 import LTX2VideoConfig
from fastvideo.configs.models.dits.magi_human import MagiHumanVideoConfig
from fastvideo.configs.models.dits.minimax_h3 import MiniMaxH3Config
@@ -22,7 +18,6 @@ from fastvideo.configs.models.dits.mmaudio import (
)
from fastvideo.configs.models.dits.stable_audio import StableAudioConfig
from fastvideo.models.wan.config import WanVideoConfig
from fastvideo.configs.models.dits.zimage import ZImageDiTConfig
from fastvideo.configs.models.dits.hyworld import HYWorldConfig
from fastvideo.configs.models.dits.kandinsky5 import Kandinsky5VideoConfig
from fastvideo.configs.models.dits.kandinsky6 import Kandinsky6ArchConfig, Kandinsky6VideoAudioConfig
@@ -32,12 +27,11 @@ from fastvideo.configs.models.dits.lingbotworld_fast import LingBotWorldFastVide
from fastvideo.configs.models.dits.lingbot_video import LingBotVideoConfig
__all__ = [
"HunyuanVideoConfig", "HunyuanVideo15Config", "HunyuanGameCraftConfig", "WanVideoConfig", "DreamXWorldConfig",
"DreamXWorldARConfig", "CosmosVideoConfig", "Cosmos25VideoConfig", "FluxDiTConfig", "Flux2Config",
"LongCatVideoConfig", "LTX2VideoConfig", "HYWorldConfig", "Kandinsky5VideoConfig", "Kandinsky6ArchConfig",
"HunyuanVideoConfig", "HunyuanVideo15Config", "WanVideoConfig", "CosmosVideoConfig", "Cosmos25VideoConfig",
"FluxDiTConfig", "Flux2Config", "LTX2VideoConfig", "HYWorldConfig", "Kandinsky5VideoConfig", "Kandinsky6ArchConfig",
"Kandinsky6VideoAudioConfig", "Kandinsky6SRArchConfig", "Kandinsky6SRConfig", "MagiHumanVideoConfig",
"StableAudioConfig", "GlmImageDiTConfig", "HeliosConfig", "LingBotWorld2CausalFastVideoConfig",
"LingBotWorldFastVideoConfig", "LingBotVideoConfig", "MiniMaxH3Config", "ZImageDiTConfig", "MMAudioArchConfig",
"MMAudioTransformerConfig", "MMAUDIO_VARIANT_ARCHITECTURES", "MMAUDIO_TRAINING_VARIANTS",
"MMAUDIO_44K_TRAINING_VARIANTS", "get_mmaudio_transformer_config"
"StableAudioConfig", "HeliosConfig", "LingBotWorld2CausalFastVideoConfig", "LingBotWorldFastVideoConfig",
"LingBotVideoConfig", "MiniMaxH3Config", "MMAudioArchConfig", "MMAudioTransformerConfig",
"MMAUDIO_VARIANT_ARCHITECTURES", "MMAUDIO_TRAINING_VARIANTS", "MMAUDIO_44K_TRAINING_VARIANTS",
"get_mmaudio_transformer_config"
]
@@ -1,71 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
from dataclasses import dataclass, field
from fastvideo.configs.models.dits.base import DiTArchConfig, DiTConfig
from fastvideo.models.wan.config import WanVideoArchConfig
@dataclass
class DreamXWorldArchConfig(WanVideoArchConfig):
"""DreamX-World DiT config with camera PRoPE control fields."""
add_control_adapter: bool = True
cam_method: str | None = "prope"
attn_compress: int = 1
cam_self_attn_layers: tuple[int, ...] | None = None
@dataclass
class DreamXWorldConfig(DiTConfig):
arch_config: DiTArchConfig = field(default_factory=DreamXWorldArchConfig)
prefix: str = "Wan"
@dataclass
class DreamXWorldARArchConfig(DreamXWorldArchConfig):
"""DreamX-World-5B autoregressive causal DiT config."""
model_type: str = "ti2v"
patch_size: tuple[int, int, int] = (1, 2, 2)
text_len: int = 512
text_dim: int = 4096
freq_dim: int = 256
attn_compress: int = 4
cam_self_attn_layers: tuple[int, ...] | None = tuple(range(30))
local_attn_size: int = 12
sink_size: int = 3
num_frames_per_block: int = 3
rope_cache_policy: str = "block_relativistic"
# The official AR checkpoint (AMAP-ML/DreamX-World ``model.safetensors``)
# already uses FastVideo's native key names and the converter copies the
# tensors verbatim, so every rule is an identity. The rules enumerate the
# full state-dict surface of ``DreamXWorldARTransformer3DModel`` (norm1 /
# norm2 / head.norm are affine-free and have no parameters).
param_names_mapping: dict = field(
default_factory=lambda: {
r"^patch_embedding\.(.*)$": r"patch_embedding.\1",
r"^text_embedding\.([02])\.(.*)$": r"text_embedding.\1.\2",
r"^time_embedding\.([02])\.(.*)$": r"time_embedding.\1.\2",
r"^time_projection\.1\.(.*)$": r"time_projection.1.\1",
r"^blocks\.(\d+)\.self_attn\.(q|k|v|o)\.(.*)$": r"blocks.\1.self_attn.\2.\3",
r"^blocks\.(\d+)\.self_attn\.norm_(q|k)\.weight$": r"blocks.\1.self_attn.norm_\2.weight",
r"^blocks\.(\d+)\.cross_attn\.(q|k|v|o)\.(.*)$": r"blocks.\1.cross_attn.\2.\3",
r"^blocks\.(\d+)\.cross_attn\.norm_(q|k)\.weight$": r"blocks.\1.cross_attn.norm_\2.weight",
r"^blocks\.(\d+)\.cam_self_attn\.(q_proj|k_proj|v_proj|out_proj)\.(.*)$": r"blocks.\1.cam_self_attn.\2.\3",
r"^blocks\.(\d+)\.cam_self_attn\.norm_(q|k)\.weight$": r"blocks.\1.cam_self_attn.norm_\2.weight",
r"^blocks\.(\d+)\.norm3\.(.*)$": r"blocks.\1.norm3.\2",
r"^blocks\.(\d+)\.ffn\.([02])\.(.*)$": r"blocks.\1.ffn.\2.\3",
r"^blocks\.(\d+)\.modulation$": r"blocks.\1.modulation",
r"^head\.head\.(.*)$": r"head.head.\1",
r"^head\.modulation$": r"head.modulation",
})
reverse_param_names_mapping: dict = field(default_factory=dict)
lora_param_names_mapping: dict = field(default_factory=dict)
@dataclass
class DreamXWorldARConfig(DreamXWorldConfig):
arch_config: DiTArchConfig = field(default_factory=DreamXWorldARArchConfig)
prefix: str = "Wan"
@@ -1,61 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
from dataclasses import dataclass, field
from fastvideo.configs.models.dits.base import DiTArchConfig, DiTConfig
def is_blocks(n: str, m) -> bool:
return "transformer_blocks" in n and str.isdigit(n.split(".")[-1])
@dataclass
class GlmImageDiTArchConfig(DiTArchConfig):
_fsdp_shard_conditions: list = field(default_factory=lambda: [is_blocks])
hidden_size: int = 4096
num_attention_heads: int = 32
attention_head_dim: int = 128
in_channels: int = 16
out_channels: int = 16
num_layers: int = 30
text_embed_dim: int = 1472
time_embed_dim: int = 512
condition_dim: int = 256
prior_vq_quantizer_codebook_size: int = 16384
patch_size: int = 2
max_height: int = 2048
max_width: int = 2048
qk_norm: str = "layer_norm"
eps: float = 1e-5
exclude_lora_layers: list[str] = field(
default_factory=lambda: ["image_projector", "glyph_projector", "prior_token_embedding"])
param_names_mapping: dict = field(
default_factory=lambda: {
r"^glyph_projector\.net\.0\.proj\.(.*)$": r"glyph_projector.fc_in.\1",
r"^glyph_projector\.net\.2\.(.*)$": r"glyph_projector.fc_out.\1",
r"^prior_projector\.net\.0\.proj\.(.*)$": r"prior_projector.fc_in.\1",
r"^prior_projector\.net\.2\.(.*)$": r"prior_projector.fc_out.\1",
r"^transformer_blocks\.(\d+)\.ff\.net\.0\.proj\.(.*)$": r"transformer_blocks.\1.ff.fc_in.\2",
r"^transformer_blocks\.(\d+)\.ff\.net\.2\.(.*)$": r"transformer_blocks.\1.ff.fc_out.\2",
})
reverse_param_names_mapping: dict = field(default_factory=dict)
lora_param_names_mapping: dict = field(default_factory=dict)
def __post_init__(self):
super().__post_init__()
self.num_channels_latents = self.out_channels
@dataclass
class GlmImageDiTConfig(DiTConfig):
arch_config: DiTArchConfig = field(default_factory=GlmImageDiTArchConfig)
prefix: str = "GlmImage"
@@ -1,143 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""
Configuration for HunyuanGameCraft transformer model.
HunyuanGameCraft extends HunyuanVideo with:
1. CameraNet for camera/action conditioning
2. 33 input channels (16 latent + 16 gt_latent + 1 mask)
3. Mask-based conditioning for autoregressive generation
"""
from dataclasses import dataclass, field
import torch
from fastvideo.configs.models.dits.base import DiTArchConfig, DiTConfig
def is_double_block(n: str, m) -> bool:
return "double" in n and str.isdigit(n.split(".")[-1])
def is_single_block(n: str, m) -> bool:
return "single" in n and str.isdigit(n.split(".")[-1])
def is_refiner_block(n: str, m) -> bool:
return "refiner" in n and str.isdigit(n.split(".")[-1])
def is_txt_in(n: str, m) -> bool:
return n.split(".")[-1] == "txt_in"
def is_camera_net(n: str, m) -> bool:
return "camera_net" in n
@dataclass
class HunyuanGameCraftArchConfig(DiTArchConfig):
"""Architecture config for HunyuanGameCraft transformer."""
# Version field for compatibility with saved config.json
_fastvideo_version: str = "0.1.0"
# Camera net flag (for config.json compatibility)
camera_net: bool = True
_fsdp_shard_conditions: list = field(
default_factory=lambda: [is_double_block, is_single_block, is_refiner_block, is_camera_net])
_compile_conditions: list = field(default_factory=lambda: [is_double_block, is_single_block, is_txt_in])
# Parameter names mapping from official checkpoint to FastVideo naming
# GameCraft weights are already close to FastVideo format with minor adjustments
param_names_mapping: dict = field(
default_factory=lambda: {
# MLP naming: fc1 -> fc_in, fc2 -> fc_out
r"^(.*)\.img_mlp\.fc1\.(.*)$": r"\1.img_mlp.fc_in.\2",
r"^(.*)\.img_mlp\.fc2\.(.*)$": r"\1.img_mlp.fc_out.\2",
r"^(.*)\.txt_mlp\.fc1\.(.*)$": r"\1.txt_mlp.fc_in.\2",
r"^(.*)\.txt_mlp\.fc2\.(.*)$": r"\1.txt_mlp.fc_out.\2",
# Single block MLP naming
r"^single_blocks\.(\d+)\.mlp\.fc1\.(.*)$": r"single_blocks.\1.mlp.fc_in.\2",
r"^single_blocks\.(\d+)\.mlp\.fc2\.(.*)$": r"single_blocks.\1.mlp.fc_out.\2",
# Token refiner naming
r"^txt_in\.individual_token_refiner\.blocks\.(\d+)\.(.*)$": r"txt_in.refiner_blocks.\1.\2",
# Vector in naming
r"^vector_in\.in_layer\.(.*)$": r"vector_in.fc_in.\1",
r"^vector_in\.out_layer\.(.*)$": r"vector_in.fc_out.\1",
# Time embedder naming
r"^time_in\.mlp\.0\.(.*)$": r"time_in.mlp.fc_in.\1",
r"^time_in\.mlp\.2\.(.*)$": r"time_in.mlp.fc_out.\1",
# Guidance embedder naming (if present)
r"^guidance_in\.mlp\.0\.(.*)$": r"guidance_in.mlp.fc_in.\1",
r"^guidance_in\.mlp\.2\.(.*)$": r"guidance_in.mlp.fc_out.\1",
# Final layer adaLN modulation
r"^final_layer\.adaLN_modulation\.1\.(.*)$": r"final_layer.adaLN_modulation.linear.\1",
# Refiner block MLP naming
r"^txt_in\.refiner_blocks\.(\d+)\.mlp\.fc1\.(.*)$": r"txt_in.refiner_blocks.\1.mlp.fc_in.\2",
r"^txt_in\.refiner_blocks\.(\d+)\.mlp\.fc2\.(.*)$": r"txt_in.refiner_blocks.\1.mlp.fc_out.\2",
# Camera net weights are already correctly named
})
# Reverse mapping for saving checkpoints
reverse_param_names_mapping: dict = field(default_factory=lambda: {})
# Model architecture parameters
# patch_size can be int or tuple - if tuple, it's [T, H, W]
patch_size: int | tuple[int, int, int] = 2
patch_size_t: int = 1
in_channels: int = 33 # 16 latent + 16 gt_latent + 1 mask
out_channels: int = 16
num_attention_heads: int = 24
attention_head_dim: int = 128
mlp_ratio: float = 4.0
num_layers: int = 20 # Double stream blocks
num_single_layers: int = 40 # Single stream blocks
num_refiner_layers: int = 2
rope_axes_dim: tuple[int, int, int] = (16, 56, 56)
guidance_embeds: bool = False # GameCraft doesn't use guidance
dtype: torch.dtype | None = None
text_embed_dim: int = 4096 # LLaMA-3 hidden size
pooled_projection_dim: int = 768 # CLIP pooled output dim
rope_theta: int = 256
qk_norm: str = "rms_norm"
# Camera net parameters
camera_in_channels: int = 6 # Plücker coordinates
camera_downscale_coef: int = 8
camera_out_channels: int = 16
# Layers to exclude from LoRA
exclude_lora_layers: list[str] = field(
default_factory=lambda: ["img_in", "txt_in", "time_in", "vector_in", "camera_net"])
def __post_init__(self):
super().__post_init__()
self.hidden_size: int = self.attention_head_dim * self.num_attention_heads
self.num_channels_latents: int = 16 # Output is 16 channels
# Convert patch_size list to tuple if needed (from JSON deserialization)
if isinstance(self.patch_size, list):
self.patch_size = tuple(self.patch_size)
# Convert rope_axes_dim list to tuple if needed
if isinstance(self.rope_axes_dim, list):
self.rope_axes_dim = tuple(self.rope_axes_dim)
@dataclass
class HunyuanGameCraftConfig(DiTConfig):
"""Full config for HunyuanGameCraft model."""
arch_config: DiTArchConfig = field(default_factory=HunyuanGameCraftArchConfig)
prefix: str = "HunyuanGameCraft"
-133
View File
@@ -1,133 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""
LongCat Video DiT configuration for native FastVideo implementation.
"""
from dataclasses import dataclass, field
from fastvideo.configs.models.dits.base import DiTArchConfig, DiTConfig
from fastvideo.platforms import AttentionBackendEnum
def is_longcat_blocks(n: str, m) -> bool:
"""FSDP shard condition for LongCat transformer blocks."""
return "blocks" in n and str.isdigit(n.split(".")[-1])
@dataclass
class LongCatVideoArchConfig(DiTArchConfig):
"""Architecture configuration for native LongCat Video DiT."""
_fsdp_shard_conditions: list = field(default_factory=lambda: [is_longcat_blocks])
# Enable torch.compile for transformer blocks (major speedup!)
_compile_conditions: list = field(default_factory=lambda: [is_longcat_blocks])
# Parameter name mapping for weight conversion
param_names_mapping: dict = field(
default_factory=lambda: {
# Embedders
r"^x_embedder\.(.*)$": r"patch_embed.\1",
r"^t_embedder\.mlp\.0\.(.*)$": r"time_embedder.linear_1.\1",
r"^t_embedder\.mlp\.2\.(.*)$": r"time_embedder.linear_2.\1",
r"^y_embedder\.y_proj\.0\.(.*)$": r"caption_embedder.linear_1.\1",
r"^y_embedder\.y_proj\.2\.(.*)$": r"caption_embedder.linear_2.\1",
# Transformer blocks - AdaLN modulation
r"^blocks\.(\d+)\.adaLN_modulation\.1\.(.*)$": r"blocks.\1.adaln_linear_1.\2",
# Transformer blocks - Normalization
r"^blocks\.(\d+)\.mod_norm_attn\.(.*)$": r"blocks.\1.norm_attn.\2",
r"^blocks\.(\d+)\.mod_norm_ffn\.(.*)$": r"blocks.\1.norm_ffn.\2",
r"^blocks\.(\d+)\.pre_crs_attn_norm\.(.*)$": r"blocks.\1.norm_cross.\2",
# Self-attention: QKV fused -> separate (will need splitting in converter)
# Original has attn.qkv.weight -> need to split into to_q, to_k, to_v
r"^blocks\.(\d+)\.attn\.qkv\.(.*)$": r"blocks.\1.self_attn.qkv_fused.\2", # Marker for splitting
r"^blocks\.(\d+)\.attn\.proj\.(.*)$": r"blocks.\1.self_attn.to_out.\2",
r"^blocks\.(\d+)\.attn\.q_norm\.(.*)$": r"blocks.\1.self_attn.q_norm.\2",
r"^blocks\.(\d+)\.attn\.k_norm\.(.*)$": r"blocks.\1.self_attn.k_norm.\2",
# Cross-attention
r"^blocks\.(\d+)\.cross_attn\.q_linear\.(.*)$": r"blocks.\1.cross_attn.to_q.\2",
r"^blocks\.(\d+)\.cross_attn\.kv_linear\.(.*)$":
r"blocks.\1.cross_attn.kv_fused.\2", # Marker for splitting
r"^blocks\.(\d+)\.cross_attn\.proj\.(.*)$": r"blocks.\1.cross_attn.to_out.\2",
r"^blocks\.(\d+)\.cross_attn\.q_norm\.(.*)$": r"blocks.\1.cross_attn.q_norm.\2",
r"^blocks\.(\d+)\.cross_attn\.k_norm\.(.*)$": r"blocks.\1.cross_attn.k_norm.\2",
# FFN (SwiGLU)
r"^blocks\.(\d+)\.ffn\.w1\.(.*)$": r"blocks.\1.ffn.w1.\2", # gate
r"^blocks\.(\d+)\.ffn\.w2\.(.*)$": r"blocks.\1.ffn.w2.\2", # down
r"^blocks\.(\d+)\.ffn\.w3\.(.*)$": r"blocks.\1.ffn.w3.\2", # up
# Final layer
r"^final_layer\.adaLN_modulation\.1\.(.*)$": r"final_layer.adaln_linear.\1",
r"^final_layer\.norm_final\.(.*)$": r"final_layer.norm.\1",
r"^final_layer\.linear\.(.*)$": r"final_layer.proj.\1",
})
# Reverse mapping for saving checkpoints: custom -> hf
reverse_param_names_mapping: dict = field(default_factory=lambda: {})
# LoRA parameter name mapping
lora_param_names_mapping: dict = field(default_factory=lambda: {})
# Model architecture parameters
hidden_size: int = 4096
depth: int = 48 # Number of transformer blocks
num_attention_heads: int = 32
attention_head_dim: int = 128 # hidden_size / num_attention_heads
in_channels: int = 16 # Latent space channels
out_channels: int = 16
num_channels_latents: int = 16
# Patch embedding
patch_size: tuple[int, int, int] = (1, 2, 2) # [T, H, W] - no temporal compression
# Text/caption embedding
caption_channels: int = 4096 # UMT5 d_model
# Timestep embedding
adaln_tembed_dim: int = 512
frequency_embedding_size: int = 256
# FFN
mlp_ratio: int = 4
# Attention backend support
_supported_attention_backends: tuple = field(default_factory=lambda: (
AttentionBackendEnum.FLASH_ATTN,
AttentionBackendEnum.TORCH_SDPA,
))
# Text padding behavior
text_tokens_zero_pad: bool = True
# Block Sparse Attention (BSA)
enable_bsa: bool = False
bsa_params: dict | None = field(default_factory=lambda: {
"sparsity": 0.9375,
"cdf_threshold": None,
"chunk_3d_shape_q": [4, 4, 4],
"chunk_3d_shape_k": [4, 4, 4],
})
# LoRA exclusions
exclude_lora_layers: list[str] = field(default_factory=lambda: [])
def __post_init__(self):
super().__post_init__()
self.out_channels = self.out_channels or self.in_channels
# Ensure attention_head_dim matches
self.attention_head_dim = self.hidden_size // self.num_attention_heads
@dataclass
class LongCatVideoConfig(DiTConfig):
"""Main configuration for LongCat Video DiT."""
arch_config: DiTArchConfig = field(default_factory=LongCatVideoArchConfig)
prefix: str = "longcat"

Some files were not shown because too many files have changed in this diff Show More