Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
25940a7e86 | ||
|
|
3f45f2da21 | ||
|
|
6212e7262e | ||
|
|
ba371e33ed | ||
|
|
06c8c5f4b8 | ||
|
|
c535797e3d | ||
|
|
59fddb00f2 | ||
|
|
9f59519959 |
@@ -52,8 +52,8 @@ in one commit.
|
||||
script that supports a global dtype downcast flag MUST own an explicit
|
||||
allowlist of fp32-keep tensors, documented at the top of the file.
|
||||
|
||||
2. **The `add-model-conversion` skill** should enforce two checks for any
|
||||
converter that ships a `--cast-bf16`-style flag:
|
||||
2. **Check two things** for any converter that ships a `--cast-bf16`-style
|
||||
flag:
|
||||
- Run the parity test for **every** variant of the model (base, distill,
|
||||
SR, etc.), not just the headline variant. Different variants exercise
|
||||
different code paths.
|
||||
|
||||
@@ -62,10 +62,10 @@ parity rationale:
|
||||
2. **Use layer-by-layer activation hooks** when DiT parity is close-but-not-
|
||||
bit-exact and the gap grows with depth. The
|
||||
`fastvideo/hooks/activation_trace.py` infra exists exactly for this case
|
||||
(`add-model-trace` skill). In MagiHuman's case it would have localized the
|
||||
(see `docs/contributing/activation_trace.md`). In MagiHuman's case it would have localized the
|
||||
first divergence point in one pass.
|
||||
|
||||
3. **The `add-model-port-dit` skill** should explicitly call out:
|
||||
3. **DiT ports must account for:**
|
||||
- SDPA input dtype must match the upstream kernel's internal cast.
|
||||
- Post-attention upcast happens **before** any per-head gate, not after.
|
||||
- Residual stream dtype across block boundaries is a parity invariant.
|
||||
|
||||
@@ -64,6 +64,5 @@ because both sides of the parity check are now correct.
|
||||
`(pT pH pW C)`) at the call site and assert the layout matches the
|
||||
downstream consumer's expectation.
|
||||
|
||||
4. The `add-model-port-dit` skill's parity gate should require an E2E hash
|
||||
check for any DiT that does video token packing, not just latent
|
||||
bit-exactness.
|
||||
4. DiT parity for any model that does video token packing should include an
|
||||
E2E hash check, not just latent bit-exactness.
|
||||
|
||||
@@ -1,173 +0,0 @@
|
||||
---
|
||||
name: add-model-01-prep
|
||||
description: Use at the start of a FastVideo model port to gather required inputs, inspect/download HF weights, clone and install the official reference repo in the current environment, create a local_tests README skeleton, and produce a handoff before conversion or implementation.
|
||||
---
|
||||
|
||||
# Add Model Prep
|
||||
|
||||
## Goal
|
||||
|
||||
Prepare external assets and the shared parity-test environment for a FastVideo
|
||||
model port. Stop before writing conversion scripts, model components, pipeline
|
||||
code, registry entries, or executable parity tests.
|
||||
|
||||
## Ask First
|
||||
|
||||
Ask once, then proceed if the HF token is already exported:
|
||||
|
||||
```text
|
||||
Before prep: (1) official reference repo or Diffusers pipeline URL, (2) HF repo
|
||||
id or local weights path and whether it has a root model_index.json, (3) target
|
||||
model_family, (4) workload types, (5) which token env var is exported:
|
||||
HF_TOKEN, HUGGINGFACE_HUB_TOKEN, or HF_API_KEY, (6) may I stage clone and
|
||||
weights under the FastVideo repo root, and (7) may I install official reference
|
||||
dependencies into the current FastVideo conda/env for parity tests?
|
||||
```
|
||||
|
||||
Useful optional inputs: `pipeline_class`, `reference_dir`, `hf_revision`,
|
||||
`official_revision`, `reuse_hints`, `download_scope`.
|
||||
|
||||
## Rules
|
||||
|
||||
- Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
|
||||
escape hatches, and skip/pass semantics.
|
||||
- Run from the FastVideo repo root.
|
||||
- Use repo-relative defaults: `<ReferenceDir>/`,
|
||||
`official_weights/<model_family>/`, `converted_weights/<model_family>/`.
|
||||
- Install official reference deps into the current FastVideo environment, not a
|
||||
new venv/conda env, so parity tests run both implementations with one shared
|
||||
numeric stack.
|
||||
- If the reference is a Diffusers class/package instead of a cloneable repo,
|
||||
record import path and version instead of cloning.
|
||||
- Prep may create only the local-test README and `PORT_STATUS.md` skeletons;
|
||||
executable `.py` parity tests belong to `../add-model-02-parity/SKILL.md`.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md`. Prep-specific ask cases include
|
||||
overwriting an existing clone or weight directory, installing untrusted/private
|
||||
deps, choosing between incompatible official references, large downloads outside
|
||||
the agreed scope, or missing gated-repo auth setup by env var name.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Verify the repo:
|
||||
|
||||
```bash
|
||||
git rev-parse --show-toplevel
|
||||
```
|
||||
|
||||
Expected markers: `fastvideo/`, `scripts/checkpoint_conversion/`,
|
||||
`scripts/huggingface/download_hf.py`, `fastvideo/registry.py`.
|
||||
|
||||
2. Inspect HF or local weight layout:
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \
|
||||
"Org/Model" \
|
||||
--revision "<revision>" \
|
||||
--json
|
||||
```
|
||||
|
||||
For a local path, replace `Org/Model` with `/path/to/weights`. Record
|
||||
`source_layout`, `needs_conversion`, `model_index_class`, and
|
||||
`components_seen`.
|
||||
|
||||
3. Download HF weights if needed:
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \
|
||||
"Org/Model" \
|
||||
"official_weights/<model_family>" \
|
||||
--revision "<revision>"
|
||||
```
|
||||
|
||||
For selected files, repeat `--file-name`. For partial snapshots, repeat
|
||||
`--allow-pattern` or `--ignore-pattern`. If the user provided a local path,
|
||||
record it instead of copying large weights by default.
|
||||
|
||||
4. Clone the official reference repo if applicable:
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \
|
||||
"<official_repo_url>" \
|
||||
"<ReferenceDir>" \
|
||||
--branch "<tag-or-branch>" \
|
||||
--commit "<commit-sha>" \
|
||||
--update-gitignore
|
||||
```
|
||||
|
||||
Omit `--branch`, `--commit`, or `--update-gitignore` when not needed. The
|
||||
helper refuses to overwrite existing paths and prints remote/HEAD instead.
|
||||
|
||||
5. Keep prep assets ignored. Ensure `.gitignore` includes relevant entries:
|
||||
|
||||
```gitignore
|
||||
/<ReferenceDir>/
|
||||
/official_weights/
|
||||
/converted_weights/
|
||||
```
|
||||
|
||||
6. Follow the official repo's setup instructions in the current environment.
|
||||
Inspect dependency files and README install docs before installing anything:
|
||||
|
||||
- `README*`, install docs, or model-card instructions.
|
||||
- `requirements*.txt`, `pyproject.toml`, `setup.py`, `environment.yml`.
|
||||
|
||||
Use the current FastVideo conda/env. Do not create a new env even if upstream
|
||||
docs recommend one; translate the needed install commands into the active env.
|
||||
Prefer editable/no-deps first so the official source is importable without
|
||||
changing shared pins:
|
||||
|
||||
```bash
|
||||
uv pip install --no-deps -e ./<ReferenceDir>
|
||||
```
|
||||
|
||||
Then install only missing official deps needed for parity imports. Stop before
|
||||
installing requirements that would change FastVideo's core stack. If upstream
|
||||
requires private/non-PyPI deps, record that parity needs a local stub helper
|
||||
rather than pretending setup is complete.
|
||||
|
||||
7. Create the model-family local test skeleton and top-level port state file:
|
||||
|
||||
```bash
|
||||
mkdir -p tests/local_tests/<model_family>
|
||||
cp ".agents/skills/add-model-01-prep/templates/local_tests_readme.md" \
|
||||
tests/local_tests/<model_family>/README.md
|
||||
cp ".agents/skills/add-model-01-prep/templates/port_status.md" \
|
||||
tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
```
|
||||
|
||||
Edit every placeholder in the README and `PORT_STATUS.md`. The README gives
|
||||
later review agents enough information to reproduce the shared environment and
|
||||
run/review parity work:
|
||||
|
||||
- official code URL or import path, local clone path, and commit/version;
|
||||
- HF URL or local weight path, revision, access notes, and token env var name
|
||||
only;
|
||||
- commands already run and any blocked official dependency installs;
|
||||
- shared-env install commands to re-run without changing core pins;
|
||||
- expected local parity test paths and pytest commands;
|
||||
- private-dependency stubs or known setup gaps;
|
||||
- PR/review notes explaining which parity tests are required before handoff.
|
||||
|
||||
Do not include raw tokens, absolute cache paths that are not repo-reproducible,
|
||||
or large generated outputs. If prep is blocked before imports work, still create
|
||||
the README with `official_env_status=blocked` and the exact blocker.
|
||||
|
||||
`PORT_STATUS.md` must follow `../add-model/contracts/port_state.md`. Record open
|
||||
questions and prep issues immediately, using stable IDs such as `Q001` and
|
||||
`I001`. Keep resolved questions/issues in the table with a resolution instead of
|
||||
deleting them.
|
||||
|
||||
## Handoff
|
||||
|
||||
End with the canonical prep handoff contract from
|
||||
`../add-model/contracts/prep_handoff.md` and update the shared state files before
|
||||
handoff.
|
||||
|
||||
## Helper Scripts
|
||||
|
||||
- `scripts/inspect_hf_layout.py`: classify HF/local layout.
|
||||
- `scripts/download_hf_weights.py`: download HF snapshot or selected files.
|
||||
- `scripts/clone_reference_repo.py`: clone reference repo safely.
|
||||
@@ -1,119 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Clone an official reference repo without overwriting existing paths."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Clone a reference repo for FastVideo parity tests.")
|
||||
parser.add_argument("repo_url", help="Official reference repository URL")
|
||||
parser.add_argument("target_dir", help="Directory to clone into")
|
||||
parser.add_argument("--branch", help="Branch or tag to clone")
|
||||
parser.add_argument("--commit", help="Commit SHA to check out after clone")
|
||||
parser.add_argument(
|
||||
"--update-gitignore",
|
||||
action="store_true",
|
||||
help="Add the target directory to .gitignore if missing",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--gitignore",
|
||||
default=".gitignore",
|
||||
help="Path to gitignore file when --update-gitignore is used",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def run(command: list[str], check: bool = True) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
command,
|
||||
check=check,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
|
||||
def print_existing_repo_info(target: Path) -> int:
|
||||
print(f"target_exists: {target}")
|
||||
if not (target / ".git").exists():
|
||||
print("error: target exists but is not a git repo", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
remote = run(["git", "-C", str(target), "remote", "-v"], check=False)
|
||||
head = run(["git", "-C", str(target), "rev-parse", "HEAD"], check=False)
|
||||
if remote.stdout:
|
||||
print("remote_v:")
|
||||
print(remote.stdout.rstrip())
|
||||
if head.stdout:
|
||||
print(f"head: {head.stdout.strip()}")
|
||||
print("not_overwritten: true")
|
||||
return 0
|
||||
|
||||
|
||||
def gitignore_entry_for(target: Path) -> str:
|
||||
root = Path.cwd().resolve()
|
||||
resolved = target.resolve()
|
||||
try:
|
||||
relative = resolved.relative_to(root)
|
||||
except ValueError as exc:
|
||||
raise ValueError("--update-gitignore requires target_dir to be under the current directory") from exc
|
||||
|
||||
text = relative.as_posix().rstrip("/")
|
||||
return "/" + text + "/"
|
||||
|
||||
|
||||
def update_gitignore(path: Path, target: Path) -> bool:
|
||||
entry = gitignore_entry_for(target)
|
||||
existing = path.read_text().splitlines() if path.exists() else []
|
||||
if entry in existing:
|
||||
return False
|
||||
|
||||
new_text = "\n".join(existing).rstrip("\n")
|
||||
if new_text:
|
||||
new_text += "\n"
|
||||
new_text += entry + "\n"
|
||||
path.write_text(new_text)
|
||||
return True
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
target = Path(args.target_dir)
|
||||
|
||||
if target.exists():
|
||||
return print_existing_repo_info(target)
|
||||
|
||||
command = ["git", "clone", "--depth", "1"]
|
||||
if args.branch:
|
||||
command.extend(["--branch", args.branch])
|
||||
command.extend([args.repo_url, str(target)])
|
||||
|
||||
try:
|
||||
run(command)
|
||||
if args.commit:
|
||||
run(["git", "-C", str(target), "fetch", "--depth", "1", "origin", args.commit])
|
||||
run(["git", "-C", str(target), "checkout", args.commit])
|
||||
except subprocess.CalledProcessError as exc:
|
||||
if exc.stdout:
|
||||
print(exc.stdout, end="")
|
||||
if exc.stderr:
|
||||
print(exc.stderr, end="", file=sys.stderr)
|
||||
return exc.returncode
|
||||
|
||||
head = run(["git", "-C", str(target), "rev-parse", "HEAD"])
|
||||
print(f"cloned: {target}")
|
||||
print(f"head: {head.stdout.strip()}")
|
||||
|
||||
if args.update_gitignore:
|
||||
changed = update_gitignore(Path(args.gitignore), target)
|
||||
print(f"gitignore_updated: {str(changed).lower()}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,103 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Download HF weights using the standard FastVideo token env vars."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Download a HF model snapshot or selected files into a local directory.")
|
||||
parser.add_argument("repo_id", help="HF repo id, for example Org/Model")
|
||||
parser.add_argument("local_dir", help="Destination directory")
|
||||
parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)")
|
||||
parser.add_argument("--revision", help="HF branch, tag, or commit")
|
||||
parser.add_argument(
|
||||
"--file-name",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Download one file; may be repeated. If omitted, download full snapshot.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow-pattern",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Snapshot allow pattern; may be repeated. Ignored when --file-name is used.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ignore-pattern",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Snapshot ignore pattern; may be repeated. Ignored when --file-name is used.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def resolve_token() -> tuple[str | None, str | None]:
|
||||
for key in HF_TOKEN_ENV_KEYS:
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
return key, value
|
||||
return None, None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
token_env, token = resolve_token()
|
||||
local_dir = Path(args.local_dir).expanduser()
|
||||
|
||||
if token_env:
|
||||
print(f"token_env: {token_env}")
|
||||
else:
|
||||
print("token_env: none", file=sys.stderr)
|
||||
|
||||
try:
|
||||
if local_dir.exists() and not local_dir.is_dir():
|
||||
print(
|
||||
f"error: destination exists and is not a directory: {local_dir}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
local_dir.mkdir(parents=True, exist_ok=True)
|
||||
if args.file_name:
|
||||
from huggingface_hub import hf_hub_download
|
||||
|
||||
for file_name in args.file_name:
|
||||
path = hf_hub_download(
|
||||
repo_id=args.repo_id,
|
||||
filename=file_name,
|
||||
repo_type=args.repo_type,
|
||||
revision=args.revision,
|
||||
local_dir=str(local_dir),
|
||||
token=token,
|
||||
)
|
||||
print(f"downloaded_file: {path}")
|
||||
else:
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
path = snapshot_download(
|
||||
repo_id=args.repo_id,
|
||||
repo_type=args.repo_type,
|
||||
revision=args.revision,
|
||||
local_dir=str(local_dir),
|
||||
token=token,
|
||||
allow_patterns=args.allow_pattern or None,
|
||||
ignore_patterns=args.ignore_pattern or None,
|
||||
)
|
||||
print(f"downloaded_snapshot: {path}")
|
||||
except Exception as exc: # noqa: BLE001 - CLI should print concise failures.
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"local_dir: {local_dir.resolve()}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,260 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Inspect a Hugging Face repo or local weight directory layout."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY")
|
||||
RAW_WEIGHT_SUFFIXES = (".safetensors", ".pt", ".pth", ".ckpt", ".bin")
|
||||
KNOWN_COMPONENTS = {
|
||||
"audio_vae",
|
||||
"conditioner",
|
||||
"feature_extractor",
|
||||
"image_encoder",
|
||||
"scheduler",
|
||||
"text_encoder",
|
||||
"text_encoder_2",
|
||||
"tokenizer",
|
||||
"tokenizer_2",
|
||||
"transformer",
|
||||
"transformer_2",
|
||||
"unet",
|
||||
"upsampler",
|
||||
"vae",
|
||||
"vocoder",
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Classify a HF repo or local directory as Diffusers, raw, custom, or unknown.")
|
||||
parser.add_argument("source", help="HF repo id or local weights directory")
|
||||
parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)")
|
||||
parser.add_argument("--revision", help="HF revision to inspect")
|
||||
parser.add_argument(
|
||||
"--max-local-files",
|
||||
type=int,
|
||||
default=20000,
|
||||
help="Maximum local files to scan recursively (default: 20000)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sample-limit",
|
||||
type=int,
|
||||
default=80,
|
||||
help="Number of file paths to print in human output (default: 80)",
|
||||
)
|
||||
parser.add_argument("--json", action="store_true", help="Emit JSON only")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def resolve_token() -> tuple[str | None, str | None]:
|
||||
for key in HF_TOKEN_ENV_KEYS:
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
return key, value
|
||||
return None, None
|
||||
|
||||
|
||||
def load_local_files(root: Path, max_files: int) -> tuple[list[str], bool]:
|
||||
files: list[str] = []
|
||||
truncated = False
|
||||
for path in root.rglob("*"):
|
||||
if not path.is_file():
|
||||
continue
|
||||
files.append(path.relative_to(root).as_posix())
|
||||
if len(files) >= max_files:
|
||||
truncated = True
|
||||
break
|
||||
return sorted(files), truncated
|
||||
|
||||
|
||||
def load_local_model_index(root: Path) -> tuple[dict[str, Any] | None, str | None]:
|
||||
index_path = root / "model_index.json"
|
||||
if not index_path.is_file():
|
||||
return None, None
|
||||
try:
|
||||
return json.loads(index_path.read_text()), None
|
||||
except Exception as exc: # noqa: BLE001 - surface malformed JSON clearly.
|
||||
return None, f"failed to parse local model_index.json: {exc}"
|
||||
|
||||
|
||||
def load_remote_files(
|
||||
repo_id: str,
|
||||
repo_type: str,
|
||||
revision: str | None,
|
||||
token: str | None,
|
||||
) -> list[str]:
|
||||
from huggingface_hub import list_repo_files
|
||||
|
||||
return sorted(list_repo_files(
|
||||
repo_id,
|
||||
repo_type=repo_type,
|
||||
revision=revision,
|
||||
token=token,
|
||||
))
|
||||
|
||||
|
||||
def load_remote_model_index(
|
||||
repo_id: str,
|
||||
repo_type: str,
|
||||
revision: str | None,
|
||||
token: str | None,
|
||||
) -> tuple[dict[str, Any] | None, str | None]:
|
||||
from huggingface_hub import hf_hub_download
|
||||
|
||||
try:
|
||||
path = hf_hub_download(
|
||||
repo_id=repo_id,
|
||||
filename="model_index.json",
|
||||
repo_type=repo_type,
|
||||
revision=revision,
|
||||
token=token,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - missing/inaccessible file is data.
|
||||
return None, f"failed to download model_index.json: {exc}"
|
||||
|
||||
try:
|
||||
return json.loads(Path(path).read_text()), None
|
||||
except Exception as exc: # noqa: BLE001 - surface malformed JSON clearly.
|
||||
return None, f"failed to parse remote model_index.json: {exc}"
|
||||
|
||||
|
||||
def root_file_names(files: list[str]) -> set[str]:
|
||||
return {name for name in files if "/" not in name}
|
||||
|
||||
|
||||
def component_names(files: list[str], model_index: dict[str, Any] | None) -> list[str]:
|
||||
components: set[str] = set()
|
||||
for name in files:
|
||||
parts = name.split("/", 1)
|
||||
if len(parts) != 2:
|
||||
continue
|
||||
top, rest = parts
|
||||
if top in KNOWN_COMPONENTS or rest == "config.json":
|
||||
components.add(top)
|
||||
|
||||
if model_index:
|
||||
for key, value in model_index.items():
|
||||
if key.startswith("_"):
|
||||
continue
|
||||
if isinstance(value, list) and len(value) == 2:
|
||||
components.add(key)
|
||||
|
||||
return sorted(components)
|
||||
|
||||
|
||||
def classify_layout(
|
||||
files: list[str],
|
||||
model_index: dict[str, Any] | None,
|
||||
components: list[str],
|
||||
) -> tuple[str, str]:
|
||||
roots = root_file_names(files)
|
||||
raw_weight_files = [name for name in roots if name.endswith(RAW_WEIGHT_SUFFIXES)]
|
||||
has_model_index = "model_index.json" in roots or model_index is not None
|
||||
|
||||
if has_model_index and components:
|
||||
return "diffusers", "no"
|
||||
if has_model_index:
|
||||
return "custom", "unknown"
|
||||
if raw_weight_files:
|
||||
return "raw_official", "yes"
|
||||
if any(name.endswith(RAW_WEIGHT_SUFFIXES) for name in files):
|
||||
return "custom", "yes"
|
||||
return "unknown", "unknown"
|
||||
|
||||
|
||||
def build_result(args: argparse.Namespace) -> dict[str, Any]:
|
||||
token_env, token = resolve_token()
|
||||
source_path = Path(args.source).expanduser()
|
||||
is_local = source_path.exists()
|
||||
|
||||
if is_local:
|
||||
root = source_path.resolve()
|
||||
if not root.is_dir():
|
||||
raise ValueError(f"local source is not a directory: {root}")
|
||||
files, truncated = load_local_files(root, args.max_local_files)
|
||||
model_index, model_index_error = load_local_model_index(root)
|
||||
source_kind = "local"
|
||||
source = str(root)
|
||||
else:
|
||||
files = load_remote_files(args.source, args.repo_type, args.revision, token)
|
||||
truncated = False
|
||||
model_index, model_index_error = load_remote_model_index(
|
||||
args.source,
|
||||
args.repo_type,
|
||||
args.revision,
|
||||
token,
|
||||
)
|
||||
source_kind = "hf"
|
||||
source = args.source
|
||||
|
||||
components = component_names(files, model_index)
|
||||
source_layout, needs_conversion = classify_layout(files, model_index, components)
|
||||
|
||||
return {
|
||||
"source": source,
|
||||
"source_kind": source_kind,
|
||||
"repo_type": None if is_local else args.repo_type,
|
||||
"revision": args.revision,
|
||||
"token_env": token_env,
|
||||
"source_layout": source_layout,
|
||||
"needs_conversion": needs_conversion,
|
||||
"model_index_class": (model_index or {}).get("_class_name"),
|
||||
"model_index_diffusers_version": (model_index or {}).get("_diffusers_version"),
|
||||
"model_index_error": model_index_error,
|
||||
"components_seen": components,
|
||||
"file_count": len(files),
|
||||
"file_scan_truncated": truncated,
|
||||
"files_sample": files[:args.sample_limit],
|
||||
}
|
||||
|
||||
|
||||
def print_human(result: dict[str, Any]) -> None:
|
||||
for key in (
|
||||
"source",
|
||||
"source_kind",
|
||||
"repo_type",
|
||||
"revision",
|
||||
"token_env",
|
||||
"source_layout",
|
||||
"needs_conversion",
|
||||
"model_index_class",
|
||||
"model_index_diffusers_version",
|
||||
"model_index_error",
|
||||
"file_count",
|
||||
"file_scan_truncated",
|
||||
):
|
||||
value = result.get(key)
|
||||
if value is not None:
|
||||
print(f"{key}: {value}")
|
||||
|
||||
components = result["components_seen"]
|
||||
print("components_seen: " + (", ".join(components) if components else "none"))
|
||||
print("files_sample:")
|
||||
for name in result["files_sample"]:
|
||||
print(f" {name}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
try:
|
||||
result = build_result(args)
|
||||
except Exception as exc: # noqa: BLE001 - CLI should print concise failures.
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(result, indent=2, sort_keys=True))
|
||||
else:
|
||||
print_human(result)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,122 +0,0 @@
|
||||
# <Model Family> Local Tests
|
||||
|
||||
Local-only parity and smoke tests for the `<model_family>` FastVideo port. These
|
||||
tests compare FastVideo against the official reference implementation and are
|
||||
not expected to run in CI unless explicitly promoted later.
|
||||
|
||||
Port progress, open questions, issues, and handoff notes live in
|
||||
`tests/local_tests/<model_family>/PORT_STATUS.md`.
|
||||
|
||||
## Reference Assets
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Model family | `<model_family>` |
|
||||
| Workload types | `<T2V/I2V/V2V/T2I/or compatibility shim with rationale>` |
|
||||
| Official reference | `<url or import path>` |
|
||||
| Local reference dir | `<ReferenceDir or none>` |
|
||||
| Official commit/version | `<sha, tag, package version, or unknown>` |
|
||||
| HF weights | `<HF repo id/url or local path>` |
|
||||
| HF revision | `<revision or default>` |
|
||||
| Local weights dir | `<official_weights/model_family or local path>` |
|
||||
| Source layout | `<diffusers/raw_official/monolithic/separate_components/mixed/custom/unknown>` |
|
||||
| Needs conversion | `<yes/no/unknown>` |
|
||||
|
||||
Do not write token values in this file. Use only the token env var name:
|
||||
`<HF_TOKEN or HUGGINGFACE_HUB_TOKEN or HF_API_KEY>`.
|
||||
|
||||
## Shared Environment Setup
|
||||
|
||||
Run from the FastVideo repo root in the same conda/env used for FastVideo.
|
||||
Do not create a separate upstream environment for parity tests.
|
||||
|
||||
```bash
|
||||
# Official reference source, if cloneable.
|
||||
python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \
|
||||
"<official_repo_url>" \
|
||||
"<ReferenceDir>" \
|
||||
--commit "<commit-sha>" \
|
||||
--update-gitignore
|
||||
|
||||
# Editable install without changing shared core pins.
|
||||
uv pip install --no-deps -e ./<ReferenceDir>
|
||||
|
||||
# Additional official deps installed or required for imports:
|
||||
# <package list or none>
|
||||
```
|
||||
|
||||
Do not change core dependency versions (`torch`, `diffusers`, `transformers`,
|
||||
`flash-attn`, `triton`, CUDA packages) without explicit approval.
|
||||
|
||||
## Official Environment Status
|
||||
|
||||
```text
|
||||
dependency_changes: <none | installed no-deps editable | installed official deps in current env | blocked on user>
|
||||
official_env_status: <imports_ok | private_deps_need_stubs | blocked>
|
||||
private_dep_stubs: <none or tests/local_tests/helpers/<model_family>_upstream.py>
|
||||
blocked_on: <none or exact blocker>
|
||||
```
|
||||
|
||||
## Weight Setup
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \
|
||||
"<Org/Model>" \
|
||||
"official_weights/<model_family>" \
|
||||
--revision "<revision>"
|
||||
```
|
||||
|
||||
If weights are local-only, record the local path and do not copy large files into
|
||||
the repository.
|
||||
|
||||
## Prototype And Conversion Artifacts
|
||||
|
||||
State-dict key/shape dumps are generated after FastVideo native prototypes exist
|
||||
and are used to build the conversion mapping.
|
||||
|
||||
```text
|
||||
official_key_dumps:
|
||||
<component>: converted_weights/<model_family>/_mapping/<component>_official_keys.json
|
||||
fastvideo_key_dumps:
|
||||
<component>: converted_weights/<model_family>/_mapping/<component>_fastvideo_keys.json
|
||||
conversion_script: scripts/checkpoint_conversion/<model_family>_to_diffusers.py
|
||||
conversion_source_layout: <diffusers | separate_components | monolithic | mixed | custom>
|
||||
converted_weights_dir: converted_weights/<model_family>
|
||||
strict_load_status: <not_run | pass | pass_with_documented_exclusions | blocked>
|
||||
```
|
||||
|
||||
For monolithic official checkpoints, record the component prefix split here. For
|
||||
example, a single checkpoint may contain transformer, VAE/pretransform,
|
||||
conditioner, and scheduler/vocoder keys that the conversion script writes into
|
||||
separate FastVideo component subfolders.
|
||||
|
||||
## Expected Parity Tests
|
||||
|
||||
Planned local tests for this family:
|
||||
|
||||
| Component | Official files / args | Test | Concerns | Status |
|
||||
|---|---|---|---|---|
|
||||
| `<component>` | `<definition path; instantiation path + args>` | `tests/local_tests/<bucket>/test_<model_family>_<component>_parity.py` | `<prototype or setup concerns>` | `<planned/scaffold_skip/debug_red/non_skip_pass/blocked>` |
|
||||
| `pipeline` | `<official pipeline call>` | `tests/local_tests/pipelines/test_<model_family>_pipeline_parity.py` | `<pipeline concerns>` | `<planned/scaffold_skip/debug_red/non_skip_pass/blocked>` |
|
||||
|
||||
Include reused components in this table. Reuse is accepted only after the
|
||||
FastVideo component definition and official instantiation arguments have both
|
||||
been checked and the component parity test passes non-skip.
|
||||
|
||||
Run the relevant tests with:
|
||||
|
||||
```bash
|
||||
pytest tests/local_tests/<bucket>/test_<model_family>_<component>_parity.py -v -s
|
||||
pytest tests/local_tests/pipelines/test_<model_family>_pipeline_parity.py -v -s
|
||||
```
|
||||
|
||||
## Review Notes
|
||||
|
||||
- Required before handoff: non-skip PASS for each required component parity
|
||||
test, including reused components that own weights or numerical behavior.
|
||||
- Pipeline parity may start as a scaffold, but final handoff requires non-skip
|
||||
PASS or an explicit blocker accepted through the escape-hatch process.
|
||||
- User decisions and pause points are tracked as `E###` rows in
|
||||
`PORT_STATUS.md`; do not rely on chat history for escape-hatch context.
|
||||
- Review agents should verify this README's setup commands still match the PR,
|
||||
then run the listed parity tests or report the exact blocker.
|
||||
@@ -1,69 +0,0 @@
|
||||
# <Model Family> Port Status
|
||||
|
||||
## Summary
|
||||
|
||||
- model_family: `<model_family>`
|
||||
- workload_types: `<T2V/I2V/V2V/T2I/or compatibility shim with rationale>`
|
||||
- official_ref: `<url or import path>`
|
||||
- official_ref_dir: `<ReferenceDir or none>`
|
||||
- hf_weights_path: `<HF repo id/url or local path>`
|
||||
- local_weights_dir: `<official_weights/model_family or local path>`
|
||||
- source_layout: `<diffusers/raw_official/monolithic/separate_components/mixed/custom/unknown>`
|
||||
- local_tests_readme: `tests/local_tests/<model_family>/README.md`
|
||||
|
||||
## Current Phase
|
||||
|
||||
- phase: `prep`
|
||||
- status: `in_progress`
|
||||
- owner: `prep`
|
||||
- last_updated: `<YYYY-MM-DD>`
|
||||
|
||||
## Component Matrix
|
||||
|
||||
| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| `<component>` | `<dit/vae/encoder/generic>` | `<unknown/reuse/port>` | `<path + symbols>` | `<path + args>` | `<target files>` | `<not_started/in_progress/pass/blocked>` | `<not_started/pass/blocked>` | `<not_started/scaffold_skip/debug_red/non_skip_pass/blocked>` | `<none or IDs>` |
|
||||
|
||||
## Conversion State
|
||||
|
||||
- conversion_script: `scripts/checkpoint_conversion/<model_family>_to_diffusers.py`
|
||||
- converted_weights_dir: `converted_weights/<model_family>`
|
||||
- source_layout: `<diffusers/separate_components/monolithic/mixed/custom/unknown>`
|
||||
- strict_load_status: `not_run`
|
||||
- passthrough_components: `<none or list>`
|
||||
- retry_history: `<none>`
|
||||
|
||||
## Parity Commands
|
||||
|
||||
| Scope | Command | Last Result | Notes |
|
||||
|---|---|---|---|
|
||||
| component | `pytest tests/local_tests/<bucket>/test_<model_family>_<component>_parity.py -v -s` | `not_run` | `<notes>` |
|
||||
| pipeline | `pytest tests/local_tests/pipelines/test_<model_family>_pipeline_parity.py -v -s` | `not_run` | `<notes>` |
|
||||
|
||||
## Open Questions
|
||||
|
||||
| ID | Question | Owner | Needed By Phase | Status | Resolution |
|
||||
|---|---|---|---|---|---|
|
||||
| Q001 | `<question>` | `<owner>` | `<phase>` | `<open/resolved>` | `<resolution or blank>` |
|
||||
|
||||
## Issues And Blockers
|
||||
|
||||
| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| I001 | `<phase>` | `<component or all>` | `<low/medium/high/blocker>` | `<issue>` | `<logs/paths/commands>` | `<owner>` | `<open/resolved>` | `<resolution or blank>` |
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution |
|
||||
|---|---|---|---|---|---|---|
|
||||
| E001 | `<phase>` | `<scope/dependency/auth/cost/destructive/ambiguity/blocker>` | `<one precise question>` | `<safe recommended option>` | `<open/resolved>` | `<resolution or blank>` |
|
||||
|
||||
## Decisions
|
||||
|
||||
| Date | Decision | Rationale | Impact |
|
||||
|---|---|---|---|
|
||||
| `<YYYY-MM-DD>` | `<decision>` | `<why>` | `<affected components/phases>` |
|
||||
|
||||
## Handoff Notes
|
||||
|
||||
- `<short notes for the next agent>`
|
||||
@@ -1,234 +0,0 @@
|
||||
---
|
||||
name: add-model-02-parity
|
||||
description: Use during /add-model after reference/architecture study to scaffold and later activate local FastVideo component parity tests. Emphasizes early test creation, official-reference loading, standardized FastVideo loading, and non-skip handoff gates.
|
||||
---
|
||||
|
||||
# Add Model Parity
|
||||
|
||||
## Goal
|
||||
|
||||
Create parity tests as early as possible in a FastVideo port. The first pass can
|
||||
land before conversion or component implementation as an executable scaffold;
|
||||
handoff is blocked until the same tests become non-skip PASS with real weights.
|
||||
|
||||
## When To Run
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
|
||||
escape hatches, and skip/pass semantics.
|
||||
|
||||
Run immediately after `/add-model` Phase 1 has identified:
|
||||
|
||||
- official component classes and call signatures;
|
||||
- FastVideo target component buckets/classes/configs;
|
||||
- local reference clone or import path from `add-model-01-prep`;
|
||||
- local raw or Diffusers weight path;
|
||||
- `official_env_status=imports_ok`, or private deps that will be stubbed
|
||||
locally in tests;
|
||||
- `local_tests_readme` documenting setup and planned review/test commands;
|
||||
- expected component inputs and output tensors.
|
||||
|
||||
Do not wait for all FastVideo components to be implemented. Write the tests
|
||||
first, then let component-porting subagents make them pass.
|
||||
|
||||
## Outputs
|
||||
|
||||
- One component parity test per required component, including reused components:
|
||||
`tests/local_tests/<bucket>/test_<family>_<component>_parity.py`.
|
||||
- Optional helper for upstream private deps:
|
||||
`tests/local_tests/helpers/<family>_upstream.py`.
|
||||
- Pipeline parity is owned later by `../add-model-09-pipeline/SKILL.md` after all
|
||||
component parity tests pass non-skip.
|
||||
- A parity status block for the `/add-model` parity verification phase.
|
||||
|
||||
## Early Scaffold Rules
|
||||
|
||||
- A scaffold may skip while the FastVideo class, converted weights, or official
|
||||
import is missing.
|
||||
- A scaffold must already encode the real official load path, FastVideo load
|
||||
path, deterministic inputs, expected output extraction, and tolerance target.
|
||||
- Each parity test must declare its coverage scope in the file docstring or a
|
||||
module constant: `production_loader`, `implementation_subcomponent`, or `both`.
|
||||
Implementation/subcomponent parity may bypass production loaders deliberately,
|
||||
but final handoff still needs production-loader coverage somewhere before the
|
||||
pipeline depends on that component.
|
||||
- Official reference imports must run in the current FastVideo environment; do
|
||||
not create or assume a separate upstream venv/conda env.
|
||||
- A scaffold is not evidence of correctness. It becomes evidence only after a
|
||||
local non-skip PASS.
|
||||
- Prefer env-var path overrides with repo-relative defaults.
|
||||
- Keep tests local-only under `tests/local_tests/`; package/CI quality tests are
|
||||
added later.
|
||||
- Update shared state files as described in
|
||||
`../add-model/shared/common_rules.md` whenever adding or activating parity
|
||||
tests.
|
||||
|
||||
## Component Template
|
||||
|
||||
Copy `templates/component_parity_test.py` and fill every `TODO` marker. The
|
||||
template is distilled from:
|
||||
|
||||
- `tests/local_tests/transformers/test_ltx2.py`
|
||||
- `tests/local_tests/transformers/test_gamecraft_parity.py`
|
||||
- `tests/local_tests/encoders/test_ltx2_gemma_parity.py`
|
||||
- `tests/local_tests/vaes/test_oobleck_vae_parity.py`
|
||||
- `tests/local_tests/sd35/test_sd35_component_parity.py`
|
||||
|
||||
The template supports three states:
|
||||
|
||||
| State | Meaning |
|
||||
|---|---|
|
||||
| Scaffold skip | Test is committed early, but official import, FastVideo class, or weights are not available yet. |
|
||||
| Debug red | Both sides load and the test fails numerically. This is useful: porting can chase the first drift. |
|
||||
| Non-skip pass | Required before `/add-model` handoff. |
|
||||
|
||||
## Subagent Dispatch Pattern
|
||||
|
||||
After Phase 1, dispatch one parity subagent per component before or alongside
|
||||
component implementation:
|
||||
|
||||
```text
|
||||
Create a local parity test scaffold for <family> <component>.
|
||||
|
||||
Use the prep handoff:
|
||||
- official_ref_dir/import: <...>
|
||||
- local_weights_dir: <...>
|
||||
- source_layout: <...>
|
||||
- needs_conversion: <yes/no>
|
||||
- official_env_status: <imports_ok | private_deps_need_stubs>
|
||||
- local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
- port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
- official_definition_files: <paths + classes/functions>
|
||||
- official_instantiation_files: <paths + factory/pipeline/config call sites + args>
|
||||
- concerns_or_unknowns: <known ambiguous inputs, outputs, deps, or args>
|
||||
|
||||
The complete per-component packet must match
|
||||
`../add-model/contracts/component_context.md`.
|
||||
|
||||
Read the official component call path and the planned FastVideo component API.
|
||||
Add tests/local_tests/<bucket>/test_<family>_<component>_parity.py based on
|
||||
add-model-02-parity/templates/component_parity_test.py.
|
||||
|
||||
The scaffold must load the official model with real weights when available,
|
||||
load the FastVideo model through the standardized config/class/loader path when
|
||||
available, create deterministic inputs, compare concrete outputs, and skip only
|
||||
when a dependency is genuinely missing. Do not make an unconditional skip or a
|
||||
shape-only test.
|
||||
```
|
||||
|
||||
## FastVideo Load Patterns
|
||||
|
||||
Pick the narrowest load path that matches the component:
|
||||
|
||||
| Component | Preferred FastVideo load path |
|
||||
|---|---|
|
||||
| DiT / transformer | Bucket config + model class, or `TransformerLoader` when testing converted Diffusers component dirs. |
|
||||
| VAE | VAE class `from_pretrained(...)` when implemented, or bucket config + class for local converted dirs. |
|
||||
| Text/image encoder | Bucket config + model class; pass HF subpaths from `local_weights_dir` or converted component dirs. |
|
||||
| Scheduler/conditioner | Native class/config plus exact official kwargs. |
|
||||
|
||||
For early scaffolds, an import of the planned FastVideo class may be inside a
|
||||
helper that calls `pytest.skip` if the class does not exist yet. Replace that
|
||||
skip with a real import once the component PR adds the class.
|
||||
|
||||
Direct class/config construction is allowed for implementation or subcomponent
|
||||
parity, such as connector-only encoder checks or official monolithic-checkpoint
|
||||
mapping tests. Label that scope explicitly and add separate production-loader
|
||||
coverage when converted component dirs are available.
|
||||
|
||||
## Official Load Patterns
|
||||
|
||||
- Clone/reference repo path: add its source dir to `sys.path` before imports.
|
||||
- HF/Diffusers reference: import only inside the test, not production code.
|
||||
- Private deps: add a helper under `tests/local_tests/helpers/` to install
|
||||
stubs before importing upstream modules; do not rely on an external upstream
|
||||
environment.
|
||||
- Gated HF repos: resolve `HF_TOKEN`, `HUGGINGFACE_HUB_TOKEN`, or `HF_API_KEY`
|
||||
under the token rules in `../add-model/shared/common_rules.md`.
|
||||
|
||||
## Non-Skip Activation Checklist
|
||||
|
||||
Before `/add-model` handoff, each scaffolded test must be activated:
|
||||
|
||||
```text
|
||||
[ ] Official side imports and loads real weights.
|
||||
[ ] FastVideo side imports and loads the converted or original weights.
|
||||
[ ] Test executes at least one real forward call on both sides.
|
||||
[ ] Test compares output tensors, not only shapes or state-dict keys.
|
||||
[ ] Local pytest output contains PASSED, not SKIPPED or XFAIL.
|
||||
[ ] Tolerance is justified for the component scope and kernel alignment.
|
||||
```
|
||||
|
||||
## Component Parity Details
|
||||
|
||||
Reference imports:
|
||||
|
||||
- Import from `official_ref_dir` or the recorded package/import path.
|
||||
- If upstream has private deps, add a helper under
|
||||
`tests/local_tests/helpers/<family>_upstream.py` that installs minimal stubs
|
||||
before importing upstream modules.
|
||||
- Common stubs: identity compile/op-registration decorators, CP world size set to
|
||||
1, identity scatter/gather, and test-friendly custom-op kernels.
|
||||
- Stub decorators that register `torch.ops.<ns>.<op>` must preserve the
|
||||
`torch.library` registration side-effect. Identity decorators alone are not
|
||||
enough.
|
||||
- Delete stub helpers and every `install_stubs()` call as soon as the real deps
|
||||
become required installs. No-op shims are dead code.
|
||||
|
||||
Kernel and wrapper pitfalls:
|
||||
|
||||
- If parity routes flash-attn GQA through SDPA, expand KV heads manually on the
|
||||
SDPA side with `repeat_interleave` along the head axis.
|
||||
- If upstream VAE `decode()` denormalizes internally but FastVideo/Diffusers
|
||||
expects pre-denormalized latents, apply `z = z * std + mean` only on the
|
||||
FastVideo side in the parity test.
|
||||
- Per-channel VAE `latents_mean` / `latents_std` must be reshaped explicitly,
|
||||
e.g. `.view(1, z_dim, 1, 1, 1)` for 5D video latents.
|
||||
|
||||
Tolerance guide:
|
||||
|
||||
| Scope | Start `atol` / `rtol` | Notes |
|
||||
|---|---|---|
|
||||
| Single block, same kernel | `1e-4` / `1e-4` | Tight default. |
|
||||
| Full DiT, aligned kernels | `1e-2` / `1e-2` | Cross-layer accumulation. |
|
||||
| Full DiT, cross-kernel bf16 | `0.1` / `0.1` | Also require abs-mean drift below 5% and per-modality diagnostics. |
|
||||
| VAE decode fp32 | `5e-2` / `5e-2` | After normalization alignment. |
|
||||
| Encoder wrapper around same HF class | `1e-3` / `1e-3` | Should be near-zero. |
|
||||
|
||||
Element-wise `assert_close` alone is not enough for deep full-DiT parity. Also
|
||||
log global abs-mean drift and per-modality summaries.
|
||||
|
||||
When a non-skip component parity run is numerically red after weight/input
|
||||
checks, invoke `../add-model-08-trace/SKILL.md` before adding bespoke forward
|
||||
hooks. Use `docs/contributing/activation_trace.md` to keep
|
||||
`FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and `FASTVIDEO_TRACE_STEPS`
|
||||
identical across FastVideo and upstream traces.
|
||||
|
||||
Useful local commands:
|
||||
|
||||
```bash
|
||||
pytest tests/local_tests/<bucket>/test_<family>_*parity*.py -v -s
|
||||
pytest tests/local_tests -k "<family> and parity" -v -s
|
||||
```
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md`. Parity-specific ask cases include
|
||||
private dependency approval, choosing between incompatible official references,
|
||||
accepting a shape-only substitute, or loosening required tolerances.
|
||||
|
||||
## Pipeline Parity
|
||||
|
||||
Pipeline parity is later than component parity because it needs stages, presets,
|
||||
registry wiring, converted weights, and green component parity. Record official
|
||||
pipeline call notes in `local_tests_readme`, but do not treat pipeline parity as
|
||||
owned by this skill.
|
||||
|
||||
Use `../add-model-09-pipeline/SKILL.md` and its
|
||||
`templates/pipeline_parity_test.py` for pipeline parity scaffolding and
|
||||
debugging. Compare denoised latents or decoded media, not just successful
|
||||
generation.
|
||||
|
||||
## Handoff Status Block
|
||||
|
||||
Return `../add-model/contracts/parity_status.md` to `/add-model` and update the
|
||||
shared state files before handoff.
|
||||
@@ -1,188 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Component parity scaffold for <FAMILY> <COMPONENT>.
|
||||
|
||||
This file is intended to be created early in a port. It may skip until the
|
||||
official reference, FastVideo class, and real weights are available, but it must
|
||||
never become an unconditional skip or shape-only test.
|
||||
|
||||
Fill every TODO before considering this test active.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from torch.testing import assert_close
|
||||
|
||||
os.environ.setdefault("MASTER_ADDR", "localhost")
|
||||
os.environ.setdefault("MASTER_PORT", "29519")
|
||||
os.environ.setdefault("DISABLE_SP", "1")
|
||||
os.environ.setdefault("FASTVIDEO_ATTENTION_BACKEND", "TORCH_SDPA")
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
FAMILY = "<family>" # TODO: snake_case family name.
|
||||
COMPONENT = "<component>" # TODO: transformer | vae | encoder | conditioner | ...
|
||||
PARITY_SCOPE = "implementation_subcomponent" # TODO: production_loader | implementation_subcomponent | both
|
||||
OFFICIAL_MODULE = "<official.module>" # TODO: e.g. "ltx_core.model.transformer".
|
||||
OFFICIAL_CLASS = "<OfficialClass>" # TODO: official class/factory name.
|
||||
FASTVIDEO_CONFIG_MODULE = "fastvideo.configs.models.<bucket>" # TODO.
|
||||
FASTVIDEO_CONFIG_CLASS = "<FastVideoConfig>" # TODO.
|
||||
FASTVIDEO_MODEL_MODULE = "fastvideo.models.<bucket>.<module>" # TODO.
|
||||
FASTVIDEO_MODEL_CLASS = "<FastVideoModel>" # TODO.
|
||||
|
||||
OFFICIAL_REF_DIR = Path(os.getenv("<FAMILY_UPPER>_OFFICIAL_REF_DIR", REPO_ROOT / "<ReferenceDir>"))
|
||||
LOCAL_WEIGHTS_DIR = Path(os.getenv("<FAMILY_UPPER>_LOCAL_WEIGHTS_DIR", REPO_ROOT / "official_weights" / FAMILY))
|
||||
CONVERTED_WEIGHTS_DIR = Path(os.getenv("<FAMILY_UPPER>_CONVERTED_WEIGHTS_DIR",
|
||||
REPO_ROOT / "converted_weights" / FAMILY))
|
||||
|
||||
|
||||
def _resolve_hf_token() -> str | None:
|
||||
for key in ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY"):
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _add_official_to_path() -> None:
|
||||
"""Add the official source path before importing upstream modules."""
|
||||
# TODO: adjust for the official repo layout. Common examples:
|
||||
# OFFICIAL_REF_DIR / "src"
|
||||
# OFFICIAL_REF_DIR / "packages" / "<pkg>" / "src"
|
||||
# OFFICIAL_REF_DIR
|
||||
official_src = OFFICIAL_REF_DIR / "src"
|
||||
if not official_src.exists():
|
||||
official_src = OFFICIAL_REF_DIR
|
||||
if official_src.exists() and str(official_src) not in sys.path:
|
||||
sys.path.insert(0, str(official_src))
|
||||
|
||||
|
||||
def _import_or_skip(module_name: str, attr_name: str | None = None):
|
||||
if "<" in module_name or (attr_name is not None and "<" in attr_name):
|
||||
pytest.skip(f"Template import placeholder not filled: {module_name}.{attr_name}")
|
||||
try:
|
||||
module = importlib.import_module(module_name)
|
||||
except Exception as exc: # noqa: BLE001 - local parity should skip missing refs.
|
||||
pytest.skip(f"Cannot import {module_name}: {exc}")
|
||||
if attr_name is None:
|
||||
return module
|
||||
try:
|
||||
return getattr(module, attr_name)
|
||||
except AttributeError:
|
||||
pytest.skip(f"{module_name} has no attribute {attr_name}")
|
||||
|
||||
|
||||
def _load_official_model(device: torch.device, dtype: torch.dtype) -> torch.nn.Module:
|
||||
"""Load the official component with real weights."""
|
||||
_add_official_to_path()
|
||||
if not OFFICIAL_REF_DIR.exists():
|
||||
pytest.skip(f"Official reference missing: {OFFICIAL_REF_DIR}")
|
||||
if not LOCAL_WEIGHTS_DIR.exists():
|
||||
pytest.skip(f"Local weights missing: {LOCAL_WEIGHTS_DIR}")
|
||||
|
||||
# TODO: import official class/factory and load real weights strictly.
|
||||
# Examples in-tree:
|
||||
# - LTX2: SingleGPUModelBuilder(...).build(device=device, dtype=dtype)
|
||||
# - GameCraft: torch.load(...)["module"] -> official_model.load_state_dict(...)
|
||||
# - Oobleck: create_model_from_config(config) + ckpt state_dict
|
||||
OfficialClass = _import_or_skip(OFFICIAL_MODULE, OFFICIAL_CLASS)
|
||||
model = OfficialClass() # TODO: pass official config kwargs.
|
||||
state_dict = {} # TODO: load official state dict from LOCAL_WEIGHTS_DIR.
|
||||
missing, unexpected = model.load_state_dict(state_dict, strict=True)
|
||||
assert not missing and not unexpected, (f"official load mismatch missing={missing[:5]} unexpected={unexpected[:5]}")
|
||||
return model.to(device=device, dtype=dtype).eval()
|
||||
|
||||
|
||||
def _load_fastvideo_model(device: torch.device, dtype: torch.dtype) -> torch.nn.Module:
|
||||
"""Load the FastVideo component with the same tensor content."""
|
||||
if not CONVERTED_WEIGHTS_DIR.exists() and not LOCAL_WEIGHTS_DIR.exists():
|
||||
pytest.skip(f"No FastVideo loadable weights: {CONVERTED_WEIGHTS_DIR} or {LOCAL_WEIGHTS_DIR}")
|
||||
|
||||
# TODO: replace with the bucket-specific FastVideo config/class/loader.
|
||||
# DiT examples:
|
||||
# from fastvideo.configs.models.dits import <Config>
|
||||
# from fastvideo.models.dits.<module> import <Model>
|
||||
# VAE examples:
|
||||
# from fastvideo.models.vaes.<module> import <VAE>
|
||||
# model = <VAE>.from_pretrained(...)
|
||||
FastVideoConfig = _import_or_skip(FASTVIDEO_CONFIG_MODULE, FASTVIDEO_CONFIG_CLASS)
|
||||
FastVideoModel = _import_or_skip(FASTVIDEO_MODEL_MODULE, FASTVIDEO_MODEL_CLASS)
|
||||
|
||||
config = FastVideoConfig()
|
||||
model = FastVideoModel(config=config)
|
||||
state_dict = {} # TODO: load converted or directly mapped state dict.
|
||||
missing, unexpected = model.load_state_dict(state_dict, strict=True)
|
||||
assert not missing and not unexpected, (
|
||||
f"FastVideo load mismatch missing={missing[:5]} unexpected={unexpected[:5]}")
|
||||
return model.to(device=device, dtype=dtype).eval()
|
||||
|
||||
|
||||
def _make_inputs(device: torch.device, dtype: torch.dtype) -> dict[str, torch.Tensor]:
|
||||
"""Create deterministic inputs matching the official component call."""
|
||||
torch.manual_seed(0)
|
||||
# TODO: replace with component-specific tensors and metadata.
|
||||
return {
|
||||
"hidden_states": torch.randn(1, 4, 16, device=device, dtype=dtype),
|
||||
"timestep": torch.tensor([10], device=device),
|
||||
}
|
||||
|
||||
|
||||
def _run_official(model: torch.nn.Module, inputs: dict[str, torch.Tensor]) -> torch.Tensor:
|
||||
"""Run official component and return the tensor to compare."""
|
||||
with torch.inference_mode():
|
||||
output = model(**inputs) # TODO: adapt official call signature.
|
||||
if isinstance(output, dict):
|
||||
sample = output.get("sample")
|
||||
output = sample if sample is not None else output.get("x")
|
||||
elif hasattr(output, "sample"):
|
||||
output = output.sample
|
||||
elif isinstance(output, tuple):
|
||||
output = output[0]
|
||||
assert torch.is_tensor(output), f"official output is not tensor: {type(output)}"
|
||||
return output.detach().float().cpu()
|
||||
|
||||
|
||||
def _run_fastvideo(model: torch.nn.Module, inputs: dict[str, torch.Tensor]) -> torch.Tensor:
|
||||
"""Run FastVideo component and return the tensor to compare."""
|
||||
with torch.inference_mode():
|
||||
output = model(**inputs) # TODO: adapt FastVideo call signature.
|
||||
if isinstance(output, dict):
|
||||
sample = output.get("sample")
|
||||
output = sample if sample is not None else output.get("x")
|
||||
elif hasattr(output, "sample"):
|
||||
output = output.sample
|
||||
elif isinstance(output, tuple):
|
||||
output = output[0]
|
||||
assert torch.is_tensor(output), f"FastVideo output is not tensor: {type(output)}"
|
||||
return output.detach().float().cpu()
|
||||
|
||||
|
||||
@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required for this parity test.")
|
||||
def test_component_parity():
|
||||
"""Compare official and FastVideo outputs on identical inputs."""
|
||||
device = torch.device("cuda:0")
|
||||
dtype = torch.bfloat16
|
||||
|
||||
official = _load_official_model(device, dtype)
|
||||
fastvideo = _load_fastvideo_model(device, dtype)
|
||||
inputs = _make_inputs(device, dtype)
|
||||
|
||||
official_out = _run_official(official, inputs)
|
||||
fastvideo_out = _run_fastvideo(fastvideo, inputs)
|
||||
|
||||
assert official_out.shape == fastvideo_out.shape
|
||||
diff = (official_out - fastvideo_out).abs()
|
||||
print(f"official abs_mean={official_out.abs().mean().item():.6f} "
|
||||
f"fastvideo abs_mean={fastvideo_out.abs().mean().item():.6f} "
|
||||
f"diff_max={diff.max().item():.6f} diff_mean={diff.mean().item():.6f}")
|
||||
|
||||
# TODO: pick tolerance by scope:
|
||||
# - single block / same kernel: 1e-4
|
||||
# - full DiT aligned kernels: 1e-2
|
||||
# - full DiT cross-kernel bf16: 1e-1 + abs_mean drift check
|
||||
# - VAE decode fp32: 5e-2 after normalization alignment
|
||||
assert_close(fastvideo_out, official_out, atol=1e-4, rtol=1e-4)
|
||||
@@ -1,114 +0,0 @@
|
||||
---
|
||||
name: add-model-03-port-dit
|
||||
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native DiT/transformer component.
|
||||
---
|
||||
|
||||
# Add Model Port DiT
|
||||
|
||||
## Goal
|
||||
|
||||
Prototype or parity-debug one diffusion transformer in FastVideo-native code.
|
||||
This skill is for one component only; do not work on the VAE, encoders,
|
||||
pipeline, or unrelated conversion code unless the current component cannot load
|
||||
without a minimal fix there.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/component_skill_common.md` and require the complete
|
||||
packet from `../add-model/contracts/component_context.md`.
|
||||
|
||||
DiT-specific packet fields:
|
||||
|
||||
- `component`: transformer or DiT name.
|
||||
- `parity_test`: `tests/local_tests/<bucket>/test_<family>_<component>_parity.py`.
|
||||
- `weights`: converted transformer dir or local official path.
|
||||
- `target_files`: `fastvideo/models/dits/<family>.py` and
|
||||
`fastvideo/configs/models/dits/<family>.py`.
|
||||
|
||||
## Modes
|
||||
|
||||
Use the common prototype and parity-debug modes from
|
||||
`../add-model/shared/component_skill_common.md`.
|
||||
|
||||
DiT-specific prototype concerns include ambiguous official flags, shape
|
||||
mismatches, missing FastVideo layer equivalents, and dedicated output heads.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
Apply the shared reuse proof. DiT-specific comparison must include attention
|
||||
algorithm, positional embeddings, RoPE/patching, timestep/guidance embeddings,
|
||||
scaling constants, dtype casts, state-dict names, and every output head.
|
||||
|
||||
## Existing FastVideo Patterns
|
||||
|
||||
- Base class: `fastvideo/models/dits/base.py::BaseDiT`.
|
||||
- Config bases: `DiTConfig` and `DiTArchConfig` in
|
||||
`fastvideo/configs/models/dits/base.py`.
|
||||
- Use the matching DiT config bucket. Wrong bucket inheritance can typecheck but
|
||||
fail during pipeline wiring.
|
||||
- Config export: add the config to
|
||||
`fastvideo/configs/models/dits/__init__.py`.
|
||||
- Registry discovery: set `EntryClass = <ClassName>` in the model file.
|
||||
- Loader path: `TransformerLoader` reads `transformer/config.json`, calls
|
||||
`dit_config.update_model_arch(config)`, resolves `_class_name` through
|
||||
`ModelRegistry`, and constructs the class with `config` and `hf_config`.
|
||||
- Reference examples: `stable_audio.py`, `wanvideo.py`, `sd3.py`, `longcat.py`,
|
||||
and `ltx2.py`.
|
||||
- Layer guidance: `fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Use FastVideo-native layers by default: `ReplicatedLinear` for DiT hot-path
|
||||
linears, `DistributedAttention` for standard full-sequence attention, and
|
||||
`LocalAttention` for local/window attention or simple single-GPU parity paths.
|
||||
- Raw SDPA is acceptable for cross-modality flat streams when no FastVideo
|
||||
distributed equivalent exists; document the SP gap in the module docstring.
|
||||
- Mirror official tensor contracts exactly: latent packing, patch ordering,
|
||||
timestep embedding scale, RoPE/positional embedding, guidance embedding,
|
||||
cross-attention context order, output head order, and dtype casts.
|
||||
- Preserve all output heads that the official DiT emits. Do not silently drop
|
||||
audio, depth, pose, mask, or auxiliary heads.
|
||||
- Put architecture fields on `DiTArchConfig`; keep inference steps, CFG scales,
|
||||
FPS, flow shift, and sampling defaults out of the arch config.
|
||||
- Define `_fsdp_shard_conditions`, `_compile_conditions`,
|
||||
`param_names_mapping`, and `reverse_param_names_mapping` where needed.
|
||||
- Follow the production import boundary in
|
||||
`../add-model/shared/common_rules.md`.
|
||||
|
||||
## Prototype Checks
|
||||
|
||||
Follow the shared prototype success criteria. A useful one-off check is:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
# Import the target config/class, instantiate with random weights, and print
|
||||
# state_dict names/shapes for the conversion mapping.
|
||||
PY
|
||||
```
|
||||
|
||||
## Parity-Debug Loop
|
||||
|
||||
Run the shared parity-debug loop. The component test command is:
|
||||
|
||||
```bash
|
||||
pytest <parity_test> -v -s
|
||||
```
|
||||
|
||||
For numerical drift, use `../add-model-08-trace/SKILL.md` before writing bespoke
|
||||
hooks. Start with FastVideo's activation trace (`fastvideo/hooks/activation_trace.py`;
|
||||
`docs/contributing/activation_trace.md`) and a block-level regex such as
|
||||
`FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$"`. Only fall back to custom
|
||||
per-block hooks if the needed boundary or statistic is not exposed by
|
||||
`FASTVIDEO_TRACE_STATS`.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
|
||||
in `../add-model/shared/component_skill_common.md`. DiT-specific ask cases include
|
||||
dropping an output head/modality, accepting an unsupported kernel/private op, or
|
||||
choosing between incompatible official transformer definitions.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/component_skill_handoff.md` following the common
|
||||
handoff rules in `../add-model/shared/component_skill_common.md`.
|
||||
@@ -1,100 +0,0 @@
|
||||
---
|
||||
name: add-model-04-port-vae
|
||||
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native VAE component.
|
||||
---
|
||||
|
||||
# Add Model Port VAE
|
||||
|
||||
## Goal
|
||||
|
||||
Prototype or parity-debug one VAE or autoencoder in FastVideo-native code. This
|
||||
skill covers video, image, and audio VAEs.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/component_skill_common.md` and require the complete
|
||||
packet from `../add-model/contracts/component_context.md`.
|
||||
|
||||
VAE-specific packet fields:
|
||||
|
||||
- `component`: VAE or autoencoder name.
|
||||
- `parity_test`: `tests/local_tests/vaes/test_<family>_<component>_parity.py`.
|
||||
- `weights`: converted VAE dir, HF subfolder, or local official path.
|
||||
- `target_files`: `fastvideo/models/vaes/<arch_or_family>.py` and
|
||||
`fastvideo/configs/models/vaes/<arch_or_family>.py`.
|
||||
|
||||
## Modes
|
||||
|
||||
Use the common prototype and parity-debug modes from
|
||||
`../add-model/shared/component_skill_common.md`.
|
||||
|
||||
VAE-specific prototype concerns include latent normalization, stochastic
|
||||
posterior behavior, tiling incompatibility, temporal/spatial/audio layout, and
|
||||
decode output containers.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
Apply the shared reuse proof. VAE-specific comparison must include latent layout,
|
||||
temporal/spatial/audio compression, scaling factor, mean/std normalization,
|
||||
posterior behavior, encode/decode output objects, tiling flags, and cropping.
|
||||
|
||||
## Existing FastVideo Patterns
|
||||
|
||||
- Shared tiling wrapper: `fastvideo/models/vaes/common.py::ParallelTiledVAE`.
|
||||
- Config bases: `VAEConfig` and `VAEArchConfig` in
|
||||
`fastvideo/configs/models/vaes/base.py`.
|
||||
- Use the matching VAE config bucket. Wrong bucket inheritance can typecheck but
|
||||
fail during pipeline wiring.
|
||||
- Config export: add the config to
|
||||
`fastvideo/configs/models/vaes/__init__.py`.
|
||||
- Registry discovery: set `EntryClass = <ClassName>` in the model file.
|
||||
- Loader path: VAE loaders resolve `_class_name` through `ModelRegistry` and
|
||||
load converted component weights from the VAE subdir.
|
||||
- Reference examples: `oobleck.py`, `autoencoder_kl.py`, `wanvae.py`,
|
||||
`ltx2vae.py`, and `gamecraftvae.py`.
|
||||
- Layer guidance: `fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Name reusable VAE architectures by architecture (`oobleck.py`,
|
||||
`autoencoder_kl.py`); name family-specific VAEs by family.
|
||||
- Match official encode/decode contracts exactly: input layout, latent layout,
|
||||
temporal/spatial/audio compression, scaling factor, mean/std normalization,
|
||||
posterior sampling behavior, decode output object, and frame/sample cropping.
|
||||
- Compare deterministic outputs in parity: decode outputs, encode mean/mode, or
|
||||
round-trip tensors. Do not compare stochastic samples unless the RNG path is
|
||||
explicitly controlled.
|
||||
- Use FastVideo tiling only when it preserves official numerics for the tested
|
||||
shape; disable it in config for audio or unsupported dimensions.
|
||||
- Put architecture constants on `VAEArchConfig`; put `load_encoder`,
|
||||
`load_decoder`, tiling, dtype, and pretrained path fields on `VAEConfig`.
|
||||
- Follow the production import boundary in
|
||||
`../add-model/shared/common_rules.md`.
|
||||
|
||||
## Prototype Checks
|
||||
|
||||
Follow the shared prototype success criteria.
|
||||
|
||||
## Parity-Debug Loop
|
||||
|
||||
Run the shared parity-debug loop. The component test command is:
|
||||
|
||||
```bash
|
||||
pytest <parity_test> -v -s
|
||||
```
|
||||
|
||||
For numerical drift, check normalization, latent scaling, posterior mode vs
|
||||
sample, channel order, and temporal/spatial/audio cropping before changing
|
||||
layers.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
|
||||
in `../add-model/shared/component_skill_common.md`. VAE-specific ask cases include
|
||||
dropping an encode/decode path, accepting an unsupported private op, or choosing
|
||||
between incompatible official VAE definitions.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/component_skill_handoff.md` following the common
|
||||
handoff rules in `../add-model/shared/component_skill_common.md`.
|
||||
@@ -1,119 +0,0 @@
|
||||
---
|
||||
name: add-model-05-port-encoder
|
||||
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native text, image, audio, or compound encoder component.
|
||||
---
|
||||
|
||||
# Add Model Port Encoder
|
||||
|
||||
## Goal
|
||||
|
||||
Prototype or parity-debug one encoder or encoder-like conditioner in
|
||||
FastVideo-native code. Use this for text encoders, image encoders, audio
|
||||
encoders, and compound conditioners that fit the encoder config/loader bucket.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/component_skill_common.md` and require the complete
|
||||
packet from `../add-model/contracts/component_context.md`.
|
||||
|
||||
Encoder-specific packet fields:
|
||||
|
||||
- `component`: encoder or encoder-like conditioner name.
|
||||
- `parity_test`: `tests/local_tests/encoders/test_<family>_<component>_parity.py`.
|
||||
- `weights`: converted encoder dir, HF subfolder, or external HF id.
|
||||
- `target_files`: `fastvideo/models/encoders/<arch_or_family>.py` and
|
||||
`fastvideo/configs/models/encoders/<arch_or_family>.py`.
|
||||
|
||||
## Modes
|
||||
|
||||
Use the common prototype and parity-debug modes from
|
||||
`../add-model/shared/component_skill_common.md`.
|
||||
|
||||
Encoder-specific prototype concerns include tokenizer kwargs, hidden-state
|
||||
extraction, output packing, connector order, and external/passthrough weight
|
||||
needs.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
Apply the shared reuse proof. Encoder-specific comparison must include tokenizer
|
||||
contracts, hidden-state extraction, masks, positional IDs, output packing,
|
||||
connector/projection ordering, passthrough paths, and returned dataclass shape.
|
||||
|
||||
## Existing FastVideo Patterns
|
||||
|
||||
- Base classes: `TextEncoder` and `ImageEncoder` in
|
||||
`fastvideo/models/encoders/base.py`.
|
||||
- Output type: `BaseEncoderOutput`.
|
||||
- Config bases: `TextEncoderConfig`, `ImageEncoderConfig`,
|
||||
`TextEncoderArchConfig`, and `ImageEncoderArchConfig` in
|
||||
`fastvideo/configs/models/encoders/base.py`.
|
||||
- Use the matching encoder config bucket. Wrong bucket inheritance can typecheck
|
||||
but fail during pipeline wiring.
|
||||
- Config export: add the config to
|
||||
`fastvideo/configs/models/encoders/__init__.py`.
|
||||
- Registry discovery: set `EntryClass = <ClassName>` or a list of class names in
|
||||
the model file.
|
||||
- Reference examples: native `t5.py`, `clip.py`, `siglip.py`, `llama.py`,
|
||||
`qwen2_5.py`, `gemma.py`, and compound `stable_audio_conditioner.py`.
|
||||
- Layer guidance: `fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Reuse tokenizers and pure data utilities when needed, but do not add runtime
|
||||
third-party model-class imports as a placeholder for a component that owns
|
||||
weights or numerical behavior.
|
||||
- For LLM-style encoders, follow existing tensor-parallel patterns such as
|
||||
`QKVParallelLinear`, `MergedColumnParallelLinear`, `RowParallelLinear`,
|
||||
`VocabParallelEmbedding`, and `RMSNorm` when matching native examples.
|
||||
- Match official hidden-state extraction exactly: layer index, pooled output,
|
||||
attention mask dtype, padding side, truncation, special tokens, final norm,
|
||||
output_hidden_states, and returned tuple/dataclass shape.
|
||||
- For connector or conditioner modules, preserve sub-conditioner order and the
|
||||
exact packing of cross-attention tokens, masks, and global conditioning.
|
||||
- Put tokenizer kwargs and architecture constants on the arch config when they
|
||||
affect numerical behavior.
|
||||
- If an external HF encoder is explicitly accepted as a lazy wrapper, keep it
|
||||
isolated, document why it is not a native port, and still require parity for
|
||||
the wrapper's output contract.
|
||||
|
||||
Hybrid external-HF encoder checklist:
|
||||
|
||||
- Put external model folders in passthrough subfolders such as
|
||||
`text_encoder/<external_name>/`, or record a root `model_index.json` path field
|
||||
that the loader resolves to a local directory.
|
||||
- Keep external model parameters out of the FastVideo-owned state-dict surface
|
||||
when the external model is loaded lazily from its own HF files.
|
||||
- Convert and strict-check only the FastVideo-owned connector/projection weights;
|
||||
document external model weights as passthrough.
|
||||
- Add parity for the wrapper's final output contract and, when useful, a narrower
|
||||
connector-only parity test that labels its scope as
|
||||
`implementation_subcomponent`.
|
||||
- Verify the production loader resolves the same external path used by the
|
||||
pipeline, not just the direct class used in the parity test.
|
||||
|
||||
## Prototype Checks
|
||||
|
||||
Follow the shared prototype success criteria.
|
||||
|
||||
## Parity-Debug Loop
|
||||
|
||||
Run the shared parity-debug loop. The component test command is:
|
||||
|
||||
```bash
|
||||
pytest <parity_test> -v -s
|
||||
```
|
||||
|
||||
For numerical drift, check tokenization, masks, hidden-state selection,
|
||||
positional IDs, dtype/autocast, and output packing before changing layers.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
|
||||
in `../add-model/shared/component_skill_common.md`. Encoder-specific ask cases
|
||||
include accepting private model-code execution, choosing between incompatible
|
||||
tokenizer/encoder references, or dropping a required conditioning stream.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/component_skill_handoff.md` following the common
|
||||
handoff rules in `../add-model/shared/component_skill_common.md`.
|
||||
@@ -1,111 +0,0 @@
|
||||
---
|
||||
name: add-model-06-port-generic
|
||||
description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one non-DiT, non-VAE, non-encoder FastVideo component.
|
||||
---
|
||||
|
||||
# Add Model Port Generic
|
||||
|
||||
## Goal
|
||||
|
||||
Prototype or parity-debug one scheduler, conditioner, upsampler, vocoder,
|
||||
adapter, preprocessor, or unknown component in FastVideo-native code.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/component_skill_common.md` and require the complete
|
||||
packet from `../add-model/contracts/component_context.md`.
|
||||
|
||||
Generic-component packet fields:
|
||||
|
||||
- `component`: component name.
|
||||
- `component_type`: scheduler, conditioner, upsampler, vocoder, adapter,
|
||||
preprocessor, or unknown.
|
||||
- `parity_test`: `tests/local_tests/<bucket>/test_<family>_<component>_parity.py`.
|
||||
- `weights`: converted component dir, HF subfolder, or none.
|
||||
- `target_files`: matching `fastvideo/models/` and `fastvideo/configs/models/`
|
||||
bucket files when applicable.
|
||||
|
||||
## Modes
|
||||
|
||||
Use the common prototype and parity-debug modes from
|
||||
`../add-model/shared/component_skill_common.md`.
|
||||
|
||||
Generic-component prototype concerns include stateless/stateful ambiguity,
|
||||
missing loader buckets, source prefixes, mutable scheduler state, and output
|
||||
container shape.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
Apply the shared reuse proof. Generic-component comparison must include mutable
|
||||
state, scaling constants, scheduler/conditioner semantics, output containers, and
|
||||
whether the component owns state or is stateless.
|
||||
|
||||
## Existing FastVideo Patterns
|
||||
|
||||
- Schedulers live under `fastvideo/models/schedulers/` and expose `EntryClass`.
|
||||
- Upsamplers use `fastvideo/models/upsamplers/` plus configs under
|
||||
`fastvideo/configs/models/upsamplers/`; see `hunyuan15.py`.
|
||||
- Vocoders and audio-specific modules can live under `fastvideo/models/audio/`
|
||||
with configs under `fastvideo/configs/models/audio/`; see `ltx2_audio_vae.py`.
|
||||
- Compound conditioners may fit the encoder bucket when the pipeline loader uses
|
||||
`ConditionerLoader`; see `stable_audio_conditioner.py`.
|
||||
- Registry discovery uses `EntryClass`; config bucket exports are required when
|
||||
pipeline configs import them by bucket.
|
||||
- Use the narrowest matching config bucket. Wrong bucket inheritance can typecheck
|
||||
but fail during pipeline wiring.
|
||||
- Layer guidance: `fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Bucket Decision
|
||||
|
||||
- If the component is a transformer/DiT, stop and use `add-model-03-port-dit`.
|
||||
- If the component is a VAE/autoencoder, stop and use `add-model-04-port-vae`.
|
||||
- If the component is a text/image/audio encoder or encoder-like conditioner,
|
||||
stop and use `add-model-05-port-encoder` unless the loader requires a different
|
||||
bucket.
|
||||
- Otherwise choose the narrowest existing bucket. Add a new bucket only when no
|
||||
existing loader/config shape can represent the component without misleading
|
||||
names or unsafe runtime behavior.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Match official behavior, not just shapes: constructor args, default values,
|
||||
runtime flags, RNG use, dtype/autocast, scaling constants, masks, and output
|
||||
containers all matter.
|
||||
- Keep the implementation minimal and native. Do not keep a runtime import of
|
||||
the official implementation as the production component.
|
||||
- For schedulers, compare timesteps, sigmas/noise levels, step outputs, shift
|
||||
handling, prediction type, and any mutable internal state.
|
||||
- For upsamplers, compare resize mode, align_corners, residual branches,
|
||||
causal padding, normalization, and exact target-shape behavior.
|
||||
- For vocoders/audio components, compare waveform shape, sample-rate contract,
|
||||
channel order, hop length, normalization, and dtype.
|
||||
- If private upstream deps are required only for tests, keep stubs under
|
||||
`tests/local_tests/helpers/` and do not import them from production code.
|
||||
|
||||
## Prototype Checks
|
||||
|
||||
Follow the shared prototype success criteria.
|
||||
|
||||
## Parity-Debug Loop
|
||||
|
||||
Run the shared parity-debug loop. The component test command is:
|
||||
|
||||
```bash
|
||||
pytest <parity_test> -v -s
|
||||
```
|
||||
|
||||
For numerical drift, add targeted intermediate comparisons in the test to
|
||||
identify the first divergent operation.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` and the component-specific guidance
|
||||
in `../add-model/shared/component_skill_common.md`. Generic-component ask cases
|
||||
include creating a new loader bucket, accepting an unsupported private op,
|
||||
choosing between incompatible official definitions, or dropping a required
|
||||
component.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/component_skill_handoff.md` following the common
|
||||
handoff rules in `../add-model/shared/component_skill_common.md`.
|
||||
@@ -1,186 +0,0 @@
|
||||
---
|
||||
name: add-model-07-conversion
|
||||
description: Use during /add-model Phase 5 to write and verify a FastVideo checkpoint conversion script after native component prototypes expose FastVideo state-dict keys/shapes.
|
||||
---
|
||||
|
||||
# Add Model Conversion
|
||||
|
||||
## Goal
|
||||
|
||||
Convert official weights into a FastVideo-loadable component layout after Phase 4
|
||||
native prototypes exist. The conversion script owns parameter mapping, component
|
||||
splitting, passthrough assets, config emission, and strict-load verification.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
|
||||
escape hatches, production boundaries, and skip/pass semantics.
|
||||
|
||||
Require the initial request from
|
||||
`../add-model/contracts/conversion_request.md`.
|
||||
|
||||
If the FastVideo key/shape dump is missing, return to `/add-model` Phase 4. Do
|
||||
not write a final mapping against an unimplemented component.
|
||||
|
||||
For Phase 6 retry requests from component skills, also require the retry shape
|
||||
from `../add-model/contracts/conversion_request.md`.
|
||||
|
||||
## Output
|
||||
|
||||
- `scripts/checkpoint_conversion/<family>_to_diffusers.py`.
|
||||
- `converted_weights/<family>/` with `model_index.json` and per-component
|
||||
subfolders.
|
||||
- Updated `tests/local_tests/<model_family>/README.md` with conversion command,
|
||||
source layout, output path, and strict-load status.
|
||||
- Updated `tests/local_tests/<model_family>/PORT_STATUS.md` with conversion
|
||||
state, retry history, open questions, and issues/blockers.
|
||||
|
||||
## Reference Scripts
|
||||
|
||||
- `scripts/checkpoint_conversion/convert_ltx2_weights.py`: component prefix
|
||||
splitting, metadata config extraction, passthrough Gemma/tokenizer assets, and
|
||||
optional component-only output.
|
||||
- `scripts/checkpoint_conversion/stable_audio_to_diffusers.py`: monolithic
|
||||
`model.safetensors` split into transformer/VAE/conditioner, plus copied
|
||||
passthrough subfolders. Use this shape for single-checkpoint official repos.
|
||||
- `scripts/checkpoint_conversion/convert_gamecraft_full.py`: separate official
|
||||
sources for transformer, VAE, text encoders, tokenizers, scheduler, and root
|
||||
`model_index.json`.
|
||||
- `scripts/checkpoint_conversion/longcat_to_fastvideo.py`: fused QKV/KV split,
|
||||
renamed native transformer weights, and copied existing Diffusers components.
|
||||
- `scripts/checkpoint_conversion/pt_to_safetensors.py`: simple `.pt` extraction
|
||||
helper for nested checkpoint dictionaries.
|
||||
|
||||
## Source Layout Decision
|
||||
|
||||
Choose exactly one primary layout:
|
||||
|
||||
| Layout | Conversion behavior |
|
||||
|---|---|
|
||||
| `diffusers` | Usually no tensor remap; verify configs/classes and copy or update `_class_name` only when needed. |
|
||||
| `raw_official` | Convert a raw official checkpoint file or directory. Choose explicit component ownership before writing output. |
|
||||
| `separate_components` | Convert/copy each component from its own file or directory. |
|
||||
| `monolithic` | Load one model checkpoint and split state dict by authoritative prefixes into component buckets. |
|
||||
| `mixed` | Convert some components and copy passthrough components such as tokenizers, text encoders, schedulers, or already-Diffusers VAE dirs. |
|
||||
| `custom` | Document why none of the above fits before writing conversion code. |
|
||||
|
||||
Monolithic checkpoints need explicit prefix ownership. For example, Stable Audio
|
||||
uses one `model.safetensors` with DiT, pretransform/VAE, and conditioner keys;
|
||||
the converter splits those keys into FastVideo component subfolders and writes
|
||||
per-component configs.
|
||||
|
||||
## Script Shape
|
||||
|
||||
Start from `templates/family_to_diffusers.py` or the closest reference script.
|
||||
Keep the script explicit and reviewable:
|
||||
|
||||
- `COMPONENT_SPECS` or `COMPONENT_PREFIXES` declares component ownership.
|
||||
- `PARAM_NAME_MAP` declares key renames.
|
||||
- `SKIP_PATTERNS` declares intentionally dropped training-only keys.
|
||||
- tensor split/fuse helpers are named by operation, e.g. `split_qkv`.
|
||||
- `build_component_configs(...)` writes loader-compatible config files. Most
|
||||
model components use `config.json`; schedulers use `scheduler_config.json`.
|
||||
- `build_model_index(...)` writes a root `model_index.json` matching the target
|
||||
FastVideo pipeline and component classes.
|
||||
- verification reports missing, unexpected, skipped, unchanged, renamed, and
|
||||
shape-mismatched keys.
|
||||
|
||||
`model_index.json` library tokens must match FastVideo loaders:
|
||||
|
||||
- standard native DiT/VAE/audio/vocoder/upsampler components loaded by existing
|
||||
Diffusers-style loaders usually use `"diffusers"` with a FastVideo
|
||||
`_class_name` in the component `config.json`;
|
||||
- text encoders, tokenizers, image encoders, processors, and feature extractors
|
||||
usually use `"transformers"`;
|
||||
- `conditioner` currently expects `"fastvideo"`;
|
||||
- use fully qualified `"fastvideo.<module>"` only when intentionally relying on
|
||||
the custom fastvideo-library escape path;
|
||||
- do not write bare `"fastvideo"` for transformer, VAE, or other loaders that
|
||||
expect `"diffusers"` unless the loader explicitly expects it.
|
||||
|
||||
## Mapping Rules
|
||||
|
||||
- Use Phase 4 key/shape dumps to derive mappings. Do not guess from official key
|
||||
names alone.
|
||||
- Preserve each component's official file paths, parity test path, and prototype
|
||||
concerns in comments or structured constants near the mapping that uses them.
|
||||
- Every official inference parameter should be mapped, copied through, or listed
|
||||
as intentionally skipped with a reason.
|
||||
- Every FastVideo prototype parameter should receive a tensor or be listed as an
|
||||
intentional external/passthrough parameter.
|
||||
- Shape matches are necessary but not sufficient; check semantic pairing for
|
||||
Q/K/V, gate/up/down, norm scale/bias, LoRA/base, and modality-specific heads.
|
||||
- If official and FastVideo fuse or split tensors differently, convert tensors in
|
||||
the script rather than changing production code to match checkpoint quirks.
|
||||
|
||||
## Verification
|
||||
|
||||
Run conversion locally, then verify before returning to Phase 6. For retry
|
||||
requests, update the mapping, rerun conversion, and refresh only the implicated
|
||||
converted component when safe; otherwise rerun the full conversion.
|
||||
|
||||
```bash
|
||||
python scripts/checkpoint_conversion/<family>_to_diffusers.py \
|
||||
--src <official_weights> \
|
||||
--revision <hf_revision> \
|
||||
--dst converted_weights/<model_family>
|
||||
```
|
||||
|
||||
Omit `--revision` for local sources or when prep recorded `default` / `none`.
|
||||
|
||||
Minimum output layout:
|
||||
|
||||
```text
|
||||
converted_weights/<family>/
|
||||
model_index.json
|
||||
transformer/config.json
|
||||
transformer/*.safetensors
|
||||
vae/config.json
|
||||
vae/*.safetensors
|
||||
scheduler/scheduler_config.json as needed
|
||||
text_encoder/... as needed
|
||||
```
|
||||
|
||||
Required checks:
|
||||
|
||||
- `model_index.json` exists and lists every required component.
|
||||
- Each converted component has the config filename its loader expects and
|
||||
safetensors weights when it owns weights. Scheduler dirs require
|
||||
`scheduler_config.json`; most other native model dirs use `config.json`.
|
||||
- Weight filenames may vary by loader: transformer and VAE loaders glob all
|
||||
`*.safetensors`; text encoders may load `*.safetensors`, `*.bin`, and
|
||||
sometimes `*.pt`; `conditioner` currently expects
|
||||
`diffusion_pytorch_model.safetensors`. Use the loader's actual accepted layout
|
||||
rather than assuming one global filename.
|
||||
- Passthrough components are copied or referenced deliberately.
|
||||
- Each emitted component config validates through the same path production
|
||||
loaders use. Instantiate the relevant config and call `update_model_arch(...)`
|
||||
or `update_model_config(...)` with the emitted JSON so unknown keys fail during
|
||||
conversion, not at pipeline load time.
|
||||
- Record production loader strictness for every stateful component. If the loader
|
||||
intentionally uses non-strict loading, add explicit missing/unexpected-key
|
||||
assertions in the parity test and document exactly which keys are allowed.
|
||||
- Each new FastVideo component strict-loads converted weights where its production
|
||||
loader is strict. If strict loading is impossible, record the exact allowed
|
||||
missing/unexpected keys and why they are not inference weights.
|
||||
- Retry fixes include the original component parity evidence and the new
|
||||
strict-load result in `local_tests_readme` so the component subagent can resume
|
||||
without rediscovering context.
|
||||
- `local_tests_readme` records the command, output directory, and strict-load
|
||||
result.
|
||||
|
||||
Do not chase numerical parity in this skill except to identify a conversion
|
||||
mapping bug. Long parity-debug loops belong to `/add-model` Phase 6.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md`. Conversion-specific ask cases
|
||||
include selecting between incompatible official checkpoints, publishing/uploading
|
||||
weights, overwriting an existing converted repo not created by this run,
|
||||
accepting non-strict missing inference weights, or dropping a component/output
|
||||
from scope.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/conversion_handoff.md` and update the shared state
|
||||
files before handoff.
|
||||
@@ -1,293 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Convert <model_family> official weights to a FastVideo Diffusers-style tree.
|
||||
|
||||
This template supports both separate component sources and a monolithic pipeline
|
||||
checkpoint that must be split by component prefix. Replace every TODO before
|
||||
using it for a real port.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
from collections import OrderedDict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
from safetensors import safe_open
|
||||
from safetensors.torch import load_file, save_file
|
||||
|
||||
try:
|
||||
from huggingface_hub import snapshot_download
|
||||
except ImportError: # pragma: no cover - optional local conversion dependency
|
||||
snapshot_download = None
|
||||
|
||||
# TODO: fill with authoritative component prefixes for monolithic checkpoints.
|
||||
# Example: {"model.model.": "transformer", "pretransform.model.": "vae"}
|
||||
COMPONENT_PREFIXES: dict[str, str] = {}
|
||||
|
||||
# TODO: fill with component-specific source paths for separate-component repos.
|
||||
# Example: {"transformer": "transformer/model.safetensors", "vae": "vae/"}
|
||||
SEPARATE_COMPONENT_PATHS: dict[str, str] = {}
|
||||
|
||||
# TODO: copy passthrough dirs that are already loadable by FastVideo/Diffusers.
|
||||
PASSTHROUGH_SUBFOLDERS: tuple[str, ...] = ("tokenizer", "scheduler")
|
||||
|
||||
# TODO: add regex renames derived from Phase 4 key/shape dumps.
|
||||
PARAM_NAME_MAP: dict[str, str] = {}
|
||||
|
||||
# TODO: include training-only or dynamically-computed keys that must not load.
|
||||
SKIP_PATTERNS: tuple[str, ...] = ()
|
||||
|
||||
|
||||
def _hf_token() -> str | None:
|
||||
return (os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACE_HUB_TOKEN") or os.environ.get("HF_API_KEY"))
|
||||
|
||||
|
||||
def resolve_src(src: str, revision: str | None) -> Path:
|
||||
if os.path.exists(src):
|
||||
return Path(src)
|
||||
if snapshot_download is None:
|
||||
raise RuntimeError("huggingface_hub is required when --src is a repo id")
|
||||
return Path(snapshot_download(repo_id=src, revision=revision, token=_hf_token()))
|
||||
|
||||
|
||||
def load_checkpoint(path: Path) -> dict[str, torch.Tensor]:
|
||||
if path.is_dir():
|
||||
weights: dict[str, torch.Tensor] = {}
|
||||
for shard in sorted(path.glob("*.safetensors")):
|
||||
weights.update(load_file(str(shard)))
|
||||
if weights:
|
||||
return weights
|
||||
raise FileNotFoundError(f"No safetensors found in {path}")
|
||||
|
||||
if path.suffix == ".safetensors":
|
||||
return load_file(str(path))
|
||||
|
||||
checkpoint = torch.load(path, map_location="cpu", weights_only=True)
|
||||
if isinstance(checkpoint, dict):
|
||||
for key in ("state_dict", "model_state_dict", "model", "module", "ema"):
|
||||
if key in checkpoint and isinstance(checkpoint[key], dict):
|
||||
return checkpoint[key]
|
||||
return checkpoint
|
||||
raise TypeError(f"Unsupported checkpoint type: {type(checkpoint)!r}")
|
||||
|
||||
|
||||
def should_skip_key(key: str) -> bool:
|
||||
return any(re.search(pattern, key) for pattern in SKIP_PATTERNS)
|
||||
|
||||
|
||||
def apply_mapping(key: str) -> str | None:
|
||||
if should_skip_key(key):
|
||||
return None
|
||||
for pattern, replacement in PARAM_NAME_MAP.items():
|
||||
if re.match(pattern, key):
|
||||
return re.sub(pattern, replacement, key)
|
||||
return key
|
||||
|
||||
|
||||
def split_monolithic(state: dict[str, torch.Tensor], ) -> dict[str, OrderedDict[str, torch.Tensor]]:
|
||||
components: dict[str, OrderedDict[str, torch.Tensor]] = {
|
||||
name: OrderedDict()
|
||||
for name in set(COMPONENT_PREFIXES.values())
|
||||
}
|
||||
intentionally_skipped: list[str] = []
|
||||
unowned: list[str] = []
|
||||
for key, value in state.items():
|
||||
if should_skip_key(key):
|
||||
intentionally_skipped.append(key)
|
||||
continue
|
||||
for prefix, component in COMPONENT_PREFIXES.items():
|
||||
if key.startswith(prefix):
|
||||
mapped = apply_mapping(key[len(prefix):])
|
||||
if mapped is not None:
|
||||
components[component][mapped] = value
|
||||
break
|
||||
else:
|
||||
unowned.append(key)
|
||||
if unowned:
|
||||
sample = ", ".join(unowned[:10])
|
||||
raise ValueError(f"Unowned monolithic keys: {len(unowned)}. "
|
||||
f"Add COMPONENT_PREFIXES or SKIP_PATTERNS entries. Sample: {sample}")
|
||||
if intentionally_skipped:
|
||||
print(f"Intentionally skipped {len(intentionally_skipped)} keys")
|
||||
return {name: weights for name, weights in components.items() if weights}
|
||||
|
||||
|
||||
def load_separate_components(src_dir: Path) -> dict[str, OrderedDict[str, torch.Tensor]]:
|
||||
components: dict[str, OrderedDict[str, torch.Tensor]] = {}
|
||||
for component, rel_path in SEPARATE_COMPONENT_PATHS.items():
|
||||
state = load_checkpoint(src_dir / rel_path)
|
||||
converted: OrderedDict[str, torch.Tensor] = OrderedDict()
|
||||
for key, value in state.items():
|
||||
mapped = apply_mapping(key)
|
||||
if mapped is not None:
|
||||
converted[mapped] = value
|
||||
components[component] = converted
|
||||
return components
|
||||
|
||||
|
||||
def build_component_configs(_src_dir: Path) -> dict[str, dict[str, Any]]:
|
||||
# TODO: emit config content accepted by FastVideo loaders. Most components use
|
||||
# config.json; schedulers use scheduler_config.json.
|
||||
return {
|
||||
"transformer": {
|
||||
"_class_name": "<FastVideoTransformerClass>"
|
||||
},
|
||||
"vae": {
|
||||
"_class_name": "<FastVideoVAEClass>"
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def config_filename(component: str) -> str:
|
||||
if component == "scheduler":
|
||||
return "scheduler_config.json"
|
||||
return "config.json"
|
||||
|
||||
|
||||
def source_label(src: str) -> str:
|
||||
if os.path.exists(src):
|
||||
return Path(src).name
|
||||
return src
|
||||
|
||||
|
||||
def build_model_index(
|
||||
src: str,
|
||||
revision: str | None,
|
||||
available_components: set[str],
|
||||
) -> dict[str, Any]:
|
||||
# TODO: match the target pipeline and every required component.
|
||||
index: dict[str, Any] = {
|
||||
"_class_name": "<FastVideoPipelineClass>",
|
||||
"_diffusers_version": "0.30.0",
|
||||
"_fastvideo_converted_from": source_label(src),
|
||||
# Existing transformer/VAE loaders expect "diffusers" even when
|
||||
# _class_name names a FastVideo-native class registered in FastVideo.
|
||||
"transformer": ["diffusers", "<FastVideoTransformerClass>"],
|
||||
"vae": ["diffusers", "<FastVideoVAEClass>"],
|
||||
}
|
||||
if revision:
|
||||
index["_fastvideo_converted_revision"] = revision
|
||||
return {key: value for key, value in index.items() if key.startswith("_") or key in available_components}
|
||||
|
||||
|
||||
def validate_component_configs(configs: dict[str, dict[str, Any]]) -> None:
|
||||
# TODO: instantiate each FastVideo config and call update_model_arch(...) or
|
||||
# update_model_config(...) with this JSON so unknown emitted keys fail here.
|
||||
placeholder_configs = [name for name, config in configs.items() if "<" in json.dumps(config)]
|
||||
if placeholder_configs:
|
||||
raise ValueError(f"Replace config placeholders for: {placeholder_configs}")
|
||||
|
||||
|
||||
def verify_conversion(
|
||||
dst_dir: Path,
|
||||
components: dict[str, OrderedDict[str, torch.Tensor]],
|
||||
) -> None:
|
||||
del dst_dir, components
|
||||
# TODO: load each emitted stateful component through its production loader and
|
||||
# assert strict load, or document exact allowed missing/unexpected keys.
|
||||
raise NotImplementedError("Implement production config validation and strict-load checks")
|
||||
|
||||
|
||||
def write_component(
|
||||
dst_dir: Path,
|
||||
name: str,
|
||||
state: dict[str, torch.Tensor],
|
||||
config: dict[str, Any] | None,
|
||||
) -> None:
|
||||
component_dir = dst_dir / name
|
||||
if component_dir.exists() and any(component_dir.iterdir()):
|
||||
shutil.rmtree(component_dir)
|
||||
component_dir.mkdir(parents=True, exist_ok=True)
|
||||
save_file(dict(state), str(component_dir / "diffusion_pytorch_model.safetensors"))
|
||||
if config is not None:
|
||||
config_path = component_dir / config_filename(name)
|
||||
with config_path.open("w", encoding="utf-8") as f:
|
||||
json.dump(config, f, indent=2)
|
||||
f.write("\n")
|
||||
print(f"Wrote {name}: {len(state)} tensors")
|
||||
|
||||
|
||||
def copy_passthrough(src_dir: Path, dst_dir: Path) -> list[str]:
|
||||
copied: list[str] = []
|
||||
for subfolder in PASSTHROUGH_SUBFOLDERS:
|
||||
src = src_dir / subfolder
|
||||
if not src.is_dir():
|
||||
continue
|
||||
dst = dst_dir / subfolder
|
||||
if dst.exists():
|
||||
shutil.rmtree(dst)
|
||||
shutil.copytree(src, dst)
|
||||
copied.append(subfolder)
|
||||
print(f"Copied {subfolder}/")
|
||||
return copied
|
||||
|
||||
|
||||
def default_monolithic_checkpoint(src_path: Path) -> Path:
|
||||
if src_path.is_file():
|
||||
return src_path
|
||||
return src_path / "model.safetensors"
|
||||
|
||||
|
||||
def convert(
|
||||
src: str,
|
||||
dst: str,
|
||||
layout: str,
|
||||
revision: str | None,
|
||||
) -> None:
|
||||
src_path = resolve_src(src, revision)
|
||||
dst_dir = Path(dst)
|
||||
dst_dir.mkdir(parents=True, exist_ok=True)
|
||||
model_index_path = dst_dir / "model_index.json"
|
||||
|
||||
if layout in {"monolithic", "raw_official"}:
|
||||
# TODO: replace model.safetensors with the official monolithic file name.
|
||||
components = split_monolithic(load_checkpoint(default_monolithic_checkpoint(src_path)))
|
||||
elif layout in {"separate_components", "mixed"}:
|
||||
if not src_path.is_dir():
|
||||
raise ValueError(f"{layout} layout requires a source directory: {src_path}")
|
||||
components = load_separate_components(src_path)
|
||||
else:
|
||||
raise ValueError(f"Unsupported template layout: {layout}")
|
||||
|
||||
copied = (copy_passthrough(src_path, dst_dir) if src_path.is_dir() else [])
|
||||
configs = build_component_configs(src_path if src_path.is_dir() else src_path.parent)
|
||||
validate_component_configs(configs)
|
||||
for name, state in components.items():
|
||||
write_component(dst_dir, name, state, configs.get(name))
|
||||
|
||||
available = set(components) | set(copied)
|
||||
with model_index_path.open("w", encoding="utf-8") as f:
|
||||
json.dump(build_model_index(src, revision, available), f, indent=2)
|
||||
f.write("\n")
|
||||
print(f"Wrote {dst_dir / 'model_index.json'}")
|
||||
verify_conversion(dst_dir, components)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--src", required=True, help="HF repo id, local dir, or checkpoint path")
|
||||
parser.add_argument("--revision", help="HF branch, tag, or commit for repo sources")
|
||||
parser.add_argument(
|
||||
"--dst",
|
||||
required=True,
|
||||
help="Output converted_weights/<model_family> directory",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--layout",
|
||||
choices=("raw_official", "monolithic", "separate_components", "mixed"),
|
||||
required=True,
|
||||
help="Official source layout",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
convert(args.src, args.dst, args.layout, args.revision)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,214 +0,0 @@
|
||||
---
|
||||
name: add-model-08-trace
|
||||
description: Use during /add-model Phase 6 when component parity has failed and root cause requires layer-by-layer divergence analysis. Uses FastVideo activation trace first, falling back to custom hooks only for boundaries or stats the utility cannot observe.
|
||||
---
|
||||
|
||||
# Add-Model Trace
|
||||
|
||||
## Manual Invocation
|
||||
|
||||
Load this skill when `/add-model` Phase 6 component parity has failed and the
|
||||
root cause requires layer-by-layer divergence analysis. This skill is not
|
||||
auto-fired. The calling subagent (DiT, VAE, encoder, or generic port skill)
|
||||
loads it when its standard parity-debug loop hits a wall and cannot isolate
|
||||
the divergence from end-to-end tensor comparisons alone.
|
||||
|
||||
Do not load this skill for first-pass parity failures. Try weight-diff and
|
||||
end-to-end tensor comparison first. Load this skill only when those do not
|
||||
isolate the cause.
|
||||
|
||||
## Goal
|
||||
|
||||
Find the first numerical divergence point between FastVideo's port and the
|
||||
official reference, layer by layer, by instrumenting both sides at matching
|
||||
tensor boundaries. The investigation must leave zero source residue in
|
||||
production code when it closes.
|
||||
|
||||
## When To Run
|
||||
|
||||
After a component parity test FAILS at a bf16-noise-realistic tolerance AND
|
||||
the calling subagent's first-pass debug (weight-diff, end-to-end tensor
|
||||
compare) does not isolate the cause.
|
||||
|
||||
Required inputs before starting:
|
||||
|
||||
- A working FastVideo loader for the component under investigation.
|
||||
- A working official loader, typically via
|
||||
`tests/local_tests/helpers/<family>_upstream.py::load_upstream_<component>`.
|
||||
- Shared deterministic test inputs (same tensors on both sides).
|
||||
- The component parity test file path and its current failure output.
|
||||
|
||||
## Primary Path: FastVideo Activation Trace
|
||||
|
||||
Use FastVideo's first-class activation trace before writing custom hooks:
|
||||
`fastvideo/hooks/activation_trace.py`, documented in
|
||||
`docs/contributing/activation_trace.md`.
|
||||
|
||||
Pipeline runs attach trace to the transformer during pipeline initialization.
|
||||
Component-only parity harnesses may call `attach_activation_trace(model)` from
|
||||
local test/debug code; do not add trace calls to production model code.
|
||||
|
||||
Prefix the failing parity command with a tight layer regex:
|
||||
|
||||
```bash
|
||||
FASTVIDEO_TRACE_ACTIVATIONS=1 \
|
||||
FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$" \
|
||||
FASTVIDEO_TRACE_STATS="abs_mean,sum,max,shape" \
|
||||
FASTVIDEO_TRACE_STEPS="0" \
|
||||
FASTVIDEO_TRACE_OUTPUT="/tmp/opencode/fv_trace.jsonl" \
|
||||
pytest tests/local_tests -k "parity" -v -s
|
||||
```
|
||||
|
||||
Match the layer regex to the actual `model.named_modules()` names. Empty or
|
||||
broad regexes are expensive; prefer block-level names first, then narrow to
|
||||
submodules after the first divergent block is known.
|
||||
|
||||
## Trace Compare Contract
|
||||
|
||||
One JSONL file per side. FastVideo output should use `FASTVIDEO_TRACE_OUTPUT`;
|
||||
the upstream harness should emit the same JSONL shape:
|
||||
|
||||
```json
|
||||
{"module":"block.layers.0","tensor":"out","step":0,"abs_mean":0.0123,"sum":1.0,"max":0.5,"shape":[1,16,32]}
|
||||
```
|
||||
|
||||
Compare rows by `(module, step, tensor)`. The first row whose `shape`,
|
||||
`abs_mean`, or `max` diverges beyond the component tolerance is the first broken
|
||||
boundary. Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
|
||||
`FASTVIDEO_TRACE_STEPS` identical between sides; if row order differs, sort or
|
||||
normalize before diffing.
|
||||
|
||||
## Drill-Down Loop
|
||||
|
||||
**Initial run:** trace every top-level block (`^block\.layers\.[0-9]+$` or the
|
||||
family's equivalent). Identify the first block index where `abs_mean` or `max`
|
||||
drifts beyond tolerance while earlier blocks match.
|
||||
|
||||
**Drill run:** tighten `FASTVIDEO_TRACE_LAYERS` to submodules inside the first
|
||||
divergent block: attention output, MLP projections, norm outputs, modality
|
||||
adapters, or other named boundaries exposed by `named_modules()`.
|
||||
|
||||
**Iterate:** if the first divergent operation is a free function or tensor op not
|
||||
visible as an `nn.Module`, use the fallback instrumentation hierarchy below.
|
||||
|
||||
The loop ends when the first divergent submodule or operation is identified with
|
||||
a file:line citation in the official source.
|
||||
|
||||
## Fallback Instrumentation Hierarchy
|
||||
|
||||
Use these only when activation trace cannot observe the needed boundary or
|
||||
statistic.
|
||||
|
||||
### (1) Custom forward hooks
|
||||
|
||||
`module.register_forward_hook(...)` and `register_forward_pre_hook(...)`.
|
||||
Always within `try/finally` with `handle.remove()`. Zero source residue.
|
||||
|
||||
### (2) Runtime monkey-patch
|
||||
|
||||
`module.attr = wrapped_func` or `cls.method = wrapped_method`, restored via
|
||||
`try/finally` (save original first). Use for free functions and non-Module sites
|
||||
such as activation functions (`swiglu`, `apply_rotary_emb`).
|
||||
|
||||
### (3) Source edits in FastVideo's own code
|
||||
|
||||
Only when (1) and (2) are insufficient. Track all edits within a single named
|
||||
`git stash` boundary OR a temporary branch. Run `git diff` before closing the
|
||||
investigation to confirm cleanup.
|
||||
|
||||
### (4) Source edits in official repo source
|
||||
|
||||
Allowed only when hook and monkey-patch approaches cannot capture the site.
|
||||
For git-tracked or editable official clones, use `git diff` in the clone path to
|
||||
verify cleanup. For non-editable site-packages, back up the target file before
|
||||
editing and restore it before handoff.
|
||||
|
||||
## Hypothesis Toggles
|
||||
|
||||
Use env-var-gated monkey-patches to A/B test suspect implementations without
|
||||
source edits. Pattern: `<FAMILY>_DEBUG_PATCH_<HYPOTHESIS>=1`.
|
||||
|
||||
Example from the magi-human investigation:
|
||||
|
||||
```
|
||||
MAGI_DEBUG_PATCH_LINEAR=1
|
||||
```
|
||||
|
||||
This patched `PackedExpertLinear.forward` to mirror upstream's
|
||||
`_BF16ComputeLinear` explicit-cast pattern, isolating a dtype-cast difference
|
||||
as the root cause.
|
||||
|
||||
Document all toggles in the script docstring. Each toggle must:
|
||||
|
||||
- save the original before patching;
|
||||
- restore the original in a `try/finally` block;
|
||||
- print a `[debug] Patched <ClassName>.<method>` line to stdout when active.
|
||||
|
||||
## Cleanup Gate
|
||||
|
||||
The calling agent MUST report `[cleanup-gate] PASS` on all five items before
|
||||
handoff. Do not hand off with any item unresolved.
|
||||
|
||||
1. `git diff` in the FastVideo repo: empty. No stray prints, hooks, or
|
||||
monkey-patches in production code.
|
||||
2. `git diff` in the official-repo clone (if used): empty. For non-editable
|
||||
site-packages installs: `diff original.py original.py.trace-backup` is
|
||||
empty OR `pip install --force-reinstall <pkg>` succeeded and the installed
|
||||
file matches the original.
|
||||
3. `git stash list`: only the named investigation stash (or empty). No
|
||||
unnamed stashes left from this session.
|
||||
4. No new untracked files outside `/tmp/opencode/` (logs) and the existing
|
||||
debug script directory (`tests/local_tests/transformers/` or equivalent).
|
||||
5. `mypy` clean on any production files touched during the investigation.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Escalate to the calling bucket skill when:
|
||||
|
||||
- A forward hook on an official module raises because of a custom `forward`
|
||||
signature or varlen handler args that the hook closure cannot satisfy. The
|
||||
bucket skill has component-specific knowledge to work around this.
|
||||
- The first divergent layer is `block[0]`, meaning the divergence is in the
|
||||
adapter, modality dispatcher, coordinate embedding, or packing step before
|
||||
any block runs. Check those sites first; the bug is not in attention or MLP.
|
||||
- Per-block drift is never zero anywhere across all blocks. This usually means
|
||||
the inputs are not bit-identical between sides. Verify with a state-dict
|
||||
compare (weight-diff script) AND confirm the input tensors are the same
|
||||
object or have identical values before the forward call.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return to the calling subagent with:
|
||||
|
||||
- FastVideo trace JSONL path and upstream trace JSONL path.
|
||||
- Trace settings used: `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
|
||||
`FASTVIDEO_TRACE_STEPS`.
|
||||
- The first divergent `(module, step, tensor)` row and observed drift.
|
||||
- The upstream file:line citation where the divergence originates.
|
||||
- Fallback hook/patch verdict if activation trace could not observe the boundary.
|
||||
- Hypothesis verdict if an A/B toggle was used, for example `PATCH_LINEAR=1`.
|
||||
- Cleanup-gate status: `[cleanup-gate] PASS` or a list of unresolved items.
|
||||
|
||||
The calling agent uses this to scope the production fix in the FastVideo
|
||||
component file.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/contributing/activation_trace.md` for canonical activation-trace env vars,
|
||||
JSONL output, cost model, and troubleshooting.
|
||||
- `fastvideo/hooks/activation_trace.py` for the implementation and
|
||||
`attach_activation_trace(model)` entry point.
|
||||
- `templates/block_trace_debug.py` in this skill directory: fallback custom-hook
|
||||
template when activation trace cannot observe the needed boundary or stat.
|
||||
- `tests/local_tests/transformers/_debug_magi_human_block_parity.py` in the
|
||||
FastVideo3 repo: historical worked example for custom hook/patch debugging.
|
||||
- `add-model/SKILL.md` Phase 6: the calling context for this skill.
|
||||
- `add-model-03-port-dit/SKILL.md`, `add-model-04-port-vae/SKILL.md`,
|
||||
`add-model-05-port-encoder/SKILL.md`, `add-model-06-port-generic/SKILL.md`:
|
||||
bucket-specific debug language and component-specific escape-hatch knowledge.
|
||||
|
||||
## Changelog
|
||||
|
||||
| Date | Change |
|
||||
|---|---|
|
||||
| 2026-05-01 | Initial skill extracted from `_debug_magi_human_block_parity.py` pattern. |
|
||||
@@ -1,334 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Per-block divergence debugger template for FastVideo model ports.
|
||||
|
||||
Run directly (not a pytest test):
|
||||
python tests/local_tests/transformers/_debug_<family>_<component>_parity.py
|
||||
|
||||
Generalizes: tests/local_tests/transformers/_debug_magi_human_block_parity.py
|
||||
|
||||
Fill FAMILY, COMPONENT, and the two loader functions. Run once for the initial
|
||||
drift table, then set <FAMILY>_DEBUG_DRILL_LAYER=NN to drill into submodules.
|
||||
Add <FAMILY>_DEBUG_PATCH_<HYPOTHESIS>=1 to A/B test a suspect implementation.
|
||||
|
||||
CLEANUP: all hooks removed in try/finally; monkey-patches restored in
|
||||
try/finally; source edits tracked in a named git stash. Zero source residue.
|
||||
See add-model-08-trace/SKILL.md for the full cleanup gate checklist.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import gc
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
FAMILY: str = "<family>" # e.g. "magi_human", "ltx2", "wan"
|
||||
COMPONENT: str = "<component>" # e.g. "dit", "vae", "encoder"
|
||||
DRILL_LAYER_ENV: str = "<FAMILY>_DEBUG_DRILL_LAYER"
|
||||
HYPOTHESIS_ENV: str = "<FAMILY>_DEBUG_PATCH_<HYPOTHESIS>"
|
||||
REL_THRESHOLD: float = 0.005 # 0.5% abs_mean drift flags a block as divergent
|
||||
LOG_DIR: Path = Path("/tmp/opencode")
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
|
||||
def load_official(device: torch.device) -> torch.nn.Module:
|
||||
"""Load the official upstream model. TODO: implement for your family.
|
||||
|
||||
Example (magi-human):
|
||||
from tests.local_tests.helpers.magi_human_upstream import install_stubs, load_upstream_dit
|
||||
install_stubs()
|
||||
return load_upstream_dit(base_shard_dir, device=device, dtype=None)
|
||||
"""
|
||||
raise NotImplementedError(f"Fill load_official() for {FAMILY}/{COMPONENT}.")
|
||||
|
||||
|
||||
def load_fastvideo(device: torch.device) -> torch.nn.Module:
|
||||
"""Load the FastVideo-native model. TODO: implement for your family.
|
||||
|
||||
Example (magi-human):
|
||||
from fastvideo.configs.models.dits.magi_human import MagiHumanVideoConfig
|
||||
from fastvideo.models.dits.magi_human import MagiHumanDiT
|
||||
from safetensors.torch import load_file; import glob
|
||||
fv = MagiHumanDiT(MagiHumanVideoConfig())
|
||||
state = {}
|
||||
for shard in sorted(glob.glob(str(transformer_dir / "*.safetensors"))): state.update(load_file(shard))
|
||||
fv.load_state_dict(state, strict=False); return fv.to(device).eval()
|
||||
"""
|
||||
raise NotImplementedError(f"Fill load_fastvideo() for {FAMILY}/{COMPONENT}.")
|
||||
|
||||
|
||||
def build_inputs(device: torch.device) -> dict[str, Any]:
|
||||
"""Return deterministic inputs shared by both sides. TODO: replace.
|
||||
|
||||
Both sides must receive the SAME tensors (clone before each forward call).
|
||||
Non-identical inputs cause non-zero drift everywhere.
|
||||
"""
|
||||
torch.manual_seed(0)
|
||||
return {"x": torch.randn(64, 1024, dtype=torch.bfloat16, device=device)}
|
||||
|
||||
|
||||
def _stat(name: str, t: torch.Tensor) -> dict:
|
||||
f = t.detach().float()
|
||||
return {
|
||||
"name": name,
|
||||
"shape": tuple(t.shape),
|
||||
"abs_mean": f.abs().mean().item(),
|
||||
"sum": f.sum().item(),
|
||||
"min": f.min().item(),
|
||||
"max": f.max().item(),
|
||||
}
|
||||
|
||||
|
||||
def _attach_block_hooks(
|
||||
model: torch.nn.Module,
|
||||
label: str,
|
||||
log: list[dict],
|
||||
tensors: dict[str, torch.Tensor] | None = None,
|
||||
drill_layer: int | None = None,
|
||||
) -> list[Any]:
|
||||
"""Return hook handles. Caller MUST remove them in try/finally."""
|
||||
handles: list[Any] = []
|
||||
|
||||
def _hook(name: str):
|
||||
|
||||
def fn(_module, _inputs, outputs):
|
||||
t = outputs[0] if isinstance(outputs, tuple) else outputs
|
||||
if not torch.is_tensor(t):
|
||||
return
|
||||
log.append({"side": label, **_stat(name, t)})
|
||||
if tensors is not None:
|
||||
tensors[name] = t.detach().float().cpu()
|
||||
|
||||
return fn
|
||||
|
||||
def _pre_hook(name: str):
|
||||
# Pre-hooks observe a free function's output by intercepting the next
|
||||
# module's input (useful when the activation is not an nn.Module).
|
||||
def fn(_module, inputs):
|
||||
t = inputs[0] if isinstance(inputs, tuple) else inputs
|
||||
if not torch.is_tensor(t):
|
||||
return
|
||||
key = f"{name}<in>"
|
||||
log.append({"side": label, **_stat(key, t)})
|
||||
if tensors is not None:
|
||||
tensors[key] = t.detach().float().cpu()
|
||||
|
||||
return fn
|
||||
|
||||
# TODO: adapt attribute paths to your model. Remove adapter block if absent.
|
||||
if hasattr(model, "adapter"):
|
||||
handles.append(model.adapter.register_forward_hook(_hook("adapter")))
|
||||
|
||||
# TODO: adapt model.block.layers to your block container.
|
||||
# Alternatives: model.transformer.layers, model.blocks, model.layers
|
||||
block_layers = model.block.layers # type: ignore[attr-defined]
|
||||
for i, layer in enumerate(block_layers):
|
||||
handles.append(layer.register_forward_hook(_hook(f"block[{i:02d}]")))
|
||||
if drill_layer is not None and i == drill_layer:
|
||||
tag = f"L{i:02d}"
|
||||
# TODO: adapt submodule names to your layer's attributes.
|
||||
# magi-human uses: attention, mlp.pre_norm, mlp.up_gate_proj,
|
||||
# mlp.down_proj (pre+post), mlp, attn_post_norm, mlp_post_norm.
|
||||
if hasattr(layer, "attention"):
|
||||
handles.append(layer.attention.register_forward_hook(_hook(f"{tag}.attention")))
|
||||
if hasattr(layer, "mlp"):
|
||||
mlp = layer.mlp
|
||||
if hasattr(mlp, "pre_norm"):
|
||||
handles.append(mlp.pre_norm.register_forward_hook(_hook(f"{tag}.mlp.pre_norm")))
|
||||
if hasattr(mlp, "up_gate_proj"):
|
||||
handles.append(mlp.up_gate_proj.register_forward_hook(_hook(f"{tag}.mlp.up_gate_proj")))
|
||||
if hasattr(mlp, "down_proj"):
|
||||
handles.append(mlp.down_proj.register_forward_pre_hook(_pre_hook(f"{tag}.mlp.down_proj")))
|
||||
handles.append(mlp.down_proj.register_forward_hook(_hook(f"{tag}.mlp.down_proj")))
|
||||
handles.append(mlp.register_forward_hook(_hook(f"{tag}.mlp")))
|
||||
if hasattr(layer, "attn_post_norm"):
|
||||
handles.append(layer.attn_post_norm.register_forward_hook(_hook(f"{tag}.attn_post_norm")))
|
||||
if hasattr(layer, "mlp_post_norm"):
|
||||
handles.append(layer.mlp_post_norm.register_forward_hook(_hook(f"{tag}.mlp_post_norm")))
|
||||
return handles
|
||||
|
||||
|
||||
def _apply_hypothesis_patch() -> bool:
|
||||
"""Apply an optional monkey-patch gated by HYPOTHESIS_ENV. TODO: implement.
|
||||
|
||||
Pattern: save original on the class, patch, restore in _restore_hypothesis_patch().
|
||||
"""
|
||||
if os.getenv(HYPOTHESIS_ENV) != "1":
|
||||
return False
|
||||
# TODO: import FastVideo class, save original, apply patch.
|
||||
print(f"[debug] Hypothesis patch {HYPOTHESIS_ENV}=1 applied.")
|
||||
return True
|
||||
|
||||
|
||||
def _restore_hypothesis_patch() -> None:
|
||||
if os.getenv(HYPOTHESIS_ENV) != "1":
|
||||
return
|
||||
# TODO: restore original, e.g.: _mod.TargetClass.method = _mod._ORIGINAL_METHOD
|
||||
|
||||
|
||||
def _write_log(entries: list[dict], path: Path) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(path, "w") as f:
|
||||
for e in entries:
|
||||
f.write(f"{e['name']} {e['shape']} "
|
||||
f"{e['abs_mean']:.8f} {e['sum']:.4f} "
|
||||
f"{e['min']:.6f} {e['max']:.6f}\n")
|
||||
|
||||
|
||||
def _sort_key(name: str, drill_layer: int) -> tuple:
|
||||
if name == "adapter":
|
||||
return (0, "")
|
||||
if name.startswith(f"L{drill_layer:02d}."):
|
||||
sub_order = {
|
||||
"attention": 0,
|
||||
"attn_post_norm": 1,
|
||||
"mlp.pre_norm": 2,
|
||||
"mlp.up_gate_proj": 3,
|
||||
"mlp.down_proj<in>": 4,
|
||||
"mlp.down_proj": 5,
|
||||
"mlp": 6,
|
||||
"mlp_post_norm": 7,
|
||||
}.get(name.split(".", 1)[1], 9)
|
||||
return (1, f"block[{drill_layer:02d}]", sub_order)
|
||||
if name.startswith("block["):
|
||||
return (1, name, 99)
|
||||
return (2, name, 0)
|
||||
|
||||
|
||||
def _print_table(by_name: dict[str, dict], drill_layer: int) -> int | None:
|
||||
hdr = (f"{'name':<18} {'up_shape':<22} {'up_absmean':>12} {'fv_absmean':>12} "
|
||||
f"{'absmean_diff':>14} {'rel%':>8} {'up_sum':>14} {'fv_sum':>14} {'sum_diff':>12}")
|
||||
print(f"\n{hdr}\n{'-' * len(hdr)}")
|
||||
first_div: int | None = None
|
||||
for name in sorted(by_name.keys(), key=lambda n: _sort_key(n, drill_layer)):
|
||||
d = by_name[name]
|
||||
up, fv = d.get("up"), d.get("fv")
|
||||
if up is None or fv is None:
|
||||
continue
|
||||
am_diff = abs(up["abs_mean"] - fv["abs_mean"])
|
||||
am_rel = am_diff / max(up["abs_mean"], 1e-9)
|
||||
sum_diff = abs(up["sum"] - fv["sum"])
|
||||
flag = ""
|
||||
if name.startswith("block[") and am_rel > REL_THRESHOLD:
|
||||
flag = " <<< DIVERGE"
|
||||
if first_div is None:
|
||||
first_div = int(name[len("block["):-1])
|
||||
print(f"{name:<18} {str(up['shape']):<22} {up['abs_mean']:>12.6f} "
|
||||
f"{fv['abs_mean']:>12.6f} {am_diff:>14.6f} {am_rel * 100:>7.3f}% "
|
||||
f"{up['sum']:>14.4f} {fv['sum']:>14.4f} {sum_diff:>12.4f}{flag}")
|
||||
return first_div
|
||||
|
||||
|
||||
def _print_elementwise(up_t: dict[str, torch.Tensor], fv_t: dict[str, torch.Tensor], drill_layer: int) -> None:
|
||||
common = set(up_t.keys()) & set(fv_t.keys())
|
||||
if not common:
|
||||
return
|
||||
hdr = f"{'name':<30} {'shape':<22} {'diff_max':>12} {'diff_mean':>12} {'diff_rel%':>10}"
|
||||
print(f"\nElement-wise diffs for drilled L{drill_layer:02d} submodules:\n{hdr}\n{'-' * len(hdr)}")
|
||||
for name in sorted(common):
|
||||
a, b = up_t[name], fv_t[name]
|
||||
if a.shape != b.shape:
|
||||
continue
|
||||
diff = (a - b).abs()
|
||||
rel = (diff.mean().item() / max(a.abs().mean().item(), 1e-9)) * 100
|
||||
print(f"{name:<30} {str(tuple(a.shape)):<22} "
|
||||
f"{diff.max().item():>12.6f} {diff.mean().item():>12.6f} {rel:>9.4f}%")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if not torch.cuda.is_available():
|
||||
print("Need CUDA. Skipping.")
|
||||
return
|
||||
|
||||
# TODO: add precondition checks (official clone present, weights available).
|
||||
|
||||
drill_layer = int(os.getenv(DRILL_LAYER_ENV, "0"))
|
||||
device = torch.device("cuda:0")
|
||||
patched = _apply_hypothesis_patch()
|
||||
try:
|
||||
inputs = build_inputs(device)
|
||||
|
||||
print("Loading official model...")
|
||||
official = load_official(device)
|
||||
up_log: list[dict] = []
|
||||
up_t: dict[str, torch.Tensor] = {}
|
||||
up_handles = _attach_block_hooks(official, "up", up_log, up_t, drill_layer)
|
||||
print("Running official forward (with hooks)...")
|
||||
try:
|
||||
with torch.inference_mode():
|
||||
# TODO: adapt forward call signature to your component.
|
||||
ref_out = official(**{k: v.clone() for k, v in inputs.items()})
|
||||
if isinstance(ref_out, dict):
|
||||
sample = ref_out.get("sample")
|
||||
ref_out = sample if sample is not None else ref_out.get("x")
|
||||
elif hasattr(ref_out, "sample"):
|
||||
ref_out = ref_out.sample
|
||||
elif isinstance(ref_out, tuple):
|
||||
ref_out = ref_out[0]
|
||||
assert torch.is_tensor(ref_out), f"official output is not tensor: {type(ref_out)}"
|
||||
ref_out = ref_out.detach().float().cpu()
|
||||
finally:
|
||||
for h in up_handles:
|
||||
h.remove()
|
||||
del official
|
||||
gc.collect()
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
print("Loading FastVideo model...")
|
||||
fv = load_fastvideo(device)
|
||||
fv_log: list[dict] = []
|
||||
fv_t: dict[str, torch.Tensor] = {}
|
||||
fv_handles = _attach_block_hooks(fv, "fv", fv_log, fv_t, drill_layer)
|
||||
print("Running FastVideo forward (with hooks)...")
|
||||
try:
|
||||
with torch.inference_mode():
|
||||
# TODO: adapt forward call signature to your component.
|
||||
fv_out = fv(**{k: v.clone() for k, v in inputs.items()})
|
||||
if isinstance(fv_out, dict):
|
||||
sample = fv_out.get("sample")
|
||||
fv_out = sample if sample is not None else fv_out.get("x")
|
||||
elif hasattr(fv_out, "sample"):
|
||||
fv_out = fv_out.sample
|
||||
elif isinstance(fv_out, tuple):
|
||||
fv_out = fv_out[0]
|
||||
assert torch.is_tensor(fv_out), f"FastVideo output is not tensor: {type(fv_out)}"
|
||||
fv_out = fv_out.detach().float().cpu()
|
||||
finally:
|
||||
for h in fv_handles:
|
||||
h.remove()
|
||||
finally:
|
||||
_restore_hypothesis_patch()
|
||||
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
up_path = LOG_DIR / f"{FAMILY}_{COMPONENT}_up_layers.log"
|
||||
fv_path = LOG_DIR / f"{FAMILY}_{COMPONENT}_fv_layers.log"
|
||||
_write_log(up_log, up_path)
|
||||
_write_log(fv_log, fv_path)
|
||||
print(f"\nLogs: {up_path} {fv_path}\nDiff: diff {up_path} {fv_path}")
|
||||
|
||||
by_name: dict[str, dict] = {}
|
||||
for entry in up_log + fv_log:
|
||||
by_name.setdefault(entry["name"], {})[entry["side"]] = entry
|
||||
first_div = _print_table(by_name, drill_layer)
|
||||
|
||||
print()
|
||||
if first_div is not None:
|
||||
print(f"First block exceeding {REL_THRESHOLD * 100:.2f}% drift: block[{first_div:02d}]")
|
||||
print(f"Re-run with {DRILL_LAYER_ENV}={first_div} to drill submodules.")
|
||||
else:
|
||||
print(f"No block exceeded {REL_THRESHOLD * 100:.2f}% -- divergence is amortized or pre-block.")
|
||||
|
||||
diff = (ref_out - fv_out).abs()
|
||||
print(f"\nFinal ref_abs={ref_out.abs().mean():.6f} fv_abs={fv_out.abs().mean():.6f} "
|
||||
f"diff_max={diff.max():.6f} diff_mean={diff.mean():.6f}")
|
||||
_print_elementwise(up_t, fv_t, drill_layer)
|
||||
if patched:
|
||||
print(f"\n[debug] Hypothesis {HYPOTHESIS_ENV}=1 was active this run.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,255 +0,0 @@
|
||||
---
|
||||
name: add-model-09-pipeline
|
||||
description: Use during /add-model Phase 7 after all required component parity tests pass to define FastVideo pipeline wiring, configs, presets, registry entries, examples, smoke tests, and pipeline parity tests.
|
||||
---
|
||||
|
||||
# Add Model Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Implement and verify the end-to-end FastVideo pipeline after the native
|
||||
components and converted weights have passed non-skip component parity. This
|
||||
skill owns pipeline class/stage wiring, pipeline configs, presets, registry
|
||||
entries, examples, smoke tests, and pipeline parity-debug.
|
||||
|
||||
FastVideo has one pipeline architecture: stage-based composition through
|
||||
`ComposedPipelineBase`. Add or specialize stages only when existing stages cannot
|
||||
represent the official behavior safely.
|
||||
|
||||
## Hard Gate
|
||||
|
||||
Do not start pipeline work until every required component, including reused
|
||||
components, has a non-skip local parity PASS.
|
||||
|
||||
If any component row is missing, skipped, red, or blocked, return to `/add-model`
|
||||
Phase 6. Pipeline parity cannot distinguish stage wiring mistakes from broken
|
||||
component numerics when component parity is still unresolved.
|
||||
|
||||
## Inputs
|
||||
|
||||
Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
|
||||
escape hatches, production boundaries, and skip/pass semantics.
|
||||
|
||||
Require a complete packet matching
|
||||
`../add-model/contracts/pipeline_context.md`.
|
||||
|
||||
The packet must include:
|
||||
|
||||
- official pipeline files and official call/default sources;
|
||||
- workload types, input/output modalities, and output contract;
|
||||
- converted or source `model_index.json` path;
|
||||
- component parity rows, all `non_skip_pass`;
|
||||
- target FastVideo pipeline/config/preset/registry/example/test paths;
|
||||
- `local_tests_readme` and `port_state_file` paths.
|
||||
|
||||
## Outputs
|
||||
|
||||
- Pipeline package under `fastvideo/pipelines/basic/<family>/`.
|
||||
- Pipeline config under `fastvideo/configs/pipelines/<family>.py` or a documented
|
||||
family-local config file when that matches existing project style.
|
||||
- Presets under `fastvideo/pipelines/basic/<family>/presets.py`.
|
||||
- Registry updates in `fastvideo/registry.py`.
|
||||
- Basic example under `examples/inference/basic/basic_<family>*.py`.
|
||||
- Local smoke and parity tests under `tests/local_tests/pipelines/`.
|
||||
- Updated `tests/local_tests/<model_family>/README.md`.
|
||||
- Updated `tests/local_tests/<model_family>/PORT_STATUS.md`.
|
||||
- Handoff matching `../add-model/contracts/pipeline_handoff.md`.
|
||||
|
||||
## Mode: Pipeline Definition
|
||||
|
||||
Use this mode first.
|
||||
|
||||
1. Read the official pipeline call path before editing FastVideo code.
|
||||
2. Compare official defaults against the planned FastVideo config and presets:
|
||||
steps, CFG scales, secondary CFG, flow shift, schedulers, sigmas, seed/RNG,
|
||||
resolution, frames, FPS, duration, VAE scaling, decode slicing, negative
|
||||
prompt defaults, and output heads.
|
||||
3. Create or update the pipeline class with `_required_config_modules` matching
|
||||
the emitted `model_index.json` and `ComposedPipelineBase.load_modules`.
|
||||
Runtime pipeline resolution is exact: `model_index.json["_class_name"]` must
|
||||
match a registered `EntryClass.__name__`, or a wrapper/alias class in
|
||||
`EntryClass`. Registry detectors do not select the executable pipeline class.
|
||||
4. Add new public generation kwargs to `fastvideo/api/sampling_param.py` before
|
||||
examples or presets use them. `SamplingParam.update()` ignores unknown keys
|
||||
except for logging, and preset defaults apply only to declared fields. Add CLI
|
||||
args when the option should be available from command-line entrypoints.
|
||||
5. Put loader-time changes in `load_modules()` or earlier, not
|
||||
`initialize_pipeline()`. `ComposedPipelineBase.__init__` loads modules before
|
||||
`post_init()` calls `initialize_pipeline()`, so process-global flags, loader
|
||||
path rewrites, dtype overrides, and tokenizer path changes needed for loading
|
||||
cannot be introduced there.
|
||||
6. Use `self.get_module("transformer_2", None)` and similar optional accessors
|
||||
for truly optional modules. Do not hard-require optional modules by accident.
|
||||
7. Avoid mutating class-level `_required_config_modules` in custom code. If a
|
||||
pipeline needs dynamic modules, copy the list to an instance-owned value or
|
||||
pass `required_config_modules` explicitly so one pipeline instance cannot leak
|
||||
module requirements into another.
|
||||
8. Create the stage chain in official execution order. Prefer existing shared
|
||||
stages for standard text encoding, timestep preparation, latent preparation,
|
||||
denoising, and decoding.
|
||||
9. Add model-specific stages only for family-specific behavior that does not fit
|
||||
the shared stage contracts.
|
||||
10. Add pipeline config classes for wiring and runtime defaults. Do not duplicate
|
||||
component architecture fields unless a loader requires them in the subconfig.
|
||||
Family-local config files such as
|
||||
`fastvideo/pipelines/basic/<family>/pipeline_configs.py` are valid only when
|
||||
`fastvideo/registry.py` imports and registers the classes explicitly.
|
||||
11. Add `InferencePreset` objects with `model_family`, `name`, `version`,
|
||||
`defaults`, optional validation-only `stage_schemas`, and an `ALL_PRESETS`
|
||||
tuple. `stage_schemas` validates user-facing `stage_overrides` names; it does
|
||||
not drive `create_pipeline_stages()` execution.
|
||||
12. Register config classes and presets in `fastvideo/registry.py`: add
|
||||
`register_configs(...)`, import the family's `ALL_PRESETS`, and append it to
|
||||
`_register_presets()`. Detectors should cover HF paths and `_class_name`
|
||||
strings for config/preset lookup, but not as a replacement for exact pipeline
|
||||
class-name resolution.
|
||||
13. Add a basic example with a user-story docstring and normal file-path inputs
|
||||
for image, audio, or video references. Keep orchestration glue in the
|
||||
pipeline or a helper, not in the example.
|
||||
14. Add a separate smoke test
|
||||
`tests/local_tests/pipelines/test_<family>_pipeline_smoke.py` that proves
|
||||
imports, `EntryClass`, registry, presets, config defaults, and at least one
|
||||
real load/generate path when weights are local. Older local tests sometimes
|
||||
colocate smoke checks in parity files; new ports should use the separate file
|
||||
convention.
|
||||
15. Add or update pipeline parity test scaffolding with
|
||||
`templates/pipeline_parity_test.py`.
|
||||
16. Update `local_tests_readme` and `port_state_file` with commands, statuses,
|
||||
default sources, decisions, and blockers.
|
||||
|
||||
Production import boundaries are defined in
|
||||
`../add-model/shared/common_rules.md`.
|
||||
|
||||
## Mode: Pipeline Parity Debug
|
||||
|
||||
Run after pipeline definition and after smoke can execute far enough to load the
|
||||
pipeline. Loop until pipeline parity is a non-skip PASS or a precise blocker is
|
||||
returned.
|
||||
|
||||
Mandatory order:
|
||||
|
||||
```bash
|
||||
pytest tests/local_tests/pipelines/test_<family>_pipeline_smoke.py -v -s
|
||||
DISABLE_SP=1 pytest tests/local_tests/pipelines/test_<family>_pipeline_parity.py -v -s
|
||||
python examples/inference/basic/basic_<family>.py
|
||||
```
|
||||
|
||||
Pipeline parity must compare real outputs, not only successful generation:
|
||||
|
||||
- denoised latents when decode parity is expensive or nondeterministic;
|
||||
- decoded videos/images when visual output should be deterministic enough;
|
||||
- decoded waveform or audio features for audio pipelines;
|
||||
- separate video and audio targets for joint AV pipelines unless a validated
|
||||
joint metric exists.
|
||||
|
||||
Debug pipeline drift in this order:
|
||||
|
||||
1. Confirm both sides use the same component weights and component parity PASS
|
||||
results are still valid.
|
||||
2. Align official and FastVideo call arguments, presets, and default values.
|
||||
3. Align scheduler timesteps, sigmas/noise levels, prediction type, flow shift,
|
||||
guidance math, and secondary-guidance branches.
|
||||
4. Align RNG: initial latents/noise, generator device, seed, per-step noise, VAE
|
||||
sampling, and any official `+1 frame` or crop/slice behavior.
|
||||
5. Align conditioning: prompt templates, negative prompts, masks, image/audio
|
||||
preprocessing, modality packing, text truncation, and dtype/autocast.
|
||||
6. Align decode: latent scaling, per-channel mean/std, tiling flags, output
|
||||
channel order, sample rate, FPS, and final slicing.
|
||||
7. Add targeted stage-level diagnostics to identify the first divergent stage.
|
||||
|
||||
If stage diagnostics show the first bad stage is transformer/denoising or a
|
||||
mid-DiT block, enable activation trace before adding ad hoc pipeline prints; see
|
||||
`docs/contributing/activation_trace.md` and `../add-model-08-trace/SKILL.md`.
|
||||
Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
|
||||
`FASTVIDEO_TRACE_STEPS` identical across reruns so pipeline parity traces diff
|
||||
one-to-one.
|
||||
|
||||
If the first divergence belongs to component implementation, strict loading, or
|
||||
conversion mapping, stop pipeline edits and return `next_step=return_to_phase_6`
|
||||
with the exact failing evidence. Do not patch conversion from this skill.
|
||||
|
||||
## Stage And Variant Rules
|
||||
|
||||
- Canonical video T2V order: `InputValidationStage`, `TextEncodingStage`,
|
||||
`ConditioningStage`, `TimestepPreparationStage`, `LatentPreparationStage`,
|
||||
`DenoisingStage`, `DecodingStage`.
|
||||
- Canonical video I2V delta adds image loading/encoding and image VAE encoding in
|
||||
the official order, commonly: `TextEncodingStage`, `ImageEncodingStage`,
|
||||
`ConditioningStage`, `TimestepPreparationStage`, `LatentPreparationStage`,
|
||||
`ImageVAEEncodingStage`, `DenoisingStage`, `DecodingStage`.
|
||||
- Treat `ConditioningStage` as default-present for Wan-style pipelines, but still
|
||||
follow the reference if another family truly skips or replaces it.
|
||||
- T2V video pipelines usually use validation, text encoding, conditioning,
|
||||
timestep preparation, latent preparation, denoising, and decoding.
|
||||
- I2V adds image loading/encoding and image-latent preparation according to the
|
||||
official pipeline, not by assuming CLIP or Wan-specific branches.
|
||||
- Pick image, audio, and video encoders from the reference. Do not assume CLIP or
|
||||
any other common encoder unless the reference uses it.
|
||||
- Cross-attention class names are not prescribed; match the family style and
|
||||
preserve the official tensor contract.
|
||||
- `WorkloadType` currently has no `T2A`, `A2A`, or `AV` values. Until that enum
|
||||
is extended, audio-only pipelines may register with `WorkloadType.T2V` and
|
||||
preset `workload_type="t2v"` as a compatibility shim, but must document the
|
||||
rationale in code and `PORT_STATUS.md`.
|
||||
- Audio-only pipelines should not force real video semantics into presets. Use
|
||||
minimal video-shaped placeholders such as small `height`/`width` and
|
||||
`num_frames=1` only when shared `VideoGenerator`/validation paths require them,
|
||||
and document that the real output is audio.
|
||||
- Record modality-specific shape knobs and output contract in the pipeline
|
||||
handoff: video uses `height`, `width`, `num_frames`, and `fps`; audio uses
|
||||
`audio_seconds` and `sampling_rate`; joint AV records both plus whether output
|
||||
is muxed or paired files.
|
||||
- Use sibling pipeline classes/configs when required modules, HF repo layout,
|
||||
stage chains, or inputs differ materially.
|
||||
- Use one kwargs-driven pipeline class only when variants share weights,
|
||||
modules, stage chain, and safe call semantics.
|
||||
- Split later if components diverge, workload tags require separate discovery,
|
||||
signatures become unsafe, or stage branches become substantial.
|
||||
- If the DiT branches on `added_kv_proj_dim`, document the T2V/I2V split.
|
||||
- If the reference uses `transformer_2`, `boundary_ratio`, `guidance_scale_2`, or
|
||||
DMD step lists, keep those on config, presets, or stages deliberately.
|
||||
- Support every official output head in scope. If a head is out of scope, record
|
||||
explicit user approval in `PORT_STATUS.md`.
|
||||
|
||||
Pipeline verification order:
|
||||
|
||||
```bash
|
||||
pytest tests/local_tests/pipelines/test_<family>_pipeline_smoke.py -v -s
|
||||
DISABLE_SP=1 pytest -v -s tests/local_tests/pipelines/test_<family>_pipeline_parity.py
|
||||
python examples/inference/basic/basic_<family>.py
|
||||
```
|
||||
|
||||
Smoke tests prove loadability only. They are not a substitute for numerical
|
||||
component or pipeline parity.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `../add-model/shared/common_rules.md`. Pipeline-specific ask cases include
|
||||
dropping a public mode, modality, or output head; adding a new workload enum;
|
||||
changing official defaults for user-facing behavior; accepting a known pipeline
|
||||
parity blocker; running GPU-heavy quality work outside the agreed scope; or
|
||||
publishing/uploading generated references or converted weights.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../add-model/contracts/pipeline_handoff.md` and update the shared state
|
||||
files before handoff.
|
||||
|
||||
Do not hand back a green pipeline if smoke or parity skipped locally. A skip is a
|
||||
setup gap, not a pass.
|
||||
|
||||
## References
|
||||
|
||||
- `fastvideo/pipelines/composed_pipeline_base.py` for module loading and stage
|
||||
execution.
|
||||
- `fastvideo/pipelines/basic/wan/` for standard video T2V/I2V/DMD variants.
|
||||
- `fastvideo/pipelines/basic/stable_audio/` for audio-specific stage composition.
|
||||
- `fastvideo/configs/pipelines/stable_audio.py` and
|
||||
`fastvideo/pipelines/basic/stable_audio/presets.py` for config/preset shape.
|
||||
- `fastvideo/registry.py` for `register_configs(...)` and preset registration.
|
||||
- `tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py` for latent
|
||||
parity structure.
|
||||
- `tests/local_tests/pipelines/test_stable_audio_pipeline_parity.py` for audio
|
||||
parity structure.
|
||||
- `tests/local_tests/pipelines/test_stable_audio_pipeline_smoke.py` for no-GPU
|
||||
import/registry/preset preflight shape.
|
||||
@@ -1,147 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Pipeline parity scaffold for TODO_MODEL_FAMILY.
|
||||
|
||||
Copy this file to
|
||||
`tests/local_tests/pipelines/test_<family>_pipeline_parity.py` and replace every
|
||||
TODO before treating it as an executable scaffold.
|
||||
|
||||
The filled test should compare denoised latents, decoded media, audio waveform,
|
||||
or another concrete output from the official pipeline against FastVideo. A
|
||||
successful generation without tensor/media comparison is not parity.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from torch.testing import assert_close
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
_MODEL_FAMILY = "TODO_MODEL_FAMILY"
|
||||
_OFFICIAL_REF_ENV = "TODO_OFFICIAL_REF_PATH"
|
||||
_OFFICIAL_REF_DEFAULT = _REPO_ROOT / "TODO_OFFICIAL_REF_DIR"
|
||||
_FASTVIDEO_MODEL_ENV = "TODO_FASTVIDEO_MODEL_PATH"
|
||||
_FASTVIDEO_MODEL_DEFAULT = _REPO_ROOT / "converted_weights" / _MODEL_FAMILY
|
||||
|
||||
|
||||
def _path_from_env(env_name: str, default: Path) -> Path:
|
||||
return Path(os.getenv(env_name, str(default))).expanduser()
|
||||
|
||||
|
||||
def _add_official_to_path() -> Path:
|
||||
official_path = _path_from_env(_OFFICIAL_REF_ENV, _OFFICIAL_REF_DEFAULT)
|
||||
if not official_path.exists():
|
||||
pytest.skip(f"Official reference not found at {official_path}")
|
||||
if str(official_path) not in sys.path:
|
||||
sys.path.insert(0, str(official_path))
|
||||
return official_path
|
||||
|
||||
|
||||
def _log_tensor_stats(label: str, tensor: torch.Tensor) -> None:
|
||||
value = tensor.detach().float()
|
||||
print(f"[{_MODEL_FAMILY} PIPELINE] {label}: shape={tuple(tensor.shape)} "
|
||||
f"dtype={tensor.dtype} device={tensor.device} "
|
||||
f"min={value.min().item():.6f} max={value.max().item():.6f} "
|
||||
f"mean={value.mean().item():.6f} std={value.std().item():.6f}")
|
||||
|
||||
|
||||
def _extract_tensor(output: Any, key: str) -> torch.Tensor:
|
||||
if isinstance(output, dict):
|
||||
value = output.get(key)
|
||||
else:
|
||||
value = getattr(output, key, None)
|
||||
if value is None:
|
||||
raise AssertionError(f"Pipeline output did not contain {key!r}")
|
||||
if not torch.is_tensor(value):
|
||||
try:
|
||||
import numpy as np
|
||||
value = torch.from_numpy(np.asarray(value))
|
||||
except Exception as exc: # pragma: no cover - scaffold guard
|
||||
raise AssertionError(f"Could not convert {key!r} to tensor") from exc
|
||||
return value.detach().float().cpu()
|
||||
|
||||
|
||||
def _run_official_pipeline(
|
||||
official_path: Path,
|
||||
params: dict[str, Any],
|
||||
device: torch.device,
|
||||
) -> Any:
|
||||
del official_path, params, device
|
||||
pytest.skip("TODO: import the official pipeline/factory, load official weights, "
|
||||
"run with params, and return the comparison target.")
|
||||
|
||||
|
||||
def _run_fastvideo_pipeline(model_path: Path, params: dict[str, Any]) -> Any:
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
str(model_path),
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False,
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=False,
|
||||
text_encoder_cpu_offload=False,
|
||||
)
|
||||
try:
|
||||
return generator.generate_video(
|
||||
prompt=params["prompt"],
|
||||
negative_prompt=params.get("negative_prompt"),
|
||||
output_path=f"outputs_{_MODEL_FAMILY}/pipeline_parity",
|
||||
save_video=False,
|
||||
height=params.get("height"),
|
||||
width=params.get("width"),
|
||||
num_frames=params.get("num_frames"),
|
||||
fps=params.get("fps"),
|
||||
num_inference_steps=params["num_inference_steps"],
|
||||
guidance_scale=params.get("guidance_scale"),
|
||||
seed=params["seed"],
|
||||
)
|
||||
finally:
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not torch.cuda.is_available(),
|
||||
reason="TODO_MODEL_FAMILY pipeline parity requires CUDA.",
|
||||
)
|
||||
def test_todo_model_family_pipeline_official_parity() -> None:
|
||||
official_path = _add_official_to_path()
|
||||
fastvideo_model_path = _path_from_env(
|
||||
_FASTVIDEO_MODEL_ENV,
|
||||
_FASTVIDEO_MODEL_DEFAULT,
|
||||
)
|
||||
if not fastvideo_model_path.exists():
|
||||
pytest.skip(f"FastVideo model path not found at {fastvideo_model_path}")
|
||||
|
||||
device = torch.device("cuda:0")
|
||||
params = {
|
||||
"prompt": "TODO: stable parity prompt",
|
||||
"negative_prompt": "",
|
||||
"height": 64,
|
||||
"width": 64,
|
||||
"num_frames": 9,
|
||||
"fps": 8,
|
||||
"num_inference_steps": 4,
|
||||
"guidance_scale": 1.0,
|
||||
"seed": 0,
|
||||
}
|
||||
|
||||
official_output = _run_official_pipeline(official_path, params, device)
|
||||
fastvideo_output = _run_fastvideo_pipeline(fastvideo_model_path, params)
|
||||
|
||||
comparison_key = "TODO_COMPARISON_KEY"
|
||||
official_tensor = _extract_tensor(official_output, comparison_key)
|
||||
fastvideo_tensor = _extract_tensor(fastvideo_output, comparison_key)
|
||||
|
||||
_log_tensor_stats("official", official_tensor)
|
||||
_log_tensor_stats("fastvideo", fastvideo_tensor)
|
||||
assert official_tensor.shape == fastvideo_tensor.shape
|
||||
|
||||
diff = (official_tensor - fastvideo_tensor).abs()
|
||||
print(f"diff max={diff.max().item():.6f} "
|
||||
f"mean={diff.mean().item():.6f} median={diff.median().item():.6f}")
|
||||
assert_close(fastvideo_tensor, official_tensor, atol=1e-2, rtol=1e-2)
|
||||
@@ -1,141 +0,0 @@
|
||||
---
|
||||
name: add-model-10-pr-review
|
||||
description: Review rubric for FastVideo PRs that add or modify model families, variants, first-class components, checkpoint conversion, pipelines, parity coverage, or generated-media quality baselines. Use when reviewing a PR whose diff touches fastvideo/models/, fastvideo/pipelines/basic/, fastvideo/registry.py, scripts/checkpoint_conversion/, fastvideo/tests/ssim/, or related model-port surfaces. Pairs with review-pr-link as a project-scoped review pass; produces findings, not fixes.
|
||||
---
|
||||
|
||||
# Add-Model PR Review
|
||||
|
||||
Use this skill when a reviewed PR appears to add, port, or substantially modify
|
||||
a FastVideo model family, model variant, first-class model component,
|
||||
checkpoint conversion, model pipeline, or local parity coverage.
|
||||
|
||||
This is a review skill, not an implementation workflow. Do not run `/add-model`
|
||||
or start writing missing port code during review. Use the add-model skill stack
|
||||
as a rubric for findings.
|
||||
|
||||
## Trigger Paths
|
||||
|
||||
Trigger this skill if `git diff --name-only <base>...HEAD` includes any of:
|
||||
|
||||
- `fastvideo/models/dits/`, `fastvideo/configs/models/dits/`
|
||||
- `fastvideo/models/vaes/`, `fastvideo/configs/models/vaes/`
|
||||
- `fastvideo/models/encoders/`, `fastvideo/configs/models/encoders/`
|
||||
- `fastvideo/models/schedulers/`, `fastvideo/configs/models/schedulers/`
|
||||
- `fastvideo/models/upsamplers/`, `fastvideo/configs/models/upsamplers/`
|
||||
- `fastvideo/models/audio/`, `fastvideo/configs/models/audio/`
|
||||
- `fastvideo/pipelines/basic/`, `fastvideo/configs/pipelines/`
|
||||
- `fastvideo/registry.py`, `fastvideo/api/sampling_param.py`
|
||||
- `scripts/checkpoint_conversion/`
|
||||
- `examples/inference/basic/`
|
||||
- `tests/local_tests/`, especially component or pipeline parity tests
|
||||
- `fastvideo/tests/ssim/` or other quality-regression tests for generated media
|
||||
|
||||
Also trigger when the PR title/body claims a new model, model variant, VAE,
|
||||
encoder, scheduler, conditioner, pipeline, conversion script, or generated-media
|
||||
quality baseline even if the path list is incomplete.
|
||||
|
||||
## Review Inputs
|
||||
|
||||
Read these add-model references as review checklists:
|
||||
|
||||
- `../add-model/SKILL.md`: phase gates and final handoff requirements.
|
||||
- `../add-model/shared/common_rules.md`: token/auth safety, production import
|
||||
boundaries, state files, and skip/pass semantics.
|
||||
- `../add-model/contracts/final_handoff.md`: final evidence expected from a
|
||||
complete port.
|
||||
- `../add-model/contracts/component_context.md` and
|
||||
`../add-model/contracts/component_skill_handoff.md`: component evidence and
|
||||
parity-debug expectations.
|
||||
- `../add-model/contracts/conversion_request.md` and
|
||||
`../add-model/contracts/conversion_handoff.md`: conversion evidence,
|
||||
strict-load status, config validation, and retry context.
|
||||
- `../add-model/contracts/pipeline_context.md` and
|
||||
`../add-model/contracts/pipeline_handoff.md`: pipeline class/stage/config/
|
||||
preset/registry/example evidence.
|
||||
|
||||
Then read only the satellite skill(s) that match touched areas:
|
||||
|
||||
- DiT/transformer changes: `../add-model-03-port-dit/SKILL.md`.
|
||||
- VAE changes: `../add-model-04-port-vae/SKILL.md`.
|
||||
- Encoder/conditioner changes: `../add-model-05-port-encoder/SKILL.md`.
|
||||
- Scheduler/upsampler/vocoder/other components:
|
||||
`../add-model-06-port-generic/SKILL.md`.
|
||||
- Component parity tests: `../add-model-02-parity/SKILL.md`.
|
||||
- Checkpoint conversion: `../add-model-07-conversion/SKILL.md`.
|
||||
- Pipeline/config/presets/registry/examples:
|
||||
`../add-model-09-pipeline/SKILL.md`.
|
||||
- Prep/state docs: `../add-model-01-prep/SKILL.md`.
|
||||
|
||||
## Required Review Lanes
|
||||
|
||||
For a full model-family or model-variant PR, cover all lanes. For a
|
||||
component-only PR, cover the component, conversion/parity as applicable, and the
|
||||
documented downstream consumer.
|
||||
|
||||
1. Scope and source-of-truth lane:
|
||||
Verify the PR clearly identifies the official reference, weights/revision,
|
||||
supported variants, modalities, output heads, and any approved scope cuts.
|
||||
|
||||
2. Component lane:
|
||||
Verify each required component is FastVideo-native or has a documented and
|
||||
accepted lazy-wrapper exception. Check bucket/config inheritance, `EntryClass`,
|
||||
state-dict surface, reused-component evidence, and output heads.
|
||||
|
||||
3. Conversion lane:
|
||||
Verify mappings are derived from prototype key/shape dumps, source layout is
|
||||
supported, skipped keys are intentional, emitted configs validate through
|
||||
production paths, component strict-load status is recorded, `model_index.json`
|
||||
library tokens match loaders, and revisions are pinned when converting from
|
||||
HF.
|
||||
|
||||
4. Component parity lane:
|
||||
Verify local parity tests exist for every required component, including reused
|
||||
components. Scaffolds may skip in CI, but the PR must provide local non-skip
|
||||
PASS evidence or an explicit accepted blocker.
|
||||
|
||||
5. Pipeline lane:
|
||||
Verify stage order, required modules, `_class_name` / `EntryClass.__name__`
|
||||
resolution, config defaults, presets, `SamplingParam` fields, registry
|
||||
registration, examples, smoke tests, and pipeline parity.
|
||||
|
||||
6. Quality and evidence lane:
|
||||
Verify media quality regression is added or explicitly deferred, examples run,
|
||||
generated outputs are non-corrupt, `tests/local_tests/<family>/README.md` and
|
||||
`PORT_STATUS.md` are current, and final blockers are surfaced in the review.
|
||||
|
||||
## Findings To Prioritize
|
||||
|
||||
Prioritize review findings in this order:
|
||||
|
||||
- Missing or skipped required component parity without accepted blocker.
|
||||
- Pipeline parity/smoke/example missing or skipped for a pipeline PR.
|
||||
- Conversion emits unloadable or unvalidated configs/weights.
|
||||
- Wrong `model_index.json` `_class_name`, component library token, or registry
|
||||
class resolution.
|
||||
- Runtime diffusers/transformers model-class imports for components that own
|
||||
weights or numerical behavior.
|
||||
- Dropped modalities, output heads, variants, or conditioning streams without
|
||||
explicit approval.
|
||||
- Reused FastVideo component lacks exact definition/instantiation proof or
|
||||
non-skip parity.
|
||||
- Public generation kwargs/preset defaults missing from `SamplingParam`.
|
||||
- Tests only check shapes, importability, or successful generation without
|
||||
numerical/media comparison.
|
||||
- Tokens, credentials, reference clones, staged weights, or generated bulk assets
|
||||
committed to the PR.
|
||||
|
||||
## Output Format
|
||||
|
||||
Write normal code-review findings first, ordered by severity. Include file and
|
||||
line references from the PR diff when possible.
|
||||
|
||||
Use this phrasing for missing add-model evidence:
|
||||
|
||||
```text
|
||||
This PR does not satisfy the add-model <component|conversion|pipeline|final>
|
||||
gate because <specific required evidence> is missing. The risk is <runtime load,
|
||||
numerical parity, dropped output, registry resolution, etc.>.
|
||||
```
|
||||
|
||||
Keep the summary short. Mention which lanes were reviewed and which could not be
|
||||
verified because assets, GPU time, or external credentials were unavailable.
|
||||
@@ -1,438 +0,0 @@
|
||||
---
|
||||
name: add-model
|
||||
description: Manual /add-model workflow for implementing a FastVideo model or first-class component port after add-model-01-prep has staged reference code and weights. Organizes the port into numbered phases with conversion rules, component policies, parity gates, and handoff checks.
|
||||
---
|
||||
|
||||
# Add Model
|
||||
|
||||
## Manual Invocation
|
||||
|
||||
This skill is for explicit `/add-model` use only. Do not auto-start it from a
|
||||
casual model-port mention. The setup-only workflow is
|
||||
`../add-model-01-prep/SKILL.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
Port a new FastVideo model family, model variant, or first-class reusable
|
||||
component so it can be loaded through FastVideo's native model, config, stage,
|
||||
registry, preset, and test infrastructure.
|
||||
|
||||
FastVideo has one pipeline architecture: stage-based composition via
|
||||
`ComposedPipelineBase`. Vary the stages and modules, not the architecture.
|
||||
|
||||
## Scope Shapes
|
||||
|
||||
Use this skill for either shape:
|
||||
|
||||
| Shape | Required output |
|
||||
|---|---|
|
||||
| Full model family or variant | Native components, conversion if needed, pipeline config/class, presets, registry, smoke test, local parity tests, example, quality regression. |
|
||||
| First-class component contribution | Native component class/config, bucket export, component parity test, and a documented downstream pipeline that will consume it. Skip pipeline/preset/registry rows only when the contribution is intentionally component-only. |
|
||||
|
||||
If upstream ships many variants, lock scope before coding. "Base model" means
|
||||
checkpoint variant, not a modality subset. If the base checkpoint produces
|
||||
audio, pose, depth, masks, or other output heads, either support those outputs
|
||||
or get explicit user agreement to drop them.
|
||||
|
||||
## Required Input
|
||||
|
||||
Start from an `add-model-01-prep` handoff, or equivalent fields matching
|
||||
`contracts/prep_handoff.md`.
|
||||
|
||||
Before Phase 0, read the shared rules and all relevant schemas:
|
||||
|
||||
- `shared/common_rules.md`
|
||||
- `contracts/prep_handoff.md`
|
||||
- `contracts/port_state.md`
|
||||
- `contracts/escape_hatch.md`
|
||||
- `contracts/component_context.md`
|
||||
- `contracts/parity_status.md`
|
||||
- `contracts/conversion_request.md`
|
||||
- `contracts/conversion_handoff.md`
|
||||
- `contracts/component_skill_handoff.md`
|
||||
- `contracts/pipeline_context.md`
|
||||
- `contracts/pipeline_handoff.md`
|
||||
- `contracts/final_handoff.md`
|
||||
|
||||
## Hard Rules
|
||||
|
||||
- Follow `shared/common_rules.md` for token/auth safety, state files, escape
|
||||
hatches, production import boundaries, and skip/pass semantics.
|
||||
- If the prep handoff is missing or ambiguous, stop and run
|
||||
`../add-model-01-prep/SKILL.md`.
|
||||
- If a needed component is not ported, do not ship the pipeline that needs it.
|
||||
- Wan is grandfathered for missing local parity; do not copy its missing-test
|
||||
precedent for new work.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `shared/common_rules.md` and `contracts/escape_hatch.md`. The main
|
||||
orchestrator should ask only when no phase skill can safely continue under the
|
||||
shared rules.
|
||||
|
||||
## Files Map
|
||||
|
||||
| Area | Paths |
|
||||
|---|---|
|
||||
| DiT | `fastvideo/models/dits/<family>.py`, `fastvideo/configs/models/dits/<family>.py`, bucket `__init__.py`. |
|
||||
| VAE | `fastvideo/models/vaes/<arch_or_family>.py`, `fastvideo/configs/models/vaes/<arch_or_family>.py`, bucket `__init__.py`. Name by shared arch when reusable (`oobleck.py`, `autoencoder_kl.py`), otherwise by family (`wanvae.py`). |
|
||||
| Encoder / conditioner / scheduler / upsampler | Native class/config in the matching `fastvideo/models/<bucket>/` and `fastvideo/configs/models/<bucket>/` bucket. |
|
||||
| Lazy loader wrapper | Optional `fastvideo/models/<bucket>/<family>_loader.py` or similar thin `nn.Module` wrapper when a component is fetched from an external HF repo and should be hidden from host-pipeline state-dict matching. |
|
||||
| Conversion | `scripts/checkpoint_conversion/<family>_to_diffusers.py` only when `needs_conversion=yes`. |
|
||||
| Pipeline | `fastvideo/pipelines/basic/<family>/<family>_pipeline.py` plus sibling files for variants whose components or required modules differ. |
|
||||
| Pipeline config | `fastvideo/configs/pipelines/<family>.py` or `fastvideo/pipelines/basic/<family>/pipeline_configs.py`. |
|
||||
| Stages | `fastvideo/pipelines/basic/<family>/stages/` only for model-specific stage subclasses. |
|
||||
| Presets / registry | `fastvideo/pipelines/basic/<family>/presets.py`, `fastvideo/registry.py`. |
|
||||
| Tests | Component parity under `tests/local_tests/<bucket>/`; pipeline smoke/parity under `tests/local_tests/pipelines/`; CI-backed quality tests under `fastvideo/tests/`. |
|
||||
| Example | `examples/inference/basic/basic_<family>*.py`, one per public mode/variant. |
|
||||
|
||||
## Phase 0: Scope And Handoff Gate
|
||||
|
||||
1. Validate every required handoff field.
|
||||
2. Resolve `needs_conversion=unknown` before component work:
|
||||
|
||||
```bash
|
||||
python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \
|
||||
"<hf-or-local-path>" \
|
||||
--json
|
||||
```
|
||||
|
||||
3. List first-PR scope across both axes:
|
||||
- Variant axis: base, distill, SR/refine, causal, DMD, I2V, V2V, etc.
|
||||
- Modality axis: video, image, audio, pose, depth, masks, text, etc.
|
||||
4. For component-only work, explicitly name the downstream full-pipeline PR or
|
||||
planned consumer.
|
||||
5. Confirm `official_env_status` is `imports_ok` or
|
||||
`private_deps_need_stubs`. If it is `blocked`, return to
|
||||
`../add-model-01-prep/SKILL.md` before parity scaffolding.
|
||||
6. Confirm `local_tests_readme` exists and records official setup, HF weights,
|
||||
dependency changes, and planned parity commands for reviewers.
|
||||
7. Confirm `port_state_file` exists, follows `contracts/port_state.md`, and has
|
||||
rows for open questions/issues found during prep.
|
||||
8. If there are multiple official implementations, choose the one whose
|
||||
architecture matches the published weights. A blessed library port can be a
|
||||
better parity reference than a highly configurable research repo; document
|
||||
the choice in tests.
|
||||
|
||||
## Phase 1: Reference And Architecture Study
|
||||
|
||||
Read the official pipeline call path before writing code.
|
||||
|
||||
Record:
|
||||
|
||||
- Required modules from `model_index.json` or equivalent: transformer, VAE,
|
||||
text encoders, tokenizers, scheduler, image encoders, audio VAE, vocoder,
|
||||
conditioners, upsamplers.
|
||||
- Input/output modalities and every dedicated DiT output head.
|
||||
- Text/image/audio encoding flow, latent shape, dtype, scaling, packing,
|
||||
scheduler/timestep math, guidance math, VAE normalization, and decode flow.
|
||||
- Whether the official code relies on private deps, custom ops, or special
|
||||
kernels that parity tests must stub.
|
||||
|
||||
Arch config rule:
|
||||
|
||||
- `ArchConfig` fields must match the emitted per-component config, especially
|
||||
`transformer/config.json`, one-to-one.
|
||||
- Pipeline knobs do not belong on the DiT arch config: inference steps, CFG
|
||||
scales, flow shift, FPS, VAE stride, text target length, data-proxy knobs,
|
||||
eval defaults, and sampling defaults go on `PipelineConfig`, presets, or
|
||||
stages.
|
||||
- If the HF repo is raw or has empty configs, synthesize
|
||||
`transformer/config.json` from the official Python model-config class, not
|
||||
from data/eval config classes.
|
||||
|
||||
## Phase 2: Early Parity Scaffolding
|
||||
|
||||
Create component parity tests before or alongside implementation. Use
|
||||
`../add-model-02-parity/SKILL.md` and its `templates/component_parity_test.py`.
|
||||
The official reference must import in the current FastVideo environment, or the
|
||||
prep handoff must identify private deps that will be stubbed locally for tests.
|
||||
Use `local_tests_readme` as the reviewer-facing source for setup commands and
|
||||
update its planned test table as parity scaffolds are added.
|
||||
|
||||
This phase is early by design:
|
||||
|
||||
- Official loading can be implemented from the reference study.
|
||||
- FastVideo loading can target planned standardized class/config/loader paths.
|
||||
- Tests may initially skip because the FastVideo class or converted weights do
|
||||
not exist yet.
|
||||
- The scaffold must still contain real official loading, deterministic inputs,
|
||||
output extraction, and concrete tensor comparisons. No unconditional skips,
|
||||
no shape-only tests.
|
||||
|
||||
Use subagents here: dispatch one parity-test subagent per required component,
|
||||
including components that may be reused. Their output becomes the red/skip
|
||||
target that porting or reuse-verification subagents make pass later.
|
||||
|
||||
## Phase 3: Reuse Gate And Component Dispatch
|
||||
|
||||
Build a component inventory before implementation:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| Component | transformer, VAE, text encoder, image encoder, scheduler, conditioner, upsampler, vocoder, etc. |
|
||||
| Official definition | Repo-relative source file, class/function name, and relevant line/range if known. |
|
||||
| Official instantiation | Repo-relative pipeline/config/factory call site plus constructor args and runtime flags. |
|
||||
| FastVideo target | Existing class to reuse or new bucket/file/config to add. |
|
||||
| Parity test | Required local test path, including reused components. |
|
||||
| Status | `reuse_pending`, `reuse_proven`, `port_pending`, `non_skip_pass`, or `blocked`. |
|
||||
|
||||
Reuse is allowed only from the checked-out FastVideo tree. Do not wait for or
|
||||
depend on an open PR adding a native class; add the native port directly in this
|
||||
PR if the current tree cannot be reused.
|
||||
|
||||
Reuse decision:
|
||||
|
||||
1. Record exact official definition and instantiation evidence for every
|
||||
component.
|
||||
2. If an existing FastVideo class and config match both definition and
|
||||
instantiation, pass that reused target to the bucket-specific skill in
|
||||
`mode=prototype` and require reuse evidence plus key/shape dumps.
|
||||
3. If either definition or instantiation differs, port the component directly as
|
||||
FastVideo-native code through the bucket-specific skill.
|
||||
4. Reused components still require non-skip component parity against the exact
|
||||
official instantiation used by the target pipeline.
|
||||
|
||||
Porting subagent dispatch:
|
||||
|
||||
- Dispatch one subagent per component after Phase 2 parity scaffolds exist.
|
||||
- Use `../add-model-03-port-dit/SKILL.md` for DiTs/transformers.
|
||||
- Use `../add-model-04-port-vae/SKILL.md` for VAEs.
|
||||
- Use `../add-model-05-port-encoder/SKILL.md` for text, image, audio, or compound
|
||||
encoders/conditioners that fit the encoder config bucket.
|
||||
- Use `../add-model-06-port-generic/SKILL.md` for schedulers, upsamplers,
|
||||
vocoders, adapters, preprocessors, or unknown components.
|
||||
- Each subagent owns one component only and must loop on that component's local
|
||||
parity test until it produces a non-skip PASS or returns a precise blocker.
|
||||
|
||||
Every component subagent must receive a complete packet matching
|
||||
`contracts/component_context.md`. If any required path is unknown, pass `unknown`
|
||||
plus the exact search already performed. Do not silently omit ambiguous official
|
||||
files or prototype concerns.
|
||||
|
||||
Bucket, layer, and attention rules live in the bucket-specific skills and
|
||||
`fastvideo/layers/AGENTS.md`.
|
||||
|
||||
## Phase 4: Native Component Prototype
|
||||
|
||||
Conversion needs a FastVideo state-dict surface. Use the Phase 3
|
||||
bucket-specific skill in `mode=prototype` for every required component, including
|
||||
reused components.
|
||||
|
||||
Prototype success criteria:
|
||||
|
||||
- the FastVideo-native or reused class/config can import and instantiate with the
|
||||
exact official architecture args;
|
||||
- official and FastVideo key/shape dumps exist for every stateful component;
|
||||
- `local_tests_readme` and `port_state_file` record prototype status and concerns;
|
||||
- the returned handoff matches `contracts/component_skill_handoff.md`.
|
||||
|
||||
Do not chase numerical parity in Phase 4. Prototype mode ends when conversion has
|
||||
the key/shape surface it needs, or when the component skill returns a precise
|
||||
blocker or escape hatch.
|
||||
|
||||
## Phase 5: Param Mapping And Weight Conversion
|
||||
|
||||
Use `../add-model-07-conversion/SKILL.md` after Phase 4 prototypes exist.
|
||||
Send a request matching `contracts/conversion_request.md`; consume the returned
|
||||
`contracts/conversion_handoff.md` update before Phase 6.
|
||||
|
||||
Use the prep handoff's `needs_conversion` value:
|
||||
|
||||
- `no`: verify the source already has the component layout FastVideo loaders can
|
||||
consume, then record any passthrough components.
|
||||
- `yes`: write `scripts/checkpoint_conversion/<family>_to_diffusers.py` and
|
||||
output `converted_weights/<family>/`.
|
||||
- `unknown`: return to Phase 0.
|
||||
|
||||
The conversion skill owns source-layout handling, mapping derivation, config and
|
||||
`model_index.json` emission, passthrough assets, strict-load verification, and
|
||||
Phase 6 retry requests. Component skills must not patch conversion scripts or
|
||||
converted weights ad hoc.
|
||||
|
||||
## Phase 6: Component Parity Debug
|
||||
|
||||
This is the expected expensive loop. Dispatch one subagent per required
|
||||
component, including reused components, using the bucket-specific skill in
|
||||
`mode=parity-debug`.
|
||||
|
||||
Each subagent gets:
|
||||
|
||||
- the complete component context packet from Phase 3/4;
|
||||
- updated conversion mapping notes and strict-load result from Phase 5;
|
||||
- any prototype concerns or unknowns that were not resolved before conversion.
|
||||
|
||||
The bucket-specific skills own parity-debug tactics. If a failure belongs to
|
||||
conversion, route it through `../add-model-07-conversion/SKILL.md` with a retry
|
||||
request matching `contracts/conversion_request.md`, then resume the component
|
||||
skill with the updated conversion handoff.
|
||||
|
||||
When a component failure narrows to layer-by-layer numerical drift, load
|
||||
`../add-model-08-trace/SKILL.md` before writing custom hooks. It uses
|
||||
`fastvideo/hooks/activation_trace.py`; canonical env vars and JSONL format are
|
||||
documented in `docs/contributing/activation_trace.md`.
|
||||
|
||||
Phase 6 ends only when every required component handoff reports
|
||||
`parity_status=non_skip_pass`, or when a precise blocker or escape hatch is
|
||||
recorded in `port_state_file`.
|
||||
|
||||
## Phase 7: Pipeline, Stages, And Variants
|
||||
|
||||
Do not start Phase 7 until every required component, reused or ported, has a
|
||||
non-skip local parity PASS from Phase 6. If any component parity test is still
|
||||
`scaffold_skip`, `debug_red`, `blocked`, or missing, resume Phase 6 first.
|
||||
|
||||
Use `../add-model-09-pipeline/SKILL.md` for pipeline definition and parity-debug.
|
||||
Send a complete packet matching `contracts/pipeline_context.md`; consume the
|
||||
returned `contracts/pipeline_handoff.md` before moving to quality regression or
|
||||
final handoff.
|
||||
|
||||
The pipeline skill owns:
|
||||
|
||||
- pipeline class, stage chain, and optional model-specific stages;
|
||||
- pipeline config, presets, registry updates, and examples;
|
||||
- official args/defaults/presets comparison before setting FastVideo defaults;
|
||||
- pipeline smoke and parity tests;
|
||||
- continuous pipeline parity-debug until non-skip PASS or precise blocker;
|
||||
- updates to `local_tests_readme` and `port_state_file`.
|
||||
|
||||
The pipeline handoff must explicitly cover stage order, variants, modality and
|
||||
output-head handling, config/preset/registry/example status, smoke/parity tests,
|
||||
and any return-to-Phase-6 evidence.
|
||||
|
||||
## Phase 8: PipelineConfig, Presets, Registry, Examples
|
||||
|
||||
This phase is implemented through `../add-model-09-pipeline/SKILL.md` after the
|
||||
Phase 7 component-parity gate passes. Accept the pipeline handoff only if it
|
||||
covers configs, presets, registry detection/exact class resolution, examples,
|
||||
new `SamplingParam` fields for public kwargs/defaults, and local smoke/parity
|
||||
status. Detailed rules live in `../add-model-09-pipeline/SKILL.md`.
|
||||
|
||||
## Phase 9: Parity Activation And Local Verification
|
||||
|
||||
Local parity is author-run, not CI-enforced. CI may only run package-level
|
||||
quality tests later. Before handoff, Phase 2 scaffolds must be activated into
|
||||
non-skip PASS results.
|
||||
|
||||
Order is mandatory:
|
||||
|
||||
1. Run conversion if needed.
|
||||
2. Run component parity for every required component, including reused ones.
|
||||
3. Run pipeline smoke.
|
||||
4. Run pipeline parity.
|
||||
5. Run the basic example.
|
||||
|
||||
If pipeline smoke or parity points back to component implementation,
|
||||
strict-load, or conversion mapping, return to Phase 6 or Phase 5 rather than
|
||||
patching around the issue in the pipeline.
|
||||
|
||||
Skip policy:
|
||||
|
||||
- Follow `shared/common_rules.md`: a committed local test may skip for absent
|
||||
clones/weights, but a local skip is not a verified pass.
|
||||
|
||||
Use the commands and tolerance guidance from `../add-model-02-parity/SKILL.md` for
|
||||
component checks and from `../add-model-09-pipeline/SKILL.md` for pipeline smoke,
|
||||
pipeline parity, and examples. Record exact commands, status, and blockers in
|
||||
`local_tests_readme` and `port_state_file`.
|
||||
|
||||
## Phase 10: Quality Regression
|
||||
|
||||
Video outputs:
|
||||
|
||||
- Add `fastvideo/tests/ssim/test_<family>_similarity.py` when output video
|
||||
quality must be preserved.
|
||||
- Seed references through `seed-ssim-references` after the test exists.
|
||||
|
||||
Audio outputs:
|
||||
|
||||
- SSIM does not apply. Use an audio-specific regression metric such as
|
||||
mel-spectrogram L1, multi-resolution STFT, CLAP cosine, or a project-approved
|
||||
learned metric.
|
||||
- Document the metric and hardware/runtime assumptions in the test.
|
||||
|
||||
Joint AV outputs:
|
||||
|
||||
- Keep video and audio regression checks separate unless there is a validated
|
||||
joint metric.
|
||||
|
||||
## Phase 11: Post-Parity Review And Handoff
|
||||
|
||||
After parity is green, run a hot-path review before handoff:
|
||||
|
||||
- Hoist constant tensor allocations out of sampler/denoising loops.
|
||||
- Replace per-step `randn_like` churn with preallocated buffers plus
|
||||
`.normal_()` when safe.
|
||||
- Move `torch.backends.*` flag changes to one-shot setup/load paths.
|
||||
- Delete `batch.extra` writes that nothing reads.
|
||||
- Derive magic constants from configs when possible.
|
||||
|
||||
Pre-handoff checklist:
|
||||
|
||||
```text
|
||||
[ ] Prep handoff is complete and committed nowhere with token values.
|
||||
[ ] Conversion was run if needed and output loads with real weights.
|
||||
[ ] Every required component, reused or newly ported, has a non-skip local parity PASS.
|
||||
[ ] `local_tests_readme` lists every component parity test, command, status, and blocker if any.
|
||||
[ ] `port_state_file` has every open question/issue either resolved or listed as an explicit blocker.
|
||||
[ ] Any `next_step=ask_user` has a matching `escape_hatch` block and `E###` row.
|
||||
[ ] Pipeline smoke has a non-skip local PASS.
|
||||
[ ] Pipeline parity has a non-skip local PASS against the official reference.
|
||||
[ ] Basic example runs and writes a non-corrupt output.
|
||||
[ ] Video SSIM or audio-specific quality regression is added or explicitly deferred.
|
||||
[ ] Runtime production code has no diffusers/transformers model-class imports.
|
||||
[ ] Production comments are WHY-focused; examples have user-story docstrings.
|
||||
[ ] Post-parity hot-path pass is complete.
|
||||
```
|
||||
|
||||
Ask before deleting any reference clone or staged weights created by
|
||||
`add-model-01-prep`. Leave `.gitignore` entries so future parity assets stay
|
||||
untracked. Never commit the clone, weights, `.env`, credentials, or anything
|
||||
matching `*secret*`.
|
||||
|
||||
## References
|
||||
|
||||
- `../add-model-01-prep/SKILL.md` for user-input collection, HF inspection,
|
||||
weight staging, reference cloning, and setup handoff.
|
||||
- `contracts/` for canonical handoff schemas used by prep, parity, conversion,
|
||||
component porting, escape hatches, and final handoff.
|
||||
- `../add-model-02-parity/SKILL.md` for early component parity scaffolds and
|
||||
activation templates.
|
||||
- `../add-model-07-conversion/SKILL.md` for Phase 5 mapping, conversion scripts,
|
||||
monolithic checkpoint splitting, and strict-load checks.
|
||||
- `../add-model-03-port-dit/SKILL.md`, `../add-model-04-port-vae/SKILL.md`,
|
||||
`../add-model-05-port-encoder/SKILL.md`, and
|
||||
`../add-model-06-port-generic/SKILL.md` for component subagent implementation
|
||||
and parity-debug loops.
|
||||
- `../add-model-09-pipeline/SKILL.md` for pipeline definition, config/preset/
|
||||
registry/example wiring, smoke tests, and pipeline parity-debug.
|
||||
- `fastvideo/layers/AGENTS.md` for native layer selection and state-dict surface
|
||||
guidance.
|
||||
- `docs/contributing/coding_agents.md` for narrative context.
|
||||
- `docs/design/overview.md` for pipeline/config/registry architecture.
|
||||
- `fastvideo/pipelines/basic/wan/` for standard T2V/I2V/DMD/Causal variants.
|
||||
- `fastvideo/pipelines/basic/ltx2/` for non-standard stages and audio/video
|
||||
patterns.
|
||||
- `tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py` for pipeline
|
||||
parity shape.
|
||||
- `tests/local_tests/transformers/test_ltx2.py`,
|
||||
`tests/local_tests/vaes/test_ltx2_vae.py`, and
|
||||
`tests/local_tests/encoders/test_ltx2_gemma_parity.py` for component parity.
|
||||
- `scripts/checkpoint_conversion/convert_ltx2_weights.py` for modern conversion
|
||||
script shape.
|
||||
- `scripts/checkpoint_conversion/wan_to_diffusers.py` for legacy regex mapping
|
||||
reference only.
|
||||
|
||||
## Changelog
|
||||
|
||||
| Date | Change |
|
||||
|---|---|
|
||||
| 2026-04-24 | Initial FastVideo add-model workflow. |
|
||||
| 2026-04-30 | Split external setup into `add-model-01-prep`. |
|
||||
| 2026-04-30 | Rewrote as manual `/add-model` phase workflow and incorporated prior review decisions. |
|
||||
| 2026-04-30 | Extracted early parity scaffolding into `add-model-02-parity` and moved it before conversion/component implementation. |
|
||||
| 2026-04-30 | Added component reuse proof gate, bucket-specific porting skills, and parity PASS requirement for reused components. |
|
||||
| 2026-04-30 | Split prototype, conversion, and parity-debug phases; added conversion skill for monolithic and separate checkpoint layouts. |
|
||||
| 2026-04-30 | Extracted handoff schemas into `contracts/` for shared use across skills. |
|
||||
| 2026-04-30 | Added pipeline skill contract and Phase 7 component-parity gate. |
|
||||
| 2026-04-30 | Added escape-hatch contract for user decisions and `ask_user` handoffs. |
|
||||
@@ -1,29 +0,0 @@
|
||||
# Add Model Contracts
|
||||
|
||||
Canonical handoff schemas for the `/add-model` workflow. When a skill needs to
|
||||
send or receive structured context, use these files instead of inventing a local
|
||||
schema.
|
||||
|
||||
| Contract | Use |
|
||||
|---|---|
|
||||
| `prep_handoff.md` | `add-model-01-prep` output and `/add-model` Phase 0 input. |
|
||||
| `port_state.md` | Per-port `PORT_STATUS.md` file tracking progress, open questions, and issues. |
|
||||
| `escape_hatch.md` | Shared pause-and-ask schema for user decisions the workflow cannot safely choose. |
|
||||
| `component_context.md` | Per-component packet passed to parity, prototype, conversion, and parity-debug subagents. |
|
||||
| `parity_status.md` | `add-model-02-parity` scaffold/activation status returned to `/add-model`. |
|
||||
| `conversion_request.md` | Phase 5 conversion input and Phase 6 conversion retry request. |
|
||||
| `conversion_handoff.md` | `add-model-07-conversion` output back to `/add-model` and component subagents. |
|
||||
| `component_skill_handoff.md` | Component porting skill output in prototype or parity-debug mode. |
|
||||
| `pipeline_context.md` | Phase 7 packet passed to `add-model-09-pipeline` after component parity is green. |
|
||||
| `pipeline_handoff.md` | `add-model-09-pipeline` output back to `/add-model` after pipeline definition or parity-debug. |
|
||||
| `final_handoff.md` | Final `/add-model` pre-handoff checklist summary. |
|
||||
|
||||
Rules:
|
||||
|
||||
- Do not omit required fields. Use `unknown` plus the search already performed
|
||||
when the value is not known yet.
|
||||
- Do not include raw token values. Use env var names only.
|
||||
- Keep model-specific mapping details in conversion scripts and the local tests
|
||||
README/status notes, not in generic skill docs.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block matching
|
||||
`escape_hatch.md`.
|
||||
@@ -1,55 +0,0 @@
|
||||
# Component Context Contract
|
||||
|
||||
Canonical per-component packet passed from `/add-model` to parity, prototype,
|
||||
conversion, and parity-debug subagents.
|
||||
|
||||
```text
|
||||
component_context:
|
||||
model_family: <snake_case>
|
||||
component: <name>
|
||||
component_type: <dit|vae|encoder|scheduler|conditioner|upsampler|vocoder|generic>
|
||||
mode: parity-scaffold | prototype | parity-debug
|
||||
official_ref_dir: <path or import path>
|
||||
official_definition_files:
|
||||
- path: <repo-relative or absolute path in official repo>
|
||||
symbols: <class/function names>
|
||||
notes: <layer graph, output contract, state-dict owner>
|
||||
official_instantiation_files:
|
||||
- path: <repo-relative or absolute path in official repo>
|
||||
symbols: <factory/pipeline/config names>
|
||||
args: <constructor args, config values, runtime flags>
|
||||
official_weight_source: <checkpoint file, subfolder, prefix, or passthrough source>
|
||||
fastvideo_target_files:
|
||||
- fastvideo/models/<bucket>/<file>.py
|
||||
- fastvideo/configs/models/<bucket>/<file>.py
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
parity_test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py
|
||||
prototype_key_dumps:
|
||||
official: converted_weights/<family>/_mapping/<component>_official_keys.json | planned | unknown
|
||||
fastvideo: converted_weights/<family>/_mapping/<component>_fastvideo_keys.json | planned | unknown
|
||||
conversion:
|
||||
script: scripts/checkpoint_conversion/<family>_to_diffusers.py | not_created | not_needed | unknown
|
||||
converted_component_dir: converted_weights/<family>/<component> | not_created | not_needed | unknown
|
||||
model_index_library: <diffusers|transformers|fastvideo|fastvideo.*|unknown|none>
|
||||
config_file: <config.json|scheduler_config.json|none|unknown>
|
||||
mapping_notes: <key prefixes, split/fuse concerns, skipped keys, not_created, not_needed, or unknown>
|
||||
production_loader_strictness: <strict|non_strict_with_allowed_keys|stateless|unknown>
|
||||
strict_load: <not_run | pass | pass_with_documented_exclusions | blocked>
|
||||
concerns_or_unknowns:
|
||||
- <prototype mismatch, ambiguous arg, missing op, dtype concern, output head, etc.>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- If any required path is unknown, pass `unknown` plus the exact search already
|
||||
performed.
|
||||
- Do not silently omit ambiguous official files, instantiation args, or prototype
|
||||
concerns.
|
||||
- For reused components, still fill every field and set `fastvideo_target_files`
|
||||
to the reused class/config.
|
||||
- In `mode=parity-scaffold`, prototype and conversion fields may be `planned`,
|
||||
`not_created`, `not_needed`, or `unknown`; do not invent paths or statuses that
|
||||
do not exist yet.
|
||||
- Update `port_state_file` when concerns, issues, conversion status, or parity
|
||||
status change.
|
||||
@@ -1,41 +0,0 @@
|
||||
# Component Skill Handoff Contract
|
||||
|
||||
Returned by `add-model-03-port-dit`, `add-model-04-port-vae`,
|
||||
`add-model-05-port-encoder`, and `add-model-06-port-generic`.
|
||||
|
||||
```text
|
||||
component: <name>
|
||||
mode: prototype | parity-debug
|
||||
files_changed: <model/config/export/test/readme paths>
|
||||
official_files_used: <definition files, instantiation files>
|
||||
prototype_key_dumps: <official path, fastvideo path, or none>
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
concerns_or_unknowns: <remaining or newly discovered concerns>
|
||||
parity_test: <path>
|
||||
parity_status: scaffold_skip | debug_red | non_skip_pass | blocked
|
||||
production_loader_strictness: strict | non_strict_with_allowed_keys | stateless
|
||||
strict_load: pass | pass_with_documented_exclusions | blocked | not_run
|
||||
pytest_output: <command + short result>
|
||||
blocker: <none or exact missing dependency/weights/numeric mismatch>
|
||||
conversion_retry_request: <none or failing keys/shapes/prefixes/evidence for add-model-07-conversion>
|
||||
readme_updated: yes | no
|
||||
next_step: phase_5_conversion | phase_5_conversion_retry | phase_6_continue | ask_user | blocked
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- In `mode=prototype`, parity may be `scaffold_skip` or `blocked`; key dumps are
|
||||
the required artifact. Successful prototype handoff should use
|
||||
`next_step=phase_5_conversion`.
|
||||
- In `mode=parity-debug`, final success requires `parity_status=non_skip_pass`.
|
||||
- If conversion is implicated, return `conversion_retry_request` and do not edit
|
||||
conversion scripts or converted weights directly.
|
||||
- If production loading is non-strict, list allowed missing/unexpected keys in
|
||||
the parity test or handoff and mark `strict_load=pass_with_documented_exclusions`.
|
||||
- Update `port_state_file` before returning: component row, open questions,
|
||||
issues/blockers, decisions, and handoff notes.
|
||||
- Return an `escape_hatch` only for user decisions, not for normal component
|
||||
implementation or parity-debug failures.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
|
||||
`PORT_STATUS.md` row.
|
||||
@@ -1,41 +0,0 @@
|
||||
# Conversion Handoff Contract
|
||||
|
||||
Returned by `../add-model-07-conversion/SKILL.md` to `/add-model` and component
|
||||
parity-debug subagents.
|
||||
|
||||
```text
|
||||
conversion_script: scripts/checkpoint_conversion/<family>_to_diffusers.py
|
||||
source_layout: <diffusers|raw_official|separate_components|monolithic|mixed|custom>
|
||||
converted_weights_dir: converted_weights/<model_family>
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
components_written: <list>
|
||||
passthrough_components: <list>
|
||||
strict_load: pass | pass_with_documented_exclusions | blocked
|
||||
component_context_updates:
|
||||
- component: <name>
|
||||
converted_component_dir: <path>
|
||||
model_index_library: <diffusers|transformers|fastvideo|fastvideo.*>
|
||||
config_file: <path or none>
|
||||
config_validation: pass | blocked | not_applicable
|
||||
mapping_notes: <prefixes, split/fuse ops, skipped keys>
|
||||
production_loader_strictness: strict | non_strict_with_allowed_keys | stateless
|
||||
strict_load: pass | pass_with_documented_exclusions | blocked | not_run
|
||||
retry_resolved: <yes | no | not_a_retry>
|
||||
concerns_or_unknowns: <remaining list>
|
||||
blocked_on: <none or exact blocker>
|
||||
next_step: phase_6_component_parity_debug | ask_user
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Include strict-load evidence for every stateful converted component.
|
||||
- If a component intentionally loads non-strictly, list the exact missing or
|
||||
unexpected keys and why they are safe.
|
||||
- Include the actual `model_index.json` library token, config filename, and config
|
||||
validation result for every emitted component.
|
||||
- Preserve retry evidence so the requesting component subagent can resume with
|
||||
updated context.
|
||||
- Keep `port_state_file` synchronized with `component_context_updates`.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
|
||||
`PORT_STATUS.md` row.
|
||||
@@ -1,54 +0,0 @@
|
||||
# Conversion Request Contract
|
||||
|
||||
Consumed by `../add-model-07-conversion/SKILL.md` in Phase 5 and during Phase 6
|
||||
conversion retries.
|
||||
|
||||
Initial conversion request:
|
||||
|
||||
```text
|
||||
model_family: <snake_case>
|
||||
source_layout: diffusers | raw_official | monolithic | separate_components | mixed | custom
|
||||
official_weights: <HF repo, local dir, or checkpoint file>
|
||||
hf_revision: <revision | default | none>
|
||||
converted_weights_dir: converted_weights/<model_family>
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
components:
|
||||
- name: <transformer|vae|text_encoder|conditioner|scheduler|...>
|
||||
component_type: <dit|vae|encoder|scheduler|conditioner|upsampler|vocoder|generic>
|
||||
official_definition_files: <paths + symbols>
|
||||
official_instantiation_files: <paths + call sites + args>
|
||||
official_weight_source: <checkpoint file, prefix, subfolder, or passthrough source>
|
||||
official_keys: <path to official key/shape dump>
|
||||
fastvideo_keys: <path to FastVideo prototype key/shape dump>
|
||||
fastvideo_class: <class name>
|
||||
model_index_library: <diffusers|transformers|fastvideo|fastvideo.*>
|
||||
config_filename: <config.json|scheduler_config.json|none>
|
||||
production_loader_strictness: <strict|non_strict_with_allowed_keys|stateless>
|
||||
source_prefix_or_path: <prefix or path>
|
||||
parity_test: <component parity test path>
|
||||
prototype_concerns_or_unknowns: <short list>
|
||||
```
|
||||
|
||||
Retry request from a component skill:
|
||||
|
||||
```text
|
||||
conversion_retry_request:
|
||||
component: <name>
|
||||
parity_test: <path>
|
||||
failing_keys: <official and FastVideo keys, if known>
|
||||
expected_actual_shapes: <expected vs actual shapes, if known>
|
||||
source_prefix_or_path: <prefix/path implicated by the failure>
|
||||
evidence: <strict-load error, first divergent tensor, parity log excerpt>
|
||||
suspected_fix: <rename | split | fuse | skip | component bucket | config | unknown>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Phase 5 conversion requires Phase 4 official/FastVideo key dumps.
|
||||
- Component skills must use the retry request instead of editing conversion
|
||||
scripts or converted weights directly.
|
||||
- Conversion must update `port_state_file` with conversion status, retry history,
|
||||
strict-load status, new issues, and resolved issues.
|
||||
- Conversion must validate emitted config keys through the production config
|
||||
update path and record the config filename expected by each loader.
|
||||
@@ -1,51 +0,0 @@
|
||||
# Escape Hatch Contract
|
||||
|
||||
Canonical pause-and-ask schema for `/add-model` skills. Use this when the next
|
||||
action requires user input instead of autonomous debugging.
|
||||
|
||||
```text
|
||||
escape_hatch:
|
||||
needs_user_input: yes | no
|
||||
decision_type: scope | dependency | auth | cost | destructive | ambiguity | blocker
|
||||
question: <one precise question>
|
||||
recommended_option: <safe recommended choice>
|
||||
options:
|
||||
- <option + consequence>
|
||||
safe_default: <what the agent will do after approval, or none>
|
||||
blocked_until_answered: yes | no
|
||||
state_snapshot:
|
||||
phase: <phase or skill mode>
|
||||
files_changed:
|
||||
- <paths>
|
||||
command_or_test: <last relevant command, or not_run>
|
||||
evidence: <short logs, paths, error text, or blocker ID>
|
||||
```
|
||||
|
||||
Use `needs_user_input=no` when the handoff is green or the next step is already
|
||||
specified by the workflow.
|
||||
|
||||
Ask the user only for decisions the workflow cannot safely choose:
|
||||
|
||||
- product or PR scope changes, including dropping a modality, output head, or
|
||||
variant;
|
||||
- core dependency changes, version pin changes, or installing untrusted/private
|
||||
dependencies;
|
||||
- auth setup for gated repos, using env var names only and never token values;
|
||||
- large downloads, publishing weights, SSIM/reference uploads, or GPU-heavy work
|
||||
where cost/runtime approval is needed;
|
||||
- destructive file/git operations, overwriting existing clones/weights, or
|
||||
deleting staged assets;
|
||||
- ambiguous official sources of truth with incompatible behavior;
|
||||
- accepting a known blocker, loosening parity/quality tolerances, or shipping
|
||||
without required non-skip parity.
|
||||
|
||||
Do not ask for normal recoverable failures:
|
||||
|
||||
- missing imports, missing local paths, skipped tests, failing parity, conversion
|
||||
mapping errors, strict-load failures, format/lint failures, or implementation
|
||||
bugs covered by the skill workflow.
|
||||
|
||||
Before returning `next_step=ask_user`, update
|
||||
`tests/local_tests/<model_family>/PORT_STATUS.md` with the blocker/question ID,
|
||||
include the exact evidence, and provide one recommended option plus at most three
|
||||
alternatives.
|
||||
@@ -1,38 +0,0 @@
|
||||
# Final Handoff Contract
|
||||
|
||||
Completed by `/add-model` before handing work back to the user or opening a PR.
|
||||
|
||||
```text
|
||||
final_handoff:
|
||||
prep_handoff_complete: yes | no
|
||||
conversion_status: not_needed | pass | blocked
|
||||
components:
|
||||
- name: <component>
|
||||
reuse_or_port: reused | ported
|
||||
parity_test: <path>
|
||||
parity_status: non_skip_pass | blocked
|
||||
concerns_or_unknowns: <none or list>
|
||||
pipeline_smoke: pass | blocked | not_run
|
||||
pipeline_parity: pass | blocked | not_run
|
||||
example_status: pass | blocked | not_run
|
||||
quality_regression: added | deferred_with_reason | not_applicable
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
token_values_committed: no
|
||||
runtime_third_party_model_imports: none | listed_with_rationale
|
||||
blockers: <none or list>
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Required before handoff:
|
||||
|
||||
- Every required component, reused or ported, has non-skip local parity PASS.
|
||||
- Pipeline smoke and pipeline parity are non-skip PASS, or a blocker is explicit.
|
||||
- Basic example runs and writes a non-corrupt output.
|
||||
- `local_tests_readme` lists every component parity command/status/blocker.
|
||||
- `port_state_file` has no unresolved blocker that is omitted from the final
|
||||
response or PR notes.
|
||||
- No raw HF token values, credentials, `.env`, reference clone, or staged weight
|
||||
blobs are committed.
|
||||
- If final handoff is blocked on user input, include an `escape_hatch` block and
|
||||
matching `PORT_STATUS.md` row.
|
||||
@@ -1,44 +0,0 @@
|
||||
# Parity Status Contract
|
||||
|
||||
Returned by `../add-model-02-parity/SKILL.md` to `/add-model` and later updated by
|
||||
component parity-debug subagents.
|
||||
|
||||
```text
|
||||
component_parity:
|
||||
- component: <name>
|
||||
test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py
|
||||
status: scaffold_skip | debug_red | non_skip_pass | blocked
|
||||
missing: <none | fastvideo_class | converted_weights | official_import | ...>
|
||||
coverage_scope: production_loader | implementation_subcomponent | both
|
||||
official_definition_files: <paths>
|
||||
official_instantiation_files: <paths>
|
||||
concerns_or_unknowns: <short list>
|
||||
pipeline_parity:
|
||||
test: <path or not-created>
|
||||
status: not_started | scaffold_skip | debug_red | non_skip_pass | blocked
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
notes: <short list>
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Status meanings:
|
||||
|
||||
- `scaffold_skip`: test is present but skips for a specific missing dependency,
|
||||
FastVideo class, or weights.
|
||||
- `debug_red`: both sides load and the test fails numerically.
|
||||
- `non_skip_pass`: required before final handoff for every required component,
|
||||
including reused components.
|
||||
- `blocked`: a precise missing dependency, weight, official call path, or
|
||||
component/conversion regression prevents local activation.
|
||||
|
||||
Coverage meanings:
|
||||
|
||||
- `production_loader`: FastVideo side loads through the same loader/path used by
|
||||
a pipeline.
|
||||
- `implementation_subcomponent`: FastVideo side constructs classes or remaps
|
||||
tensors directly to isolate implementation behavior.
|
||||
- `both`: the test covers both production loading and implementation behavior.
|
||||
|
||||
Use `escape_hatch` only when blocked status requires a user decision. Normal
|
||||
skips or red parity should be debugged by the workflow without asking.
|
||||
@@ -1,82 +0,0 @@
|
||||
# Pipeline Context Contract
|
||||
|
||||
Canonical packet passed from `/add-model` to `add-model-09-pipeline` for pipeline
|
||||
definition and pipeline parity-debug work.
|
||||
|
||||
```text
|
||||
pipeline_context:
|
||||
model_family: <snake_case>
|
||||
mode: pipeline-definition | pipeline-parity-debug
|
||||
workload_types:
|
||||
- <T2V|I2V|V2V|T2I|compatibility-shim-with-rationale>
|
||||
modalities:
|
||||
inputs: <text/image/video/audio/pose/depth/mask/etc.>
|
||||
outputs: <video/image/audio/joint-av/latents/etc.>
|
||||
official_ref_dir: <path or import path>
|
||||
official_pipeline_files:
|
||||
- path: <repo-relative or absolute path in official repo>
|
||||
symbols: <pipeline/factory/sample functions>
|
||||
notes: <stage order, mutable state, output contract>
|
||||
official_call:
|
||||
command_or_api: <official CLI, Python call, or package entrypoint>
|
||||
args_and_defaults: <height, width, frames, fps, duration, steps, CFG, scheduler, seeds, etc.>
|
||||
preset_source: <model card, config file, official script, or unknown>
|
||||
scheduler_and_rng: <timestep/sigma/noise/generator behavior>
|
||||
output_contract: <decoded media, denoised latents, waveform, dict keys, etc.>
|
||||
model_index:
|
||||
class_name: <FastVideo pipeline class name to emit in model_index.json>
|
||||
entry_class_names: <registered EntryClass.__name__ values that must include class_name>
|
||||
required_modules: <text_encoder, tokenizer, vae, transformer, scheduler, etc.>
|
||||
passthrough_modules: <tokenizer, scheduler, processor, external HF dirs, or none>
|
||||
sampling_param:
|
||||
new_fields: <none or list of public kwargs/preset defaults to add to SamplingParam>
|
||||
cli_fields: <none or list of fields that need CLI args>
|
||||
placeholder_fields: <none or video-shaped compatibility placeholders with rationale>
|
||||
components:
|
||||
- name: <component>
|
||||
component_type: <dit|vae|encoder|scheduler|conditioner|upsampler|vocoder|generic>
|
||||
parity_test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py
|
||||
parity_status: non_skip_pass
|
||||
fastvideo_target_files: <model/config/export files>
|
||||
converted_component_dir: converted_weights/<family>/<component>
|
||||
conversion:
|
||||
converted_weights_dir: converted_weights/<family>
|
||||
source_layout: <diffusers|raw_official|monolithic|separate_components|mixed|custom>
|
||||
model_index_path: converted_weights/<family>/model_index.json
|
||||
fastvideo_targets:
|
||||
pipeline_files:
|
||||
- fastvideo/pipelines/basic/<family>/<family>_pipeline.py
|
||||
stage_files:
|
||||
- fastvideo/pipelines/basic/<family>/stages/<stage>.py
|
||||
pipeline_config_files:
|
||||
- fastvideo/configs/pipelines/<family>.py
|
||||
preset_file: fastvideo/pipelines/basic/<family>/presets.py
|
||||
registry_file: fastvideo/registry.py
|
||||
example_files:
|
||||
- examples/inference/basic/basic_<family>.py
|
||||
smoke_test: tests/local_tests/pipelines/test_<family>_pipeline_smoke.py
|
||||
parity_test: tests/local_tests/pipelines/test_<family>_pipeline_parity.py
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
concerns_or_unknowns:
|
||||
- <pipeline branch, unsupported workload, output head, preset ambiguity, etc.>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Start only after every required component, reused or ported, has
|
||||
`parity_status=non_skip_pass`. If any row is missing or skipped, return to
|
||||
`/add-model` Phase 6.
|
||||
- Record official call arguments and default sources before writing FastVideo
|
||||
presets. Do not invent inference defaults from memory.
|
||||
- `model_index.class_name` must match a registered pipeline `EntryClass.__name__`;
|
||||
registry detectors are not sufficient for executable pipeline resolution.
|
||||
- Every public generation kwarg or preset default must be represented in
|
||||
`SamplingParam`, or documented as an intentional internal-only field.
|
||||
- `T2A`, `A2A`, and `AV` may be used only after `WorkloadType` supports them;
|
||||
otherwise record the compatibility shim and rationale explicitly.
|
||||
- Keep token values out of the packet. Use only token environment variable names.
|
||||
- If a target path is unknown, use `unknown` plus the exact search already
|
||||
performed.
|
||||
- Update `local_tests_readme` and `port_state_file` whenever pipeline smoke,
|
||||
parity, presets, registry, examples, or blockers change.
|
||||
@@ -1,71 +0,0 @@
|
||||
# Pipeline Handoff Contract
|
||||
|
||||
Returned by `add-model-09-pipeline` to `/add-model` after pipeline definition or
|
||||
pipeline parity-debug work.
|
||||
|
||||
```text
|
||||
pipeline_handoff:
|
||||
model_family: <snake_case>
|
||||
mode: pipeline-definition | pipeline-parity-debug
|
||||
files_changed:
|
||||
- <pipeline/config/preset/registry/stage/example/test/readme/status paths>
|
||||
official_files_used:
|
||||
- <definition/call/default source paths>
|
||||
required_config_modules:
|
||||
emitted: <list from pipeline class>
|
||||
model_index: <list from converted or source model_index.json>
|
||||
status: match | mismatch | blocked
|
||||
pipeline_class_resolution:
|
||||
model_index_class_name: <_class_name>
|
||||
entry_class_names: <registered EntryClass.__name__ values>
|
||||
status: exact_match | alias_added | blocked
|
||||
sampling_param:
|
||||
fields_added: <none or list>
|
||||
cli_fields_added: <none or list>
|
||||
unknown_kwargs_checked: yes | no | blocked
|
||||
stage_chain:
|
||||
- <stage names in execution order>
|
||||
pipeline_config:
|
||||
file: <path>
|
||||
classes: <class names>
|
||||
official_defaults_checked: yes | no | blocked
|
||||
presets:
|
||||
file: <path>
|
||||
names: <preset names>
|
||||
status: pass | blocked | not_run
|
||||
registry:
|
||||
status: pass | blocked | not_run
|
||||
detectors: <HF paths and model_index _class_name strings covered>
|
||||
smoke_test:
|
||||
path: tests/local_tests/pipelines/test_<family>_pipeline_smoke.py
|
||||
status: non_skip_pass | blocked | not_run
|
||||
pytest_output: <command + short result>
|
||||
pipeline_parity:
|
||||
path: tests/local_tests/pipelines/test_<family>_pipeline_parity.py
|
||||
status: scaffold_skip | debug_red | non_skip_pass | blocked
|
||||
pytest_output: <command + short result>
|
||||
comparison_target: <latents|decoded video|audio|joint outputs>
|
||||
example:
|
||||
path: examples/inference/basic/basic_<family>.py
|
||||
status: pass | blocked | not_run
|
||||
output: <path or none>
|
||||
readme_updated: yes | no
|
||||
port_state_updated: yes | no
|
||||
blockers: <none or exact blocker list>
|
||||
next_step: phase_10_quality_regression | return_to_phase_6 | ask_user
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `pipeline-definition` may return with parity still `scaffold_skip` only if the
|
||||
exact missing dependency, weight, or call-path blocker is recorded.
|
||||
- Final `/add-model` handoff requires `smoke_test.status=non_skip_pass` and
|
||||
`pipeline_parity.status=non_skip_pass`, unless the user explicitly accepts a
|
||||
documented blocker.
|
||||
- If parity failure traces to a component, conversion, or strict-load issue,
|
||||
return `next_step=return_to_phase_6` and include the exact failing evidence.
|
||||
- Keep `local_tests_readme` and `port_state_file` synchronized with this
|
||||
handoff before returning.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
|
||||
`PORT_STATUS.md` row.
|
||||
@@ -1,89 +0,0 @@
|
||||
# Port State Contract
|
||||
|
||||
Canonical per-port state file created during prep and updated by every
|
||||
`/add-model` phase.
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
```
|
||||
|
||||
Purpose:
|
||||
|
||||
- Single source of truth for resumable port progress.
|
||||
- Tracks component status, conversion status, parity status, open questions,
|
||||
blockers, escape hatches, and issue history.
|
||||
- Lets review agents run the same setup/tests without reconstructing handoffs
|
||||
from conversation history.
|
||||
|
||||
Required sections:
|
||||
|
||||
```text
|
||||
# <Model Family> Port Status
|
||||
|
||||
## Summary
|
||||
- model_family:
|
||||
- workload_types:
|
||||
- official_ref:
|
||||
- official_ref_dir:
|
||||
- hf_weights_path:
|
||||
- local_weights_dir:
|
||||
- source_layout:
|
||||
- local_tests_readme:
|
||||
|
||||
## Current Phase
|
||||
- phase:
|
||||
- status: not_started | in_progress | blocked | complete
|
||||
- owner: orchestrator | prep | parity | conversion | component:<name> | pipeline
|
||||
- last_updated:
|
||||
|
||||
## Component Matrix
|
||||
| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
|
||||
## Conversion State
|
||||
- conversion_script:
|
||||
- converted_weights_dir:
|
||||
- source_layout:
|
||||
- strict_load_status:
|
||||
- passthrough_components:
|
||||
- retry_history:
|
||||
|
||||
## Parity Commands
|
||||
| Scope | Command | Last Result | Notes |
|
||||
|---|---|---|---|
|
||||
|
||||
## Open Questions
|
||||
| ID | Question | Owner | Needed By Phase | Status | Resolution |
|
||||
|---|---|---|---|---|---|
|
||||
|
||||
## Issues And Blockers
|
||||
| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
|
||||
## Escape Hatches
|
||||
| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution |
|
||||
|---|---|---|---|---|---|---|
|
||||
|
||||
## Decisions
|
||||
| Date | Decision | Rationale | Impact |
|
||||
|---|---|---|---|
|
||||
|
||||
## Handoff Notes
|
||||
- <short notes for the next agent>
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Update this file whenever a phase starts, blocks, resolves an issue, or hands
|
||||
off to another skill.
|
||||
- Record open questions and issues immediately. Do not leave blockers only in
|
||||
chat history or subagent responses.
|
||||
- Use stable IDs: `Q001`, `Q002`, `I001`, `I002`, etc.
|
||||
- Use stable escape-hatch IDs: `E001`, `E002`, etc. Link them from handoff
|
||||
`escape_hatch.state_snapshot.evidence` when returning `next_step=ask_user`.
|
||||
- Do not include raw token values, machine-local cache internals, or large output
|
||||
dumps. Use repo-relative paths when possible.
|
||||
- If a question or issue is resolved, keep the row and fill `Resolution` instead
|
||||
of deleting it.
|
||||
@@ -1,42 +0,0 @@
|
||||
# Prep Handoff Contract
|
||||
|
||||
Produced by `../add-model-01-prep/SKILL.md` and consumed by `/add-model` Phase 0.
|
||||
|
||||
```text
|
||||
model_family: <snake_case>
|
||||
workload_types: <T2V/I2V/V2V/T2I/or compatibility shim with rationale>
|
||||
official_ref: <url or import path>
|
||||
official_ref_dir: <ReferenceDir or none>
|
||||
official_ref_commit: <sha or unknown>
|
||||
hf_weights_path: <HF id or local path>
|
||||
hf_revision: <revision or default>
|
||||
local_weights_dir: official_weights/<model_family> or <local path>
|
||||
source_layout: diffusers | raw_official | monolithic | separate_components | mixed | custom | unknown
|
||||
model_index_class: <_class_name or none>
|
||||
components_seen: <components>
|
||||
needs_conversion: yes | no | unknown
|
||||
hf_token_env: <env var name only>
|
||||
dependency_changes: none | installed no-deps editable | installed official deps in current env | blocked on user
|
||||
official_env_status: imports_ok | private_deps_need_stubs | blocked
|
||||
local_tests_readme: tests/local_tests/<model_family>/README.md
|
||||
port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
|
||||
gitignore_entries_added: <list>
|
||||
next_step: add-model | ask_user
|
||||
open_questions: <short list>
|
||||
escape_hatch: <none or block matching contracts/escape_hatch.md>
|
||||
```
|
||||
|
||||
Validation:
|
||||
|
||||
- `official_env_status` must be `imports_ok` or `private_deps_need_stubs` before
|
||||
component parity scaffolding.
|
||||
- `local_tests_readme` must exist and describe official setup, HF weights,
|
||||
dependency changes, planned parity commands, and review notes.
|
||||
- `port_state_file` must exist and follow `contracts/port_state.md`.
|
||||
- Prep does not go directly to conversion; `/add-model` must run component
|
||||
prototype/key-dump Phase 4 before Phase 5 conversion.
|
||||
- `T2A`, `A2A`, and `AV` may be used only after `WorkloadType` supports them;
|
||||
otherwise record the compatibility shim and rationale explicitly.
|
||||
- Never include HF token values.
|
||||
- Use `next_step=ask_user` only with an `escape_hatch` block and a matching
|
||||
`PORT_STATUS.md` row.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Shared Add-Model Rules
|
||||
|
||||
These rules apply to every `add-model` related skill: prep, parity, conversion,
|
||||
component porting, pipeline, and the main `/add-model` orchestrator.
|
||||
|
||||
## Token And Auth Safety
|
||||
|
||||
- Never accept, print, echo, log, hard-code, or commit raw HF token values.
|
||||
- Refer only to token environment variable names: `HF_TOKEN`,
|
||||
`HUGGINGFACE_HUB_TOKEN`, or `HF_API_KEY`.
|
||||
- Scripts may read those environment variables but must not print their values.
|
||||
- Ask for auth setup only by env var name. Do not ask the user to paste a token.
|
||||
- Read scope is needed for gated repos during conversion/load. Write scope is
|
||||
needed for publishing converted weights or seeding generated references.
|
||||
|
||||
## Shared State Files
|
||||
|
||||
- `tests/local_tests/<model_family>/README.md` is the reviewer-facing setup and
|
||||
verification log. Keep it current with setup commands, dependency blockers,
|
||||
parity commands, conversion commands, and pass/blocker status.
|
||||
- `tests/local_tests/<model_family>/PORT_STATUS.md` is the per-port state file.
|
||||
It must follow `../contracts/port_state.md` and keep stable `Q###`, `I###`, and
|
||||
`E###` IDs.
|
||||
- Keep resolved questions/issues in `PORT_STATUS.md` with the resolution instead
|
||||
of deleting them.
|
||||
- Before returning a handoff, update both state files when the skill changed
|
||||
setup, tests, conversion, parity status, blockers, or decisions.
|
||||
- Do not include raw tokens, non-reproducible absolute cache paths, large
|
||||
generated outputs, `.env`, credentials, or anything matching `*secret*`.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Continue autonomously for recoverable setup, implementation, conversion,
|
||||
strict-load, smoke, parity-debug, lint, or test failures. Stop and ask the user
|
||||
only when the next action requires a product, cost, safety, auth, dependency, or
|
||||
scope decision the workflow cannot safely choose.
|
||||
|
||||
Use `../contracts/escape_hatch.md` whenever returning `next_step=ask_user`.
|
||||
|
||||
Ask for user input only for:
|
||||
|
||||
- scope changes, such as dropping a modality, output head, variant, component, or
|
||||
public mode;
|
||||
- core dependency changes, untrusted/private dependency installs, or version pin
|
||||
changes;
|
||||
- auth setup for gated repos, using env var names only;
|
||||
- large downloads, publishing weights, SSIM/reference uploads, or GPU-heavy work
|
||||
where cost/runtime approval is needed;
|
||||
- destructive file/git operations, overwriting existing clones/weights, or
|
||||
deleting staged assets;
|
||||
- incompatible official sources of truth where no reference can be chosen from
|
||||
published weights and docs;
|
||||
- accepting a blocker, loosening tolerances, using shape-only substitutes, or
|
||||
shipping without required non-skip parity.
|
||||
|
||||
Do not ask for normal recoverable failures: missing imports, missing local paths,
|
||||
skipped tests, failing parity, conversion mapping bugs, strict-load errors,
|
||||
format/lint failures, smoke failures, registry import issues, example failures,
|
||||
or implementation bugs covered by the phase workflow.
|
||||
|
||||
Before asking:
|
||||
|
||||
- update `PORT_STATUS.md` with an `E###` escape-hatch row plus any linked `Q###`
|
||||
or `I###` row;
|
||||
- include exact evidence: command, path, short error text, parity or strict-load
|
||||
excerpt, or blocker ID;
|
||||
- provide one recommended option and at most three alternatives;
|
||||
- set the relevant handoff `next_step=ask_user` and include the `escape_hatch`
|
||||
block.
|
||||
|
||||
Skill-specific escape-hatch sections may add extra examples, but they must not
|
||||
weaken these shared rules.
|
||||
|
||||
## Production Boundary
|
||||
|
||||
- No runtime `from diffusers import <model class>` or
|
||||
`from transformers import <model class>` in `fastvideo/` production code.
|
||||
- Components that own weights or numerical behavior must be FastVideo-native
|
||||
unless the user explicitly accepts a documented lazy-wrapper exception.
|
||||
- Allowed third-party runtime exceptions are tokenizers and pure data utilities
|
||||
when they match existing project patterns.
|
||||
- Tests may import diffusers/transformers as parity references.
|
||||
- Production comments explain why, not what or provenance. Avoid narrative
|
||||
comments like `vendored from`, `matches upstream`, `REVIEW`, or session-history
|
||||
commentary.
|
||||
|
||||
## Verification Semantics
|
||||
|
||||
- A committed local test may skip when clones, weights, or private deps are absent
|
||||
so CI and other contributors are not blocked.
|
||||
- On the porter's machine, a skip is not a pass. Fix the missing import, weights,
|
||||
or path before claiming verification.
|
||||
- New ports require local non-skip parity for required components and pipeline
|
||||
parity when a pipeline is in scope.
|
||||
- Smoke tests prove loadability only. They are not a substitute for numerical
|
||||
component or pipeline parity.
|
||||
@@ -1,117 +0,0 @@
|
||||
# Component Skill Common Instructions
|
||||
|
||||
These instructions apply to `add-model-03-port-dit`, `add-model-04-port-vae`,
|
||||
`add-model-05-port-encoder`, and `add-model-06-port-generic`. Bucket-specific skills
|
||||
add target paths, implementation patterns, drift checks, and scope questions.
|
||||
|
||||
## Required Context
|
||||
|
||||
Require the complete packet from `../contracts/component_context.md`.
|
||||
|
||||
Do not start if the official definition files, official instantiation files, or
|
||||
parity test path are missing. Ask the `/add-model` orchestrator for the complete
|
||||
component context packet instead of rediscovering broad scope silently.
|
||||
|
||||
If the parity scaffold is missing, create it first with
|
||||
`../../add-model-02-parity/templates/component_parity_test.py`.
|
||||
|
||||
## Prototype Mode
|
||||
|
||||
Prototype mode runs before conversion:
|
||||
|
||||
- implement or prove reuse for the minimal native component, config, export, and
|
||||
`EntryClass` surface needed by the relevant loader;
|
||||
- instantiate with random weights using the exact official architecture args, or
|
||||
instantiate/document stateless components with no weights;
|
||||
- dump official and FastVideo `state_dict()` names/shapes for every stateful
|
||||
component so conversion can derive mappings from real surfaces;
|
||||
- return concerns discovered during prototype work, such as ambiguous official
|
||||
flags, shape mismatches, private ops, missing loader buckets, passthrough
|
||||
weights, or output heads;
|
||||
- update `local_tests_readme` with prototype status and key-dump paths;
|
||||
- update `port_state_file` with prototype status, open questions, issues, and
|
||||
handoff notes;
|
||||
- do not chase numerical parity and do not block on converted weights.
|
||||
|
||||
Prototype mode succeeds when the component imports, instantiates with official
|
||||
args, and required key/shape dumps exist. Converted weights and parity PASS are
|
||||
not required yet.
|
||||
|
||||
## Parity-Debug Mode
|
||||
|
||||
Parity-debug mode runs after conversion:
|
||||
|
||||
- strict-load converted weights through the same path the pipeline will use, or
|
||||
document that the component is stateless or an approved passthrough;
|
||||
- use `conversion_context` and `concerns_or_unknowns` to decide whether a failure
|
||||
belongs to mapping, loading, implementation, tokenization, normalization,
|
||||
scheduler semantics, or the parity test;
|
||||
- run only the component parity test first with `pytest <parity_test> -v -s`;
|
||||
- if it skips, fix the missing official import, FastVideo class, tokenizer,
|
||||
converted weights, or path;
|
||||
- if it fails numerically, add targeted intermediate comparisons to identify the
|
||||
first divergent operation or tensor;
|
||||
- update component implementation only when the failure is a component
|
||||
layer/config/forward/contract bug;
|
||||
- update `local_tests_readme` with the command, result, and blocker or PASS;
|
||||
- update `port_state_file` with parity status, resolved/new issues, open
|
||||
questions, and handoff notes;
|
||||
- keep iterating until the test is a non-skip PASS or return a precise blocker.
|
||||
|
||||
## Conversion Boundary
|
||||
|
||||
Component skills must not patch conversion scripts or converted weights ad hoc.
|
||||
If the first drift or strict-load failure points to wrong keys, missing tensors,
|
||||
shape mismatches, component prefixes, split/fuse logic, skipped-key policy, or
|
||||
config emission, return a conversion retry request for
|
||||
`../../add-model-07-conversion/SKILL.md` matching
|
||||
`../contracts/conversion_request.md`.
|
||||
Resume parity-debug only after conversion returns an updated handoff.
|
||||
|
||||
## Reuse Proof
|
||||
|
||||
When `fastvideo_target_files` point to existing FastVideo code instead of a new
|
||||
port:
|
||||
|
||||
- compare the official definition against the FastVideo target: graph/operation
|
||||
structure, parameter or state shapes, normalization, activation, positional or
|
||||
temporal behavior, scaling constants, dtype behavior, state-dict names, output
|
||||
containers, and output tensors;
|
||||
- compare the official instantiation against the FastVideo config and loader
|
||||
args: constructor args, config values, defaults, variant flags, optional
|
||||
submodules, checkpoint metadata, tokenizer/media paths, and loader path;
|
||||
- treat a matching class instantiated with different args as not reusable;
|
||||
- record reuse evidence in `local_tests_readme` and keep the reused component in
|
||||
`prototype_key_dumps` when it owns state so conversion and parity-debug use the
|
||||
same surface;
|
||||
- still run parity-debug to a non-skip PASS. If mismatch is found, return the
|
||||
concern so `/add-model` can switch the component to a native port.
|
||||
|
||||
## Handoff
|
||||
|
||||
Return `../contracts/component_skill_handoff.md`.
|
||||
|
||||
Mode-specific expectations:
|
||||
|
||||
- In `mode=prototype`, `parity_status` may be `scaffold_skip` or `blocked`; key
|
||||
dumps are the required artifact and successful prototype handoff should use
|
||||
`next_step=phase_5_conversion`.
|
||||
- In `mode=parity-debug`, final success requires
|
||||
`parity_status=non_skip_pass`.
|
||||
- If conversion is implicated, return `conversion_retry_request` and leave
|
||||
conversion edits to `add-model-07-conversion`.
|
||||
- If production loading is non-strict, list allowed missing/unexpected keys in
|
||||
the parity test or handoff and mark
|
||||
`strict_load=pass_with_documented_exclusions`.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
Follow `common_rules.md`. Do not ask for normal prototype or parity-debug
|
||||
failures such as missing imports, tokenizer/path issues, red parity,
|
||||
strict-load failures, key mismatches, shape mismatches, or implementation bugs.
|
||||
Return conversion retry requests or precise blockers as directed by the workflow.
|
||||
|
||||
Ask only when component work requires a scope or safety decision, such as
|
||||
dropping a required stream/output/path, changing core dependencies, accepting
|
||||
private model code or unsupported private ops, choosing between incompatible
|
||||
official definitions, creating a new loader bucket, or loosening required parity.
|
||||
@@ -1,338 +0,0 @@
|
||||
---
|
||||
name: decompose-pipeline-pr
|
||||
description: Decompose an oversized FastVideo pipeline PR into a stack of independently-reviewable PRs. Tiers the diff by blast radius (invisible / dead code / cross-cutting infra / activation), produces a branch graph and worktree bootstrap, drafts the AGENTS.md manifest, flags missing tests on cross-cutting infra changes, and extracts lessons from the PR body.
|
||||
---
|
||||
|
||||
# Decompose Pipeline PR
|
||||
|
||||
## Purpose
|
||||
|
||||
When a PR adds a new pipeline (or first-class component port) and crosses
|
||||
~3,000 LOC, single-shot review converges to rubber-stamping. This skill
|
||||
decomposes such a PR into a stack of independently-reviewable PRs without
|
||||
disturbing `main`.
|
||||
|
||||
It is the inverse of `add-model`: where `add-model` walks adding a new
|
||||
pipeline as a fresh PR, this skill walks decomposing an existing oversized
|
||||
pipeline PR.
|
||||
|
||||
**Worked example:** PR #1280 (daVinci-MagiHuman, 9,812 LOC, 56 files) →
|
||||
2 prerequisite PRs off main + 8-PR stack:
|
||||
- #1293 `will/activation-trace` (prerequisite)
|
||||
- #1294 `will/loader-infra` (prerequisite)
|
||||
- #1295 (1/8) housekeeping
|
||||
- #1296 (2/8) t5gemma encoder
|
||||
- #1297 (3/8) DiT
|
||||
- #1298 (4/8) pipeline stages
|
||||
- #1299 (5/8) pipeline orchestrator
|
||||
- #1300 (6/8) provenance (AGENTS.md, JOURNAL.md, lessons)
|
||||
- #1301 (7/8) conversion scripts
|
||||
- #1302 (8/8) registry activation
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Open PR number on `hao-ai-lab/FastVideo` (or any FastVideo fork)
|
||||
- `gh` CLI authenticated against the target remote
|
||||
- Local git worktree support (`git worktree`)
|
||||
- Git config `user.name` / `user.email` set
|
||||
- Pre-commit installed (`pre-commit install --hook-type pre-commit --hook-type commit-msg`)
|
||||
- The target PR's branch fetched locally as `origin/<feature-branch>`
|
||||
|
||||
## Inputs
|
||||
|
||||
| Parameter | Required | Description |
|
||||
|-----------|----------|-------------|
|
||||
| PR number or URL | Yes | E.g. `1280` or `https://github.com/hao-ai-lab/FastVideo/pull/1280` |
|
||||
| Max desired PR size | No | Defaults to ~2,500 LOC of code per stack PR (excluding generated/journal files) |
|
||||
| Output directory | No | Defaults to `.agents/tmp/decompose-<pr-number>/` (gitignored) |
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Verify ground truth (do not trust `gh pr diff --name-only`)
|
||||
|
||||
`gh pr diff <N> --name-only` has been observed to emit phantom file entries.
|
||||
Always cross-check against the authoritative `git diff`:
|
||||
|
||||
```bash
|
||||
mkdir -p .agents/tmp/decompose-<N>
|
||||
git fetch origin pull/<N>/head:<feature-branch>
|
||||
git diff origin/main..origin/<feature-branch> --name-status \
|
||||
> .agents/tmp/decompose-<N>/files.txt
|
||||
git diff origin/main..origin/<feature-branch> --stat
|
||||
```
|
||||
|
||||
Use the `--name-status` output as the authoritative file list. If it
|
||||
disagrees with `gh pr diff --name-only`, trust the git diff.
|
||||
|
||||
### 2. Tier the diff by blast radius
|
||||
|
||||
Classify every changed file into one of four tiers:
|
||||
|
||||
| Tier | Description | Examples |
|
||||
|---|---|---|
|
||||
| **Tier 0 — Invisible** | Lint/style/CI configs that don't affect runtime | `.gitignore`, `pyproject.toml` (codespell only), agent documentation |
|
||||
| **Tier 1 — Dead code** | New files in their own dirs; aggregator one-liners | `fastvideo/models/dits/<new>/`, `fastvideo/pipelines/basic/<new>/`, `examples/inference/basic/basic_<new>*.py`, `tests/local_tests/<new>/`, `__init__.py` exports |
|
||||
| **Tier 2 — Cross-cutting infra** | Modifications to files used by every pipeline | See protected-paths list below |
|
||||
| **Tier 3 — Activation switch** | `register_configs(...)` calls + the example scripts that demo them | `fastvideo/registry.py` |
|
||||
|
||||
**FastVideo Tier 2 protected paths:**
|
||||
```
|
||||
fastvideo/utils.py
|
||||
fastvideo/pipelines/composed_pipeline_base.py
|
||||
fastvideo/models/loader/component_loader.py
|
||||
fastvideo/configs/models/dits/__init__.py
|
||||
fastvideo/configs/models/encoders/__init__.py
|
||||
fastvideo/configs/models/vaes/__init__.py
|
||||
fastvideo/envs.py
|
||||
fastvideo/fastvideo_args.py
|
||||
fastvideo/distributed/**
|
||||
fastvideo/layers/**
|
||||
fastvideo/attention/**
|
||||
fastvideo/registry.py # treat as Tier 3 if change is the activation
|
||||
```
|
||||
|
||||
Tier 3 detection (mechanical):
|
||||
```bash
|
||||
git diff origin/main..origin/<feature-branch> -- fastvideo/registry.py | \
|
||||
grep -E "^\+.*register_configs\("
|
||||
```
|
||||
|
||||
If `registry.py` only contains `register_configs` additions, treat it as
|
||||
Tier 3. If it modifies existing behavior, treat it as Tier 2 (rare).
|
||||
|
||||
### 3. Identify reusable Tier-1 components
|
||||
|
||||
Within Tier 1, look for sub-trees that are **not** model-specific and could
|
||||
land separately:
|
||||
|
||||
- Encoders matching a known multi-model base (T5/T5-Gemma/Llama/Gemma/CLIP variants)
|
||||
- New stage classes that subclass shared bases without referencing the new model
|
||||
- Hook/profiler/debug infra under `fastvideo/hooks/`
|
||||
- New helpers that have no model-specific dependencies
|
||||
|
||||
These get split into their own PRs (e.g. PR 4 `t5gemma-encoder` in the
|
||||
MagiHuman example).
|
||||
|
||||
### 4. Hunt for missing test coverage on Tier 2 changes
|
||||
|
||||
For every Tier-2 file modified, check whether the original PR added unit
|
||||
tests for the new behavior:
|
||||
|
||||
```bash
|
||||
for f in <list-of-tier-2-files>; do
|
||||
echo "=== Tests for $f ==="
|
||||
git diff origin/main..origin/<feature-branch> -- \
|
||||
"$(echo $f | sed 's|fastvideo/|fastvideo/tests/|; s|\.py|*|')"
|
||||
done
|
||||
```
|
||||
|
||||
If a Tier-2 PR has no accompanying tests, **emit a "must-add tests" list**
|
||||
with a sketch of the case grid. Tier-2 PRs do not ship without those tests.
|
||||
|
||||
The MagiHuman example required this for PR-B (`utils.py`): the original PR
|
||||
shipped no `test_utils_loader.py`, so the decomposition added 9 unit-test
|
||||
cases covering the umbrella-detector boundary, the optional-component-dirs
|
||||
relaxation, and regression coverage on every existing 2-segment HF id.
|
||||
|
||||
### 5. Build the dependency DAG and topo-sort
|
||||
|
||||
Edges:
|
||||
- Tier 2 infra → Tier 1 code that imports it
|
||||
- Reusable Tier 1 components → model-specific Tier 1 code that uses them
|
||||
(encoder before DiT before pipeline)
|
||||
- Tier 1 → Tier 3 (activation always last)
|
||||
- Tier 0 has no dependents (lands first as a freebie)
|
||||
|
||||
Topo-sort produces the stack ordering. Pull Tier-2 PRs **out of the stack**
|
||||
when they have no model-specific dependency — they should land off main
|
||||
with their own focused review, not buried in a model port.
|
||||
|
||||
Render as a tree (markdown):
|
||||
|
||||
```
|
||||
main
|
||||
├─ <prereq-A>
|
||||
│ └─ <prereq-B>
|
||||
│ ├─ <stack-01-housekeeping>
|
||||
│ │ └─ <stack-02-encoder>
|
||||
│ │ └─ <stack-03-dit>
|
||||
│ │ └─ ...
|
||||
│ │ └─ <stack-N-activate>
|
||||
│ └─ (parallel) <skill-pr> off main
|
||||
```
|
||||
|
||||
### 6. Detect mis-shelved docs and debug scratch
|
||||
|
||||
Two categories to flag:
|
||||
|
||||
- **Mis-shelved docs**: Markdown files under `tests/local_tests/` are
|
||||
journals, not tests. Flag for relocation to the package dir as
|
||||
`JOURNAL.md`.
|
||||
- **Debug scratch**: files starting with `_debug_`, `_scratch_`, or
|
||||
`_explore_`. Flag for drop (do not carry into any output PR).
|
||||
|
||||
For MagiHuman: `tests/local_tests/magi-human.md` → relocate. Two
|
||||
`_debug_magi_human_*.py` files → drop.
|
||||
|
||||
### 7. Author the AGENTS.md manifest skeleton
|
||||
|
||||
For the new pipeline package, generate a 6-section `AGENTS.md` scaffold
|
||||
with the file table pre-populated from the diff:
|
||||
|
||||
1. **Manifest** — file table by role
|
||||
2. **Parity invariants** — load-bearing rules with one-paragraph each + lesson refs
|
||||
3. **Cross-refs** — "If you change X, re-run Y" matrix
|
||||
4. **Run book** — single pytest command + prereqs (HF tokens, GPU, wall-time)
|
||||
5. **Open questions** — known issues (e.g. tolerance carve-outs)
|
||||
6. **Provenance** — PR table with branch names and source SHA
|
||||
|
||||
The provenance section is filled incrementally during stack execution and
|
||||
finalized in the activation PR.
|
||||
|
||||
### 8. Extract lessons from the PR body
|
||||
|
||||
Scan the PR body for sections titled "Key implementation work", "Bug hunt",
|
||||
"Lessons", or sentences with patterns like "took N waves to localize",
|
||||
"silent regression", "investigation revealed". Each becomes a candidate
|
||||
`.agents/lessons/<YYYY-MM-DD>_<slug>.md` draft.
|
||||
|
||||
Lessons MUST follow the existing template in
|
||||
`.agents/lessons/README.md`:
|
||||
- YAML frontmatter: `date`, `experiment`, `category`, `severity`
|
||||
- Sections: What Happened, Root Cause, Fix / Workaround, Prevention
|
||||
- Filename: `<YYYY-MM-DD>_<short-slug>.md`
|
||||
|
||||
Lessons co-locate with the code they concern: a conversion-script lesson
|
||||
lands in the same PR as the conversion script, not in the docs PR.
|
||||
|
||||
### 9. Emit the commit-footer convention
|
||||
|
||||
Every commit in the stack ends with:
|
||||
|
||||
```
|
||||
<Feature>-Stack: N/M
|
||||
```
|
||||
|
||||
E.g. `Magi-Stack: 5/8`. Use the package directory name as the feature key.
|
||||
After all PRs squash-merge, `git log --grep='^<Feature>-Stack:'` reconstructs
|
||||
the lineage even if PR numbers later get renumbered.
|
||||
|
||||
### 10. Produce the worktree bootstrap
|
||||
|
||||
Generate a runnable bash script:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
REPO=/home/<user>/FastVideo
|
||||
WORKTREE=/home/<user>/FastVideoMagi # NB: directory name must be a valid
|
||||
# Python identifier (no hyphens) so
|
||||
# mypy doesn't choke
|
||||
SOURCE_PR=<N>
|
||||
SOURCE_BRANCH=will/<feature>
|
||||
SOURCE_SHA=$(git -C "$REPO" rev-parse "origin/$SOURCE_BRANCH")
|
||||
|
||||
git -C "$REPO" fetch origin main:main
|
||||
git -C "$REPO" fetch "origin/$SOURCE_BRANCH"
|
||||
git -C "$REPO" worktree add "$WORKTREE" origin/main
|
||||
|
||||
# Capture baseline for provenance. Everything under .agents/tmp is transient
|
||||
# and ignored by git.
|
||||
OUTPUT_DIR="$REPO/.agents/tmp/decompose-$SOURCE_PR"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
cat > "$OUTPUT_DIR/<feature>-baseline-${SOURCE_SHA:0:8}.txt" <<EOF
|
||||
Source PR: <repo>#$SOURCE_PR
|
||||
Source SHA: $SOURCE_SHA
|
||||
Authoritative file count: $(git -C "$REPO" diff origin/main..origin/$SOURCE_BRANCH --name-only | wc -l)
|
||||
Date captured: $(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
EOF
|
||||
```
|
||||
|
||||
### 11. Author preserve via `git checkout`, not `cherry-pick`
|
||||
|
||||
For each stack PR:
|
||||
|
||||
```bash
|
||||
git -C "$WORKTREE" switch -c <new-branch> <base-branch>
|
||||
git -C "$WORKTREE" checkout origin/<source-branch> -- <file1> <file2> ...
|
||||
git -C "$WORKTREE" commit -m "[<scope>]: <subject>
|
||||
|
||||
<body>
|
||||
|
||||
<Feature>-Stack: N/M"
|
||||
git -C "$WORKTREE" push -u origin <new-branch>
|
||||
gh pr create --base <base-branch> --head <new-branch> --title "..." --body "$(cat <<EOF ... EOF)"
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `git checkout origin/<source> -- <files>` extracts only the named files,
|
||||
preserving the diff. The original PR's author is **not** preserved on the
|
||||
new commit (it's authored by whoever runs the script). Reference the
|
||||
original PR + source SHA in every commit body and PR description for
|
||||
authorship attribution.
|
||||
- **Never use `git cherry-pick`** for this workflow — cherry-pick applies
|
||||
whole commits, which mixes concerns across PR boundaries.
|
||||
|
||||
## Outputs
|
||||
|
||||
The skill produces all transient planning artifacts under
|
||||
`.agents/tmp/decompose-<pr>/`:
|
||||
|
||||
1. A markdown decomposition plan (`plan.md`)
|
||||
2. A proposed branch graph
|
||||
3. A worktree-bootstrap script (`bootstrap.sh`)
|
||||
4. Per-PR file allocation lists (under `stack/`)
|
||||
5. AGENTS.md scaffolds for any new pipeline packages
|
||||
6. Draft lesson files (placed alongside the PR that owns the code they concern)
|
||||
7. A finalized provenance table for the package AGENTS.md
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
The skill should warn against:
|
||||
|
||||
- **"Just rebase the megaPR into smaller commits."** Doesn't help review;
|
||||
reviewer still sees one PR.
|
||||
- **Co-locating tests under the new package.** FastVideo's convention is
|
||||
by-kind under `fastvideo/tests/` and `tests/local_tests/<family>/`. Don't
|
||||
invent a new layout per pipeline.
|
||||
- **Splitting Tier 2 changes into "one file per PR."** Tier 2 PRs are
|
||||
about semantic units (e.g., "loader umbrella + optional component dirs"
|
||||
together because they jointly define the new diffusers-format contract),
|
||||
not file-count.
|
||||
- **Landing the activation switch first** ("just register, the code can
|
||||
be empty"). The skill enforces activation-last so every intermediate
|
||||
state is dead code, not broken code.
|
||||
- **Trusting `gh pr diff --name-only`.** Cross-check against
|
||||
`git diff origin/main..origin/<feature-branch> --name-status` —
|
||||
`gh`'s output has been observed to include phantom entries.
|
||||
- **Worktree dir names with hyphens.** mypy interprets them as invalid
|
||||
Python package names and refuses to run. Use CamelCase or underscores.
|
||||
- **Skipping the lesson-extraction step.** PR bodies contain the most
|
||||
expensive learnings of the original implementation. Losing them to a
|
||||
squash-merge is the silent decay of institutional knowledge.
|
||||
|
||||
## Example Usage
|
||||
|
||||
```
|
||||
User: split PR 1280
|
||||
Agent: [invokes decompose-pipeline-pr]
|
||||
→ produces .agents/tmp/decompose-1280/plan.md with:
|
||||
- tiered file table (56 files: 3 tier-0, 35 tier-1, 9 tier-2,
|
||||
9 tier-3)
|
||||
- branch graph (PR-A + PR-B + 8-PR stack)
|
||||
- worktree bootstrap script
|
||||
- per-PR file lists
|
||||
- AGENTS.md scaffold for fastvideo/pipelines/basic/magi_human/
|
||||
- 3 draft lessons extracted from the PR body
|
||||
→ asks user to confirm before opening branches
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- The MagiHuman decomposition (worked example):
|
||||
`fastvideo/pipelines/basic/magi_human/AGENTS.md` (after PR #1302 merges)
|
||||
- Existing skill: `.agents/skills/add-model/SKILL.md` (the inverse — adding
|
||||
a new pipeline as a fresh PR)
|
||||
- Lesson template: `.agents/lessons/README.md`
|
||||
- Skill template: `.agents/skills/SKILL_TEMPLATE.md`
|
||||
@@ -105,8 +105,8 @@ Detect artefact type by inspecting the file's imports / helper call:
|
||||
- **pixel** (`.mp4`) — file imports
|
||||
`run_text_to_video_similarity_test` / `run_image_to_video_similarity_test`
|
||||
from `fastvideo.tests.ssim.inference_similarity_utils`, OR uses the
|
||||
legacy custom-inline helper pattern (see `test_gamecraft`,
|
||||
`test_longcat`, etc.). Default to pixel when both heuristics fail.
|
||||
legacy custom-inline helper pattern (see `test_gen3c`). Default to pixel
|
||||
when both heuristics fail.
|
||||
|
||||
Record `ARTEFACT_TYPE ∈ {pixel, latent}` for use in step 4. Steps 2, 3, 5,
|
||||
and 6 are artefact-type-agnostic — `_iter_reference_files`,
|
||||
|
||||
@@ -75,7 +75,7 @@ if [[ $machine_arch =~ ^(aarch64|arm64)$ ]]; then
|
||||
npm run e2e -- \
|
||||
--project=chromium \
|
||||
--project=mobile-chromium \
|
||||
--grep-invert='streams, plays, and surfaces a downloadable clip|starts a new project and switches back to the prior session|saved projects persist across a page reload'
|
||||
--grep-invert='streams, plays, and surfaces a downloadable clip|starts a new project and switches back to the prior session|saved projects persist across a page reload|sends validated assets and plays a clearly labeled sample'
|
||||
else
|
||||
BACKEND_HOST=127.0.0.1 BACKEND_PORT="$BACKEND_PORT" CI=1 \
|
||||
npm run e2e -- \
|
||||
|
||||
@@ -2,4 +2,4 @@
|
||||
# Canonical Slurm CI selection for the VMoBA-inference lane.
|
||||
set -euo pipefail
|
||||
|
||||
exec python fastvideo/tests/inference/vmoba/test_vmoba_inference.py
|
||||
exec pytest ./fastvideo/tests/inference/vmoba/test_vmoba_inference.py -vs
|
||||
|
||||
@@ -9,9 +9,12 @@ mkdir -p "$PERF_REPORTS_DIR"
|
||||
|
||||
if [[ ${BUILDKITE_PULL_REQUEST:-false} =~ ^[1-9][0-9]*$ ]]; then
|
||||
export PERF_RUN_SOURCE=pr
|
||||
export PERF_UPLOAD_POLICY=pass
|
||||
export PERF_UPLOAD_POLICY=never
|
||||
elif [ "${BUILDKITE_BRANCH:-}" = main ] \
|
||||
&& { [ "${BUILDKITE_SOURCE:-}" = schedule ] || [ "${TEST_SCOPE:-}" = full ]; }; then
|
||||
&& { [ "${BUILDKITE_SOURCE:-}" = schedule ] || [ "${TEST_SCOPE:-}" = full ] \
|
||||
|| { [ "${BUILDKITE_SOURCE:-}" = api ] \
|
||||
&& [ "${BUILDKITE_PIPELINE_SLUG:-}" = fastvideo-performance-lane ] \
|
||||
&& [ "${TEST_SCOPE:-}" = direct ]; }; }; then
|
||||
export PERF_RUN_SOURCE=scheduled_main
|
||||
export PERF_UPLOAD_POLICY=always
|
||||
elif [ "${TEST_SCOPE:-}" = direct ]; then
|
||||
|
||||
@@ -205,40 +205,6 @@ pull_request_rules:
|
||||
label:
|
||||
add: ["scope: model"]
|
||||
|
||||
# ============================================================
|
||||
# Pre-commit failure help comment
|
||||
# ============================================================
|
||||
|
||||
- name: comment on pre-commit failure
|
||||
conditions:
|
||||
- check-failure~=pre-commit
|
||||
- -closed
|
||||
actions:
|
||||
comment:
|
||||
message: |
|
||||
## Pre-commit checks failed
|
||||
|
||||
Hi @{{author}}, the pre-commit checks have failed. To fix them locally:
|
||||
|
||||
```bash
|
||||
# Install pre-commit if you haven't already
|
||||
uv pip install pre-commit
|
||||
pre-commit install
|
||||
|
||||
# Run all checks and auto-fix what's possible
|
||||
pre-commit run --all-files
|
||||
```
|
||||
|
||||
Common fixes:
|
||||
- **yapf**: `yapf -i <file>` (formatting)
|
||||
- **ruff**: `ruff check --fix <file>` (linting)
|
||||
- **codespell**: `codespell --write-changes <file>` (spelling)
|
||||
|
||||
After fixing, commit and push the changes. The checks will re-run automatically.
|
||||
|
||||
For future commits, `pre-commit` will run automatically on changed files before each commit.
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Merge conflict detection
|
||||
# ============================================================
|
||||
@@ -251,16 +217,6 @@ pull_request_rules:
|
||||
actions:
|
||||
label:
|
||||
add: [needs-rebase]
|
||||
comment:
|
||||
message: |
|
||||
This PR has merge conflicts with the base branch. Please rebase:
|
||||
|
||||
```bash
|
||||
git fetch origin main
|
||||
git rebase origin/main
|
||||
# Resolve any conflicts, then:
|
||||
git push --force-with-lease
|
||||
```
|
||||
|
||||
- name: remove conflict label when resolved
|
||||
conditions:
|
||||
@@ -290,35 +246,5 @@ pull_request_rules:
|
||||
merge:
|
||||
method: squash
|
||||
|
||||
# ============================================================
|
||||
# PR title format help
|
||||
# ============================================================
|
||||
|
||||
- name: comment on invalid PR title format
|
||||
conditions:
|
||||
- -closed
|
||||
- -draft
|
||||
- "-title~=(?i)^\\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\\]"
|
||||
actions:
|
||||
comment:
|
||||
message: |
|
||||
## ⚠️ PR title format required
|
||||
|
||||
Your PR title must start with a type tag in brackets. Examples:
|
||||
- `[feat] Add new model support`
|
||||
- `[bugfix] Fix VAE tiling corruption`
|
||||
- `[refactor] Restructure training pipeline`
|
||||
- `[perf] Optimize attention kernel`
|
||||
- `[ci] Update test infrastructure`
|
||||
- `[infra] Add activation trace hooks`
|
||||
- `[docs] Add inference guide`
|
||||
- `[misc] Clean up configs`
|
||||
- `[new-model] Port Flux2 to FastVideo`
|
||||
- `[skill] Add add-model agent skill`
|
||||
|
||||
Valid tags: `feat`, `feature`, `bugfix`, `fix`, `refactor`, `perf`, `ci`, `infra`, `doc`, `docs`, `misc`, `chore`, `kernel`, `new-model`, `skill`, `skills`
|
||||
|
||||
Please update your PR title and the merge protection check will pass automatically.
|
||||
|
||||
merge_protections_settings:
|
||||
reporting_method: check-runs
|
||||
|
||||
@@ -12,6 +12,7 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import fnmatch
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import TextIO
|
||||
@@ -103,9 +104,9 @@ SAFE_PATTERNS = (
|
||||
"comfyui/**",
|
||||
"docs/**",
|
||||
"examples/**",
|
||||
"mkdocs.yml",
|
||||
"requirements-mkdocs.in",
|
||||
"requirements-mkdocs.txt",
|
||||
"docs-site/**",
|
||||
"requirements-docs.in",
|
||||
"requirements-docs.txt",
|
||||
"scripts/**",
|
||||
"tests/__init__.py",
|
||||
"tests/local_tests/**",
|
||||
@@ -130,11 +131,6 @@ class FamilyCoverage:
|
||||
|
||||
|
||||
FAMILY_COVERAGE = (
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])dreamx(_world)?([/_.-]|$)"),
|
||||
("test_dreamx.py", ),
|
||||
("test_dreamx_world_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])flux[_-]?2([/_.-]|$)"),
|
||||
("test_flux2_klein.py", ),
|
||||
@@ -145,21 +141,11 @@ FAMILY_COVERAGE = (
|
||||
("test_flux.py", ),
|
||||
("test_flux_t2i_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])(hunyuan)?gamecraft([/_.-]|$)"),
|
||||
("test_gamecraft.py", ),
|
||||
("test_gamecraft_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])gen3c([/_.-]|$)"),
|
||||
("test_gen3c.py", ),
|
||||
("test_gen3c_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])glm[_-]?image([/_.-]|$)"),
|
||||
("test_glm_image.py", ),
|
||||
("test_glm_image_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])hunyuan(video)?15([a-z0-9_-]*)([/_.-]|$)"),
|
||||
(),
|
||||
@@ -175,11 +161,6 @@ FAMILY_COVERAGE = (
|
||||
("test_lingbot.py", ),
|
||||
("test_lingbot_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])longcat([/_.-]|$)"),
|
||||
("test_longcat.py", ),
|
||||
("test_longcat_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])ltx[_-]?2([/_.-]|$)"),
|
||||
("test_ltx2.py", ),
|
||||
@@ -210,11 +191,6 @@ FAMILY_COVERAGE = (
|
||||
("test_stable_audio.py", ),
|
||||
("test_stable_audio_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])turbo(diffusion)?([/_.-]|$)"),
|
||||
(),
|
||||
("test_turbodiffusion_similarity.py", ),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])wan(video|vae)?([/_.-]|$)"),
|
||||
("test_wan_t2v.py", "test_wan_vae.py", "test_wan_causal.py", "test_wan_denoising.py"),
|
||||
@@ -225,11 +201,6 @@ FAMILY_COVERAGE = (
|
||||
"test_wan_ti2v_similarity.py",
|
||||
),
|
||||
),
|
||||
FamilyCoverage(
|
||||
re.compile(r"(^|[/_.-])z[_-]?image([/_.-]|$)"),
|
||||
("test_zimage.py", ),
|
||||
("test_zimage_similarity.py", ),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@@ -325,16 +296,22 @@ def _select_output_coverage(plan: MergePlan, path: str) -> None:
|
||||
plan.add_ssim(SSIM_SMOKE_TESTS, reason=f"shared output SSIM smoke coverage: {path}")
|
||||
|
||||
|
||||
def classify_paths(paths: list[str]) -> MergePlan:
|
||||
def _normalize_path(raw_path: str) -> str:
|
||||
path = raw_path.strip()
|
||||
while path.startswith("./"):
|
||||
path = path[2:]
|
||||
return path
|
||||
|
||||
|
||||
def classify_paths(paths: list[str], removed_paths: Iterable[str] = ()) -> MergePlan:
|
||||
"""Plan merge lanes for ``paths``.
|
||||
|
||||
``removed_paths`` lists changed paths that no longer exist at the PR head
|
||||
(deleted files and rename sources); removed golden/SSIM tests are not run.
|
||||
"""
|
||||
plan = MergePlan()
|
||||
normalized_paths: list[str] = []
|
||||
for raw_path in paths:
|
||||
path = raw_path.strip()
|
||||
while path.startswith("./"):
|
||||
path = path[2:]
|
||||
if path:
|
||||
normalized_paths.append(path)
|
||||
normalized_paths = sorted(set(normalized_paths))
|
||||
normalized_paths = sorted({path for path in map(_normalize_path, paths) if path})
|
||||
removed = {path for path in map(_normalize_path, removed_paths) if path}
|
||||
if not normalized_paths:
|
||||
plan.require_all("changed-file list was empty; failing closed")
|
||||
return plan
|
||||
@@ -344,7 +321,7 @@ def classify_paths(paths: list[str]) -> MergePlan:
|
||||
plan.require_all("changed-file API failed; failing closed")
|
||||
continue
|
||||
|
||||
if path in {"requirements-mkdocs.in", "requirements-mkdocs.txt"}:
|
||||
if path in {"requirements-docs.in", "requirements-docs.txt"}:
|
||||
plan.reasons.append(f"documentation dependencies need no GPU integration: {path}")
|
||||
continue
|
||||
|
||||
@@ -374,7 +351,10 @@ def classify_paths(paths: list[str]) -> MergePlan:
|
||||
if path.startswith("fastvideo/tests/golden_gate/"):
|
||||
name = Path(path).name
|
||||
if name.startswith("test_") and name.endswith(".py"):
|
||||
plan.add_golden((name, ), reason=f"changed golden test: {path}")
|
||||
if path in removed:
|
||||
plan.reasons.append(f"removed golden test has nothing to run: {path}")
|
||||
else:
|
||||
plan.add_golden((name, ), reason=f"changed golden test: {path}")
|
||||
elif name in {"AGENTS.md", "README.md"}:
|
||||
plan.reasons.append(f"golden documentation only: {path}")
|
||||
else:
|
||||
@@ -385,7 +365,10 @@ def classify_paths(paths: list[str]) -> MergePlan:
|
||||
if path.startswith("fastvideo/tests/ssim/"):
|
||||
name = Path(path).name
|
||||
if name.startswith("test_") and name.endswith(".py"):
|
||||
plan.add_ssim((name, ), reason=f"changed SSIM test: {path}")
|
||||
if path in removed:
|
||||
plan.reasons.append(f"removed SSIM test has nothing to run: {path}")
|
||||
else:
|
||||
plan.add_ssim((name, ), reason=f"changed SSIM test: {path}")
|
||||
elif path.endswith((".py", ".json", ".pt", ".png", ".mp4")):
|
||||
plan.ssim_all = True
|
||||
plan.add_lanes("ssim", reason=f"shared SSIM harness/reference: {path}")
|
||||
@@ -565,6 +548,7 @@ def _write_summary(output: TextIO, plan: MergePlan) -> None:
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--paths-file", type=Path, required=True)
|
||||
parser.add_argument("--removed-paths-file", type=Path)
|
||||
parser.add_argument("--github-output", type=Path)
|
||||
parser.add_argument("--summary-file", type=Path)
|
||||
return parser.parse_args()
|
||||
@@ -573,7 +557,9 @@ def parse_args() -> argparse.Namespace:
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
paths = args.paths_file.read_text(encoding="utf-8").splitlines()
|
||||
plan = classify_paths(paths)
|
||||
removed_paths = (args.removed_paths_file.read_text(encoding="utf-8").splitlines()
|
||||
if args.removed_paths_file else [])
|
||||
plan = classify_paths(paths, removed_paths)
|
||||
print(f"MERGE_TEST_PLAN={plan.encoded_lanes()}")
|
||||
print(f"MERGE_GOLDEN_TESTS={plan.encoded_golden_tests()}")
|
||||
print(f"MERGE_SSIM_TESTS={plan.encoded_ssim_tests()}")
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
name: Scheduled Performance
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Sunday 23:00 America/Los_Angeles, including daylight saving time.
|
||||
- cron: "0 6 * * 1"
|
||||
- cron: "0 7 * * 1"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
trigger:
|
||||
if: github.repository == 'hao-ai-lab/FastVideo' && github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Select the Pacific-time schedule
|
||||
id: schedule
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
SCHEDULE_EXPR: ${{ github.event.schedule }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [[ "$EVENT_NAME" == schedule ]]; then
|
||||
expected='0 7 * * 1'
|
||||
if [[ "$(TZ=America/Los_Angeles date +%z)" == -0700 ]]; then
|
||||
expected='0 6 * * 1'
|
||||
fi
|
||||
if [[ "$SCHEDULE_EXPR" != "$expected" ]]; then
|
||||
echo 'run=false' >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
echo 'run=true' >> "$GITHUB_OUTPUT"
|
||||
- name: Trigger weekly performance on Slinky Slurm
|
||||
if: steps.schedule.outputs.run == 'true'
|
||||
env:
|
||||
BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
|
||||
SOURCE_SHA: ${{ github.sha }}
|
||||
SOURCE_BRANCH: ${{ github.event.repository.default_branch }}
|
||||
BK_ORG: ${{ vars.BUILDKITE_ORG_SLUG }}
|
||||
BK_PIPELINE: fastvideo-performance-lane
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [[ ${#SOURCE_SHA} -ne 40 || "$SOURCE_SHA" == *[!0-9a-f]* ]]; then
|
||||
echo 'SOURCE_SHA must be a full immutable commit SHA' >&2
|
||||
exit 1
|
||||
fi
|
||||
payload=$(python3 - <<'PY'
|
||||
import json
|
||||
import os
|
||||
|
||||
print(json.dumps({
|
||||
"commit": os.environ["SOURCE_SHA"],
|
||||
"branch": os.environ["SOURCE_BRANCH"],
|
||||
"message": "Weekly performance on Slinky Slurm",
|
||||
"ignore_pipeline_branch_filters": True,
|
||||
"env": {
|
||||
"TEST_SCOPE": "direct",
|
||||
"FULL_SUITE": "false",
|
||||
"TEST_TYPE": "performance",
|
||||
"PR_NUMBER": "false",
|
||||
"PR_TITLE": "Scheduled performance",
|
||||
},
|
||||
}))
|
||||
PY
|
||||
)
|
||||
curl -sS --fail-with-body --connect-timeout 10 --max-time 60 -X POST \
|
||||
"https://api.buildkite.com/v2/organizations/${BK_ORG}/pipelines/${BK_PIPELINE}/builds" \
|
||||
-H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
--data-raw "$payload"
|
||||
@@ -180,6 +180,8 @@ jobs:
|
||||
set -euo pipefail
|
||||
changed_json="$RUNNER_TEMP/merge-changed-files.json"
|
||||
changed_paths="$RUNNER_TEMP/merge-changed-paths.txt"
|
||||
removed_paths="$RUNNER_TEMP/merge-removed-paths.txt"
|
||||
: > "$removed_paths"
|
||||
if gh api --paginate --slurp \
|
||||
"repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/files?per_page=100" \
|
||||
> "$changed_json"; then
|
||||
@@ -187,6 +189,10 @@ jobs:
|
||||
if [ "$observed" = "$EXPECTED_CHANGED_FILES" ]; then
|
||||
jq -r '.[][] | .filename, (.previous_filename // empty)' "$changed_json" \
|
||||
| sort -u > "$changed_paths"
|
||||
# Paths absent from the PR head, so the planner never selects a deleted test.
|
||||
jq -r '.[][] | if .status == "removed" then .filename
|
||||
elif .status == "renamed" then (.previous_filename // empty) else empty end' \
|
||||
"$changed_json" | sort -u > "$removed_paths"
|
||||
else
|
||||
echo "::warning::Changed-file API returned $observed of $EXPECTED_CHANGED_FILES paths; selecting all merge lanes."
|
||||
echo '__FASTVIDEO_CI_PLAN_ALL__' > "$changed_paths"
|
||||
@@ -202,6 +208,7 @@ jobs:
|
||||
run: |
|
||||
python3 .github/scripts/plan_merge_ci.py \
|
||||
--paths-file "$RUNNER_TEMP/merge-changed-paths.txt" \
|
||||
--removed-paths-file "$RUNNER_TEMP/merge-removed-paths.txt" \
|
||||
--github-output "$GITHUB_OUTPUT" \
|
||||
--summary-file "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
name: Close Stale Issues and PRs
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily at 1:30 AM UTC
|
||||
- cron: '30 1 * * *'
|
||||
|
||||
jobs:
|
||||
stale:
|
||||
if: github.repository == 'hao-ai-lab/FastVideo'
|
||||
permissions:
|
||||
issues: write
|
||||
pull-requests: write
|
||||
actions: write
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/stale@997185467fa4f803885201cee163a9f38240193d # v10.1.1
|
||||
with:
|
||||
operations-per-run: 500
|
||||
|
||||
exempt-draft-pr: true
|
||||
exempt-issue-labels: 'keep-open,pinned,security,Bug,RFC'
|
||||
exempt-pr-labels: 'keep-open,pinned'
|
||||
|
||||
labels-to-add-when-unstale: 'unstale'
|
||||
labels-to-remove-when-stale: 'unstale'
|
||||
|
||||
days-before-issue-stale: 90
|
||||
days-before-issue-close: 30
|
||||
stale-issue-label: 'stale'
|
||||
stale-issue-message: >
|
||||
This issue has been automatically marked as stale because it has not
|
||||
had any activity within 90 days. It will be automatically closed if
|
||||
no further activity occurs within 30 days. Leave a comment if you
|
||||
feel this issue should remain open. Thank you!
|
||||
close-issue-message: >
|
||||
This issue has been automatically closed due to inactivity. Please
|
||||
feel free to reopen if you feel it is still relevant. Thank you!
|
||||
|
||||
days-before-pr-stale: 60
|
||||
days-before-pr-close: 14
|
||||
stale-pr-label: 'stale'
|
||||
stale-pr-message: >
|
||||
This pull request has been automatically marked as stale because it
|
||||
has not had any activity within 60 days. It will be automatically
|
||||
closed if no further activity occurs within 14 days. Leave a comment
|
||||
if you feel this pull request should remain open. Thank you!
|
||||
close-pr-message: >
|
||||
This pull request has been automatically closed due to inactivity.
|
||||
Please feel free to reopen if you intend to continue working on it.
|
||||
Thank you!
|
||||
@@ -1,56 +0,0 @@
|
||||
name: Welcome First-Time Contributors
|
||||
|
||||
on:
|
||||
issues:
|
||||
types: [opened]
|
||||
pull_request_target:
|
||||
types: [opened]
|
||||
|
||||
permissions:
|
||||
issues: write
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
welcome:
|
||||
if: github.repository == 'hao-ai-lab/FastVideo'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/first-interaction@34f15e814fe48ac9312ccf29db4e74fa767cbab7 # v1.3.0
|
||||
with:
|
||||
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
issue-message: |
|
||||
Welcome to FastVideo! Thanks for opening your first issue.
|
||||
|
||||
To help us investigate, please include:
|
||||
- **FastVideo version**: `pip show fastvideo`
|
||||
- **GPU**: `nvidia-smi` output (GPU model, driver, CUDA version)
|
||||
- **Python version**: `python --version`
|
||||
- **OS**: e.g., Ubuntu 22.04
|
||||
|
||||
If this is a bug, a minimal reproduction script helps us fix it faster.
|
||||
|
||||
Useful links:
|
||||
- [Documentation](https://hao-ai-lab.github.io/FastVideo)
|
||||
- [Contributing Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
|
||||
- [Slack](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ)
|
||||
pr-message: |
|
||||
Welcome to FastVideo! Thanks for your first pull request.
|
||||
|
||||
**How our CI works:**
|
||||
|
||||
PRs run a three-tier CI system:
|
||||
1. **Pre-commit** — formatting (yapf), linting (ruff), type checking (mypy). Runs immediately on every PR.
|
||||
2. **Fastcheck** — six core GPU lanes run automatically via Buildkite (~10-15 min).
|
||||
3. **Merge gate** — a reviewer adds `ready`; changed paths select only the relevant integration, training, golden, or SSIM coverage.
|
||||
|
||||
**Before your PR is reviewed:**
|
||||
- [ ] `pre-commit run --all-files` passes locally
|
||||
- [ ] You've added or updated tests for your changes
|
||||
- [ ] The PR description explains what and why
|
||||
|
||||
If pre-commit fails, a bot comment will explain how to fix it. Fastcheck and merge-gate results appear in the Checks section below.
|
||||
|
||||
**Useful links:**
|
||||
- [Contributing Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
|
||||
- [Development Roadmap](https://github.com/hao-ai-lab/FastVideo/issues/899)
|
||||
- [Slack](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ)
|
||||
@@ -7,9 +7,11 @@ on:
|
||||
- 'docs/**'
|
||||
- 'examples/**'
|
||||
- 'scripts/inference/**'
|
||||
- 'mkdocs.yml'
|
||||
- 'requirements-mkdocs.in'
|
||||
- 'requirements-mkdocs.txt'
|
||||
- 'docs-site/**'
|
||||
- 'fastvideo/**'
|
||||
- 'README.md'
|
||||
- 'requirements-docs.in'
|
||||
- 'requirements-docs.txt'
|
||||
- 'scripts/check_docs_links.py'
|
||||
- '.github/workflows/infra-docs.yml'
|
||||
pull_request:
|
||||
@@ -18,9 +20,11 @@ on:
|
||||
- 'docs/**'
|
||||
- 'examples/**'
|
||||
- 'scripts/inference/**'
|
||||
- 'mkdocs.yml'
|
||||
- 'requirements-mkdocs.in'
|
||||
- 'requirements-mkdocs.txt'
|
||||
- 'docs-site/**'
|
||||
- 'fastvideo/**'
|
||||
- 'README.md'
|
||||
- 'requirements-docs.in'
|
||||
- 'requirements-docs.txt'
|
||||
- 'scripts/check_docs_links.py'
|
||||
- '.github/workflows/infra-docs.yml'
|
||||
|
||||
@@ -35,6 +39,7 @@ concurrency:
|
||||
|
||||
jobs:
|
||||
build:
|
||||
timeout-minutes: 15
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
@@ -51,7 +56,7 @@ jobs:
|
||||
uses: astral-sh/setup-uv@v3
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv pip install --system -r requirements-mkdocs.txt
|
||||
run: uv pip install --system -r requirements-docs.txt
|
||||
|
||||
- name: Setup Pages
|
||||
uses: actions/configure-pages@v4
|
||||
@@ -59,10 +64,16 @@ jobs:
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
node-version: '24'
|
||||
cache: npm
|
||||
cache-dependency-path: |
|
||||
docs/package-lock.json
|
||||
docs-site/package-lock.json
|
||||
|
||||
- name: Install cookbook dependencies
|
||||
run: npm ci --prefix docs
|
||||
run: |
|
||||
npm ci --prefix docs
|
||||
npm ci --prefix docs-site
|
||||
|
||||
- name: Validate and generate cookbook catalogs
|
||||
run: |
|
||||
@@ -71,8 +82,25 @@ jobs:
|
||||
python docs/tests/test_cookbook_assets.py
|
||||
npm run build:catalog --prefix docs
|
||||
|
||||
- name: Build documentation
|
||||
run: mkdocs build
|
||||
- name: Check site types and formatting
|
||||
run: |
|
||||
npm run check --prefix docs-site
|
||||
npm run format:check --prefix docs-site
|
||||
|
||||
- name: Build and validate documentation
|
||||
run: |
|
||||
npm run build --prefix docs-site
|
||||
npm run validate --prefix docs-site
|
||||
npm run test:content --prefix docs-site
|
||||
python -m unittest discover -s docs-site/tests -p 'test_*.py'
|
||||
python docs/check_cookbook_assets.py docs-site/dist
|
||||
|
||||
- name: Install test browser
|
||||
working-directory: docs-site
|
||||
run: npx playwright install --with-deps chromium
|
||||
|
||||
- name: Test documentation and cookbook interactions
|
||||
run: npm test --prefix docs-site
|
||||
|
||||
- name: Check docs links
|
||||
run: python scripts/check_docs_links.py
|
||||
@@ -80,7 +108,15 @@ jobs:
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: ./site
|
||||
path: ./docs-site/dist
|
||||
|
||||
- name: Upload browser failure traces
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: docs-browser-results
|
||||
path: docs-site/test-results
|
||||
if-no-files-found: ignore
|
||||
|
||||
deploy:
|
||||
environment:
|
||||
|
||||
+1
-4
@@ -36,7 +36,6 @@ env
|
||||
*.log
|
||||
weights/
|
||||
logs/
|
||||
/Z-Image/
|
||||
official_weights/
|
||||
converted_weights/
|
||||
|
||||
@@ -53,9 +52,8 @@ dist/
|
||||
eggs/
|
||||
.eggs/
|
||||
|
||||
# MkDocs documentation
|
||||
# Generated documentation
|
||||
site/
|
||||
docs/assets/cookbook-serving.json
|
||||
docs/assets/cookbook-config/
|
||||
examples/serving/clients/node_modules/
|
||||
docs/node_modules/
|
||||
@@ -64,7 +62,6 @@ docs/examples/
|
||||
docs/inference/examples/
|
||||
docs/training/examples/
|
||||
docs/distillation/examples/
|
||||
!requirements-mkdocs.txt
|
||||
|
||||
# VSCode
|
||||
.vscode/
|
||||
|
||||
@@ -21,6 +21,7 @@ repos:
|
||||
rev: v0.43.0
|
||||
hooks:
|
||||
- id: yapf
|
||||
require_serial: true
|
||||
args: [--in-place, --verbose]
|
||||
language_version: python3.12
|
||||
additional_dependencies: [toml] # TODO: Remove when yapf is upgraded
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
- Tests:
|
||||
- `fastvideo/tests/` for package-level tests (dataset, encoders, inference, training, SSIM, workflow).
|
||||
- `tests/local_tests/` for additional local/component checks.
|
||||
- Docs and guides: `docs/` (MkDocs source), with contributor docs in `docs/contributing/`.
|
||||
- Docs and guides: canonical Markdown in `docs/`, the Blume app in `docs-site/`, and contributor docs in `docs/contributing/`.
|
||||
- Runnable examples and scripts: `examples/` and `scripts/`.
|
||||
- Static assets: `assets/` (including `assets/images/`, `assets/videos/`, and `assets/prompts/`) and `comfyui/assets/`.
|
||||
|
||||
|
||||
@@ -3,6 +3,37 @@ import { stat } from 'node:fs/promises';
|
||||
import { test, expect, type WebSocket as PWWebSocket } from '@playwright/test';
|
||||
|
||||
test.describe('mock-backed generation smoke', () => {
|
||||
test('leave dialog blocks navigation and keeps cancel and confirm usable', async ({ page, request }) => {
|
||||
const health = await request.get('/healthz');
|
||||
const body = health.ok() ? await health.json() : {};
|
||||
test.skip(body.service !== 'ltx2-streaming-mock-server', 'Requires the local mock runtime.');
|
||||
await page.goto('/');
|
||||
await page.getByLabel('Continuation prompt').fill('A paper fox walks through a sunlit forest.');
|
||||
await page.getByRole('button', { name: 'Generate', exact: true }).click();
|
||||
const leave = page.getByRole('button', { name: 'Leave', exact: true });
|
||||
await expect(leave).toBeEnabled();
|
||||
await leave.click();
|
||||
const dialog = page.getByRole('dialog', { name: 'Leave session?' });
|
||||
await expect(dialog).toBeVisible();
|
||||
expect(await dialog.evaluate((element) => element.closest('.isolate') === null)).toBe(true);
|
||||
await expect(page.getByRole('button', { name: 'Toggle sidebar' }).click({ trial: true, timeout: 1000 })).rejects.toThrow();
|
||||
for (let index = 0; index < 4; index++) {
|
||||
await page.keyboard.press('Tab');
|
||||
await expect.poll(() => dialog.evaluate((element) => element.contains(document.activeElement))).toBe(true);
|
||||
}
|
||||
await dialog.getByRole('button', { name: 'Cancel', exact: true }).click();
|
||||
await expect(dialog).toHaveCount(0);
|
||||
await expect(leave).toBeFocused();
|
||||
await leave.click();
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(dialog).toHaveCount(0);
|
||||
await expect(leave).toBeFocused();
|
||||
await leave.click();
|
||||
await dialog.getByRole('button', { name: 'Leave session', exact: true }).click();
|
||||
await expect(dialog).toHaveCount(0);
|
||||
await expect(page.getByRole('button', { name: 'Generate', exact: true })).toBeVisible();
|
||||
});
|
||||
|
||||
test('accepts a custom prompt and enters the streaming state', async ({ page, request }) => {
|
||||
const health = await request.get('/healthz');
|
||||
const body = health.ok() ? await health.json() : {};
|
||||
|
||||
@@ -2739,7 +2739,7 @@ export default function Page() {
|
||||
<Header timeLeft={headerTimeLeft} formatTime={formatTime} onToggleSidebar={() => setSidebarOpen((prev) => !prev)} />
|
||||
|
||||
<div className={cn(
|
||||
"relative flex flex-1 min-h-0 flex-col px-4 pb-2 sm:px-6 sm:pb-12",
|
||||
"relative isolate flex flex-1 min-h-0 flex-col px-4 pb-2 sm:px-6 sm:pb-12",
|
||||
!isViewingMode && !showActiveProject && generationMode !== "t2va"
|
||||
? "justify-start overflow-y-auto pt-4"
|
||||
: "justify-center",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"use client";
|
||||
|
||||
import { useState, useEffect } from "react";
|
||||
import { useState, useEffect, useRef } from "react";
|
||||
import { Dialog as DialogPrimitive } from "radix-ui";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Checkbox } from "@/components/ui/checkbox";
|
||||
import { Label } from "@/components/ui/label";
|
||||
@@ -19,6 +20,7 @@ export default function LeaveSessionModal({
|
||||
onConfirmLeave = () => {},
|
||||
}: LeaveSessionModalProps) {
|
||||
const [suppress, setSuppress] = useState(false);
|
||||
const previousFocus = useRef<HTMLElement | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
if (open) setSuppress(false);
|
||||
@@ -36,43 +38,58 @@ export default function LeaveSessionModal({
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="fixed inset-0 z-[70] flex items-center justify-center bg-black/45 px-4 backdrop-blur-[3px]">
|
||||
<div
|
||||
role="dialog"
|
||||
aria-modal="true"
|
||||
aria-labelledby="leave-session-title"
|
||||
className="w-full max-w-md rounded-3xl border border-border/70 bg-card/95 p-6 shadow-2xl"
|
||||
>
|
||||
<div className="space-y-3">
|
||||
<div className="space-y-1">
|
||||
<h2 id="leave-session-title" className="text-lg font-semibold text-foreground">
|
||||
Leave session?
|
||||
</h2>
|
||||
<p className="text-sm text-muted-foreground">
|
||||
This will end your current session. You will need to queue again for a GPU to start a new project.
|
||||
</p>
|
||||
<DialogPrimitive.Root open={open} onOpenChange={(nextOpen) => { if (!nextOpen) onClose(); }}>
|
||||
<DialogPrimitive.Portal>
|
||||
<DialogPrimitive.Overlay asChild>
|
||||
<div className="fixed inset-0 z-[70] flex items-center justify-center bg-black/45 px-4 backdrop-blur-[3px]">
|
||||
<DialogPrimitive.Content
|
||||
asChild
|
||||
onInteractOutside={(event) => event.preventDefault()}
|
||||
onOpenAutoFocus={() => {
|
||||
previousFocus.current = document.activeElement instanceof HTMLElement ? document.activeElement : null;
|
||||
}}
|
||||
onCloseAutoFocus={(event) => {
|
||||
event.preventDefault();
|
||||
if (previousFocus.current?.isConnected) previousFocus.current.focus();
|
||||
}}
|
||||
>
|
||||
<div
|
||||
className="w-full max-w-md rounded-3xl border border-border/70 bg-card/95 p-6 shadow-2xl"
|
||||
>
|
||||
<div className="space-y-3">
|
||||
<div className="space-y-1">
|
||||
<DialogPrimitive.Title className="text-lg font-semibold text-foreground">
|
||||
Leave session?
|
||||
</DialogPrimitive.Title>
|
||||
<DialogPrimitive.Description className="text-sm text-muted-foreground">
|
||||
This will end your current session. You will need to queue again for a GPU to start a new project.
|
||||
</DialogPrimitive.Description>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<Checkbox
|
||||
id="suppress-leave-warning"
|
||||
checked={suppress}
|
||||
onCheckedChange={(v) => setSuppress(v === true)}
|
||||
/>
|
||||
<Label htmlFor="suppress-leave-warning" className="text-sm text-muted-foreground cursor-pointer">
|
||||
Do not warn again
|
||||
</Label>
|
||||
</div>
|
||||
</div>
|
||||
<div className="mt-5 flex flex-col-reverse gap-2 sm:flex-row sm:justify-end">
|
||||
<Button variant="outline" onClick={onClose}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button variant="destructive" onClick={handleConfirm}>
|
||||
Leave session
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</DialogPrimitive.Content>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<Checkbox
|
||||
id="suppress-leave-warning"
|
||||
checked={suppress}
|
||||
onCheckedChange={(v) => setSuppress(v === true)}
|
||||
/>
|
||||
<Label htmlFor="suppress-leave-warning" className="text-sm text-muted-foreground cursor-pointer">
|
||||
Do not warn again
|
||||
</Label>
|
||||
</div>
|
||||
</div>
|
||||
<div className="mt-5 flex flex-col-reverse gap-2 sm:flex-row sm:justify-end">
|
||||
<Button variant="outline" onClick={onClose}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button variant="destructive" onClick={handleConfirm}>
|
||||
Leave session
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</DialogPrimitive.Overlay>
|
||||
</DialogPrimitive.Portal>
|
||||
</DialogPrimitive.Root>
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -160,8 +160,8 @@ class FastH3Backend:
|
||||
Each clip length is a torch.compile shape, and the fullgraph regional
|
||||
route treats exceeding dynamo's recompile limit as a crash rather than
|
||||
a fallback. FastVideo's own imports lower it (`layers/lora/linear.py`
|
||||
to 16, longcat's `bsa_interface.py` to 32), so this raises it again
|
||||
and turns overflow back into a recompile.
|
||||
to 16), so this raises it again and turns overflow back into a
|
||||
recompile.
|
||||
|
||||
One pinned clip length keeps every process far under even the lowered
|
||||
limit; a varied `warmup_lengths` is where this becomes the seatbelt it
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
node_modules/
|
||||
.blume/
|
||||
dist/
|
||||
.env.local
|
||||
public/
|
||||
test-results/
|
||||
playwright-report/
|
||||
.generated/
|
||||
.venv/
|
||||
@@ -0,0 +1,10 @@
|
||||
node_modules
|
||||
.blume
|
||||
dist
|
||||
public
|
||||
package-lock.json
|
||||
test-results
|
||||
playwright-report
|
||||
.generated
|
||||
.venv
|
||||
migration-anchors.json
|
||||
@@ -0,0 +1,117 @@
|
||||
# Documentation and cookbook plan
|
||||
|
||||
Readers should be able to choose a model, find their hardware path, generate a
|
||||
result, and then customize or train it without searching several sections for
|
||||
the same instructions.
|
||||
|
||||
## One home for each kind of information
|
||||
|
||||
| Area | Question it answers | Owns |
|
||||
| --------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Fast Models | Which FastVideo release should I choose? | Our released families, checkpoint exports, collections, short capability descriptions, links to recipes |
|
||||
| Support matrix | What else can FastVideo run? | The wider supported model set and optimization compatibility |
|
||||
| Hardware and runtimes | Where can I run it? | NVIDIA RTX/workstation and data center CUDA paths, DGX Spark, Apple MLX, AMD ROCm, platform requirements and limits |
|
||||
| Documentation | How does FastVideo work? | Installation, the first generation, configuration, CLI, parallelism, offloading, integrations, training, architecture and API reference |
|
||||
| Cookbook | How do I run this model for this task? | One family home, runtime choices, runnable recipes, model-specific configuration, troubleshooting and evidence |
|
||||
|
||||
Fast Models highlights our models without implying that FastVideo only supports
|
||||
those releases. CUDA and ROCm are runtimes; RTX, data center GPUs and DGX Spark
|
||||
are hardware paths. MLX is the Apple Silicon runtime. These distinctions should
|
||||
be visible before readers choose an installation command.
|
||||
|
||||
## Included now
|
||||
|
||||
- One Hardware and runtimes overview links the five hardware paths. AMD is
|
||||
explicitly source-backed support with incomplete installation/device coverage,
|
||||
rather than a promise that CUDA recipes or kernels work on every AMD GPU.
|
||||
- The getting-started sidebar contains Quick start, Installation, Hardware and
|
||||
runtimes, and Python API basics.
|
||||
- Run inference contains shared tasks. Hardware and Integrations are expandable
|
||||
groups; individual hardware recipes do not interrupt onboarding.
|
||||
- Model-specific references sit beside their family recipes in the Cookbook
|
||||
sidebar. There is no separate Model guides list competing with the cookbook.
|
||||
Existing LTX, Kandinsky and FastH3 reference URLs remain intact.
|
||||
- Fast Models points FastH3 and FastWan readers to their cookbook families.
|
||||
FastMetal also has a direct MLX setup entry.
|
||||
- Generated examples have one visible page title, with the source link below it.
|
||||
Their previous heading fragments remain available.
|
||||
|
||||
## The intended reader flow
|
||||
|
||||
1. Start at **Fast Models** to choose a released model, or the support matrix for
|
||||
the broader catalog.
|
||||
2. Check **Hardware and runtimes**, then follow the matching installation guide.
|
||||
3. Open the model's **Cookbook family**. Choose the task and runtime, configure
|
||||
a recipe, and run it.
|
||||
4. Follow links to shared configuration, offloading or optimization docs when
|
||||
more explanation is needed.
|
||||
5. Move to training, distillation or contributor guides for development work.
|
||||
|
||||
The quick start remains a short route for readers who want a first result before
|
||||
browsing the catalog.
|
||||
|
||||
## Next editorial pass
|
||||
|
||||
1. **Consolidate onboarding.** Keep one first-video tutorial. Remove repeated
|
||||
installation blocks from the inference quick start after preserving its unique
|
||||
image-conditioning examples. Make Installation a platform chooser and keep
|
||||
installation commands in platform guides.
|
||||
2. **Give every model one family home.** Expand the cookbook family page with a
|
||||
short capability overview, runtime/task recipes, checkpoint choices, limits,
|
||||
troubleshooting and next steps. Fold overlapping model-specific guides into
|
||||
that page or link a detailed reference from it. Preserve old routes with
|
||||
redirects before deleting any page.
|
||||
3. **Separate explanation from recipes.** Hardware owns environment setup;
|
||||
cookbook owns model commands; shared docs explain the options. The RTX PRO
|
||||
6000 page's kernel internals and rejected experiments belong in research or
|
||||
contributor material, separate from its runnable recipe.
|
||||
4. **Reduce pointer pages and duplicate references.** Replace the two-sentence
|
||||
ComfyUI pointer with a useful integration guide, or link directly to its
|
||||
maintained README. Reconcile the two attention-backend development guides.
|
||||
Let Architecture own the package map instead of repeating it in the agent guide.
|
||||
5. **Split long optimization material by task.** Keep attention selection,
|
||||
quantization, distributed execution and offloading cross-linked, with one
|
||||
owner for each setting. Keep STA and mask search clearly archived.
|
||||
6. **Fill the ROCm evidence gap.** Document a complete AMD installation and model
|
||||
run with GPU architecture, ROCm/PyTorch versions, model revision, attention
|
||||
backend, expected output and measured memory. Source implementation alone
|
||||
should not turn into a broad device-support claim.
|
||||
|
||||
## Per-page structure
|
||||
|
||||
Task guides follow: outcome → requirements → runnable example → expected output
|
||||
→ configuration → troubleshooting → next step. A model recipe adds its checkpoint,
|
||||
workload, runtime, hardware evidence and measured limitations. Detailed reference
|
||||
and implementation history should not interrupt setup commands.
|
||||
|
||||
See the [authoring guide](../docs/contributing/documentation.md). Examples and
|
||||
Python reference stay source-generated and searchable; they do not need hundreds
|
||||
of sidebar entries.
|
||||
|
||||
## Publishing
|
||||
|
||||
Blume replaces MkDocs through the existing GitHub Pages workflow at
|
||||
`https://haoailab.com/FastVideo/`. Only `main` deploys. The migration manifest
|
||||
checks previous routes and heading fragments. This PR changes navigation and
|
||||
presentation while keeping the technical content available; the consolidation
|
||||
steps above remain editorial follow-ups.
|
||||
|
||||
## Cookbook model pages
|
||||
|
||||
The cookbook catalog includes the maintained video, image, audio, and world
|
||||
models, with category and family filters. FastH3 appears first. Each checkpoint
|
||||
has its own page; related task or hardware recipes become variants on that page.
|
||||
Family groups hold the pages alongside their broader runbooks.
|
||||
|
||||
FastH3 8-Step V2, FastWan2.1 1.3B, Wan2.1 I2V 14B, and LTX2 Distilled use the five
|
||||
existing editable serving deployments. Their pages have a model header, YAML
|
||||
actions, generated commands, and a compact hardware/options matrix. Other models
|
||||
publish existing source commands and YAML without inventing editable settings.
|
||||
Hardware-fit validation remains unverified. New quantization, encoder, or runtime
|
||||
choices require a backend recipe before becoming selectable.
|
||||
|
||||
The retired page-copy and tab handlers, cookbook JavaScript, and theme styles are
|
||||
removed. Canonical callouts now use Blume syntax, so builds no longer execute a
|
||||
migration codemod. Existing family URLs show React model cards with their setup
|
||||
and troubleshooting content. The homepage adds a direct FastH3 recipe link while
|
||||
keeping the existing logo, news, project cards, typography, and backgrounds.
|
||||
@@ -0,0 +1,134 @@
|
||||
# FastVideo documentation
|
||||
|
||||
Blume builds the documentation and React cookbook at
|
||||
[haoailab.com/FastVideo](https://haoailab.com/FastVideo/). This PR replaces the
|
||||
previous MkDocs site using the existing GitHub Pages publishing workflow.
|
||||
|
||||

|
||||
|
||||

|
||||
|
||||
## Develop and validate
|
||||
|
||||
Read [Documenting FastVideo](../docs/contributing/documentation.md) for setup,
|
||||
commands, source ownership, and the per-page writing template. Use Node 24 and
|
||||
Python 3.12. `npm run dev --prefix docs-site` serves under `/FastVideo`.
|
||||
`npm run build --prefix docs-site` prepares the content and writes `dist/`.
|
||||
`npm run preview --prefix docs-site` serves that static build.
|
||||
|
||||
Edit canonical guides under `docs/`, the welcome page under `content/`, the model
|
||||
catalog in `lib/fast-models.ts`, and sidebar grouping in `navigation.json`.
|
||||
Preparation regenerates `.generated/` and `public/`; neither is an editable copy.
|
||||
|
||||
The [documentation structure plan](INFORMATION_ARCHITECTURE.md) records the new
|
||||
navigation and proposed page consolidations. URLs stay stable in this PR.
|
||||
Hardware and integrations sit beneath Run inference. Model-specific references
|
||||
sit beside their family recipes in the Cookbook tab. Hardware and runtimes gives
|
||||
readers a single starting point for CUDA, MLX and ROCm.
|
||||
|
||||
## Build architecture
|
||||
|
||||
- `scripts/prepare-python-docs.py` uses Griffe to extract modules, classes,
|
||||
signatures, docstrings, and source directly from Python files. It never imports
|
||||
FastVideo or needs PyTorch. Only Griffe, Markdown, and PyYAML are installed from
|
||||
`requirements-docs.txt`. It also regenerates examples and serving metadata.
|
||||
- `scripts/transfer-docs.mjs` rewrites links, preserves fragments, copies
|
||||
assets, and overlays the handwritten application pages. Canonical guides use
|
||||
native Blume callouts and React family cards; no migration converter runs during builds.
|
||||
- `migration-anchors.json` is a frozen compatibility contract for the previous
|
||||
site's URLs and heading fragments. Tests check the published output against it.
|
||||
Do not regenerate it to hide missing routes or anchors.
|
||||
- `scripts/prepare-content.mjs` builds and copies cookbook catalogs and assets.
|
||||
React model cards replace the old family selectors. Source commands and
|
||||
serving configurations remain owned by their checked-in recipes.
|
||||
- `.github/workflows/infra-docs.yml` validates and uploads `docs-site/dist/` with
|
||||
the existing Pages actions. Deployment runs only on `main`. The domain,
|
||||
environment and `/FastVideo` prefix are unchanged.
|
||||
|
||||
## Homepage and catalog
|
||||
|
||||
The welcome page uses Blume frame mode for its centered logo and accessible
|
||||
heading. News comes from the first two dated entries under `NEWS` in the root
|
||||
README; projects come from its Awesome work section. The original introduction
|
||||
and key features come from canonical `docs/index.md`.
|
||||
|
||||
The star button fetches GitHub's public repository count on load and every five
|
||||
minutes while visible. Failed initial requests show "Star"; failed refreshes
|
||||
retain the previous count. No credentials are used. The Trendshift badge loads
|
||||
from Trendshift. The lab mark in `assets/hao-ai-lab.png` is the supplied image.
|
||||
|
||||
Fast Models uses family cards, setup links, family filters, search, and consistent
|
||||
checkpoint rows. The curated links were checked against the public Hugging Face
|
||||
API on October 16, 2026. Published weights do not establish runtime or hardware
|
||||
compatibility. Check the support matrix and matching setup guide.
|
||||
|
||||
## Extend the cookbook
|
||||
|
||||
`islands/Cookbook.tsx` owns the React UI and its loading/edit state. Shared
|
||||
configuration behavior stays in `docs/js/cookbook-config.mjs`. The typed adapter
|
||||
in `lib/cookbook.ts` describes the part of that existing contract used by this UI.
|
||||
Read [the API contract](../docs/cookbook/design.md) and
|
||||
[recipe authoring guide](../docs/contributing/cookbook_configuration.md) before
|
||||
adding fields or recipes.
|
||||
|
||||
- Keep native serving YAMLs and `docs/cookbook/recipes/` as the source of truth.
|
||||
Do not add another recipe database or hardcode field lists in the UI.
|
||||
- Use `getOptions` for field metadata and `resolveConfig` for validated YAML,
|
||||
server commands, and client instructions. Clear generated output when invalid.
|
||||
- The URL stores the selected deployment. Edit values stay local to the page;
|
||||
URL serialization of all edited values is follow-up work.
|
||||
- Reset restores the baseline. Missing fields remain inherited. Explicit `false`,
|
||||
`0`, and `null` keep their meanings. Switching deployments discards edits.
|
||||
- Installation and deployment guides resolve against the current origin and
|
||||
deployment prefix. Family runbook URLs remain available in the Blume site.
|
||||
- VRAM fit is unknown. Add a calculator or a verified-run badge only with the
|
||||
measurements and evidence contract described in #1949.
|
||||
|
||||
Blume builds the documentation shell with Astro. Components in `islands/` are
|
||||
ordinary React; `client = "load"` enables hydration. `components/ui/` contains
|
||||
editable shadcn components, `components.json` configures the CLI, and `theme.css`
|
||||
maps their tokens to Blume's light/dark palette. Blume supplies Tailwind and reads
|
||||
`tsconfig.json` aliases. Add components from this directory:
|
||||
|
||||
```bash
|
||||
npx shadcn@4.21.5 add dialog
|
||||
```
|
||||
|
||||
The two npm overrides keep Astro's React integration and React plugin aligned
|
||||
with the versions used by Blume 2.2.3. Without them, npm's resolver cycles between
|
||||
the integration's newer plugin and Blume's pinned plugin. Recheck and remove the
|
||||
overrides when upgrading Blume.
|
||||
|
||||
The initial dependency audit reports 10 transitive advisories in Blume's dependency
|
||||
tree, including its unused Vercel adapter and Markdown build dependencies. The
|
||||
ordinary `npm audit fix` does not clear them; review upstream fixes before
|
||||
production deployment. Do not apply `npm audit fix --force`, whose proposed
|
||||
replacement is `blume@0.0.0`.
|
||||
|
||||
## Verify UI changes
|
||||
|
||||
```bash
|
||||
npm run format:check --prefix docs-site
|
||||
npm run check --prefix docs-site
|
||||
npm run test:cookbook --prefix docs
|
||||
npm run build --prefix docs-site
|
||||
npm run validate --prefix docs-site
|
||||
npm run test:content --prefix docs-site
|
||||
cd docs-site
|
||||
npx playwright install chromium
|
||||
npm test
|
||||
```
|
||||
|
||||
The browser suite starts a production preview. It checks edits and downloads,
|
||||
invalid-output suppression, reset, switching runtimes, retry, obsolete loads,
|
||||
keyboard editing, and mobile overflow under `/FastVideo`. Keep shared API tests
|
||||
in `docs/tests/`; add React interaction tests here. No serving process is launched.
|
||||
|
||||
## Review and follow-up
|
||||
|
||||
This PR switches the existing Pages artifact from MkDocs to Blume when merged.
|
||||
The original routes and fragments are checked against the frozen migration
|
||||
manifest. Review CI and the catalog/sidebar on desktop and mobile before merging.
|
||||
The remaining cookbook design and editor coverage in #1949 can continue without
|
||||
keeping a second documentation build. The editorial consolidation proposals are
|
||||
listed separately in `INFORMATION_ARCHITECTURE.md`.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 7.4 KiB |
@@ -0,0 +1,30 @@
|
||||
import { defineConfig } from "blume";
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
export default defineConfig({
|
||||
title: "FastVideo",
|
||||
description: "A unified inference and post-training framework for accelerated video generation.",
|
||||
content: { root: ".generated/content" },
|
||||
logo: { image: "/icon.svg", text: "FastVideo" },
|
||||
theme: { accent: "blue", background: { light: "#ffffff", dark: "#1c1d20" }, mode: "system" },
|
||||
search: {
|
||||
popular: [
|
||||
{ label: "Quick start", href: "/getting_started/quick_start", icon: "rocket" },
|
||||
{ label: "Fast Models", href: "/fast-models", icon: "boxes" },
|
||||
{ label: "Installation", href: "/getting_started/installation", icon: "download" },
|
||||
{ label: "Supported models", href: "/inference/support_matrix", icon: "video" },
|
||||
{ label: "Python API", href: "/api", icon: "code" },
|
||||
],
|
||||
},
|
||||
navigation: {
|
||||
actions: [{ label: "GitHub", href: "https://github.com/hao-ai-lab/FastVideo" }],
|
||||
tabs: [
|
||||
{ label: "Documentation", path: "/" },
|
||||
{ label: "Cookbook", path: "/cookbook" },
|
||||
],
|
||||
sidebar: { items: JSON.parse(readFileSync(new URL("./.generated/navigation.json", import.meta.url), "utf8")) },
|
||||
},
|
||||
// Keep the production prefix in local builds so asset-path regressions are visible.
|
||||
deployment: { site: "https://haoailab.com", base: "/FastVideo" },
|
||||
seo: { robots: true, sitemap: true, og: { enabled: false } },
|
||||
});
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"$schema": "https://ui.shadcn.com/schema.json",
|
||||
"style": "new-york",
|
||||
"rsc": false,
|
||||
"tsx": true,
|
||||
"tailwind": {
|
||||
"config": "",
|
||||
"css": "theme.css",
|
||||
"baseColor": "neutral",
|
||||
"cssVariables": true
|
||||
},
|
||||
"aliases": {
|
||||
"components": "@/components",
|
||||
"utils": "@/lib/utils",
|
||||
"ui": "@/components/ui",
|
||||
"lib": "@/lib",
|
||||
"hooks": "@/hooks"
|
||||
},
|
||||
"iconLibrary": "lucide"
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
import { useState } from "react";
|
||||
|
||||
/** A UI experiment, deliberately separated from the serving schema and generated YAML. */
|
||||
export default function ComponentPlan({ onDraftChange }: { onDraftChange?: (draft: boolean) => void }) {
|
||||
const [selection, setSelection] = useState<Record<string, string>>({});
|
||||
const groups = [
|
||||
{ name: "Weights", values: ["Recipe default", "FP8", "NVFP4"] },
|
||||
{ name: "Text encoder", values: ["Recipe default", "NVFP4 encoder"] },
|
||||
{ name: "Video decoder", values: ["Recipe default", "FP16 VAE", "NVFP4 VAE"] },
|
||||
];
|
||||
return (
|
||||
<details className="component-plan">
|
||||
<summary>
|
||||
Explore components <span>Design preview</span>
|
||||
</summary>
|
||||
<div className="component-plan-fields">
|
||||
{groups.map((group) => (
|
||||
<div key={group.name}>
|
||||
<span className="component-plan-label">{group.name}</span>
|
||||
<div className="option-chips" role="group" aria-label={group.name}>
|
||||
{group.values.map((value) => (
|
||||
<button
|
||||
key={value}
|
||||
type="button"
|
||||
aria-pressed={(selection[group.name] ?? "Recipe default") === value}
|
||||
onClick={() => {
|
||||
const next = { ...selection, [group.name]: value };
|
||||
setSelection(next);
|
||||
onDraftChange?.(Object.values(next).some((value) => value !== "Recipe default"));
|
||||
}}
|
||||
>
|
||||
{value}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<p>
|
||||
Preview only. Alternate components need a compatible recipe; some releases are planned. These choices do not
|
||||
change YAML.
|
||||
</p>
|
||||
{Object.values(selection).some((value) => value !== "Recipe default") && (
|
||||
<button
|
||||
type="button"
|
||||
className="component-plan-reset"
|
||||
onClick={() => {
|
||||
setSelection({});
|
||||
onDraftChange?.(false);
|
||||
}}
|
||||
>
|
||||
Restore recipe components
|
||||
</button>
|
||||
)}
|
||||
</details>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
import { useEffect, useId, useState } from "react";
|
||||
import { Input } from "@/components/ui/input";
|
||||
|
||||
export default function HardwarePicker({
|
||||
backend,
|
||||
options,
|
||||
value,
|
||||
onChange,
|
||||
}: {
|
||||
backend: string;
|
||||
options: string[][];
|
||||
value: string;
|
||||
onChange: (value: string) => void;
|
||||
}) {
|
||||
const [search, setSearch] = useState("");
|
||||
const [adding, setAdding] = useState(false);
|
||||
const [custom, setCustom] = useState("");
|
||||
useEffect(() => {
|
||||
const requested = new URL(window.location.href).searchParams.get("hardware");
|
||||
setCustom(requested?.startsWith("custom:") ? requested.slice(7) : "");
|
||||
}, []);
|
||||
const id = useId();
|
||||
const known = options.filter((option) => option.join(" ").toLowerCase().includes(search.toLowerCase()));
|
||||
function add() {
|
||||
const name = search.trim();
|
||||
if (!name) return;
|
||||
setCustom(name);
|
||||
onChange(`custom:${name}`);
|
||||
setSearch("");
|
||||
setAdding(false);
|
||||
}
|
||||
return (
|
||||
<div className="hardware-picker">
|
||||
<fieldset className="hardware-matrix">
|
||||
<legend className="sr-only">Hardware target</legend>
|
||||
{known.map(([key, title, detail]) => (
|
||||
<label key={key} data-selected={value === key}>
|
||||
<input
|
||||
type="radio"
|
||||
name={`hardware-${id}`}
|
||||
checked={value === key}
|
||||
onChange={() => onChange(key)}
|
||||
value={key}
|
||||
/>
|
||||
<span>
|
||||
<strong>{title}</strong>
|
||||
<small>{detail}</small>
|
||||
</span>
|
||||
</label>
|
||||
))}
|
||||
{custom && (
|
||||
<label data-selected={value === `custom:${custom}`}>
|
||||
<input
|
||||
type="radio"
|
||||
name={`hardware-${id}`}
|
||||
checked={value === `custom:${custom}`}
|
||||
onChange={() => onChange(`custom:${custom}`)}
|
||||
/>
|
||||
<span>
|
||||
<strong>{custom}</strong>
|
||||
<small>Custom device</small>
|
||||
</span>
|
||||
</label>
|
||||
)}
|
||||
</fieldset>
|
||||
<button type="button" className="hardware-add" aria-expanded={adding} onClick={() => setAdding(!adding)}>
|
||||
+ Find or add {backend === "mlx" ? "a Mac" : "a GPU"}
|
||||
</button>
|
||||
{adding && (
|
||||
<form
|
||||
className="hardware-search"
|
||||
onSubmit={(event) => {
|
||||
event.preventDefault();
|
||||
add();
|
||||
}}
|
||||
>
|
||||
<label htmlFor={id}>Device name</label>
|
||||
<Input
|
||||
id={id}
|
||||
autoFocus
|
||||
value={search}
|
||||
onChange={(event) => setSearch(event.target.value)}
|
||||
placeholder={backend === "mlx" ? "e.g. M4 Max" : "e.g. RTX 4090"}
|
||||
maxLength={80}
|
||||
/>
|
||||
<p>Search the list or add your device. Memory lookup and fit checks are not connected yet.</p>
|
||||
<button type="submit" disabled={!search.trim()}>
|
||||
Add device
|
||||
</button>
|
||||
</form>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
import type { ReactNode } from "react";
|
||||
import { sitePath, type DirectoryModel } from "@/lib/cookbook-pages";
|
||||
|
||||
export default function ModelHeader({ model, actions }: { model: DirectoryModel; actions?: ReactNode }) {
|
||||
return (
|
||||
<>
|
||||
<a className="recipe-back" href={sitePath("/cookbook")}>
|
||||
← All recipes
|
||||
</a>
|
||||
<header className="recipe-model-header">
|
||||
<img src={sitePath(`/assets/logos/${model.logo}.webp`)} alt="" width="48" height="48" />
|
||||
<div>
|
||||
<p className="recipe-publisher">{model.provider}</p>
|
||||
<h1>{model.title}</h1>
|
||||
<p className="recipe-model-task">{model.tags.join(" · ")}</p>
|
||||
{!model.model.startsWith("./") && (
|
||||
<a href={`https://huggingface.co/${model.model}`}>View on Hugging Face ↗</a>
|
||||
)}
|
||||
</div>
|
||||
<div className="recipe-model-actions">{actions}</div>
|
||||
</header>
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
import * as React from "react";
|
||||
import { cva, type VariantProps } from "class-variance-authority";
|
||||
import { cn } from "cn";
|
||||
import { Slot } from "radix-ui";
|
||||
|
||||
const badgeVariants = cva(
|
||||
"inline-flex w-fit shrink-0 items-center justify-center gap-1 overflow-hidden rounded-full border border-transparent px-2 py-0.5 text-xs font-medium whitespace-nowrap transition-[color,box-shadow] focus-visible:border-ring focus-visible:ring-[3px] focus-visible:ring-ring/50 aria-invalid:border-destructive aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 [&>svg]:pointer-events-none [&>svg]:size-3",
|
||||
{
|
||||
variants: {
|
||||
variant: {
|
||||
default: "bg-primary text-primary-foreground [a&]:hover:bg-primary/90",
|
||||
secondary: "bg-secondary text-secondary-foreground [a&]:hover:bg-secondary/90",
|
||||
destructive:
|
||||
"bg-destructive text-white focus-visible:ring-destructive/20 dark:bg-destructive/60 dark:focus-visible:ring-destructive/40 [a&]:hover:bg-destructive/90",
|
||||
outline: "border-border text-foreground [a&]:hover:bg-accent [a&]:hover:text-accent-foreground",
|
||||
ghost: "[a&]:hover:bg-accent [a&]:hover:text-accent-foreground",
|
||||
link: "text-primary underline-offset-4 [a&]:hover:underline",
|
||||
},
|
||||
},
|
||||
defaultVariants: {
|
||||
variant: "default",
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
function Badge({
|
||||
className,
|
||||
variant = "default",
|
||||
asChild = false,
|
||||
...props
|
||||
}: React.ComponentProps<"span"> & VariantProps<typeof badgeVariants> & { asChild?: boolean }) {
|
||||
const Comp = asChild ? Slot.Root : "span";
|
||||
|
||||
return (
|
||||
<Comp data-slot="badge" data-variant={variant} className={cn(badgeVariants({ variant }), className)} {...props} />
|
||||
);
|
||||
}
|
||||
|
||||
export { Badge, badgeVariants };
|
||||
@@ -0,0 +1,61 @@
|
||||
import * as React from "react";
|
||||
import { cva, type VariantProps } from "class-variance-authority";
|
||||
import { cn } from "cn";
|
||||
import { Slot } from "radix-ui";
|
||||
|
||||
const buttonVariants = cva(
|
||||
"inline-flex shrink-0 items-center justify-center gap-2 rounded-md text-sm font-medium whitespace-nowrap transition-all outline-none focus-visible:border-ring focus-visible:ring-[3px] focus-visible:ring-ring/50 disabled:pointer-events-none disabled:opacity-50 aria-invalid:border-destructive aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4",
|
||||
{
|
||||
variants: {
|
||||
variant: {
|
||||
default: "bg-primary text-primary-foreground hover:bg-primary/90",
|
||||
destructive:
|
||||
"bg-destructive text-white hover:bg-destructive/90 focus-visible:ring-destructive/20 dark:bg-destructive/60 dark:focus-visible:ring-destructive/40",
|
||||
outline:
|
||||
"border bg-background shadow-xs hover:bg-accent hover:text-accent-foreground dark:border-input dark:bg-input/30 dark:hover:bg-input/50",
|
||||
secondary: "bg-secondary text-secondary-foreground hover:bg-secondary/80",
|
||||
ghost: "hover:bg-accent hover:text-accent-foreground dark:hover:bg-accent/50",
|
||||
link: "text-primary underline-offset-4 hover:underline",
|
||||
},
|
||||
size: {
|
||||
default: "h-9 px-4 py-2 has-[>svg]:px-3",
|
||||
xs: "h-6 gap-1 rounded-md px-2 text-xs has-[>svg]:px-1.5 [&_svg:not([class*='size-'])]:size-3",
|
||||
sm: "h-8 gap-1.5 rounded-md px-3 has-[>svg]:px-2.5",
|
||||
lg: "h-10 rounded-md px-6 has-[>svg]:px-4",
|
||||
icon: "size-9",
|
||||
"icon-xs": "size-6 rounded-md [&_svg:not([class*='size-'])]:size-3",
|
||||
"icon-sm": "size-8",
|
||||
"icon-lg": "size-10",
|
||||
},
|
||||
},
|
||||
defaultVariants: {
|
||||
variant: "default",
|
||||
size: "default",
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
function Button({
|
||||
className,
|
||||
variant = "default",
|
||||
size = "default",
|
||||
asChild = false,
|
||||
...props
|
||||
}: React.ComponentProps<"button"> &
|
||||
VariantProps<typeof buttonVariants> & {
|
||||
asChild?: boolean;
|
||||
}) {
|
||||
const Comp = asChild ? Slot.Root : "button";
|
||||
|
||||
return (
|
||||
<Comp
|
||||
data-slot="button"
|
||||
data-variant={variant}
|
||||
data-size={size}
|
||||
className={cn(buttonVariants({ variant, size, className }))}
|
||||
{...props}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
export { Button, buttonVariants };
|
||||
@@ -0,0 +1,20 @@
|
||||
import * as React from "react";
|
||||
import { cn } from "cn";
|
||||
|
||||
function Input({ className, type, ...props }: React.ComponentProps<"input">) {
|
||||
return (
|
||||
<input
|
||||
type={type}
|
||||
data-slot="input"
|
||||
className={cn(
|
||||
"h-9 w-full min-w-0 rounded-md border border-input bg-transparent px-3 py-1 text-base shadow-xs transition-[color,box-shadow] outline-none selection:bg-primary selection:text-primary-foreground file:inline-flex file:h-7 file:border-0 file:bg-transparent file:text-sm file:font-medium file:text-foreground placeholder:text-muted-foreground disabled:pointer-events-none disabled:cursor-not-allowed disabled:opacity-50 md:text-sm dark:bg-input/30",
|
||||
"focus-visible:border-ring focus-visible:ring-[3px] focus-visible:ring-ring/50",
|
||||
"aria-invalid:border-destructive aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40",
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
export { Input };
|
||||
@@ -0,0 +1,17 @@
|
||||
import * as React from "react";
|
||||
import { cn } from "cn";
|
||||
|
||||
function Textarea({ className, ...props }: React.ComponentProps<"textarea">) {
|
||||
return (
|
||||
<textarea
|
||||
data-slot="textarea"
|
||||
className={cn(
|
||||
"flex field-sizing-content min-h-16 w-full rounded-md border border-input bg-transparent px-3 py-2 text-base shadow-xs transition-[color,box-shadow] outline-none placeholder:text-muted-foreground focus-visible:border-ring focus-visible:ring-[3px] focus-visible:ring-ring/50 disabled:cursor-not-allowed disabled:opacity-50 aria-invalid:border-destructive aria-invalid:ring-destructive/20 md:text-sm dark:bg-input/30 dark:aria-invalid:ring-destructive/40",
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
export { Textarea };
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
title: Cookbook
|
||||
description: Recipes for video, image, audio, and world generation.
|
||||
mode: wide
|
||||
---
|
||||
|
||||
<a id="inference-cookbook"></a>
|
||||
<a id="planned-families-heading"></a>
|
||||
<a id="video-models-heading"></a>
|
||||
<a id="image-models-heading"></a>
|
||||
<a id="audio-models-heading"></a>
|
||||
<a id="world-models-heading"></a>
|
||||
|
||||
<Cookbook />
|
||||
|
||||
<noscript>
|
||||
Interactive configuration needs JavaScript. Browse the{" "}
|
||||
<a href="https://github.com/hao-ai-lab/FastVideo/tree/main/examples">runnable examples</a> directly.
|
||||
</noscript>
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
title: LTX2 Distilled
|
||||
description: Configure incremental video delivery with the native WebSocket server.
|
||||
mode: frame
|
||||
---
|
||||
|
||||
<Cookbook modelId="ltx2-distilled" />
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
title: FastH3 8-Step V2
|
||||
description: Configure distilled video generation with synchronized audio on CUDA or Apple Silicon.
|
||||
mode: frame
|
||||
---
|
||||
|
||||
<Cookbook modelId="fasth3-8step" />
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
title: FastH3 Trim
|
||||
description: Configure FastH3 Trim NVFP4 and copy inference or serving commands.
|
||||
mode: frame
|
||||
---
|
||||
|
||||
<Cookbook modelId="fasth3-trim" />
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
title: FastWan2.1 1.3B
|
||||
description: Configure distilled text-to-video generation on CUDA.
|
||||
mode: frame
|
||||
---
|
||||
|
||||
<Cookbook modelId="fastwan21" />
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
title: Wan2.1 I2V 14B
|
||||
description: Configure image-to-video generation with a reference image.
|
||||
mode: frame
|
||||
---
|
||||
|
||||
<Cookbook modelId="wan21-i2v" />
|
||||
@@ -0,0 +1,10 @@
|
||||
---
|
||||
title: Fast Models
|
||||
description: Explore FastVideo's model releases and inference recipes.
|
||||
mode: wide
|
||||
---
|
||||
|
||||
<FastModels />
|
||||
|
||||
Model links reviewed October 16, 2026. Browse all releases on
|
||||
[FastVideo's Hugging Face organization](https://huggingface.co/FastVideo).
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: Welcome to FastVideo
|
||||
description: Run, train, and accelerate video generation models with FastVideo.
|
||||
mode: frame
|
||||
---
|
||||
|
||||
<div className="home-page">
|
||||
<header className="home-hero not-prose">
|
||||
<h1 id="welcome-to-fastvideo" className="home-logo">
|
||||
<img src="/assets/logos/logo.svg" alt="Welcome to FastVideo" width={252} height={105} />
|
||||
</h1>
|
||||
<p className="home-description">Run, train, and accelerate video generation models with FastVideo.</p>
|
||||
<nav className="home-project-links" aria-label="About the project">
|
||||
<GitHubStars />
|
||||
<a className="home-lab-link" href="https://haoailab.com/">
|
||||
<img src="/assets/hao-ai-lab.png" width={40} height={24} alt="" aria-hidden="true" />
|
||||
Hao AI Lab
|
||||
</a>
|
||||
<a className="home-models-link" href="/fast-models">Fast Models <span aria-hidden="true">→</span></a>
|
||||
</nav>
|
||||
<div className="home-trendshift">
|
||||
<a href="https://trendshift.io/repositories/14509?utm_source=repository-badge&utm_medium=badge&utm_campaign=badge-repository-14509" target="_blank" rel="noopener noreferrer">
|
||||
<img src="https://trendshift.io/api/badge/repositories/14509" alt="hao-ai-lab/FastVideo | Trendshift" width={250} height={55} />
|
||||
</a>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<include>/_snippets/home-description.mdx</include>
|
||||
|
||||
<a className="home-featured-model not-prose" href="/cookbook/minimax-h3/fasth3-8step">
|
||||
<img src="/assets/logos/fastvideo.webp" alt="" width={40} height={40} />
|
||||
<span>
|
||||
<small>Start with a FastVideo model</small>
|
||||
<strong>FastH3 8-Step V2</strong>
|
||||
</span>
|
||||
<span className="home-featured-action">Configure a recipe →</span>
|
||||
</a>
|
||||
|
||||
<include>/_snippets/home-news.mdx</include>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Quick start" href="/getting_started/quick_start" icon="rocket">
|
||||
Generate your first video with the Python API.
|
||||
</Card>
|
||||
<Card title="Installation" href="/getting_started/installation" icon="download">
|
||||
Set up FastVideo on NVIDIA GPUs or Apple Silicon.
|
||||
</Card>
|
||||
<Card title="Cookbook" href="/cookbook" icon="book-open">
|
||||
Choose a model recipe, customize its settings, and download a serving configuration.
|
||||
</Card>
|
||||
<Card title="Training and distillation" href="/training/overview" icon="layers">
|
||||
Prepare data, fine-tune models, and build faster generators.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
<include>/_snippets/home-details.mdx</include>
|
||||
|
||||
<a id="documentation"></a>
|
||||
|
||||
## Projects and research
|
||||
|
||||
Explore work using FastVideo's code or the attention research behind it.
|
||||
|
||||
<include>/_snippets/home-projects.mdx</include>
|
||||
</div>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 121 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 68 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 73 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
@@ -0,0 +1,835 @@
|
||||
import { useEffect, useId, useState, type ReactNode } from "react";
|
||||
import { Tabs } from "radix-ui";
|
||||
import { modelDirectory, recipePages, sitePath, type DirectoryModel } from "@/lib/cookbook-pages";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { Textarea } from "@/components/ui/textarea";
|
||||
import SourceRecipe from "./SourceRecipe";
|
||||
import ComponentPlan from "@/components/cookbook/ComponentPlan";
|
||||
import HardwarePicker from "@/components/cookbook/HardwarePicker";
|
||||
import ModelHeader from "@/components/cookbook/ModelHeader";
|
||||
import { configAndLaunch, inferenceOutput, sdkCommand } from "@/lib/cookbook-commands";
|
||||
import {
|
||||
api,
|
||||
guideUrl,
|
||||
parseField,
|
||||
types,
|
||||
type Catalog,
|
||||
type Field,
|
||||
type Index,
|
||||
type Result,
|
||||
type Value,
|
||||
} from "@/lib/cookbook";
|
||||
|
||||
export const client = "load";
|
||||
|
||||
function groupTitle(path: string) {
|
||||
const titles: Record<string, string> = {
|
||||
server: "Server",
|
||||
"generator.engine": "GPU resources",
|
||||
generator: "Checkpoint settings",
|
||||
"generator.engine.parallelism": "Parallelism",
|
||||
"generator.engine.offload": "CPU offload",
|
||||
"generator.engine.compile": "Compilation",
|
||||
"default_request.sampling": "Request defaults",
|
||||
streaming: "Streaming",
|
||||
};
|
||||
return titles[path] ?? path.split(".").at(-1)?.replaceAll("_", " ");
|
||||
}
|
||||
|
||||
function errorText(error: unknown) {
|
||||
return error instanceof Error ? error.message : "Unable to load this recipe.";
|
||||
}
|
||||
|
||||
function FieldControl({
|
||||
field,
|
||||
edits,
|
||||
update,
|
||||
failure,
|
||||
}: {
|
||||
field: Field;
|
||||
edits: Record<string, Value>;
|
||||
update: (path: string, raw: string | undefined, field: Field) => void;
|
||||
failure?: string;
|
||||
}) {
|
||||
const id = useId();
|
||||
const kind = types(field.schema);
|
||||
const changed = Object.hasOwn(edits, field.path);
|
||||
const value = changed ? edits[field.path] : field.value;
|
||||
const label = field.schema.title ?? field.path;
|
||||
const choices = field.schema.enum ?? (kind.includes("boolean") ? [true, false] : undefined);
|
||||
const nullable = kind.includes("null");
|
||||
// Keep unfinished numeric and JSON input visible while output is invalid.
|
||||
const [raw, setRaw] = useState<string | undefined>();
|
||||
const displayed = raw ?? (value == null ? "" : typeof value === "string" ? value : JSON.stringify(value));
|
||||
const props = {
|
||||
id,
|
||||
"aria-describedby": `${id}-hint${failure ? ` ${id}-error` : ""}`,
|
||||
"aria-invalid": !!failure,
|
||||
};
|
||||
function edit(text: string) {
|
||||
setRaw(text);
|
||||
update(field.path, text, field);
|
||||
}
|
||||
return (
|
||||
<div className="matrix-field">
|
||||
<div className="matrix-field-label">
|
||||
{choices ? <span>{label}</span> : <label htmlFor={id}>{label}</label>}
|
||||
{changed && (
|
||||
<button
|
||||
type="button"
|
||||
aria-label={`Use baseline for ${label}`}
|
||||
onClick={() => {
|
||||
setRaw(undefined);
|
||||
update(field.path, undefined, field);
|
||||
}}
|
||||
>
|
||||
Reset
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
{choices ? (
|
||||
<div role="group" aria-label={label} className="option-chips">
|
||||
{value === undefined && (
|
||||
<button type="button" aria-pressed={!changed} onClick={() => update(field.path, undefined, field)}>
|
||||
Default
|
||||
</button>
|
||||
)}
|
||||
{choices.map((choice) => (
|
||||
<button
|
||||
key={JSON.stringify(choice)}
|
||||
type="button"
|
||||
aria-pressed={value === choice}
|
||||
onClick={() => update(field.path, JSON.stringify(choice), field)}
|
||||
>
|
||||
{typeof choice === "boolean" ? (choice ? "On" : "Off") : String(choice)}
|
||||
</button>
|
||||
))}
|
||||
{nullable && !choices.includes(null) && (
|
||||
<button
|
||||
type="button"
|
||||
aria-pressed={value === null}
|
||||
onClick={() => update(field.path, "null", { ...field, schema: { ...field.schema, enum: [null] } })}
|
||||
>
|
||||
null
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
) : kind.includes("object") || kind.includes("array") ? (
|
||||
<Textarea
|
||||
{...props}
|
||||
value={displayed}
|
||||
onChange={(event) => edit(event.target.value)}
|
||||
placeholder="Inherited; enter JSON"
|
||||
/>
|
||||
) : (
|
||||
<div className={field.path === "generator.engine.num_gpus" ? "resource-stepper" : ""}>
|
||||
{field.path === "generator.engine.num_gpus" && (
|
||||
<button
|
||||
type="button"
|
||||
aria-label="Remove a GPU"
|
||||
disabled={Number(value) <= (field.schema.minimum ?? 1)}
|
||||
onClick={() => edit(String(Number(value) - 1))}
|
||||
>
|
||||
−
|
||||
</button>
|
||||
)}
|
||||
<Input
|
||||
{...props}
|
||||
type={kind.includes("integer") || kind.includes("number") ? "number" : "text"}
|
||||
value={displayed}
|
||||
placeholder={value === undefined ? "Inherited" : value === null ? "null" : ""}
|
||||
min={field.schema.minimum}
|
||||
max={field.schema.maximum}
|
||||
step={field.schema.multipleOf ?? (kind.includes("integer") ? 1 : "any")}
|
||||
onChange={(event) => edit(event.target.value)}
|
||||
/>
|
||||
{field.path === "generator.engine.num_gpus" && (
|
||||
<button
|
||||
type="button"
|
||||
aria-label="Add a GPU"
|
||||
disabled={Number(value) >= (field.schema.maximum ?? Infinity)}
|
||||
onClick={() => edit(String(Number(value) + 1))}
|
||||
>
|
||||
+
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
{nullable && !choices && (
|
||||
<button
|
||||
className="matrix-null"
|
||||
type="button"
|
||||
aria-label={`Set ${label} to null`}
|
||||
onClick={() => {
|
||||
setRaw(undefined);
|
||||
update(field.path, "null", { ...field, schema: { ...field.schema, enum: [null] } });
|
||||
}}
|
||||
>
|
||||
Use null
|
||||
</button>
|
||||
)}
|
||||
<p id={`${id}-hint`} className="sr-only">
|
||||
{field.schema.description}
|
||||
{value === undefined ? " Inherited from the runtime." : ""}
|
||||
</p>
|
||||
{failure && (
|
||||
<p id={`${id}-error`} className="matrix-field-error" role="alert">
|
||||
{failure}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function Editor({
|
||||
catalog,
|
||||
deployments,
|
||||
selectRuntime,
|
||||
}: {
|
||||
catalog: Catalog;
|
||||
deployments: Index["models"][number]["deployments"];
|
||||
selectRuntime: (id: string) => void;
|
||||
}) {
|
||||
const [advanced, setAdvanced] = useState(false);
|
||||
const [componentDraft, setComponentDraft] = useState(false);
|
||||
const [runMode, setRunMode] = useState("serving");
|
||||
const [sourceImage, setSourceImage] = useState("");
|
||||
const sourceImageId = useId();
|
||||
const [showEnv, setShowEnv] = useState(false);
|
||||
const [showClient, setShowClient] = useState(false);
|
||||
const [outputTab, setOutputTab] = useState("commands");
|
||||
const [hardware, setHardware] = useState(
|
||||
() =>
|
||||
new URL(window.location.href).searchParams.get("hardware") ??
|
||||
(catalog.id.startsWith("fasth3-trim/") ? "spark" : "unspecified"),
|
||||
);
|
||||
const [edits, setEdits] = useState<Record<string, Value>>({});
|
||||
const [parseErrors, setParseErrors] = useState<Record<string, string>>({});
|
||||
const [reset, setReset] = useState(0);
|
||||
const [notice, setNotice] = useState("");
|
||||
let result: Result | undefined;
|
||||
let output: ReturnType<typeof inferenceOutput> | undefined;
|
||||
let failure = "";
|
||||
let failedPath: string | undefined;
|
||||
try {
|
||||
if (componentDraft)
|
||||
throw new Error("Component preview selected. A compatible serving recipe is needed before generating YAML.");
|
||||
if (Object.keys(parseErrors).length) throw new Error("Correct the highlighted fields to generate a configuration.");
|
||||
result = api.resolveConfig(catalog, edits);
|
||||
output =
|
||||
runMode === "serving" ? result : inferenceOutput(result, catalog.workload === "i2v" ? sourceImage : undefined);
|
||||
} catch (error) {
|
||||
failure = errorText(error);
|
||||
failedPath = (error as { path?: string }).path;
|
||||
}
|
||||
const fields = api.getOptions(catalog, catalog.base_config, edits);
|
||||
const launchCommand =
|
||||
output &&
|
||||
(runMode === "sdk"
|
||||
? output.command.replace(/fastvideo generate --config config\.yaml$/, sdkCommand)
|
||||
: output.command);
|
||||
const groups = [...new Set(fields.map((field) => field.path.split(".").slice(0, -1).join(".")))];
|
||||
function update(path: string, raw: string | undefined, field: Field) {
|
||||
setNotice("");
|
||||
const next = { ...edits };
|
||||
const errors = { ...parseErrors };
|
||||
delete errors[path];
|
||||
if (raw === undefined) delete next[path];
|
||||
else {
|
||||
try {
|
||||
next[path] = parseField(field, raw);
|
||||
} catch {
|
||||
errors[path] = "Enter a valid number or JSON value for this field.";
|
||||
}
|
||||
}
|
||||
setEdits(next);
|
||||
setParseErrors(errors);
|
||||
}
|
||||
async function copy(text: string) {
|
||||
try {
|
||||
await navigator.clipboard.writeText(text);
|
||||
setNotice("Copied to clipboard.");
|
||||
} catch {
|
||||
setNotice("Copy failed. Select and copy the text below.");
|
||||
}
|
||||
}
|
||||
function download() {
|
||||
if (!output) return;
|
||||
const url = URL.createObjectURL(new Blob([output.yaml], { type: "application/yaml" }));
|
||||
const link = document.createElement("a");
|
||||
link.href = url;
|
||||
link.download = "config.yaml";
|
||||
link.click();
|
||||
setTimeout(() => URL.revokeObjectURL(url), 1000);
|
||||
setNotice("Downloaded config.yaml.");
|
||||
}
|
||||
const mlx = catalog.runtime.backend === "mlx";
|
||||
const generator = catalog.base_config.generator as { engine?: { quantization?: { transformer_quant?: string } } };
|
||||
const nvfp4 = generator?.engine?.quantization?.transformer_quant === "NVFP4";
|
||||
const hardwareOptions = (
|
||||
mlx
|
||||
? [["apple", "Apple Silicon", "Unified memory"]]
|
||||
: [
|
||||
["b200", "B200", "Data center"],
|
||||
["gb200", "GB200", "Data center"],
|
||||
["h100", "H100", "Data center"],
|
||||
["rtx5090", "RTX 5090", "GeForce"],
|
||||
["rtx", "RTX PRO 6000", "Workstation"],
|
||||
["spark", "DGX Spark", "GB10"],
|
||||
]
|
||||
).filter(([id]) => !nvfp4 || id !== "h100");
|
||||
const selectedHardware = hardwareOptions.find(([value]) => value === hardware) ?? hardwareOptions[0];
|
||||
const model = modelDirectory.find((model) => model.id === catalog.id.split("/")[0])!;
|
||||
const renderField = (field: Field) => (
|
||||
<FieldControl
|
||||
key={`${reset}-${field.path}`}
|
||||
field={field}
|
||||
edits={edits}
|
||||
update={update}
|
||||
failure={parseErrors[field.path] || (field.path === failedPath ? failure : undefined)}
|
||||
/>
|
||||
);
|
||||
function chooseHardware(value: string) {
|
||||
setHardware(value);
|
||||
const url = new URL(window.location.href);
|
||||
url.searchParams.set("hardware", value);
|
||||
window.history.replaceState(null, "", url);
|
||||
}
|
||||
const advancedGroups = groups.filter(
|
||||
(group) => group === "server" || group.includes("parallelism") || group.startsWith("streaming"),
|
||||
);
|
||||
const standardGroups = groups.filter((group) => group !== "generator.engine" && !advancedGroups.includes(group));
|
||||
const groupFields = (group: string) =>
|
||||
fields.filter((field) => field.path.split(".").slice(0, -1).join(".") === group);
|
||||
return (
|
||||
<>
|
||||
<ModelHeader
|
||||
model={model}
|
||||
actions={
|
||||
<>
|
||||
<Button variant="outline" disabled={!output} onClick={() => output && copy(output.yaml)}>
|
||||
Copy YAML
|
||||
</Button>
|
||||
<Button disabled={!output} onClick={download}>
|
||||
Download YAML
|
||||
</Button>
|
||||
</>
|
||||
}
|
||||
/>
|
||||
{
|
||||
<div className="recipe-run-modes option-chips" role="group" aria-label="Run mode">
|
||||
{(mlx ? ["serving"] : ["sdk", "inference", "serving"]).map((mode) => (
|
||||
<button
|
||||
type="button"
|
||||
key={mode}
|
||||
aria-pressed={runMode === mode}
|
||||
onClick={() => {
|
||||
setRunMode(mode);
|
||||
setNotice("");
|
||||
}}
|
||||
>
|
||||
{mode === "sdk" ? "Python SDK" : mode === "inference" ? "CLI" : "Serving"}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
}
|
||||
<section className="recipe-command-box" aria-label="Generated configuration">
|
||||
<Tabs.Root value={outputTab} onValueChange={setOutputTab}>
|
||||
<div className="recipe-command-toolbar">
|
||||
<span className="recipe-unverified">Unverified recipe</span>
|
||||
<div className="recipe-command-tools">
|
||||
<button
|
||||
type="button"
|
||||
disabled={!output}
|
||||
onClick={() => output && copy(configAndLaunch(output.yaml, launchCommand!))}
|
||||
>
|
||||
Copy
|
||||
</button>
|
||||
{runMode === "serving" && (
|
||||
<button
|
||||
type="button"
|
||||
disabled={!result}
|
||||
aria-expanded={showClient}
|
||||
aria-controls={`client-${catalog.id.replaceAll("/", "-")}`}
|
||||
onClick={() => {
|
||||
setOutputTab("commands");
|
||||
setShowClient(!showClient);
|
||||
}}
|
||||
>
|
||||
cURL
|
||||
</button>
|
||||
)}
|
||||
{Object.keys(catalog.env).length > 0 && (
|
||||
<button type="button" aria-pressed={showEnv} onClick={() => setShowEnv(!showEnv)}>
|
||||
Env
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
<Tabs.List className="recipe-tabs" aria-label="Output format">
|
||||
<Tabs.Trigger value="commands">Commands</Tabs.Trigger>
|
||||
<Tabs.Trigger value="yaml">config.yaml</Tabs.Trigger>
|
||||
</Tabs.List>
|
||||
</div>
|
||||
{showEnv && (
|
||||
<pre className="recipe-environment" tabIndex={0} aria-label="Environment variables">
|
||||
<code>
|
||||
{Object.entries(catalog.env)
|
||||
.map(([name, value]) => `${name}=${value}`)
|
||||
.join("\n")}
|
||||
</code>
|
||||
</pre>
|
||||
)}
|
||||
{failure && (
|
||||
<div role="alert" className="recipe-error">
|
||||
<p>{failure}</p>
|
||||
<button type="button" onClick={() => setAdvanced(true)}>
|
||||
Review settings ↓
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
<Tabs.Content value="commands" className="recipe-command-content">
|
||||
{result && output && (
|
||||
<>
|
||||
<div className="recipe-code-heading">
|
||||
<span>{runMode === "sdk" ? "Python SDK" : runMode === "inference" ? "Generate" : "Serve"}</span>
|
||||
</div>
|
||||
<pre
|
||||
tabIndex={0}
|
||||
aria-label={
|
||||
runMode === "sdk" ? "SDK command" : runMode === "inference" ? "Inference command" : "Server command"
|
||||
}
|
||||
>
|
||||
<code>{launchCommand}</code>
|
||||
</pre>
|
||||
<p className="matrix-note">
|
||||
Includes your settings and writes config.yaml. Run from an installed FastVideo checkout.
|
||||
</p>
|
||||
{runMode === "serving" && (
|
||||
<details
|
||||
className="recipe-request-code"
|
||||
id={`client-${catalog.id.replaceAll("/", "-")}`}
|
||||
open={showClient}
|
||||
onToggle={(event) => setShowClient(event.currentTarget.open)}
|
||||
>
|
||||
<summary>{result.websocketUrl ? "Health check and WebSocket" : "Example request"}</summary>
|
||||
<button type="button" onClick={() => copy(result.clientCommand)}>
|
||||
Copy client command
|
||||
</button>
|
||||
<pre tabIndex={0} aria-label="Client command">
|
||||
<code>{result.clientCommand}</code>
|
||||
</pre>
|
||||
{result.websocketUrl && (
|
||||
<p>
|
||||
WebSocket URL: <code>{result.websocketUrl}</code>
|
||||
</p>
|
||||
)}
|
||||
</details>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</Tabs.Content>
|
||||
<Tabs.Content value="yaml" className="recipe-command-content">
|
||||
{output && (
|
||||
<pre tabIndex={0} aria-label="Generated YAML">
|
||||
<code>{output.yaml}</code>
|
||||
</pre>
|
||||
)}
|
||||
</Tabs.Content>
|
||||
</Tabs.Root>
|
||||
<p role="status" className="recipe-copy-status">
|
||||
{notice}
|
||||
</p>
|
||||
</section>
|
||||
<details className="recipe-install">
|
||||
<summary>
|
||||
Installation <span>{catalog.runtime.label.replace(/\s*REST/gi, "")}</span>
|
||||
</summary>
|
||||
{catalog.requirements?.length ? (
|
||||
<ul>
|
||||
{catalog.requirements.map((note) => (
|
||||
<li key={note}>{note}</li>
|
||||
))}
|
||||
</ul>
|
||||
) : null}
|
||||
<div className="recipe-guide-links">
|
||||
{catalog.runtime.install_url && <a href={guideUrl(catalog.runtime.install_url)}>Installation guide ↗</a>}
|
||||
{catalog.guide && <a href={guideUrl(catalog.guide.url)}>Model guide ↗</a>}
|
||||
{catalog.runtime.client_guide_url && <a href={guideUrl(catalog.runtime.client_guide_url)}>Client guide ↗</a>}
|
||||
</div>
|
||||
</details>
|
||||
<section className="recipe-matrix" aria-label="Configuration settings">
|
||||
<MatrixRow title="Hardware">
|
||||
<div className="option-chips hardware-backends" role="group" aria-label="Hardware backend">
|
||||
{deployments.map((deployment) => (
|
||||
<button
|
||||
key={deployment.id}
|
||||
type="button"
|
||||
aria-pressed={catalog.id === deployment.id}
|
||||
onClick={() => {
|
||||
if (catalog.id !== deployment.id) selectRuntime(deployment.id);
|
||||
}}
|
||||
>
|
||||
{deployment.runtime.includes("mlx") ? "MLX" : "CUDA"}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<div className="hardware-resource-grid">
|
||||
<HardwarePicker
|
||||
backend={mlx ? "mlx" : "nvidia"}
|
||||
options={hardwareOptions}
|
||||
value={hardware.startsWith("custom:") ? hardware : selectedHardware[0]}
|
||||
onChange={chooseHardware}
|
||||
/>
|
||||
<div className="matrix-resources">
|
||||
<span className="matrix-resource-title">Resources</span>
|
||||
{fields.filter((field) => field.path === "generator.engine.num_gpus").map(renderField)}
|
||||
{mlx && <p>1 Apple Silicon host</p>}
|
||||
<a
|
||||
href={sitePath(
|
||||
mlx
|
||||
? "/getting_started/installation/mlx"
|
||||
: selectedHardware[0] === "spark"
|
||||
? "/getting_started/installation/spark"
|
||||
: "/getting_started/installation/gpu",
|
||||
)}
|
||||
>
|
||||
Setup guide ↗
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
<p className="matrix-note">Hardware fit has not been verified. GPU count updates the YAML.</p>
|
||||
</MatrixRow>
|
||||
{catalog.workload === "i2v" && runMode !== "serving" && (
|
||||
<MatrixRow title="Inputs">
|
||||
<div className="matrix-field">
|
||||
<label htmlFor={sourceImageId} className="matrix-field-label">
|
||||
Source image path
|
||||
</label>
|
||||
<Input
|
||||
id={sourceImageId}
|
||||
value={sourceImage}
|
||||
required
|
||||
aria-invalid={!sourceImage.trim()}
|
||||
aria-describedby={`${sourceImageId}-hint`}
|
||||
placeholder="/path/to/first-frame.png"
|
||||
onChange={(event) => {
|
||||
setSourceImage(event.target.value);
|
||||
setNotice("");
|
||||
}}
|
||||
/>
|
||||
<p id={`${sourceImageId}-hint`} className="matrix-note">
|
||||
Path to an image on the machine running FastVideo. Included as request.inputs.image_path.
|
||||
</p>
|
||||
</div>
|
||||
</MatrixRow>
|
||||
)}
|
||||
{model.family === "FastH3" && (
|
||||
<MatrixRow title="Components">
|
||||
{model.id === "fasth3-trim" && (
|
||||
<div className="option-chips" aria-label="Recipe components">
|
||||
{["NVFP4 transformer", "50-layer NVFP4 encoder", "Lightweight video VAE", "H3 audio VAE"].map(
|
||||
(name) => (
|
||||
<span className="option-chip-static" key={name}>
|
||||
{name}
|
||||
</span>
|
||||
),
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
<ComponentPlan key={reset} onDraftChange={setComponentDraft} />
|
||||
</MatrixRow>
|
||||
)}
|
||||
{standardGroups.map((group) => (
|
||||
<MatrixRow title={groupTitle(group) ?? group} key={group}>
|
||||
<div className="matrix-options-grid">{groupFields(group).map(renderField)}</div>
|
||||
</MatrixRow>
|
||||
))}
|
||||
<details
|
||||
className="matrix-advanced"
|
||||
open={advanced}
|
||||
onToggle={(event) => setAdvanced(event.currentTarget.open)}
|
||||
>
|
||||
<summary>
|
||||
Advanced <span>Server, topology, and runtime settings</span>
|
||||
</summary>
|
||||
{advancedGroups.map((group) => (
|
||||
<MatrixRow title={groupTitle(group) ?? group} key={group}>
|
||||
<div className="matrix-options-grid">{groupFields(group).map(renderField)}</div>
|
||||
{group.includes("parallelism") && (
|
||||
<button
|
||||
className="matrix-auto"
|
||||
type="button"
|
||||
onClick={() => {
|
||||
const next = { ...edits };
|
||||
for (const field of fields.filter((field) => /parallelism\.(tp_size|sp_size)$/.test(field.path)))
|
||||
next[field.path] = -1;
|
||||
setEdits(next);
|
||||
setReset(reset + 1);
|
||||
setParseErrors(
|
||||
Object.fromEntries(Object.entries(parseErrors).filter(([path]) => !path.includes("parallelism"))),
|
||||
);
|
||||
setComponentDraft(false);
|
||||
setNotice("Parallelism set to automatic.");
|
||||
}}
|
||||
>
|
||||
Use automatic parallelism
|
||||
</button>
|
||||
)}
|
||||
</MatrixRow>
|
||||
))}
|
||||
</details>
|
||||
</section>
|
||||
<div className="recipe-bottom-actions">
|
||||
<span>Copy config and launch together, or download config.yaml first.</span>
|
||||
<Button
|
||||
variant="ghost"
|
||||
onClick={() => {
|
||||
setEdits({});
|
||||
setSourceImage("");
|
||||
setComponentDraft(false);
|
||||
setParseErrors({});
|
||||
setReset(reset + 1);
|
||||
setNotice("Restored the baseline configuration.");
|
||||
}}
|
||||
>
|
||||
Reset changes
|
||||
</Button>
|
||||
</div>
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
function MatrixRow({ title, children }: { title: string; children: ReactNode }) {
|
||||
return (
|
||||
<div className="matrix-row">
|
||||
<h2>{title}</h2>
|
||||
<div className="matrix-row-content">{children}</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default function Cookbook({ modelId }: { modelId?: string }) {
|
||||
const [mounted, setMounted] = useState(false);
|
||||
useEffect(() => setMounted(true), []);
|
||||
const [index, setIndex] = useState<Index>();
|
||||
const [selected, setSelected] = useState("");
|
||||
const [catalog, setCatalog] = useState<Catalog>();
|
||||
const [error, setError] = useState("");
|
||||
const [retry, setRetry] = useState(0);
|
||||
useEffect(() => {
|
||||
const controller = new AbortController();
|
||||
setError("");
|
||||
const url = new URL(
|
||||
`${import.meta.env.BASE_URL.replace(/\/$/, "")}/assets/cookbook-config/index.json`,
|
||||
window.location.origin,
|
||||
);
|
||||
api
|
||||
.loadIndex(url.href, { signal: controller.signal })
|
||||
.then((data) => {
|
||||
if (!data.models.some((model) => model.deployments.length)) {
|
||||
throw new Error("No cookbook deployments are available. Check the catalog and retry.");
|
||||
}
|
||||
setIndex(data);
|
||||
const requested = new URL(window.location.href).searchParams.get("deployment");
|
||||
const requestedModel = data.models.find((model) =>
|
||||
model.deployments.some((deployment) => deployment.id === requested),
|
||||
);
|
||||
if (!modelId && requestedModel) {
|
||||
const page = recipePages.find((page) => page.id === requestedModel.id);
|
||||
if (page) {
|
||||
window.location.replace(`${sitePath(page.path)}${window.location.search}`);
|
||||
return;
|
||||
}
|
||||
}
|
||||
const deployments = data.models
|
||||
.filter((model) => !modelId || model.id === modelId)
|
||||
.flatMap((model) => model.deployments);
|
||||
setSelected(deployments.some((item) => item.id === requested) ? requested! : (deployments[0]?.id ?? ""));
|
||||
})
|
||||
.catch((error) => {
|
||||
if (!controller.signal.aborted) setError(errorText(error));
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, [retry, modelId]);
|
||||
useEffect(() => {
|
||||
if (!index || !selected || !modelId || !recipePages.some((page) => page.id === modelId)) return;
|
||||
const controller = new AbortController();
|
||||
setCatalog(undefined);
|
||||
setError("");
|
||||
api
|
||||
.loadDeployment(index, selected, { signal: controller.signal })
|
||||
.then((data) => {
|
||||
if (!controller.signal.aborted) setCatalog(data);
|
||||
})
|
||||
.catch((error) => {
|
||||
if (!controller.signal.aborted) setError(errorText(error));
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, [index, selected, modelId]);
|
||||
function select(value: string) {
|
||||
setSelected(value);
|
||||
setCatalog(undefined);
|
||||
setError("");
|
||||
const url = new URL(window.location.href);
|
||||
url.searchParams.set("deployment", value);
|
||||
window.history.replaceState(null, "", url);
|
||||
}
|
||||
const model = index?.models.find((model) => model.id === modelId);
|
||||
if (!modelId) return <RecipeDirectory index={index} error={error} retry={() => setRetry(retry + 1)} />;
|
||||
const directoryModel = modelDirectory.find((model) => model.id === modelId);
|
||||
if (directoryModel && !directoryModel.configurable)
|
||||
return mounted ? (
|
||||
<SourceRecipe model={directoryModel} />
|
||||
) : (
|
||||
<div className="cookbook not-prose recipe-detail">
|
||||
<ModelHeader model={directoryModel} />
|
||||
<p>{directoryModel.description}</p>
|
||||
<p className="recipe-loading">Loading recipe…</p>
|
||||
<a href={`https://github.com/hao-ai-lab/FastVideo/blob/main/${directoryModel.recipes[0].source}`}>
|
||||
View maintained example ↗
|
||||
</a>
|
||||
</div>
|
||||
);
|
||||
return (
|
||||
<div className="cookbook not-prose recipe-detail">
|
||||
{error ? (
|
||||
<div role="alert" className="space-y-3 rounded-lg border p-4">
|
||||
<p>{error}</p>
|
||||
<Button
|
||||
variant="outline"
|
||||
onClick={() => {
|
||||
setCatalog(undefined);
|
||||
setIndex(undefined);
|
||||
setRetry(retry + 1);
|
||||
}}
|
||||
>
|
||||
Retry loading recipes
|
||||
</Button>
|
||||
</div>
|
||||
) : catalog?.id === selected ? (
|
||||
<Editor key={catalog.id} catalog={catalog} deployments={model?.deployments ?? []} selectRuntime={select} />
|
||||
) : (
|
||||
<div className="recipe-loading">
|
||||
{directoryModel && <ModelHeader model={directoryModel} />}
|
||||
<div className="option-chips" role="group" aria-label="Hardware backend">
|
||||
{model?.deployments.map((deployment) => (
|
||||
<button
|
||||
key={deployment.id}
|
||||
type="button"
|
||||
aria-pressed={selected === deployment.id}
|
||||
onClick={() => select(deployment.id)}
|
||||
>
|
||||
{deployment.runtime.includes("mlx") ? "MLX" : "CUDA"}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<p role="status">Loading deployment settings…</p>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function RecipeDirectory({ index, error, retry }: { index?: Index; error: string; retry: () => void }) {
|
||||
const [query, setQuery] = useState("");
|
||||
const [category, setCategory] = useState("All models");
|
||||
const [family, setFamily] = useState("All families");
|
||||
const visible = modelDirectory.filter(
|
||||
(model) =>
|
||||
(category === "All models" || model.category === category) &&
|
||||
(family === "All families" || model.family === family) &&
|
||||
`${model.title} ${model.family} ${model.tags.join(" ")}`.toLowerCase().includes(query.trim().toLowerCase()),
|
||||
);
|
||||
return (
|
||||
<div className="cookbook not-prose recipe-directory">
|
||||
<div className="recipe-directory-tools">
|
||||
<div className="recipe-family-filters" aria-label="Filter recipes by category">
|
||||
{["All models", "Video", "Image", "Audio", "World"].map((value) => (
|
||||
<button
|
||||
key={value}
|
||||
type="button"
|
||||
aria-pressed={category === value}
|
||||
onClick={() => {
|
||||
setCategory(value);
|
||||
setFamily("All families");
|
||||
}}
|
||||
>
|
||||
{value}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<Input
|
||||
aria-label="Search recipes"
|
||||
type="search"
|
||||
placeholder="Search models…"
|
||||
value={query}
|
||||
onChange={(event) => setQuery(event.target.value)}
|
||||
/>
|
||||
</div>
|
||||
<div className="recipe-directory-subbar">
|
||||
<p role="status">{visible.length} models</p>
|
||||
<select aria-label="Model family" value={family} onChange={(event) => setFamily(event.target.value)}>
|
||||
<option>All families</option>
|
||||
{[
|
||||
...new Set(
|
||||
modelDirectory
|
||||
.filter((model) => category === "All models" || model.category === category)
|
||||
.map((model) => model.family),
|
||||
),
|
||||
].map((family) => (
|
||||
<option key={family}>{family}</option>
|
||||
))}
|
||||
</select>
|
||||
</div>
|
||||
{error && (
|
||||
<div role="alert" className="recipe-error">
|
||||
<p>{error}</p>
|
||||
<Button variant="outline" onClick={retry}>
|
||||
Retry loading recipes
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
<div className="recipe-card-grid">
|
||||
{visible.map((model) => (
|
||||
<a className="recipe-card" key={model.id} href={sitePath(model.path)}>
|
||||
<div className="recipe-card-top">
|
||||
<img src={sitePath(`/assets/logos/${model.logo}.webp`)} alt="" width="40" height="40" />
|
||||
<span className="recipe-card-kicker">{model.provider}</span>
|
||||
{model.featured && <span className="recipe-featured">Featured</span>}
|
||||
</div>
|
||||
<h2>{model.title}</h2>
|
||||
<div className="recipe-tags">
|
||||
<span>{model.category}</span>
|
||||
{model.tags.slice(0, 2).map((tag) => (
|
||||
<span key={tag}>{tag}</span>
|
||||
))}
|
||||
</div>
|
||||
<footer>
|
||||
<span>
|
||||
{model.platforms.map((platform) => (platform === "Apple Silicon" ? "MLX" : platform)).join(" · ")}
|
||||
</span>
|
||||
<span>{model.configurable ? "Configure →" : "View recipe →"}</span>
|
||||
</footer>
|
||||
</a>
|
||||
))}
|
||||
</div>
|
||||
{!visible.length && (
|
||||
<div className="recipe-empty">
|
||||
<h2>No matching recipes</h2>
|
||||
<Button
|
||||
variant="outline"
|
||||
onClick={() => {
|
||||
setQuery("");
|
||||
setCategory("All models");
|
||||
setFamily("All families");
|
||||
}}
|
||||
>
|
||||
Clear filters
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
import { useId, useState } from "react";
|
||||
import { families } from "@/lib/fast-models";
|
||||
|
||||
export const client = "load";
|
||||
const local = (path: string) => `${import.meta.env.BASE_URL.replace(/\/$/, "")}${path}`;
|
||||
|
||||
export default function FastModels() {
|
||||
const [family, setFamily] = useState("All models");
|
||||
const [query, setQuery] = useState("");
|
||||
const id = useId();
|
||||
const visible = families
|
||||
.map((group) => ({
|
||||
...group,
|
||||
models: group.models.filter(
|
||||
(model) =>
|
||||
(family === "All models" || family === group.name) &&
|
||||
`${group.name} ${group.summary} ${model.id}`.toLowerCase().includes(query.trim().toLowerCase()),
|
||||
),
|
||||
}))
|
||||
.filter((group) => group.models.length);
|
||||
const count = visible.reduce((total, group) => total + group.models.length, 0);
|
||||
return (
|
||||
<div className="models-page not-prose">
|
||||
<section className="model-families" aria-label="Model families">
|
||||
{families.map((group) => (
|
||||
<article className="model-family" key={group.name}>
|
||||
<a href={group.collection} className="model-family-visual" aria-label={`${group.name} collection`}>
|
||||
<span aria-hidden="true">{group.mark}</span>
|
||||
<img src={local("/icon.svg")} alt="" width="24" height="24" />
|
||||
</a>
|
||||
<div className="model-family-body">
|
||||
<h2>{group.name}</h2>
|
||||
<p>{group.summary}</p>
|
||||
<p className="model-family-detail">{group.detail}</p>
|
||||
<div className="model-family-links">
|
||||
<a href={group.collection}>
|
||||
View collection <span aria-hidden="true">↗</span>
|
||||
</a>
|
||||
<a href={local(group.guide)}>
|
||||
{group.name === "FastMetal" ? "MLX setup" : "Recipes and setup"} <span aria-hidden="true">→</span>
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</article>
|
||||
))}
|
||||
</section>
|
||||
<section className="model-catalog" aria-labelledby="checkpoint-catalog">
|
||||
<header className="model-catalog-heading">
|
||||
<div>
|
||||
<h2 id="checkpoint-catalog">Browse checkpoints</h2>
|
||||
<p>Find the export your runtime needs.</p>
|
||||
</div>
|
||||
<a href="https://huggingface.co/collections/FastVideo/fastvideo-fasth3">
|
||||
Earlier FastH3 releases <span aria-hidden="true">↗</span>
|
||||
</a>
|
||||
</header>
|
||||
<div className="model-catalog-controls">
|
||||
<div className="model-filters" role="group" aria-label="Filter by model family">
|
||||
{["All models", ...families.map((group) => group.name)].map((name) => (
|
||||
<button key={name} type="button" aria-pressed={family === name} onClick={() => setFamily(name)}>
|
||||
{name}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<div className="model-search">
|
||||
<label htmlFor={id}>Search checkpoints</label>
|
||||
<input
|
||||
id={id}
|
||||
type="search"
|
||||
placeholder="Name, precision, or runtime…"
|
||||
value={query}
|
||||
onChange={(event) => setQuery(event.target.value)}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<p className="model-result-count" role="status">
|
||||
{count} {count === 1 ? "checkpoint" : "checkpoints"}
|
||||
</p>
|
||||
{visible.map((group) => (
|
||||
<section className="model-checkpoints" key={group.name} aria-labelledby={`catalog-${group.name}`}>
|
||||
<h3 id={`catalog-${group.name}`}>{group.name}</h3>
|
||||
<ul>
|
||||
{group.models.map((model) => (
|
||||
<li key={model.id}>
|
||||
<div>
|
||||
<a href={model.url}>{model.id.replace(/^FastVideo-/, "").replace(/-Diffusers$/, "")}</a>
|
||||
<code>FastVideo/{model.id}</code>
|
||||
</div>
|
||||
<a className="model-checkpoint-link" href={model.url} aria-label={`Open ${model.id} on Hugging Face`}>
|
||||
Hugging Face <span aria-hidden="true">↗</span>
|
||||
</a>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</section>
|
||||
))}
|
||||
{!count && (
|
||||
<div className="model-empty">
|
||||
<p>No checkpoints match these filters.</p>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => {
|
||||
setQuery("");
|
||||
setFamily("All models");
|
||||
}}
|
||||
>
|
||||
Clear filters
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
import { useEffect, useState } from "react";
|
||||
|
||||
export const client = "load";
|
||||
|
||||
const repository = "https://github.com/hao-ai-lab/FastVideo";
|
||||
const endpoint = "https://api.github.com/repos/hao-ai-lab/FastVideo";
|
||||
const compact = new Intl.NumberFormat("en", { notation: "compact", maximumFractionDigits: 1 });
|
||||
const exact = new Intl.NumberFormat("en");
|
||||
|
||||
export default function GitHubStars() {
|
||||
const [stars, setStars] = useState<number | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
const controller = new AbortController();
|
||||
async function refresh() {
|
||||
try {
|
||||
const response = await fetch(endpoint, {
|
||||
headers: { Accept: "application/vnd.github+json" },
|
||||
signal: AbortSignal.any([controller.signal, AbortSignal.timeout(5000)]),
|
||||
});
|
||||
if (!response.ok) return;
|
||||
const data = await response.json();
|
||||
if (Number.isSafeInteger(data.stargazers_count) && data.stargazers_count >= 0 && !controller.signal.aborted) {
|
||||
setStars(data.stargazers_count);
|
||||
}
|
||||
} catch {
|
||||
// Keep the link usable when offline or when the public API is rate limited.
|
||||
}
|
||||
}
|
||||
void refresh();
|
||||
const timer = setInterval(
|
||||
() => {
|
||||
if (document.visibilityState === "visible") void refresh();
|
||||
},
|
||||
5 * 60 * 1000,
|
||||
);
|
||||
return () => {
|
||||
controller.abort();
|
||||
clearInterval(timer);
|
||||
};
|
||||
}, []);
|
||||
|
||||
const label = stars === null ? "Star FastVideo on GitHub" : `FastVideo on GitHub, ${exact.format(stars)} stars`;
|
||||
return (
|
||||
<a className="home-github-stars" href={repository} aria-label={label} title={label}>
|
||||
<svg width="17" height="17" viewBox="0 0 16 16" fill="currentColor" aria-hidden="true">
|
||||
<path d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38v-1.49c-2.23.48-2.7-1.08-2.7-1.08-.36-.92-.89-1.17-.89-1.17-.73-.5.06-.49.06-.49.8.06 1.22.82 1.22.82.71 1.22 1.87.87 2.33.67.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.96 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82A7.65 7.65 0 0 1 8 3.72c.68 0 1.36.09 2 .27 1.53-1.03 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.28.82 2.15 0 3.08-1.87 3.75-3.65 3.95.29.25.54.74.54 1.49v2.21c0 .21.15.46.55.38A8.01 8.01 0 0 0 16 8c0-4.42-3.58-8-8-8Z" />
|
||||
</svg>
|
||||
<svg className="home-star" width="16" height="16" viewBox="0 0 24 24" fill="currentColor" aria-hidden="true">
|
||||
<path d="m12 2.5 2.94 5.96 6.58.96-4.76 4.64 1.12 6.55L12 17.52l-5.88 3.09 1.12-6.55L2.48 9.42l6.58-.96L12 2.5Z" />
|
||||
</svg>
|
||||
<span>{stars === null ? "Star" : compact.format(stars).toLowerCase()}</span>
|
||||
</a>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
import { modelDirectory, sitePath } from "@/lib/cookbook-pages";
|
||||
|
||||
/** Family runbooks link to the same model pages as the cookbook catalog. */
|
||||
export default function RecipeFamily({ family }: { family: string }) {
|
||||
const models = modelDirectory.filter(
|
||||
(model) =>
|
||||
model.recipes.some((recipe) => recipe.family === family) ||
|
||||
(family === "minimax_h3" && model.family === "FastH3"),
|
||||
);
|
||||
return (
|
||||
<div className="recipe-directory not-prose">
|
||||
<a className="recipe-back" href={sitePath("/cookbook")}>
|
||||
← All models
|
||||
</a>
|
||||
<div className="recipe-card-grid">
|
||||
{models.map((model) => (
|
||||
<a className="recipe-card" key={model.id} href={sitePath(model.path)}>
|
||||
<div className="recipe-card-top">
|
||||
<img src={sitePath(`/assets/logos/${model.logo}.webp`)} alt="" width="40" height="40" />
|
||||
<span className="recipe-card-kicker">{model.provider}</span>
|
||||
</div>
|
||||
<h2>{model.title}</h2>
|
||||
<div className="recipe-tags">
|
||||
<span>{model.category}</span>
|
||||
{model.tags.slice(0, 2).map((tag) => (
|
||||
<span key={tag}>{tag}</span>
|
||||
))}
|
||||
</div>
|
||||
<footer>
|
||||
<span>
|
||||
{model.platforms.map((platform) => (platform === "Apple Silicon" ? "MLX" : platform)).join(" · ")}
|
||||
</span>
|
||||
<span>{model.configurable ? "Configure →" : "View recipe →"}</span>
|
||||
</footer>
|
||||
</a>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,526 @@
|
||||
import { useEffect, useState } from "react";
|
||||
import { Tabs } from "radix-ui";
|
||||
import { parse, stringify } from "yaml";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import ModelHeader from "@/components/cookbook/ModelHeader";
|
||||
import HardwarePicker from "@/components/cookbook/HardwarePicker";
|
||||
import { configAndLaunch, fileAndLaunch, sdkCommand, pythonSource } from "@/lib/cookbook-commands";
|
||||
import { sitePath, type DirectoryModel } from "@/lib/cookbook-pages";
|
||||
import { api } from "@/lib/cookbook";
|
||||
|
||||
interface Control {
|
||||
name: string;
|
||||
label: string;
|
||||
value: string | number | boolean;
|
||||
spans?: number[][];
|
||||
path?: string[];
|
||||
flag?: string;
|
||||
integer?: boolean;
|
||||
booleanOptional?: boolean;
|
||||
choices?: (string | number | { value: string | number; label: string })[];
|
||||
}
|
||||
interface Example {
|
||||
source: string;
|
||||
controls: Control[];
|
||||
}
|
||||
const labels: Record<string, string> = {
|
||||
prompt: "Prompt",
|
||||
num_gpus: "GPU count",
|
||||
height: "Height",
|
||||
width: "Width",
|
||||
num_frames: "Frames",
|
||||
seed: "Seed",
|
||||
num_inference_steps: "Inference steps",
|
||||
guidance_scale: "Guidance scale",
|
||||
fps: "FPS",
|
||||
host: "Host",
|
||||
port: "Port",
|
||||
served_model_name: "Served model name",
|
||||
};
|
||||
|
||||
/** Every source page uses the model editor; settings modify the maintained example itself. */
|
||||
export default function SourceRecipe({ model }: { model: DirectoryModel }) {
|
||||
const [selected, setSelected] = useState(model.recipes[0].id);
|
||||
const [mode, setMode] = useState("sdk");
|
||||
const [loaded, setLoaded] = useState<{ url: string; example: Example }>();
|
||||
const [notice, setNotice] = useState("");
|
||||
const [showEnv, setShowEnv] = useState(false);
|
||||
const [showClient, setShowClient] = useState(false);
|
||||
const [outputTab, setOutputTab] = useState("commands");
|
||||
const [error, setError] = useState("");
|
||||
const [edits, setEdits] = useState<Record<number, string | number | boolean>>({});
|
||||
const [hardware, setHardware] = useState("unspecified");
|
||||
useEffect(() => {
|
||||
setHardware(new URL(window.location.href).searchParams.get("hardware") ?? "unspecified");
|
||||
}, []);
|
||||
const recipe = model.recipes.find((recipe) => recipe.id === selected)!;
|
||||
const yamlSource =
|
||||
mode === "serving" ? recipe.serving?.source : /\.ya?ml$/.test(recipe.source) ? recipe.source : undefined;
|
||||
const sourceUrl = sitePath(
|
||||
yamlSource
|
||||
? `/assets/cookbook-sources/${recipe.id}-${mode === "serving" ? "serving" : "inference"}.yaml`
|
||||
: `/assets/cookbook-examples/${recipe.id}.json`,
|
||||
);
|
||||
const example = loaded?.url === sourceUrl ? loaded.example : undefined;
|
||||
useEffect(() => {
|
||||
const controller = new AbortController();
|
||||
setError("");
|
||||
fetch(sourceUrl, { signal: controller.signal })
|
||||
.then(async (response) => {
|
||||
if (!response.ok) throw new Error(`Example could not be loaded: HTTP ${response.status}.`);
|
||||
let data: Example;
|
||||
if (!yamlSource) data = await response.json();
|
||||
else {
|
||||
const source = await response.text();
|
||||
const config = parse(source);
|
||||
const controls: Control[] = [];
|
||||
function visit(value: Record<string, unknown>, path: string[] = []) {
|
||||
for (const [name, item] of Object.entries(value)) {
|
||||
if (item && typeof item === "object" && !Array.isArray(item))
|
||||
visit(item as Record<string, unknown>, [...path, name]);
|
||||
else if (labels[name] && ["string", "number", "boolean"].includes(typeof item))
|
||||
controls.push({
|
||||
name,
|
||||
label: labels[name],
|
||||
value: item as Control["value"],
|
||||
integer: !["guidance_scale", "audio_end_in_s"].includes(name),
|
||||
path: [...path, name],
|
||||
});
|
||||
}
|
||||
}
|
||||
visit(config);
|
||||
data = { source, controls };
|
||||
}
|
||||
if (!yamlSource) {
|
||||
for (const knob of recipe.knobs ?? []) {
|
||||
const field = data.controls.find((field) => field.name === knob.key);
|
||||
if (field) field.choices = knob.options;
|
||||
else
|
||||
data.controls.push({
|
||||
name: knob.key,
|
||||
label: labels[knob.key] ?? knob.label,
|
||||
value: knob.default,
|
||||
flag: knob.flag,
|
||||
spans: [],
|
||||
choices: knob.options,
|
||||
});
|
||||
}
|
||||
}
|
||||
if (!controller.signal.aborted) setLoaded({ url: sourceUrl, example: data });
|
||||
})
|
||||
.catch((error) => {
|
||||
if (!controller.signal.aborted) setError(error.message);
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, [sourceUrl, yamlSource, recipe]);
|
||||
function selectMode(next: string) {
|
||||
setMode(next);
|
||||
if ((next === "serving") !== (mode === "serving")) setEdits({});
|
||||
setNotice("");
|
||||
}
|
||||
async function copy(value: string) {
|
||||
try {
|
||||
await navigator.clipboard.writeText(value);
|
||||
setNotice("Copied to clipboard.");
|
||||
} catch {
|
||||
setNotice("Copy failed. Select and copy the text below.");
|
||||
}
|
||||
}
|
||||
let source = example?.source ?? "";
|
||||
let failure = "";
|
||||
for (const [key, value] of Object.entries(edits)) {
|
||||
const field = example?.controls[Number(key)];
|
||||
if (
|
||||
field &&
|
||||
typeof field.value === "number" &&
|
||||
(!Number.isFinite(value) ||
|
||||
((field.integer ?? Number.isInteger(field.value)) && !Number.isInteger(value)) ||
|
||||
(field.name === "num_gpus" && Number(value) < 1))
|
||||
)
|
||||
failure = "Enter valid numeric settings before copying the example.";
|
||||
}
|
||||
if (example && yamlSource) {
|
||||
const config = parse(source);
|
||||
for (const [key, value] of Object.entries(edits)) {
|
||||
const path = example.controls[Number(key)].path!;
|
||||
const parent = path.slice(0, -1).reduce((item, name) => item[name], config);
|
||||
parent[path.at(-1)!] = value;
|
||||
}
|
||||
source = stringify(config);
|
||||
} else if (example) {
|
||||
source = pythonSource(source, example.controls, edits);
|
||||
}
|
||||
const baselineCommand =
|
||||
mode === "serving" && recipe.serving
|
||||
? `${recipe.serving.env ? `${recipe.serving.env} ` : ""}fastvideo serve --config ${recipe.serving.source}`
|
||||
: recipe.command;
|
||||
const localFile = recipe.source.replace(/[^/]+$/, `cookbook_${recipe.id.replaceAll("-", "_")}.py`);
|
||||
const command = yamlSource
|
||||
? baselineCommand.replace(/--config\s+\S+/, "--config config.yaml")
|
||||
: baselineCommand.replace(recipe.source, localFile) +
|
||||
Object.entries(edits)
|
||||
.filter(([key]) => example?.controls[Number(key)].flag)
|
||||
.map(([key, value]) => {
|
||||
const field = example!.controls[Number(key)];
|
||||
return typeof value === "boolean" && field.booleanOptional
|
||||
? ` ${value ? field.flag : field.flag!.replace(/^--/, "--no-")}`
|
||||
: ` ${field.flag} '${String(value).replaceAll("'", "'\"'\"'")}'`;
|
||||
})
|
||||
.join("");
|
||||
const sdk = mode === "sdk" && !!yamlSource;
|
||||
const config = sdk && source ? parse(source) : undefined;
|
||||
const displayedSource = config
|
||||
? stringify({ generator: config.generator, request: config.request ?? config.default_request })
|
||||
: source;
|
||||
const launch = sdk ? command.replace(/fastvideo generate --config config\.yaml$/, sdkCommand) : command;
|
||||
const ready = !!example && !failure;
|
||||
const clientCommand =
|
||||
mode === "serving" && ready
|
||||
? api.sampleRequest(
|
||||
{
|
||||
runtime: { interface: "rest", server_defaults: { host: "127.0.0.1", port: 8000 } },
|
||||
model: { id: model.model },
|
||||
workload: "t2v",
|
||||
},
|
||||
parse(source),
|
||||
).command
|
||||
: undefined;
|
||||
const launchScript = yamlSource
|
||||
? configAndLaunch(displayedSource, launch)
|
||||
: fileAndLaunch(localFile, displayedSource, launch);
|
||||
const environment = [...baselineCommand.matchAll(/(?:^|\s)([A-Z][A-Z0-9_]*=(?:'[^']*'|"[^"]*"|[^\s]+))/g)]
|
||||
.map((match) => match[1])
|
||||
.join("\n");
|
||||
const mlx = recipe.gpu_types.includes("Apple Silicon");
|
||||
const devices = mlx
|
||||
? [["apple", "Apple Silicon", "Unified memory"]]
|
||||
: [
|
||||
["b200", "B200", "Data center"],
|
||||
["h100", "H100", "Data center"],
|
||||
["rtx5090", "RTX 5090", "GeForce"],
|
||||
["rtx", "RTX PRO 6000", "Workstation"],
|
||||
["spark", "DGX Spark", "GB10"],
|
||||
];
|
||||
const chooseHardware = (value: string) => {
|
||||
setHardware(value);
|
||||
const url = new URL(window.location.href);
|
||||
url.searchParams.set("hardware", value);
|
||||
window.history.replaceState(null, "", url);
|
||||
};
|
||||
const control = (field: Control, index: number) => {
|
||||
const value = Object.hasOwn(edits, index) ? edits[index] : field.value;
|
||||
return (
|
||||
<div className="matrix-field" key={index}>
|
||||
{field.choices || typeof value === "boolean" ? (
|
||||
<span className="matrix-field-label">{field.label}</span>
|
||||
) : (
|
||||
<label htmlFor={`source-${index}`} className="matrix-field-label">
|
||||
{field.label}
|
||||
</label>
|
||||
)}
|
||||
{field.choices ? (
|
||||
<div role="group" aria-label={field.label} className="option-chips">
|
||||
{field.choices.map((option) => {
|
||||
const choice = typeof option === "object" ? option.value : option;
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
key={String(choice)}
|
||||
aria-pressed={value === choice}
|
||||
onClick={() => setEdits({ ...edits, [index]: choice })}
|
||||
>
|
||||
{typeof option === "object" ? option.label : String(option)}
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
) : typeof value === "boolean" ? (
|
||||
<div role="group" aria-label={field.label} className="option-chips">
|
||||
{[true, false].map((choice) => (
|
||||
<button
|
||||
type="button"
|
||||
key={String(choice)}
|
||||
aria-pressed={value === choice}
|
||||
onClick={() => setEdits({ ...edits, [index]: choice })}
|
||||
>
|
||||
{choice ? "On" : "Off"}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
) : (
|
||||
<Input
|
||||
id={`source-${index}`}
|
||||
type={typeof field.value === "number" ? "number" : "text"}
|
||||
value={typeof value === "number" && Number.isNaN(value) ? "" : String(value)}
|
||||
step={typeof field.value === "number" && (field.integer ?? Number.isInteger(field.value)) ? 1 : "any"}
|
||||
onChange={(event) =>
|
||||
setEdits({
|
||||
...edits,
|
||||
[index]:
|
||||
typeof field.value === "number"
|
||||
? event.target.value === ""
|
||||
? NaN
|
||||
: Number(event.target.value)
|
||||
: event.target.value,
|
||||
})
|
||||
}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
};
|
||||
return (
|
||||
<div className="cookbook not-prose recipe-detail">
|
||||
<ModelHeader
|
||||
model={model}
|
||||
actions={
|
||||
<>
|
||||
<Button variant="outline" disabled={!ready} onClick={() => copy(displayedSource)}>
|
||||
Copy {yamlSource ? "YAML" : "Python"}
|
||||
</Button>
|
||||
<Button
|
||||
disabled={!ready}
|
||||
onClick={() => {
|
||||
const url = URL.createObjectURL(new Blob([displayedSource], { type: "text/plain" }));
|
||||
const link = document.createElement("a");
|
||||
link.href = url;
|
||||
link.download = yamlSource ? "config.yaml" : localFile.split("/").at(-1)!;
|
||||
link.click();
|
||||
setTimeout(() => URL.revokeObjectURL(url), 1000);
|
||||
}}
|
||||
>
|
||||
Download {yamlSource ? "YAML" : "Python"}
|
||||
</Button>
|
||||
</>
|
||||
}
|
||||
/>
|
||||
<div className="recipe-run-modes option-chips" role="group" aria-label="Run mode">
|
||||
{["sdk", "inference", ...(recipe.serving ? ["serving"] : [])].map((entry) => (
|
||||
<button type="button" key={entry} aria-pressed={mode === entry} onClick={() => selectMode(entry)}>
|
||||
{entry === "sdk" ? "Python SDK" : entry === "inference" ? "CLI" : "Serving"}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<section className="recipe-command-box" aria-label="Source recipe">
|
||||
<Tabs.Root value={outputTab} onValueChange={setOutputTab}>
|
||||
<div className="recipe-command-toolbar">
|
||||
<span className="recipe-unverified">Unverified recipe</span>
|
||||
<div className="recipe-command-tools">
|
||||
<button type="button" disabled={!ready} onClick={() => copy(launchScript)}>
|
||||
Copy
|
||||
</button>
|
||||
{mode === "serving" && (
|
||||
<button
|
||||
type="button"
|
||||
disabled={!clientCommand}
|
||||
aria-expanded={showClient}
|
||||
aria-controls="source-client-command"
|
||||
onClick={() => {
|
||||
setOutputTab("commands");
|
||||
setShowClient(!showClient);
|
||||
}}
|
||||
>
|
||||
cURL
|
||||
</button>
|
||||
)}
|
||||
{environment && (
|
||||
<button type="button" aria-pressed={showEnv} onClick={() => setShowEnv(!showEnv)}>
|
||||
Env
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
<Tabs.List className="recipe-tabs" aria-label="Output format">
|
||||
<Tabs.Trigger value="commands">Commands</Tabs.Trigger>
|
||||
<Tabs.Trigger value="source">{yamlSource ? "config.yaml" : "Python example"}</Tabs.Trigger>
|
||||
</Tabs.List>
|
||||
</div>
|
||||
{showEnv && (
|
||||
<pre className="recipe-environment" tabIndex={0} aria-label="Environment variables">
|
||||
<code>{environment}</code>
|
||||
</pre>
|
||||
)}
|
||||
{(error || failure) && (
|
||||
<p className="recipe-error" role="alert">
|
||||
{error || failure}
|
||||
</p>
|
||||
)}
|
||||
<Tabs.Content value="commands" className="recipe-command-content">
|
||||
<div className="recipe-code-heading">
|
||||
<span>{mode === "serving" ? "Serve" : mode === "sdk" ? "Python SDK" : "Generate"}</span>
|
||||
</div>
|
||||
<pre tabIndex={0} aria-label={mode === "serving" ? "Server command" : "Inference command"}>
|
||||
<code>{launch}</code>
|
||||
</pre>
|
||||
<p className="matrix-note">Includes your settings. Run from an installed FastVideo checkout.</p>
|
||||
{clientCommand && (
|
||||
<details
|
||||
className="recipe-request-code"
|
||||
id="source-client-command"
|
||||
open={showClient}
|
||||
onToggle={(event) => setShowClient(event.currentTarget.open)}
|
||||
>
|
||||
<summary>Example request</summary>
|
||||
<button type="button" onClick={() => copy(clientCommand)}>
|
||||
Copy client command
|
||||
</button>
|
||||
<pre tabIndex={0} aria-label="Client command">
|
||||
<code>{clientCommand}</code>
|
||||
</pre>
|
||||
</details>
|
||||
)}
|
||||
</Tabs.Content>
|
||||
<Tabs.Content value="source" className="recipe-command-content">
|
||||
<pre tabIndex={0} aria-label="Source example">
|
||||
<code>{displayedSource || "Loading example…"}</code>
|
||||
</pre>
|
||||
</Tabs.Content>
|
||||
</Tabs.Root>
|
||||
<p role="status" className="recipe-copy-status">
|
||||
{notice}
|
||||
</p>
|
||||
</section>
|
||||
<details className="recipe-install">
|
||||
<summary>
|
||||
Installation and setup <span>{mlx ? "MLX" : "CUDA"}</span>
|
||||
</summary>
|
||||
{mode === "serving" && recipe.serving && (
|
||||
<pre>
|
||||
<code>{recipe.serving.install}</code>
|
||||
</pre>
|
||||
)}
|
||||
{mode === "serving" && recipe.serving?.prepare && (
|
||||
<>
|
||||
<pre>
|
||||
<code>{recipe.serving.prepare}</code>
|
||||
</pre>
|
||||
<Button variant="outline" onClick={() => copy(recipe.serving!.prepare!)}>
|
||||
Copy preparation commands
|
||||
</Button>
|
||||
</>
|
||||
)}
|
||||
<a href={sitePath(model.familyPath)}>Model setup guide ↗</a>
|
||||
</details>
|
||||
<section className="recipe-matrix" aria-label="Configuration settings">
|
||||
<div className="matrix-row">
|
||||
<h2>Hardware</h2>
|
||||
<div className="matrix-row-content">
|
||||
<div className="hardware-backends option-chips" role="group" aria-label="Hardware backend">
|
||||
{[
|
||||
...new Set(model.recipes.map((entry) => (entry.gpu_types.includes("Apple Silicon") ? "MLX" : "CUDA"))),
|
||||
].map((platform) => (
|
||||
<button
|
||||
type="button"
|
||||
key={platform}
|
||||
aria-pressed={platform === (mlx ? "MLX" : "CUDA")}
|
||||
onClick={() => {
|
||||
if (platform === (mlx ? "MLX" : "CUDA")) return;
|
||||
setNotice("");
|
||||
const target = model.recipes.find((entry) =>
|
||||
entry.gpu_types.includes(platform === "MLX" ? "Apple Silicon" : "NVIDIA"),
|
||||
)!;
|
||||
setSelected(target.id);
|
||||
setMode("sdk");
|
||||
setEdits({});
|
||||
}}
|
||||
>
|
||||
{platform}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<div className="hardware-resource-grid">
|
||||
<HardwarePicker
|
||||
backend={mlx ? "mlx" : "nvidia"}
|
||||
options={devices}
|
||||
value={
|
||||
hardware.startsWith("custom:")
|
||||
? hardware
|
||||
: (devices.find(([id]) => id === hardware)?.[0] ?? devices[0][0])
|
||||
}
|
||||
onChange={chooseHardware}
|
||||
/>
|
||||
<div className="matrix-resources">
|
||||
<span className="matrix-resource-title">Resources</span>
|
||||
{example?.controls.map((field, i) => (field.name === "num_gpus" ? control(field, i) : null))}
|
||||
{!example?.controls.some((field) => field.name === "num_gpus") && (
|
||||
<p>{mlx ? "1 Apple Silicon host" : `${recipe.hardware?.gpu_count ?? 1} GPU in the source example`}</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
<p className="matrix-note">
|
||||
Hardware fit has not been verified. Settings edit the source example; device choice is a preference.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
{model.recipes.filter((entry) => entry.gpu_types.includes(mlx ? "Apple Silicon" : "NVIDIA")).length > 1 && (
|
||||
<div className="matrix-row">
|
||||
<h2>Recipe</h2>
|
||||
<div className="matrix-row-content option-chips" role="group" aria-label="Recipe variant">
|
||||
{model.recipes
|
||||
.filter((entry) => entry.gpu_types.includes(mlx ? "Apple Silicon" : "NVIDIA"))
|
||||
.map((entry) => (
|
||||
<button
|
||||
type="button"
|
||||
key={entry.id}
|
||||
aria-pressed={selected === entry.id}
|
||||
onClick={() => {
|
||||
setNotice("");
|
||||
setSelected(entry.id);
|
||||
setMode("sdk");
|
||||
setEdits({});
|
||||
}}
|
||||
>
|
||||
{entry.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
<div className="matrix-row">
|
||||
<h2>Generation settings</h2>
|
||||
<div className="matrix-row-content">
|
||||
<div className="matrix-options-grid">
|
||||
{example?.controls.map((field, i) =>
|
||||
field.name !== "num_gpus" && field.path?.[0] !== "server" ? control(field, i) : null,
|
||||
)}
|
||||
</div>
|
||||
{example && !example.controls.length && (
|
||||
<p className="matrix-note">
|
||||
This example uses its model defaults. Open the Python example to customize its pipeline.
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
{example?.controls.some((field) => field.path?.[0] === "server") && (
|
||||
<details className="matrix-advanced">
|
||||
<summary>
|
||||
Advanced <span>Server settings</span>
|
||||
</summary>
|
||||
<div className="matrix-row">
|
||||
<h2>Server</h2>
|
||||
<div className="matrix-row-content matrix-options-grid">
|
||||
{example.controls.map((field, i) => (field.path?.[0] === "server" ? control(field, i) : null))}
|
||||
</div>
|
||||
</div>
|
||||
</details>
|
||||
)}
|
||||
</section>
|
||||
<div className="recipe-bottom-actions">
|
||||
<a
|
||||
href={`https://github.com/hao-ai-lab/FastVideo/blob/main/${mode === "serving" ? recipe.serving?.source : recipe.source}`}
|
||||
>
|
||||
View original example ↗
|
||||
</a>
|
||||
<Button
|
||||
variant="ghost"
|
||||
onClick={() => {
|
||||
setEdits({});
|
||||
setNotice("Restored the baseline example.");
|
||||
}}
|
||||
>
|
||||
Reset changes
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
import { stringify } from "yaml";
|
||||
import type { Result, Value } from "./cookbook";
|
||||
|
||||
export function pythonSource(
|
||||
source: string,
|
||||
controls: { value: string | number | boolean; spans?: number[][] }[],
|
||||
edits: Record<number, string | number | boolean> = {},
|
||||
) {
|
||||
const replacements = controls.flatMap((field, index) =>
|
||||
(field.spans ?? []).map(([start, end]) => {
|
||||
const value = Object.hasOwn(edits, index) ? edits[index] : field.value;
|
||||
return { start, end, text: typeof value === "boolean" ? (value ? "True" : "False") : JSON.stringify(value) };
|
||||
}),
|
||||
);
|
||||
for (const replacement of replacements.sort((a, b) => b.start - a.start))
|
||||
source = source.slice(0, replacement.start) + replacement.text + source.slice(replacement.end);
|
||||
return source;
|
||||
}
|
||||
|
||||
/** Quoted heredocs preserve prompts literally, including shell substitutions. */
|
||||
export function configAndLaunch(yaml: string, command: string) {
|
||||
return fileAndLaunch("config.yaml", yaml, command);
|
||||
}
|
||||
|
||||
export function fileAndLaunch(filename: string, content: string, command: string) {
|
||||
let delimiter = "FASTVIDEO_CONFIG";
|
||||
while (content.split("\n").includes(delimiter)) delimiter += "_";
|
||||
return `(\nset -e\ncat > ${filename} <<'${delimiter}'\n${content.trimEnd()}\n${delimiter}\n${command}\n)`;
|
||||
}
|
||||
|
||||
export const sdkCommand = `python - <<'FASTVIDEO_SDK'
|
||||
import yaml
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
with open("config.yaml") as file:
|
||||
config = yaml.safe_load(file)
|
||||
generator = VideoGenerator.from_config(config["generator"])
|
||||
try:
|
||||
generator.generate(config["request"])
|
||||
finally:
|
||||
generator.shutdown()
|
||||
FASTVIDEO_SDK`;
|
||||
|
||||
/** The CUDA CLI uses the same generator with request instead of server defaults. */
|
||||
export function inferenceOutput(result: Result, imagePath?: string) {
|
||||
const defaults = (result.config.default_request ?? {}) as Record<string, Value>;
|
||||
const output = (defaults.output ?? {}) as Record<string, Value>;
|
||||
const generator = result.config.generator as Record<string, Value>;
|
||||
const pipeline = generator.pipeline as Record<string, Value> | undefined;
|
||||
const inputs = { ...((defaults.inputs ?? {}) as Record<string, Value>) };
|
||||
if (imagePath !== undefined) inputs.image_path = imagePath.trim();
|
||||
const images = Array.isArray(inputs.image_path) ? inputs.image_path : [inputs.image_path];
|
||||
if (pipeline?.workload_type === "i2v" && !images.some((image) => typeof image === "string" && image.trim())) {
|
||||
throw new Error("Enter a source image path before copying a CLI or Python SDK example.");
|
||||
}
|
||||
return {
|
||||
yaml: stringify({
|
||||
generator: result.config.generator,
|
||||
request: {
|
||||
...defaults,
|
||||
...(Object.keys(inputs).length ? { inputs } : {}),
|
||||
prompt: defaults.prompt ?? "A river flowing through a peaceful forest",
|
||||
output: { ...output, save_video: true, return_frames: false, output_path: "outputs/cookbook/" },
|
||||
},
|
||||
}),
|
||||
command: result.command.replace(
|
||||
/\bfastvideo serve --config config\.yaml$/,
|
||||
"fastvideo generate --config config.yaml",
|
||||
),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
import sourceCatalog from "../../docs/assets/cookbook-recipes.json" with { type: "json" };
|
||||
|
||||
/** Presentation metadata only. Deployments and editable fields come from the cookbook backend. */
|
||||
export const recipePages = [
|
||||
{
|
||||
id: "fasth3-8step",
|
||||
family: "FastH3",
|
||||
title: "FastH3 8-Step V2",
|
||||
mark: "H3",
|
||||
path: "/cookbook/minimax-h3/fasth3-8step",
|
||||
description: "Distilled MiniMax-H3 video generation with synchronized audio.",
|
||||
tags: ["Text to video", "Audio", "8-step"],
|
||||
featured: true,
|
||||
},
|
||||
{
|
||||
id: "fasth3-trim",
|
||||
family: "FastH3",
|
||||
title: "FastH3 Trim",
|
||||
mark: "H3",
|
||||
path: "/cookbook/minimax-h3/fasth3-trim",
|
||||
description: "The smaller FastH3 model for eight-step video generation with synchronized audio.",
|
||||
tags: ["Text to video", "Audio", "NVFP4", "8-step"],
|
||||
},
|
||||
{
|
||||
id: "fastwan21",
|
||||
family: "Wan",
|
||||
title: "FastWan2.1 1.3B",
|
||||
mark: "Wan",
|
||||
path: "/cookbook/wan/fastwan21",
|
||||
description: "A compact, distilled Wan model for text-to-video generation.",
|
||||
tags: ["Text to video", "1.3B", "3-step"],
|
||||
},
|
||||
{
|
||||
id: "wan21-i2v",
|
||||
family: "Wan",
|
||||
title: "Wan2.1 I2V 14B",
|
||||
mark: "Wan",
|
||||
path: "/cookbook/wan/wan21-i2v",
|
||||
description: "Turn a reference image into video with Wan2.1.",
|
||||
tags: ["Image to video", "14B", "480p"],
|
||||
},
|
||||
{
|
||||
id: "ltx2-distilled",
|
||||
family: "LTX",
|
||||
title: "LTX2 Distilled",
|
||||
mark: "LTX",
|
||||
path: "/cookbook/ltx/ltx2-distilled",
|
||||
description: "Generate video with incremental delivery over WebSocket.",
|
||||
tags: ["Text to video", "Streaming"],
|
||||
},
|
||||
];
|
||||
export type RecipePage = (typeof recipePages)[number];
|
||||
export function sitePath(path: string) {
|
||||
return `${import.meta.env.BASE_URL.replace(/\/$/, "")}${path}`;
|
||||
}
|
||||
|
||||
export interface SourceRecipe {
|
||||
id: string;
|
||||
family: string;
|
||||
task: string;
|
||||
label: string;
|
||||
summary: string;
|
||||
model: string;
|
||||
source: string;
|
||||
command: string;
|
||||
gpu_types: string[];
|
||||
hardware?: { gpu_count?: number; platform?: string; accelerator?: string };
|
||||
serving?: { source: string; install: string; env?: string; prepare?: string };
|
||||
knobs?: {
|
||||
key: string;
|
||||
label: string;
|
||||
hint?: string;
|
||||
flag: string;
|
||||
default: string | number;
|
||||
options: (string | number | { value: string | number; label: string })[];
|
||||
}[];
|
||||
}
|
||||
export const sourceRecipes = sourceCatalog.recipes as SourceRecipe[];
|
||||
const familyInfo: Record<string, { title: string; path: string; logo: string; category: string }> = {
|
||||
minimax_h3: { title: "MiniMax H3", path: "minimax-h3", logo: "minimax", category: "Video" },
|
||||
wan: { title: "Wan", path: "wan", logo: "wan-ai", category: "Video" },
|
||||
ltx2: { title: "LTX", path: "ltx", logo: "ltx", category: "Video" },
|
||||
hunyuan: { title: "Hunyuan", path: "hunyuan", logo: "tencent-hunyuan", category: "Video" },
|
||||
kandinsky5: { title: "Kandinsky 5", path: "kandinsky5", logo: "kandinsky", category: "Video" },
|
||||
kandinsky6: { title: "Kandinsky 6", path: "kandinsky6", logo: "kandinsky", category: "Video" },
|
||||
flux: { title: "FLUX", path: "flux", logo: "black-forest-labs", category: "Image" },
|
||||
sd35: { title: "Stable Diffusion", path: "stable-diffusion", logo: "stabilityai", category: "Image" },
|
||||
stable_audio: { title: "Stable Audio", path: "stable-audio", logo: "stabilityai", category: "Audio" },
|
||||
mmaudio: { title: "MMAudio", path: "mmaudio", logo: "fastvideo", category: "Audio" },
|
||||
cosmos: { title: "Cosmos", path: "cosmos", logo: "nvidia", category: "World" },
|
||||
matrixgame: { title: "Matrix Game", path: "matrix-game", logo: "fastvideo", category: "World" },
|
||||
};
|
||||
const backendModels: Record<string, string> = {
|
||||
"FastVideo/FastVideo-FastH3-8-Step-V2": "fasth3-8step",
|
||||
"FastVideo/FastVideo-FastH3-Trim-8-Step-NVFP4": "fasth3-trim",
|
||||
"FastVideo/FastWan2.1-T2V-1.3B-Diffusers": "fastwan21",
|
||||
"Wan-AI/Wan2.1-I2V-14B-480P-Diffusers": "wan21-i2v",
|
||||
"FastVideo/LTX2-Distilled-Diffusers": "ltx2-distilled",
|
||||
};
|
||||
export const modelDirectory = [...new Set(sourceRecipes.map((recipe) => recipe.model))]
|
||||
.map((model) => {
|
||||
const recipes = sourceRecipes.filter((recipe) => recipe.model === model);
|
||||
const first = recipes[0];
|
||||
const family = familyInfo[first.family];
|
||||
const backend = recipePages.find((page) => page.id === backendModels[model]);
|
||||
const ownModel = model.startsWith("FastVideo/Fast");
|
||||
const logo = ownModel ? "fastvideo" : family.logo;
|
||||
const providers: Record<string, string> = {
|
||||
fastvideo: "FastVideo",
|
||||
minimax: "MiniMax",
|
||||
"wan-ai": "Wan AI",
|
||||
ltx: "Lightricks",
|
||||
"tencent-hunyuan": "Tencent",
|
||||
kandinsky: "Kandinsky Lab",
|
||||
"black-forest-labs": "Black Forest Labs",
|
||||
stabilityai: "Stability AI",
|
||||
nvidia: "NVIDIA",
|
||||
};
|
||||
return {
|
||||
id: backend?.id ?? first.id,
|
||||
model,
|
||||
recipes,
|
||||
title:
|
||||
backend?.title ??
|
||||
(model.includes("FastH3-4-step")
|
||||
? "FastH3 4-Step V1"
|
||||
: model.includes("FastH3-Preview-v0.2")
|
||||
? "FastH3 Preview V0.2"
|
||||
: model === "MiniMaxAI/MiniMax-H3"
|
||||
? "MiniMax H3"
|
||||
: first.label.split(" on ")[0]),
|
||||
family: first.family === "minimax_h3" ? "FastH3" : (backend?.family ?? family.title),
|
||||
familyPath: `/cookbook/${family.path}`,
|
||||
path: backend?.path ?? `/cookbook/${family.path}/models/${first.id}`,
|
||||
logo,
|
||||
provider: providers[logo],
|
||||
category: family.category,
|
||||
description: backend?.description ?? first.summary,
|
||||
tags: backend?.tags ?? [...new Set(recipes.map((recipe) => recipe.task))],
|
||||
platforms: [...new Set(recipes.flatMap((recipe) => recipe.gpu_types))],
|
||||
configurable: !!backend,
|
||||
featured: backend?.featured ?? false,
|
||||
};
|
||||
})
|
||||
.sort(
|
||||
(a, b) =>
|
||||
Number(b.featured) - Number(a.featured) ||
|
||||
Number(b.family === "FastH3") - Number(a.family === "FastH3") ||
|
||||
Number(b.configurable) - Number(a.configurable),
|
||||
);
|
||||
export type DirectoryModel = (typeof modelDirectory)[number];
|
||||
@@ -0,0 +1,107 @@
|
||||
// The public JavaScript contract lives in docs/cookbook/design.md.
|
||||
// @ts-expect-error The shared ES module has no TypeScript declarations.
|
||||
import * as shared from "../../docs/js/cookbook-config.mjs";
|
||||
|
||||
export type Value = null | boolean | number | string | Value[] | { [key: string]: Value };
|
||||
export interface Schema {
|
||||
title?: string;
|
||||
description?: string;
|
||||
type?: string | string[];
|
||||
enum?: Value[];
|
||||
anyOf?: Schema[];
|
||||
minimum?: number;
|
||||
maximum?: number;
|
||||
multipleOf?: number;
|
||||
}
|
||||
export interface Field {
|
||||
path: string;
|
||||
schema: Schema;
|
||||
value?: Value;
|
||||
}
|
||||
export interface Index {
|
||||
url: string;
|
||||
models: {
|
||||
id: string;
|
||||
title: string;
|
||||
model_id: string;
|
||||
deployments: {
|
||||
id: string;
|
||||
label: string;
|
||||
runtime: string;
|
||||
workload: string;
|
||||
}[];
|
||||
}[];
|
||||
}
|
||||
export interface Catalog {
|
||||
id: string;
|
||||
title: string;
|
||||
summary?: string;
|
||||
workload: string;
|
||||
env: Record<string, string>;
|
||||
base_config: Record<string, Value>;
|
||||
runtime: {
|
||||
label: string;
|
||||
backend: string;
|
||||
interface: string;
|
||||
install_url?: string;
|
||||
client_guide_url?: string;
|
||||
};
|
||||
requirements?: string[];
|
||||
guide?: { url: string; label?: string };
|
||||
}
|
||||
export interface Result {
|
||||
config: Record<string, Value>;
|
||||
yaml: string;
|
||||
command: string;
|
||||
clientCommand: string;
|
||||
websocketUrl?: string;
|
||||
}
|
||||
interface Api {
|
||||
sampleRequest(
|
||||
catalog: {
|
||||
runtime: { interface: string; server_defaults: { host: string; port: number } };
|
||||
model: { id: string };
|
||||
workload: string;
|
||||
},
|
||||
config: Record<string, Value>,
|
||||
): { command: string };
|
||||
loadIndex(url: string, options?: { signal?: AbortSignal }): Promise<Index>;
|
||||
loadDeployment(index: Index, id: string, options?: { signal?: AbortSignal }): Promise<Catalog>;
|
||||
getOptions(catalog: Catalog, config?: Catalog["base_config"], edits?: Record<string, Value>): Field[];
|
||||
resolveConfig(catalog: Catalog, edits?: Record<string, Value>): Result;
|
||||
}
|
||||
export const api = shared as Api;
|
||||
|
||||
export function types(schema: Schema): string[] {
|
||||
return [
|
||||
...new Set([
|
||||
...(Array.isArray(schema.type) ? schema.type : schema.type ? [schema.type] : []),
|
||||
...(schema.anyOf?.flatMap(types) ?? []),
|
||||
]),
|
||||
];
|
||||
}
|
||||
|
||||
export function parseField(field: Field, raw: string): Value {
|
||||
const kind = types(field.schema);
|
||||
if (field.schema.enum || kind.includes("boolean") || kind.includes("object") || kind.includes("array")) {
|
||||
return JSON.parse(raw) as Value;
|
||||
}
|
||||
if (kind.includes("number") || kind.includes("integer")) {
|
||||
if (!raw.trim() || !Number.isFinite(Number(raw))) throw new Error("Enter a number.");
|
||||
return Number(raw);
|
||||
}
|
||||
return raw;
|
||||
}
|
||||
|
||||
/** Keep catalog guide links on the current site and its deployment prefix. */
|
||||
export function guideUrl(path: string): string {
|
||||
const url = new URL(
|
||||
path,
|
||||
new URL(`${import.meta.env.BASE_URL.replace(/\/$/, "")}/assets/cookbook-config/index.json`, window.location.origin),
|
||||
);
|
||||
if (url.origin === "https://haoailab.com" && url.pathname.startsWith("/FastVideo/")) {
|
||||
url.host = window.location.host;
|
||||
url.protocol = window.location.protocol;
|
||||
}
|
||||
return url.href;
|
||||
}
|
||||
Vendored
+3
@@ -0,0 +1,3 @@
|
||||
interface ImportMeta {
|
||||
readonly env: { readonly BASE_URL: string };
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
export const families = [
|
||||
{
|
||||
name: "FastH3",
|
||||
mark: "H3",
|
||||
summary: "Distilled MiniMax-H3 video with synchronized audio.",
|
||||
collection: "https://huggingface.co/collections/FastVideo/fasth3-v2",
|
||||
guide: "/cookbook/minimax-h3",
|
||||
detail: "8-step V2 \u00b7 FP8 and NVFP4 exports",
|
||||
models: [
|
||||
{
|
||||
id: "FastVideo-FastH3-8-Step-V2",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2",
|
||||
},
|
||||
{
|
||||
id: "FastVideo-FastH3-8-Step-V2-NVFP4",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2-NVFP4",
|
||||
},
|
||||
{
|
||||
id: "FastVideo-FastH3-8-Step-V2-FP8",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2-FP8",
|
||||
},
|
||||
{
|
||||
id: "FastVideo-FastH3-8-Step-V2-NVFP4-Consumer",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2-NVFP4-Consumer",
|
||||
},
|
||||
{
|
||||
id: "FastVideo-FastH3-Trim-8-Step",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-Trim-8-Step",
|
||||
},
|
||||
{
|
||||
id: "FastVideo-FastH3-Trim-8-Step-NVFP4",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-Trim-8-Step-NVFP4",
|
||||
},
|
||||
{
|
||||
id: "FastVideo-FastH3-Trim-8-Step-FP8",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-Trim-8-Step-FP8",
|
||||
},
|
||||
{
|
||||
id: "FastVideo-FastH3-8-Step-V2-NVFP4-Comfy",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2-NVFP4-Comfy",
|
||||
},
|
||||
{
|
||||
id: "FastVideo-FastH3-Trim-Comfy",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-Trim-Comfy",
|
||||
},
|
||||
{
|
||||
id: "FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree",
|
||||
url: "https://huggingface.co/FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
name: "FastWan",
|
||||
mark: "Wan",
|
||||
summary: "Distilled Wan models for text-to-video and image-to-video.",
|
||||
collection: "https://huggingface.co/collections/FastVideo/fastwan",
|
||||
guide: "/cookbook/wan",
|
||||
detail: "1.3B, 5B, and 14B releases",
|
||||
models: [
|
||||
{
|
||||
id: "FastWan2.1-T2V-1.3B-Diffusers",
|
||||
url: "https://huggingface.co/FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
|
||||
},
|
||||
{
|
||||
id: "FastWan2.1-T2V-14B-Diffusers",
|
||||
url: "https://huggingface.co/FastVideo/FastWan2.1-T2V-14B-Diffusers",
|
||||
},
|
||||
{
|
||||
id: "FastWan2.2-TI2V-5B-FullAttn-Diffusers",
|
||||
url: "https://huggingface.co/FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers",
|
||||
},
|
||||
{
|
||||
id: "FastWan-QAD-FP8-1.3B",
|
||||
url: "https://huggingface.co/FastVideo/FastWan-QAD-FP8-1.3B",
|
||||
},
|
||||
{
|
||||
id: "CausalWan2.2-I2V-A14B-Preview-Diffusers",
|
||||
url: "https://huggingface.co/FastVideo/CausalWan2.2-I2V-A14B-Preview-Diffusers",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
name: "FastMetal",
|
||||
mark: "Metal",
|
||||
summary: "Quantized Wan checkpoints for Apple Silicon and MLX.",
|
||||
collection: "https://huggingface.co/collections/FastVideo/fastmetal",
|
||||
guide: "/getting_started/installation/mlx",
|
||||
detail: "INT8 QAD \u00b7 1.3B, 5B, and 14B",
|
||||
models: [
|
||||
{
|
||||
id: "FastMetal-1.3B-QAD",
|
||||
url: "https://huggingface.co/FastVideo/FastMetal-1.3B-QAD",
|
||||
},
|
||||
{
|
||||
id: "FastMetal-5B-QAD",
|
||||
url: "https://huggingface.co/FastVideo/FastMetal-5B-QAD",
|
||||
},
|
||||
{
|
||||
id: "FastMetal-14B-QAD",
|
||||
url: "https://huggingface.co/FastVideo/FastMetal-14B-QAD",
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,499 @@
|
||||
[
|
||||
{
|
||||
"label": "Welcome",
|
||||
"href": "/"
|
||||
},
|
||||
{
|
||||
"label": "Fast Models",
|
||||
"href": "/fast-models",
|
||||
"icon": "boxes"
|
||||
},
|
||||
{
|
||||
"label": "Get started",
|
||||
"icon": "rocket",
|
||||
"items": [
|
||||
{
|
||||
"label": "Quick start",
|
||||
"href": "/getting_started/quick_start"
|
||||
},
|
||||
{
|
||||
"label": "Installation",
|
||||
"href": "/getting_started/installation"
|
||||
},
|
||||
{
|
||||
"label": "Hardware and runtimes",
|
||||
"href": "/getting_started/hardware"
|
||||
},
|
||||
{
|
||||
"label": "Python API basics",
|
||||
"href": "/getting_started/v1_api"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "Run inference",
|
||||
"items": [
|
||||
{
|
||||
"label": "Inference guide",
|
||||
"href": "/inference/inference_quick_start"
|
||||
},
|
||||
{
|
||||
"label": "Configuration",
|
||||
"href": "/inference/configuration"
|
||||
},
|
||||
{
|
||||
"label": "CLI",
|
||||
"href": "/inference/cli"
|
||||
},
|
||||
{
|
||||
"label": "Supported models",
|
||||
"href": "/inference/support_matrix"
|
||||
},
|
||||
{
|
||||
"label": "Memory and offloading",
|
||||
"href": "/inference/offloading"
|
||||
},
|
||||
{
|
||||
"label": "Performance and attention",
|
||||
"href": "/inference/optimizations"
|
||||
},
|
||||
{
|
||||
"label": "LoRA adapters",
|
||||
"href": "/utilities/lora"
|
||||
},
|
||||
{
|
||||
"label": "Hardware",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "Choose your runtime",
|
||||
"href": "/getting_started/hardware"
|
||||
},
|
||||
{
|
||||
"label": "NVIDIA GPU setup",
|
||||
"href": "/getting_started/installation/gpu"
|
||||
},
|
||||
{
|
||||
"label": "Apple Silicon / MLX",
|
||||
"href": "/getting_started/installation/mlx"
|
||||
},
|
||||
{
|
||||
"label": "NVIDIA DGX Spark",
|
||||
"href": "/getting_started/installation/spark"
|
||||
},
|
||||
{
|
||||
"label": "Two DGX Sparks",
|
||||
"href": "/getting_started/installation/spark_pair"
|
||||
},
|
||||
{
|
||||
"label": "RTX PRO 6000 / FastH3",
|
||||
"href": "/inference/fasth3_rtx_pro_6000"
|
||||
},
|
||||
{
|
||||
"label": "DGX Spark tuning",
|
||||
"href": "/getting_started/installation/spark_performance"
|
||||
},
|
||||
{
|
||||
"label": "AMD / ROCm",
|
||||
"href": "/getting_started/installation/amd"
|
||||
}
|
||||
],
|
||||
"display": "group"
|
||||
},
|
||||
{
|
||||
"label": "Integrations",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "ComfyUI",
|
||||
"href": "/inference/comfyui"
|
||||
},
|
||||
{
|
||||
"label": "Server and API clients",
|
||||
"href": "/cookbook/openai-api"
|
||||
}
|
||||
],
|
||||
"display": "group"
|
||||
},
|
||||
{
|
||||
"label": "Examples",
|
||||
"href": "/inference/examples/examples_inference_index"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "Cookbook",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "All recipes",
|
||||
"href": "/cookbook"
|
||||
},
|
||||
{
|
||||
"label": "H3 server and playground",
|
||||
"href": "/cookbook/openai-api"
|
||||
},
|
||||
{
|
||||
"label": "FastH3",
|
||||
"display": "group",
|
||||
"collapsed": false,
|
||||
"items": [
|
||||
{
|
||||
"label": "FastH3 8-Step V2",
|
||||
"href": "/cookbook/minimax-h3/fasth3-8step"
|
||||
},
|
||||
{
|
||||
"label": "FastH3 Trim",
|
||||
"href": "/cookbook/minimax-h3/fasth3-trim"
|
||||
},
|
||||
{
|
||||
"label": "Recipes and setup",
|
||||
"href": "/cookbook/minimax-h3"
|
||||
},
|
||||
{
|
||||
"label": "Checkpoint schedules",
|
||||
"href": "/inference/fasth3-distilled"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "Wan",
|
||||
"display": "group",
|
||||
"collapsed": false,
|
||||
"items": [
|
||||
{
|
||||
"label": "FastWan2.1 1.3B",
|
||||
"href": "/cookbook/wan/fastwan21"
|
||||
},
|
||||
{
|
||||
"label": "Wan2.1 I2V 14B",
|
||||
"href": "/cookbook/wan/wan21-i2v"
|
||||
},
|
||||
{
|
||||
"label": "Family guide",
|
||||
"href": "/cookbook/wan"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "LTX",
|
||||
"display": "group",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "LTX2 Distilled",
|
||||
"href": "/cookbook/ltx/ltx2-distilled"
|
||||
},
|
||||
{
|
||||
"label": "Recipes and setup",
|
||||
"href": "/cookbook/ltx"
|
||||
},
|
||||
{
|
||||
"label": "LTX-2.5 reference",
|
||||
"href": "/inference/ltx2_5"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "Hunyuan",
|
||||
"href": "/cookbook/hunyuan"
|
||||
},
|
||||
{
|
||||
"label": "Kandinsky 5",
|
||||
"href": "/cookbook/kandinsky5"
|
||||
},
|
||||
{
|
||||
"label": "Kandinsky 6",
|
||||
"display": "group",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "Recipes and setup",
|
||||
"href": "/cookbook/kandinsky6"
|
||||
},
|
||||
{
|
||||
"label": "Generation reference",
|
||||
"href": "/inference/kandinsky6"
|
||||
},
|
||||
{
|
||||
"label": "Upscaling reference",
|
||||
"href": "/inference/kandinsky6_sr"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "FLUX",
|
||||
"href": "/cookbook/flux"
|
||||
},
|
||||
{
|
||||
"label": "Stable Diffusion",
|
||||
"href": "/cookbook/stable-diffusion"
|
||||
},
|
||||
{
|
||||
"label": "Stable Audio",
|
||||
"href": "/cookbook/stable-audio"
|
||||
},
|
||||
{
|
||||
"label": "MMAudio",
|
||||
"href": "/cookbook/mmaudio"
|
||||
},
|
||||
{
|
||||
"label": "Cosmos",
|
||||
"href": "/cookbook/cosmos"
|
||||
},
|
||||
{
|
||||
"label": "Matrix Game",
|
||||
"href": "/cookbook/matrix-game"
|
||||
}
|
||||
],
|
||||
"display": "group",
|
||||
"root": "/cookbook"
|
||||
},
|
||||
{
|
||||
"label": "Train and fine-tune",
|
||||
"display": "group",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "Overview",
|
||||
"href": "/training/overview"
|
||||
},
|
||||
{
|
||||
"label": "Training Infrastructure",
|
||||
"href": "/training/train_infra"
|
||||
},
|
||||
{
|
||||
"label": "Experiment Tracking",
|
||||
"href": "/training/trackers"
|
||||
},
|
||||
{
|
||||
"label": "Data Preprocessing",
|
||||
"href": "/training/data_preprocess"
|
||||
},
|
||||
{
|
||||
"label": "Fine-tuning",
|
||||
"href": "/training/finetune"
|
||||
},
|
||||
{
|
||||
"label": "Attn-QAT Training",
|
||||
"href": "/training/attn_qat"
|
||||
},
|
||||
{
|
||||
"label": "HunyuanVideo 1.5 T2V",
|
||||
"href": "/training/hunyuan15_overfit"
|
||||
},
|
||||
{
|
||||
"label": "Examples",
|
||||
"items": [
|
||||
{
|
||||
"label": "Training Index",
|
||||
"href": "/training/examples/examples_training_index"
|
||||
},
|
||||
{
|
||||
"label": "Wan T2V 1.3B",
|
||||
"href": "/training/examples/wan_t2v_1.3B"
|
||||
},
|
||||
{
|
||||
"label": "Wan I2V 14B 480p",
|
||||
"href": "/training/examples/wan_i2v_14B_480p"
|
||||
},
|
||||
{
|
||||
"label": "Wan2.1 Fun 1.3B InP",
|
||||
"href": "/training/examples/Wan2.1-Fun-1.3B-InP"
|
||||
},
|
||||
{
|
||||
"label": "Wan2.1 VSA",
|
||||
"href": "/training/examples/Wan2.1-VSA"
|
||||
},
|
||||
{
|
||||
"label": "Fine-tuning",
|
||||
"href": "/training/examples/finetune"
|
||||
}
|
||||
],
|
||||
"collapsed": true,
|
||||
"display": "group"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "Distill models",
|
||||
"display": "group",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "Data Preprocessing",
|
||||
"href": "/distillation/data_preprocess"
|
||||
},
|
||||
{
|
||||
"label": "DMD",
|
||||
"href": "/distillation/dmd"
|
||||
},
|
||||
{
|
||||
"label": "AnyFlow",
|
||||
"href": "/distillation/anyflow"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "Attention research",
|
||||
"display": "group",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "Overview",
|
||||
"href": "/attention"
|
||||
},
|
||||
{
|
||||
"label": "Video Sparse Attention",
|
||||
"href": "/attention/vsa"
|
||||
},
|
||||
{
|
||||
"label": "Sliding Tile Attention (Archived)",
|
||||
"href": "/attention/sta"
|
||||
},
|
||||
{
|
||||
"label": "Backend Development",
|
||||
"href": "/contributing/attention_backend"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "Contribute",
|
||||
"display": "group",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "Overview",
|
||||
"href": "/contributing/overview"
|
||||
},
|
||||
{
|
||||
"label": "Developer Environment",
|
||||
"items": [
|
||||
{
|
||||
"label": "Index",
|
||||
"href": "/contributing/developer_env"
|
||||
},
|
||||
{
|
||||
"label": "Docker",
|
||||
"href": "/contributing/developer_env/docker"
|
||||
},
|
||||
{
|
||||
"label": "RunPod",
|
||||
"href": "/contributing/developer_env/runpod"
|
||||
}
|
||||
],
|
||||
"collapsed": true,
|
||||
"display": "group"
|
||||
},
|
||||
{
|
||||
"label": "Pull Requests",
|
||||
"href": "/contributing/pull_requests"
|
||||
},
|
||||
{
|
||||
"label": "CI/CD Architecture",
|
||||
"href": "/contributing/ci_architecture"
|
||||
},
|
||||
{
|
||||
"label": "Testing",
|
||||
"href": "/contributing/testing"
|
||||
},
|
||||
{
|
||||
"label": "Environment Variables",
|
||||
"href": "/contributing/env_vars"
|
||||
},
|
||||
{
|
||||
"label": "Performance Benchmarks",
|
||||
"href": "/contributing/performance_benchmarks"
|
||||
},
|
||||
{
|
||||
"label": "Profiling",
|
||||
"href": "/contributing/profiling"
|
||||
},
|
||||
{
|
||||
"label": "Activation Trace",
|
||||
"href": "/contributing/activation_trace"
|
||||
},
|
||||
{
|
||||
"label": "Coding Agents",
|
||||
"href": "/contributing/coding_agents"
|
||||
},
|
||||
{
|
||||
"label": "Dreamverse Development",
|
||||
"href": "/contributing/dreamverse-development"
|
||||
},
|
||||
{
|
||||
"label": "Attention Backend Development",
|
||||
"href": "/contributing/attention_backend"
|
||||
},
|
||||
{
|
||||
"label": "Add a pipeline",
|
||||
"href": "/inference/add_pipeline"
|
||||
},
|
||||
{
|
||||
"label": "Debugging",
|
||||
"href": "/utilities/debugging"
|
||||
},
|
||||
{
|
||||
"label": "Documenting FastVideo",
|
||||
"href": "/contributing/documentation"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "Architecture",
|
||||
"display": "group",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "Overview",
|
||||
"href": "/design/overview"
|
||||
},
|
||||
{
|
||||
"label": "Training Architecture",
|
||||
"href": "/design/training_architecture"
|
||||
},
|
||||
{
|
||||
"label": "Server Contracts",
|
||||
"items": [
|
||||
{
|
||||
"label": "Overview",
|
||||
"href": "/design/server_contracts"
|
||||
},
|
||||
{
|
||||
"label": "OpenAI HTTP",
|
||||
"href": "/design/server_contracts/openai"
|
||||
},
|
||||
{
|
||||
"label": "Streaming WebSocket",
|
||||
"href": "/design/server_contracts/streaming"
|
||||
},
|
||||
{
|
||||
"label": "Dynamo Integration",
|
||||
"href": "/design/server_contracts/dynamo"
|
||||
}
|
||||
],
|
||||
"collapsed": true,
|
||||
"display": "group"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"label": "API reference",
|
||||
"collapsed": true,
|
||||
"items": [
|
||||
{
|
||||
"label": "Overview",
|
||||
"href": "/api"
|
||||
},
|
||||
{
|
||||
"label": "Python package",
|
||||
"href": "/api/fastvideo"
|
||||
}
|
||||
],
|
||||
"display": "group"
|
||||
},
|
||||
{
|
||||
"label": "Contact",
|
||||
"href": "/contact"
|
||||
}
|
||||
]
|
||||
Generated
+13588
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"name": "fastvideo-docs-site",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"prepare:content": "node scripts/prepare-content.mjs",
|
||||
"predev": "npm run prepare:content",
|
||||
"dev": "blume dev",
|
||||
"prebuild": "npm run prepare:content",
|
||||
"build": "blume build",
|
||||
"preview": "blume preview",
|
||||
"prevalidate": "npm run prepare:content",
|
||||
"validate": "blume validate --strict",
|
||||
"check": "tsc --noEmit",
|
||||
"test": "playwright test",
|
||||
"format": "prettier --write .",
|
||||
"format:check": "prettier --check .",
|
||||
"test:content": "node --test tests/content.test.mjs tests/commands.test.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"blume": "2.2.3",
|
||||
"class-variance-authority": "0.7.1",
|
||||
"cn": "^0.4.0",
|
||||
"radix-ui": "^1.8.1",
|
||||
"react": "19.3.1",
|
||||
"react-dom": "19.3.1",
|
||||
"yaml": "2.9.1"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@playwright/test": "1.64.1",
|
||||
"@types/react": "19.3.0",
|
||||
"@types/react-dom": "19.3.0",
|
||||
"prettier": "3.9.9",
|
||||
"typescript": "~5.9.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=22.19.0"
|
||||
},
|
||||
"overrides": {
|
||||
"@astrojs/react": "7.0.0",
|
||||
"@vitejs/plugin-react": "6.1.1"
|
||||
},
|
||||
"prettier": {
|
||||
"printWidth": 120
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
import { defineConfig } from "@playwright/test";
|
||||
|
||||
export default defineConfig({
|
||||
testDir: "./tests",
|
||||
testMatch: "**/*.spec.ts",
|
||||
use: { baseURL: "http://127.0.0.1:4327", trace: "retain-on-failure" },
|
||||
webServer: {
|
||||
command: "npm run preview -- --host 127.0.0.1 --port 4327",
|
||||
url: "http://127.0.0.1:4327/FastVideo/cookbook",
|
||||
reuseExistingServer: !process.env.CI,
|
||||
},
|
||||
});
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
@@ -0,0 +1,38 @@
|
||||
import { sourceRecipes } from "../lib/cookbook-pages.ts";
|
||||
/** Prepare the docs transfer and publish the existing cookbook catalogs. */
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { cp, mkdir, rm } from "node:fs/promises";
|
||||
|
||||
import { transferDocs } from "./transfer-docs.mjs";
|
||||
|
||||
await transferDocs();
|
||||
|
||||
const repo = new URL("../../", import.meta.url);
|
||||
execFileSync("npm", ["run", "build:catalog", "--prefix", "docs"], {
|
||||
cwd: repo,
|
||||
stdio: "inherit",
|
||||
});
|
||||
const publicDir = new URL("../public/", import.meta.url);
|
||||
await mkdir(publicDir, { recursive: true });
|
||||
const destination = new URL("assets/cookbook-config/", publicDir);
|
||||
await rm(destination, { recursive: true, force: true });
|
||||
await cp(new URL("docs/assets/cookbook-config/", repo), destination, {
|
||||
recursive: true,
|
||||
});
|
||||
await cp(new URL("docs/assets/logos/icon_simple.svg", repo), new URL("icon.svg", publicDir));
|
||||
|
||||
await cp(new URL("../assets/hao-ai-lab.png", import.meta.url), new URL("assets/hao-ai-lab.png", publicDir));
|
||||
await cp(new URL("../.generated/source-examples/", import.meta.url), new URL("assets/cookbook-examples/", publicDir), {
|
||||
recursive: true,
|
||||
});
|
||||
|
||||
// Publish exact checked-in YAML for inference-only model pages. No settings are synthesized.
|
||||
const sourceDir = new URL("assets/cookbook-sources/", publicDir);
|
||||
await rm(sourceDir, { recursive: true, force: true });
|
||||
await mkdir(sourceDir, { recursive: true });
|
||||
for (const recipe of sourceRecipes) {
|
||||
if (/\.ya?ml$/.test(recipe.source))
|
||||
await cp(new URL(recipe.source, repo), new URL(`${recipe.id}-inference.yaml`, sourceDir));
|
||||
if (recipe.serving?.source)
|
||||
await cp(new URL(recipe.serving.source, repo), new URL(`${recipe.id}-serving.yaml`, sourceDir));
|
||||
}
|
||||
@@ -0,0 +1,291 @@
|
||||
"""Build source-only Python reference pages with Griffe, without importing FastVideo."""
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import ast
|
||||
import json
|
||||
import re
|
||||
import runpy
|
||||
import shutil
|
||||
import shlex
|
||||
from pathlib import Path
|
||||
from urllib.parse import urljoin, urlsplit, urlunsplit
|
||||
|
||||
import griffe
|
||||
import markdown
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
SITE = ROOT / "docs-site"
|
||||
GENERATED = SITE / ".generated"
|
||||
|
||||
|
||||
def source_controls(source: str) -> list[dict]:
|
||||
"""Expose literal settings in maintained examples without importing them."""
|
||||
labels = {
|
||||
"prompt": "Prompt", "num_gpus": "GPU count", "height": "Height", "width": "Width",
|
||||
"num_frames": "Frames", "seed": "Seed", "num_inference_steps": "Inference steps",
|
||||
"guidance_scale": "Guidance scale", "fps": "FPS", "audio_end_in_s": "Audio duration",
|
||||
"text_encoder_cpu_offload": "Offload text encoder", "vae_cpu_offload": "Offload VAE",
|
||||
"dit_cpu_offload": "Offload transformer", "inference_torch_compile": "Compile transformer",
|
||||
"video_decode_backend": "Video decoder",
|
||||
}
|
||||
lines = source.splitlines(keepends=True)
|
||||
offsets = [0]
|
||||
for line in lines:
|
||||
offsets.append(offsets[-1] + len(line))
|
||||
|
||||
def position(line, column):
|
||||
prefix = source[:offsets[line - 1]] + lines[line - 1].encode()[:column].decode()
|
||||
return len(prefix.encode("utf-16-le")) // 2
|
||||
|
||||
settings = {}
|
||||
|
||||
def add(name, value):
|
||||
if name not in labels or not isinstance(value, ast.Constant) or type(value.value) not in (str, int, float, bool):
|
||||
return
|
||||
key = json.dumps([name, value.value])
|
||||
entry = settings.setdefault(key, {"name": name, "label": labels[name], "value": value.value,
|
||||
"integer": type(value.value) is int, "spans": []})
|
||||
entry["spans"].append([position(value.lineno, value.col_offset), position(value.end_lineno, value.end_col_offset)])
|
||||
|
||||
for node in ast.walk(ast.parse(source)):
|
||||
if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
|
||||
if node.func.attr in {"from_pretrained", "generate_video"}:
|
||||
for keyword in node.keywords:
|
||||
add(keyword.arg, keyword.value)
|
||||
elif node.func.attr == "add_argument":
|
||||
flags = [arg.value for arg in node.args if isinstance(arg, ast.Constant) and isinstance(arg.value, str)]
|
||||
flag = next((flag for flag in flags if flag.startswith("--")), "")
|
||||
for keyword in node.keywords:
|
||||
if keyword.arg == "default":
|
||||
add(flag.removeprefix("--").replace("-", "_"), keyword.value)
|
||||
key = json.dumps([flag.removeprefix("--").replace("-", "_"), getattr(keyword.value, "value", None)])
|
||||
if key in settings:
|
||||
settings[key]["flag"] = flag
|
||||
if any(item.arg == "type" and isinstance(item.value, ast.Name) and item.value.id == "float"
|
||||
for item in node.keywords):
|
||||
settings[key]["integer"] = False
|
||||
settings[key]["booleanOptional"] = any(
|
||||
item.arg == "action" and isinstance(item.value, ast.Attribute) and item.value.attr == "BooleanOptionalAction"
|
||||
for item in node.keywords)
|
||||
elif isinstance(node, ast.Assign):
|
||||
for target in node.targets:
|
||||
if isinstance(target, ast.Name) and target.id.isupper():
|
||||
add(target.id.lower(), node.value)
|
||||
result = list(settings.values())
|
||||
seen = {}
|
||||
for field in result:
|
||||
seen[field["label"]] = seen.get(field["label"], 0) + 1
|
||||
if seen[field["label"]] > 1:
|
||||
field["label"] += f' ({seen[field["label"]]})'
|
||||
return result
|
||||
|
||||
|
||||
def signature(obj) -> str:
|
||||
"""Preserve parameter kinds, annotations, defaults, and return annotations."""
|
||||
if obj.is_attribute:
|
||||
annotation = f": {obj.annotation}" if obj.annotation else ""
|
||||
value = f" = {obj.value}" if obj.value is not None else ""
|
||||
return f"{obj.name}{annotation}{value}"
|
||||
parameters = list(getattr(obj, "parameters", []))
|
||||
parts = []
|
||||
keyword_separator = False
|
||||
for i, param in enumerate(parameters):
|
||||
if param.name in {"self", "cls"}:
|
||||
continue
|
||||
kind = param.kind.value
|
||||
if kind == "keyword-only" and not keyword_separator:
|
||||
parts.append("*")
|
||||
keyword_separator = True
|
||||
prefix = "**" if kind == "variadic keyword" else "*" if kind == "variadic positional" else ""
|
||||
if prefix == "*":
|
||||
keyword_separator = True
|
||||
text = prefix + param.name
|
||||
if param.annotation is not None:
|
||||
text += f": {param.annotation}"
|
||||
if param.default is not None and not prefix:
|
||||
text += f" = {param.default}"
|
||||
parts.append(text)
|
||||
if kind == "positional-only" and (i + 1 == len(parameters) or parameters[i + 1].kind.value != kind):
|
||||
parts.append("/")
|
||||
result = f"{obj.name}({', '.join(parts)})"
|
||||
if obj.is_function and obj.returns is not None:
|
||||
result += f" -> {obj.returns}"
|
||||
return result
|
||||
|
||||
|
||||
def code_html(text: str) -> str:
|
||||
return html.escape(text).replace("[", "[").replace("]", "]")
|
||||
|
||||
|
||||
def prose(text: str) -> str:
|
||||
# Docstrings are trusted repository source, rendered as Markdown rather than executed.
|
||||
return markdown.markdown(text, extensions=["fenced_code", "tables", "sane_lists"])
|
||||
|
||||
|
||||
def api_overview(text: str, package) -> str:
|
||||
def reference(match):
|
||||
label, target = match.groups()
|
||||
path = (target or label).strip("`")
|
||||
obj = package if path == package.path else package[path.removeprefix(package.path + ".")]
|
||||
module = obj
|
||||
while not module.is_module:
|
||||
module = module.parent
|
||||
route = "/api/" + module.path.replace(".", "/")
|
||||
return f"[{label}]({route}#{path})"
|
||||
|
||||
text = re.sub(r"\[([^\]\n]+)\]\[([^\]\n]*)\]", reference, text)
|
||||
|
||||
def guide(match):
|
||||
label, target = match.groups()
|
||||
url = urlsplit(target)
|
||||
if url.scheme or url.netloc or not re.search(r"\.mdx?$", url.path):
|
||||
return match[0]
|
||||
url = urlsplit(urljoin("/api/index.md", target))
|
||||
route = re.sub(r"\.mdx?$", "", url.path)
|
||||
route = re.sub(r"/index$", "", route) or "/"
|
||||
return f"[{label}]({urlunsplit(('', '', route, url.query, url.fragment))})"
|
||||
|
||||
return prose(re.sub(r"\[([^\]\n]+)\]\(([^\s)]+)\)", guide, text))
|
||||
|
||||
|
||||
def docstring(obj, level: int) -> str:
|
||||
if not obj.docstring:
|
||||
return ""
|
||||
output = []
|
||||
for section in obj.docstring.parsed:
|
||||
kind = section.kind.value
|
||||
if kind == "text":
|
||||
output.append(prose(section.value))
|
||||
continue
|
||||
title = section.title or kind.replace("_", " ").capitalize()
|
||||
output.append(f"<h{level}>{html.escape(title)}</h{level}>")
|
||||
if kind == "examples":
|
||||
for example_kind, value in section.value:
|
||||
output.append(f"<pre><code>{html.escape(value)}</code></pre>" if example_kind.value == "examples" else prose(value))
|
||||
elif isinstance(section.value, list):
|
||||
output.append("<dl>")
|
||||
for item in section.value:
|
||||
name = getattr(item, "name", "") or ""
|
||||
annotation = getattr(item, "annotation", "") or ""
|
||||
label = f"{name}: {annotation}" if name and annotation else str(name or annotation)
|
||||
output.append(f"<dt><code>{html.escape(label)}</code></dt><dd>{prose(getattr(item, 'description', ''))}</dd>")
|
||||
output.append("</dl>")
|
||||
else:
|
||||
output.append(prose(getattr(section.value, "description", str(section.value))))
|
||||
return "\n".join(output)
|
||||
|
||||
|
||||
def symbol(obj, path: str, level: int = 2, members: list[str] | None = None, source: bool = True) -> str:
|
||||
level = min(level, 6)
|
||||
output = [f'<h{level} id="{html.escape(path)}"><code>{html.escape(path)}</code></h{level}>']
|
||||
output.append(f'<pre class="doc-signature"><code>{code_html(signature(obj))}</code></pre>')
|
||||
output.append(docstring(obj, min(level + 1, 6)))
|
||||
if obj.is_class:
|
||||
constructor = obj.members.get("__init__")
|
||||
if constructor is not None:
|
||||
output.append(docstring(constructor, min(level + 1, 6)))
|
||||
if source and not obj.is_alias:
|
||||
try:
|
||||
code = obj.source
|
||||
except (ValueError, KeyError):
|
||||
code = ""
|
||||
if code:
|
||||
output.append(f'<details><summary>Source</summary><pre><code>{code_html(code)}</code></pre></details>')
|
||||
if obj.is_class:
|
||||
for name, member in obj.members.items():
|
||||
if members is not None and name not in members:
|
||||
continue
|
||||
if name.startswith("_") or member.is_module:
|
||||
continue
|
||||
if member.is_attribute and not member.docstring:
|
||||
continue
|
||||
try:
|
||||
output.append(symbol(member, f"{path}.{name}", min(level + 1, 6), source=source))
|
||||
except griffe.AliasResolutionError:
|
||||
continue
|
||||
return "\n".join(output)
|
||||
|
||||
|
||||
def write_page(route: str, title: str, body: str, anchors: dict) -> None:
|
||||
# Preserve incoming fragments, including historical category labels whose layout changed.
|
||||
present = set(re.findall(r'id="([^"]+)"', body))
|
||||
aliases = "\n".join(f'<a id="{html.escape(h["id"])}"></a>' for h in anchors.get(route, []) if h["id"] not in present)
|
||||
target = GENERATED / "python-reference" / route / "index.md"
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text("---\n" + yaml.safe_dump({"title": title}, sort_keys=False) +
|
||||
f'---\n\n<div class="python-reference">\n{aliases}\n{body}\n</div>\n')
|
||||
|
||||
|
||||
def prepare() -> None:
|
||||
GENERATED.mkdir(exist_ok=True)
|
||||
examples = runpy.run_path(str(ROOT / "docs/generate_examples.py"))
|
||||
examples["validate_cookbook"]()
|
||||
examples["generate_examples"](generate_main_index=True)
|
||||
source_dir = GENERATED / "source-examples"
|
||||
shutil.rmtree(source_dir, ignore_errors=True)
|
||||
source_dir.mkdir()
|
||||
for recipe in json.loads((ROOT / "docs/assets/cookbook-recipes.json").read_text())["recipes"]:
|
||||
path = ROOT / recipe["source"]
|
||||
if path.suffix == ".py":
|
||||
source = path.read_text()
|
||||
controls = source_controls(source)
|
||||
argv = shlex.split(recipe["command"])
|
||||
for field in controls:
|
||||
flag = field.get("flag")
|
||||
positions = [i for i, argument in enumerate(argv[:-1]) if argument == flag]
|
||||
if positions and type(field["value"]) is not bool:
|
||||
converter = float if isinstance(field["value"], (int, float)) and not field["integer"] else type(field["value"])
|
||||
field["value"] = converter(argv[positions[-1] + 1])
|
||||
elif field.get("booleanOptional"):
|
||||
if flag in argv:
|
||||
field["value"] = True
|
||||
elif flag.replace("--", "--no-", 1) in argv:
|
||||
field["value"] = False
|
||||
(source_dir / f'{recipe["id"]}.json').write_text(json.dumps({"source": source, "controls": controls}))
|
||||
anchors = json.loads((SITE / "migration-anchors.json").read_text())
|
||||
shutil.rmtree(GENERATED / "python-reference", ignore_errors=True)
|
||||
package = griffe.load("fastvideo", search_paths=[ROOT], allow_inspection=False, resolve_aliases=True,
|
||||
resolve_external=False, docstring_parser="google", docstring_options={"warnings": False})
|
||||
modules = []
|
||||
|
||||
def visit(module):
|
||||
if module.is_alias or any(part.startswith("_") or part in {"tests", "third_party"} for part in module.path.split(".")):
|
||||
return
|
||||
modules.append(module)
|
||||
for child in module.modules.values():
|
||||
visit(child)
|
||||
|
||||
visit(package)
|
||||
for module in sorted(modules, key=lambda item: item.path):
|
||||
body = [f'<a id="{module.path}"></a>', docstring(module, 2)]
|
||||
for name, member in module.members.items():
|
||||
if name.startswith("_"):
|
||||
continue
|
||||
try:
|
||||
# External imports belong to their upstream reference, not FastVideo's API.
|
||||
if member.is_alias and not member.target_path.startswith("fastvideo."):
|
||||
continue
|
||||
if member.is_module or member.is_attribute and not member.docstring:
|
||||
continue
|
||||
body.append(symbol(member, f"{module.path}.{name}"))
|
||||
except griffe.AliasResolutionError:
|
||||
continue
|
||||
write_page("api/" + module.path.replace(".", "/"), module.name, "\n".join(body), anchors)
|
||||
write_page("api", "Python API reference", api_overview((ROOT / "docs/api/index.md").read_text(), package), anchors)
|
||||
v1 = (ROOT / "docs/getting_started/v1_api.md").read_text()
|
||||
|
||||
def reference(match):
|
||||
path, level, members = match.groups()
|
||||
return "\n" + symbol(package[path.removeprefix("fastvideo.")], path,
|
||||
int(level), members.split(","), source=False) + "\n"
|
||||
|
||||
v1 = re.sub(r'<api-reference path="([^"]+)" heading="(\d+)" members="([^"]+)" />', reference, v1)
|
||||
write_page("getting_started/v1_api", "V1 API", prose(v1.removeprefix("# V1 API\n")), anchors)
|
||||
(GENERATED / "old-headings.json").write_text(json.dumps(anchors))
|
||||
print(f"Rendered {len(modules) + 1} Python reference pages directly from source, without MkDocs or runtime imports.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
prepare()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user