diff --git a/.agents/lessons/2026-05-07_conversion-cast-bf16-suffix-allowlist.md b/.agents/lessons/2026-05-07_conversion-cast-bf16-suffix-allowlist.md index d32466293..652a5a213 100644 --- a/.agents/lessons/2026-05-07_conversion-cast-bf16-suffix-allowlist.md +++ b/.agents/lessons/2026-05-07_conversion-cast-bf16-suffix-allowlist.md @@ -52,8 +52,8 @@ in one commit. script that supports a global dtype downcast flag MUST own an explicit allowlist of fp32-keep tensors, documented at the top of the file. -2. **The `add-model-conversion` skill** should enforce two checks for any - converter that ships a `--cast-bf16`-style flag: +2. **Check two things** for any converter that ships a `--cast-bf16`-style + flag: - Run the parity test for **every** variant of the model (base, distill, SR, etc.), not just the headline variant. Different variants exercise different code paths. diff --git a/.agents/lessons/2026-05-07_dit-dtype-boundary-with-flash-attn.md b/.agents/lessons/2026-05-07_dit-dtype-boundary-with-flash-attn.md index d9e3fd5ec..bad819a33 100644 --- a/.agents/lessons/2026-05-07_dit-dtype-boundary-with-flash-attn.md +++ b/.agents/lessons/2026-05-07_dit-dtype-boundary-with-flash-attn.md @@ -62,10 +62,10 @@ parity rationale: 2. **Use layer-by-layer activation hooks** when DiT parity is close-but-not- bit-exact and the gap grows with depth. The `fastvideo/hooks/activation_trace.py` infra exists exactly for this case - (`add-model-trace` skill). In MagiHuman's case it would have localized the + (see `docs/contributing/activation_trace.md`). In MagiHuman's case it would have localized the first divergence point in one pass. -3. **The `add-model-port-dit` skill** should explicitly call out: +3. **DiT ports must account for:** - SDPA input dtype must match the upstream kernel's internal cast. - Post-attention upcast happens **before** any per-head gate, not after. - Residual stream dtype across block boundaries is a parity invariant. diff --git a/.agents/lessons/2026-05-07_silent-channel-major-packing-bugs.md b/.agents/lessons/2026-05-07_silent-channel-major-packing-bugs.md index 941bc2757..f5775d9c3 100644 --- a/.agents/lessons/2026-05-07_silent-channel-major-packing-bugs.md +++ b/.agents/lessons/2026-05-07_silent-channel-major-packing-bugs.md @@ -64,6 +64,5 @@ because both sides of the parity check are now correct. `(pT pH pW C)`) at the call site and assert the layout matches the downstream consumer's expectation. -4. The `add-model-port-dit` skill's parity gate should require an E2E hash - check for any DiT that does video token packing, not just latent - bit-exactness. +4. DiT parity for any model that does video token packing should include an + E2E hash check, not just latent bit-exactness. diff --git a/.agents/lessons/2026-09-29_streaming-ipc-semaphore-enoent.md b/.agents/lessons/2026-09-29_streaming-ipc-semaphore-enoent.md new file mode 100644 index 000000000..fc43540cd --- /dev/null +++ b/.agents/lessons/2026-09-29_streaming-ipc-semaphore-enoent.md @@ -0,0 +1,66 @@ +--- +date: 2026-09-29 +experiment: Wan-VACE multi-GPU inference on shared Slurm nodes +category: infrastructure +severity: important +--- + +# Spawn workers fail with ENOENT in `SemLock._rebuild` + +## What Happened + +On Slurm nodes, standard `VideoGenerator` runs with the `mp` executor sometimes +failed during worker startup. A spawn child raised `FileNotFoundError` from +`multiprocessing.synchronize.SemLock._rebuild` while unpickling its arguments, +even though the parent still held its Queue objects. Failures showed up on +several nodes and looked intermittent. + +## Root Cause + +The cluster's Slurm epilog `80-epilog-cleanup-shm-tmp.sh` runs +`find /dev/shm -maxdepth 2 -user "$SLURM_JOB_USER" -delete` at the end of +**every** job. When one of a user's jobs ends, the epilog deletes that user's +`/dev/shm` files on the node, including those that the user's *other* running +jobs are still using. Python's POSIX named semaphores (`/dev/shm/sem.mp-*`) are +among them. If the deletion lands between queue creation and a spawn child +unpickling it, the child fails with ENOENT. + +Evidence: + +- **Timing.** Every recorded deletion burst during a probe job matched, to the + second, the end of another job by the same user on the same node + (`sacct -u $USER -N ` end times). +- **Controlled reproduction.** + - Job A held a spawn `Lock` semaphore and a marker file in `/dev/shm`. + - Job B, on the same node, started and then ended. + - Job B's start deleted nothing. + - Within 1 s of job B's end, both of job A's objects were gone, and job A's + finalizer then hit the same ENOENT. +- **Still unexplained.** One historical failure has no same-user job + end in its window, so another deleter cannot be ruled out for it. logind + `RemoveIPC` was the earlier hypothesis; it is not needed to explain the other + cases. + +## Fix / Workaround + +- **FastVideo hardening.** Standard inference no longer creates the two + streaming queues it never uses. `FastVideoArgs.enable_streaming_ipc_queues` + defaults to `False`, and `StreamingVideoGenerator` sets it to `True`. This + removes the exposure for standard inference only. +- **Still exposed.** Streaming queues, NCCL shared-memory segments, and + DataLoader shared memory on the same node can still be deleted by the epilog. +- **Cluster fix (administrators).** Clean `/dev/shm` only when the user has no + other job on the node, or give each job a private `/dev/shm` with + `JobContainerType=job_container/tmpfs`. +- **User workaround.** Don't co-locate your own jobs on one node + (`--exclusive=user`), or avoid ending short jobs next to long-running ones. + +## Prevention + +- Before blaming an application for `/dev/shm` ENOENT on Slurm, read the + node-local prolog/epilog hooks (`/cm/local/apps/slurm/var/{prologs,epilogs}`). + Then correlate the failure window with `sacct -u $USER -N ` job end + times. +- Do not pass IPC primitives to spawn workers unless the worker needs them. +- `fastvideo/tests/worker/test_multiproc_executor.py` checks that standard + workers survive semaphore removal during spawn. diff --git a/.agents/skills/add-model-01-prep/SKILL.md b/.agents/skills/add-model-01-prep/SKILL.md deleted file mode 100644 index 6c4e40917..000000000 --- a/.agents/skills/add-model-01-prep/SKILL.md +++ /dev/null @@ -1,173 +0,0 @@ ---- -name: add-model-01-prep -description: Use at the start of a FastVideo model port to gather required inputs, inspect/download HF weights, clone and install the official reference repo in the current environment, create a local_tests README skeleton, and produce a handoff before conversion or implementation. ---- - -# Add Model Prep - -## Goal - -Prepare external assets and the shared parity-test environment for a FastVideo -model port. Stop before writing conversion scripts, model components, pipeline -code, registry entries, or executable parity tests. - -## Ask First - -Ask once, then proceed if the HF token is already exported: - -```text -Before prep: (1) official reference repo or Diffusers pipeline URL, (2) HF repo -id or local weights path and whether it has a root model_index.json, (3) target -model_family, (4) workload types, (5) which token env var is exported: -HF_TOKEN, HUGGINGFACE_HUB_TOKEN, or HF_API_KEY, (6) may I stage clone and -weights under the FastVideo repo root, and (7) may I install official reference -dependencies into the current FastVideo conda/env for parity tests? -``` - -Useful optional inputs: `pipeline_class`, `reference_dir`, `hf_revision`, -`official_revision`, `reuse_hints`, `download_scope`. - -## Rules - -- Follow `../add-model/shared/common_rules.md` for token/auth safety, state files, - escape hatches, and skip/pass semantics. -- Run from the FastVideo repo root. -- Use repo-relative defaults: `/`, - `official_weights//`, `converted_weights//`. -- Install official reference deps into the current FastVideo environment, not a - new venv/conda env, so parity tests run both implementations with one shared - numeric stack. -- If the reference is a Diffusers class/package instead of a cloneable repo, - record import path and version instead of cloning. -- Prep may create only the local-test README and `PORT_STATUS.md` skeletons; - executable `.py` parity tests belong to `../add-model-02-parity/SKILL.md`. - -## Escape Hatches - -Follow `../add-model/shared/common_rules.md`. Prep-specific ask cases include -overwriting an existing clone or weight directory, installing untrusted/private -deps, choosing between incompatible official references, large downloads outside -the agreed scope, or missing gated-repo auth setup by env var name. - -## Workflow - -1. Verify the repo: - -```bash -git rev-parse --show-toplevel -``` - -Expected markers: `fastvideo/`, `scripts/checkpoint_conversion/`, -`scripts/huggingface/download_hf.py`, `fastvideo/registry.py`. - -2. Inspect HF or local weight layout: - -```bash -python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \ - "Org/Model" \ - --revision "" \ - --json -``` - -For a local path, replace `Org/Model` with `/path/to/weights`. Record -`source_layout`, `needs_conversion`, `model_index_class`, and -`components_seen`. - -3. Download HF weights if needed: - -```bash -python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \ - "Org/Model" \ - "official_weights/" \ - --revision "" -``` - -For selected files, repeat `--file-name`. For partial snapshots, repeat -`--allow-pattern` or `--ignore-pattern`. If the user provided a local path, -record it instead of copying large weights by default. - -4. Clone the official reference repo if applicable: - -```bash -python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \ - "" \ - "" \ - --branch "" \ - --commit "" \ - --update-gitignore -``` - -Omit `--branch`, `--commit`, or `--update-gitignore` when not needed. The -helper refuses to overwrite existing paths and prints remote/HEAD instead. - -5. Keep prep assets ignored. Ensure `.gitignore` includes relevant entries: - -```gitignore -// -/official_weights/ -/converted_weights/ -``` - -6. Follow the official repo's setup instructions in the current environment. -Inspect dependency files and README install docs before installing anything: - -- `README*`, install docs, or model-card instructions. -- `requirements*.txt`, `pyproject.toml`, `setup.py`, `environment.yml`. - -Use the current FastVideo conda/env. Do not create a new env even if upstream -docs recommend one; translate the needed install commands into the active env. -Prefer editable/no-deps first so the official source is importable without -changing shared pins: - -```bash -uv pip install --no-deps -e ./ -``` - -Then install only missing official deps needed for parity imports. Stop before -installing requirements that would change FastVideo's core stack. If upstream -requires private/non-PyPI deps, record that parity needs a local stub helper -rather than pretending setup is complete. - -7. Create the model-family local test skeleton and top-level port state file: - -```bash -mkdir -p tests/local_tests/ -cp ".agents/skills/add-model-01-prep/templates/local_tests_readme.md" \ - tests/local_tests//README.md -cp ".agents/skills/add-model-01-prep/templates/port_status.md" \ - tests/local_tests//PORT_STATUS.md -``` - -Edit every placeholder in the README and `PORT_STATUS.md`. The README gives -later review agents enough information to reproduce the shared environment and -run/review parity work: - -- official code URL or import path, local clone path, and commit/version; -- HF URL or local weight path, revision, access notes, and token env var name - only; -- commands already run and any blocked official dependency installs; -- shared-env install commands to re-run without changing core pins; -- expected local parity test paths and pytest commands; -- private-dependency stubs or known setup gaps; -- PR/review notes explaining which parity tests are required before handoff. - -Do not include raw tokens, absolute cache paths that are not repo-reproducible, -or large generated outputs. If prep is blocked before imports work, still create -the README with `official_env_status=blocked` and the exact blocker. - -`PORT_STATUS.md` must follow `../add-model/contracts/port_state.md`. Record open -questions and prep issues immediately, using stable IDs such as `Q001` and -`I001`. Keep resolved questions/issues in the table with a resolution instead of -deleting them. - -## Handoff - -End with the canonical prep handoff contract from -`../add-model/contracts/prep_handoff.md` and update the shared state files before -handoff. - -## Helper Scripts - -- `scripts/inspect_hf_layout.py`: classify HF/local layout. -- `scripts/download_hf_weights.py`: download HF snapshot or selected files. -- `scripts/clone_reference_repo.py`: clone reference repo safely. diff --git a/.agents/skills/add-model-01-prep/scripts/clone_reference_repo.py b/.agents/skills/add-model-01-prep/scripts/clone_reference_repo.py deleted file mode 100755 index 3bc2e27e1..000000000 --- a/.agents/skills/add-model-01-prep/scripts/clone_reference_repo.py +++ /dev/null @@ -1,119 +0,0 @@ -#!/usr/bin/env python3 -"""Clone an official reference repo without overwriting existing paths.""" - -from __future__ import annotations - -import argparse -import subprocess -import sys -from pathlib import Path - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser(description="Clone a reference repo for FastVideo parity tests.") - parser.add_argument("repo_url", help="Official reference repository URL") - parser.add_argument("target_dir", help="Directory to clone into") - parser.add_argument("--branch", help="Branch or tag to clone") - parser.add_argument("--commit", help="Commit SHA to check out after clone") - parser.add_argument( - "--update-gitignore", - action="store_true", - help="Add the target directory to .gitignore if missing", - ) - parser.add_argument( - "--gitignore", - default=".gitignore", - help="Path to gitignore file when --update-gitignore is used", - ) - return parser.parse_args() - - -def run(command: list[str], check: bool = True) -> subprocess.CompletedProcess[str]: - return subprocess.run( - command, - check=check, - text=True, - capture_output=True, - ) - - -def print_existing_repo_info(target: Path) -> int: - print(f"target_exists: {target}") - if not (target / ".git").exists(): - print("error: target exists but is not a git repo", file=sys.stderr) - return 1 - - remote = run(["git", "-C", str(target), "remote", "-v"], check=False) - head = run(["git", "-C", str(target), "rev-parse", "HEAD"], check=False) - if remote.stdout: - print("remote_v:") - print(remote.stdout.rstrip()) - if head.stdout: - print(f"head: {head.stdout.strip()}") - print("not_overwritten: true") - return 0 - - -def gitignore_entry_for(target: Path) -> str: - root = Path.cwd().resolve() - resolved = target.resolve() - try: - relative = resolved.relative_to(root) - except ValueError as exc: - raise ValueError("--update-gitignore requires target_dir to be under the current directory") from exc - - text = relative.as_posix().rstrip("/") - return "/" + text + "/" - - -def update_gitignore(path: Path, target: Path) -> bool: - entry = gitignore_entry_for(target) - existing = path.read_text().splitlines() if path.exists() else [] - if entry in existing: - return False - - new_text = "\n".join(existing).rstrip("\n") - if new_text: - new_text += "\n" - new_text += entry + "\n" - path.write_text(new_text) - return True - - -def main() -> int: - args = parse_args() - target = Path(args.target_dir) - - if target.exists(): - return print_existing_repo_info(target) - - command = ["git", "clone", "--depth", "1"] - if args.branch: - command.extend(["--branch", args.branch]) - command.extend([args.repo_url, str(target)]) - - try: - run(command) - if args.commit: - run(["git", "-C", str(target), "fetch", "--depth", "1", "origin", args.commit]) - run(["git", "-C", str(target), "checkout", args.commit]) - except subprocess.CalledProcessError as exc: - if exc.stdout: - print(exc.stdout, end="") - if exc.stderr: - print(exc.stderr, end="", file=sys.stderr) - return exc.returncode - - head = run(["git", "-C", str(target), "rev-parse", "HEAD"]) - print(f"cloned: {target}") - print(f"head: {head.stdout.strip()}") - - if args.update_gitignore: - changed = update_gitignore(Path(args.gitignore), target) - print(f"gitignore_updated: {str(changed).lower()}") - - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/.agents/skills/add-model-01-prep/scripts/download_hf_weights.py b/.agents/skills/add-model-01-prep/scripts/download_hf_weights.py deleted file mode 100755 index 1890aaad6..000000000 --- a/.agents/skills/add-model-01-prep/scripts/download_hf_weights.py +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env python3 -"""Download HF weights using the standard FastVideo token env vars.""" - -from __future__ import annotations - -import argparse -import os -import sys -from pathlib import Path - -HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY") - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser( - description="Download a HF model snapshot or selected files into a local directory.") - parser.add_argument("repo_id", help="HF repo id, for example Org/Model") - parser.add_argument("local_dir", help="Destination directory") - parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)") - parser.add_argument("--revision", help="HF branch, tag, or commit") - parser.add_argument( - "--file-name", - action="append", - default=[], - help="Download one file; may be repeated. If omitted, download full snapshot.", - ) - parser.add_argument( - "--allow-pattern", - action="append", - default=[], - help="Snapshot allow pattern; may be repeated. Ignored when --file-name is used.", - ) - parser.add_argument( - "--ignore-pattern", - action="append", - default=[], - help="Snapshot ignore pattern; may be repeated. Ignored when --file-name is used.", - ) - return parser.parse_args() - - -def resolve_token() -> tuple[str | None, str | None]: - for key in HF_TOKEN_ENV_KEYS: - value = os.environ.get(key) - if value: - return key, value - return None, None - - -def main() -> int: - args = parse_args() - token_env, token = resolve_token() - local_dir = Path(args.local_dir).expanduser() - - if token_env: - print(f"token_env: {token_env}") - else: - print("token_env: none", file=sys.stderr) - - try: - if local_dir.exists() and not local_dir.is_dir(): - print( - f"error: destination exists and is not a directory: {local_dir}", - file=sys.stderr, - ) - return 1 - local_dir.mkdir(parents=True, exist_ok=True) - if args.file_name: - from huggingface_hub import hf_hub_download - - for file_name in args.file_name: - path = hf_hub_download( - repo_id=args.repo_id, - filename=file_name, - repo_type=args.repo_type, - revision=args.revision, - local_dir=str(local_dir), - token=token, - ) - print(f"downloaded_file: {path}") - else: - from huggingface_hub import snapshot_download - - path = snapshot_download( - repo_id=args.repo_id, - repo_type=args.repo_type, - revision=args.revision, - local_dir=str(local_dir), - token=token, - allow_patterns=args.allow_pattern or None, - ignore_patterns=args.ignore_pattern or None, - ) - print(f"downloaded_snapshot: {path}") - except Exception as exc: # noqa: BLE001 - CLI should print concise failures. - print(f"error: {exc}", file=sys.stderr) - return 1 - - print(f"local_dir: {local_dir.resolve()}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/.agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py b/.agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py deleted file mode 100755 index e7280ec6b..000000000 --- a/.agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py +++ /dev/null @@ -1,260 +0,0 @@ -#!/usr/bin/env python3 -"""Inspect a Hugging Face repo or local weight directory layout.""" - -from __future__ import annotations - -import argparse -import json -import os -import sys -from pathlib import Path -from typing import Any - -HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY") -RAW_WEIGHT_SUFFIXES = (".safetensors", ".pt", ".pth", ".ckpt", ".bin") -KNOWN_COMPONENTS = { - "audio_vae", - "conditioner", - "feature_extractor", - "image_encoder", - "scheduler", - "text_encoder", - "text_encoder_2", - "tokenizer", - "tokenizer_2", - "transformer", - "transformer_2", - "unet", - "upsampler", - "vae", - "vocoder", -} - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser( - description="Classify a HF repo or local directory as Diffusers, raw, custom, or unknown.") - parser.add_argument("source", help="HF repo id or local weights directory") - parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)") - parser.add_argument("--revision", help="HF revision to inspect") - parser.add_argument( - "--max-local-files", - type=int, - default=20000, - help="Maximum local files to scan recursively (default: 20000)", - ) - parser.add_argument( - "--sample-limit", - type=int, - default=80, - help="Number of file paths to print in human output (default: 80)", - ) - parser.add_argument("--json", action="store_true", help="Emit JSON only") - return parser.parse_args() - - -def resolve_token() -> tuple[str | None, str | None]: - for key in HF_TOKEN_ENV_KEYS: - value = os.environ.get(key) - if value: - return key, value - return None, None - - -def load_local_files(root: Path, max_files: int) -> tuple[list[str], bool]: - files: list[str] = [] - truncated = False - for path in root.rglob("*"): - if not path.is_file(): - continue - files.append(path.relative_to(root).as_posix()) - if len(files) >= max_files: - truncated = True - break - return sorted(files), truncated - - -def load_local_model_index(root: Path) -> tuple[dict[str, Any] | None, str | None]: - index_path = root / "model_index.json" - if not index_path.is_file(): - return None, None - try: - return json.loads(index_path.read_text()), None - except Exception as exc: # noqa: BLE001 - surface malformed JSON clearly. - return None, f"failed to parse local model_index.json: {exc}" - - -def load_remote_files( - repo_id: str, - repo_type: str, - revision: str | None, - token: str | None, -) -> list[str]: - from huggingface_hub import list_repo_files - - return sorted(list_repo_files( - repo_id, - repo_type=repo_type, - revision=revision, - token=token, - )) - - -def load_remote_model_index( - repo_id: str, - repo_type: str, - revision: str | None, - token: str | None, -) -> tuple[dict[str, Any] | None, str | None]: - from huggingface_hub import hf_hub_download - - try: - path = hf_hub_download( - repo_id=repo_id, - filename="model_index.json", - repo_type=repo_type, - revision=revision, - token=token, - ) - except Exception as exc: # noqa: BLE001 - missing/inaccessible file is data. - return None, f"failed to download model_index.json: {exc}" - - try: - return json.loads(Path(path).read_text()), None - except Exception as exc: # noqa: BLE001 - surface malformed JSON clearly. - return None, f"failed to parse remote model_index.json: {exc}" - - -def root_file_names(files: list[str]) -> set[str]: - return {name for name in files if "/" not in name} - - -def component_names(files: list[str], model_index: dict[str, Any] | None) -> list[str]: - components: set[str] = set() - for name in files: - parts = name.split("/", 1) - if len(parts) != 2: - continue - top, rest = parts - if top in KNOWN_COMPONENTS or rest == "config.json": - components.add(top) - - if model_index: - for key, value in model_index.items(): - if key.startswith("_"): - continue - if isinstance(value, list) and len(value) == 2: - components.add(key) - - return sorted(components) - - -def classify_layout( - files: list[str], - model_index: dict[str, Any] | None, - components: list[str], -) -> tuple[str, str]: - roots = root_file_names(files) - raw_weight_files = [name for name in roots if name.endswith(RAW_WEIGHT_SUFFIXES)] - has_model_index = "model_index.json" in roots or model_index is not None - - if has_model_index and components: - return "diffusers", "no" - if has_model_index: - return "custom", "unknown" - if raw_weight_files: - return "raw_official", "yes" - if any(name.endswith(RAW_WEIGHT_SUFFIXES) for name in files): - return "custom", "yes" - return "unknown", "unknown" - - -def build_result(args: argparse.Namespace) -> dict[str, Any]: - token_env, token = resolve_token() - source_path = Path(args.source).expanduser() - is_local = source_path.exists() - - if is_local: - root = source_path.resolve() - if not root.is_dir(): - raise ValueError(f"local source is not a directory: {root}") - files, truncated = load_local_files(root, args.max_local_files) - model_index, model_index_error = load_local_model_index(root) - source_kind = "local" - source = str(root) - else: - files = load_remote_files(args.source, args.repo_type, args.revision, token) - truncated = False - model_index, model_index_error = load_remote_model_index( - args.source, - args.repo_type, - args.revision, - token, - ) - source_kind = "hf" - source = args.source - - components = component_names(files, model_index) - source_layout, needs_conversion = classify_layout(files, model_index, components) - - return { - "source": source, - "source_kind": source_kind, - "repo_type": None if is_local else args.repo_type, - "revision": args.revision, - "token_env": token_env, - "source_layout": source_layout, - "needs_conversion": needs_conversion, - "model_index_class": (model_index or {}).get("_class_name"), - "model_index_diffusers_version": (model_index or {}).get("_diffusers_version"), - "model_index_error": model_index_error, - "components_seen": components, - "file_count": len(files), - "file_scan_truncated": truncated, - "files_sample": files[:args.sample_limit], - } - - -def print_human(result: dict[str, Any]) -> None: - for key in ( - "source", - "source_kind", - "repo_type", - "revision", - "token_env", - "source_layout", - "needs_conversion", - "model_index_class", - "model_index_diffusers_version", - "model_index_error", - "file_count", - "file_scan_truncated", - ): - value = result.get(key) - if value is not None: - print(f"{key}: {value}") - - components = result["components_seen"] - print("components_seen: " + (", ".join(components) if components else "none")) - print("files_sample:") - for name in result["files_sample"]: - print(f" {name}") - - -def main() -> int: - args = parse_args() - try: - result = build_result(args) - except Exception as exc: # noqa: BLE001 - CLI should print concise failures. - print(f"error: {exc}", file=sys.stderr) - return 1 - - if args.json: - print(json.dumps(result, indent=2, sort_keys=True)) - else: - print_human(result) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/.agents/skills/add-model-01-prep/templates/local_tests_readme.md b/.agents/skills/add-model-01-prep/templates/local_tests_readme.md deleted file mode 100644 index 24e317941..000000000 --- a/.agents/skills/add-model-01-prep/templates/local_tests_readme.md +++ /dev/null @@ -1,122 +0,0 @@ -# Local Tests - -Local-only parity and smoke tests for the `` FastVideo port. These -tests compare FastVideo against the official reference implementation and are -not expected to run in CI unless explicitly promoted later. - -Port progress, open questions, issues, and handoff notes live in -`tests/local_tests//PORT_STATUS.md`. - -## Reference Assets - -| Field | Value | -|---|---| -| Model family | `` | -| Workload types | `` | -| Official reference | `` | -| Local reference dir | `` | -| Official commit/version | `` | -| HF weights | `` | -| HF revision | `` | -| Local weights dir | `` | -| Source layout | `` | -| Needs conversion | `` | - -Do not write token values in this file. Use only the token env var name: -``. - -## Shared Environment Setup - -Run from the FastVideo repo root in the same conda/env used for FastVideo. -Do not create a separate upstream environment for parity tests. - -```bash -# Official reference source, if cloneable. -python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \ - "" \ - "" \ - --commit "" \ - --update-gitignore - -# Editable install without changing shared core pins. -uv pip install --no-deps -e ./ - -# Additional official deps installed or required for imports: -# -``` - -Do not change core dependency versions (`torch`, `diffusers`, `transformers`, -`flash-attn`, `triton`, CUDA packages) without explicit approval. - -## Official Environment Status - -```text -dependency_changes: -official_env_status: -private_dep_stubs: _upstream.py> -blocked_on: -``` - -## Weight Setup - -```bash -python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \ - "" \ - "official_weights/" \ - --revision "" -``` - -If weights are local-only, record the local path and do not copy large files into -the repository. - -## Prototype And Conversion Artifacts - -State-dict key/shape dumps are generated after FastVideo native prototypes exist -and are used to build the conversion mapping. - -```text -official_key_dumps: - : converted_weights//_mapping/_official_keys.json -fastvideo_key_dumps: - : converted_weights//_mapping/_fastvideo_keys.json -conversion_script: scripts/checkpoint_conversion/_to_diffusers.py -conversion_source_layout: -converted_weights_dir: converted_weights/ -strict_load_status: -``` - -For monolithic official checkpoints, record the component prefix split here. For -example, a single checkpoint may contain transformer, VAE/pretransform, -conditioner, and scheduler/vocoder keys that the conversion script writes into -separate FastVideo component subfolders. - -## Expected Parity Tests - -Planned local tests for this family: - -| Component | Official files / args | Test | Concerns | Status | -|---|---|---|---|---| -| `` | `` | `tests/local_tests//test___parity.py` | `` | `` | -| `pipeline` | `` | `tests/local_tests/pipelines/test__pipeline_parity.py` | `` | `` | - -Include reused components in this table. Reuse is accepted only after the -FastVideo component definition and official instantiation arguments have both -been checked and the component parity test passes non-skip. - -Run the relevant tests with: - -```bash -pytest tests/local_tests//test___parity.py -v -s -pytest tests/local_tests/pipelines/test__pipeline_parity.py -v -s -``` - -## Review Notes - -- Required before handoff: non-skip PASS for each required component parity - test, including reused components that own weights or numerical behavior. -- Pipeline parity may start as a scaffold, but final handoff requires non-skip - PASS or an explicit blocker accepted through the escape-hatch process. -- User decisions and pause points are tracked as `E###` rows in - `PORT_STATUS.md`; do not rely on chat history for escape-hatch context. -- Review agents should verify this README's setup commands still match the PR, - then run the listed parity tests or report the exact blocker. diff --git a/.agents/skills/add-model-01-prep/templates/port_status.md b/.agents/skills/add-model-01-prep/templates/port_status.md deleted file mode 100644 index 44392c1dc..000000000 --- a/.agents/skills/add-model-01-prep/templates/port_status.md +++ /dev/null @@ -1,69 +0,0 @@ -# Port Status - -## Summary - -- model_family: `` -- workload_types: `` -- official_ref: `` -- official_ref_dir: `` -- hf_weights_path: `` -- local_weights_dir: `` -- source_layout: `` -- local_tests_readme: `tests/local_tests//README.md` - -## Current Phase - -- phase: `prep` -- status: `in_progress` -- owner: `prep` -- last_updated: `` - -## Component Matrix - -| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues | -|---|---|---|---|---|---|---|---|---|---| -| `` | `` | `` | `` | `` | `` | `` | `` | `` | `` | - -## Conversion State - -- conversion_script: `scripts/checkpoint_conversion/_to_diffusers.py` -- converted_weights_dir: `converted_weights/` -- source_layout: `` -- strict_load_status: `not_run` -- passthrough_components: `` -- retry_history: `` - -## Parity Commands - -| Scope | Command | Last Result | Notes | -|---|---|---|---| -| component | `pytest tests/local_tests//test___parity.py -v -s` | `not_run` | `` | -| pipeline | `pytest tests/local_tests/pipelines/test__pipeline_parity.py -v -s` | `not_run` | `` | - -## Open Questions - -| ID | Question | Owner | Needed By Phase | Status | Resolution | -|---|---|---|---|---|---| -| Q001 | `` | `` | `` | `` | `` | - -## Issues And Blockers - -| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution | -|---|---|---|---|---|---|---|---|---| -| I001 | `` | `` | `` | `` | `` | `` | `` | `` | - -## Escape Hatches - -| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution | -|---|---|---|---|---|---|---| -| E001 | `` | `` | `` | `` | `` | `` | - -## Decisions - -| Date | Decision | Rationale | Impact | -|---|---|---|---| -| `` | `` | `` | `` | - -## Handoff Notes - -- `` diff --git a/.agents/skills/add-model-02-parity/SKILL.md b/.agents/skills/add-model-02-parity/SKILL.md deleted file mode 100644 index 5a96f34a5..000000000 --- a/.agents/skills/add-model-02-parity/SKILL.md +++ /dev/null @@ -1,234 +0,0 @@ ---- -name: add-model-02-parity -description: Use during /add-model after reference/architecture study to scaffold and later activate local FastVideo component parity tests. Emphasizes early test creation, official-reference loading, standardized FastVideo loading, and non-skip handoff gates. ---- - -# Add Model Parity - -## Goal - -Create parity tests as early as possible in a FastVideo port. The first pass can -land before conversion or component implementation as an executable scaffold; -handoff is blocked until the same tests become non-skip PASS with real weights. - -## When To Run - -Follow `../add-model/shared/common_rules.md` for token/auth safety, state files, -escape hatches, and skip/pass semantics. - -Run immediately after `/add-model` Phase 1 has identified: - -- official component classes and call signatures; -- FastVideo target component buckets/classes/configs; -- local reference clone or import path from `add-model-01-prep`; -- local raw or Diffusers weight path; -- `official_env_status=imports_ok`, or private deps that will be stubbed - locally in tests; -- `local_tests_readme` documenting setup and planned review/test commands; -- expected component inputs and output tensors. - -Do not wait for all FastVideo components to be implemented. Write the tests -first, then let component-porting subagents make them pass. - -## Outputs - -- One component parity test per required component, including reused components: - `tests/local_tests//test___parity.py`. -- Optional helper for upstream private deps: - `tests/local_tests/helpers/_upstream.py`. -- Pipeline parity is owned later by `../add-model-09-pipeline/SKILL.md` after all - component parity tests pass non-skip. -- A parity status block for the `/add-model` parity verification phase. - -## Early Scaffold Rules - -- A scaffold may skip while the FastVideo class, converted weights, or official - import is missing. -- A scaffold must already encode the real official load path, FastVideo load - path, deterministic inputs, expected output extraction, and tolerance target. -- Each parity test must declare its coverage scope in the file docstring or a - module constant: `production_loader`, `implementation_subcomponent`, or `both`. - Implementation/subcomponent parity may bypass production loaders deliberately, - but final handoff still needs production-loader coverage somewhere before the - pipeline depends on that component. -- Official reference imports must run in the current FastVideo environment; do - not create or assume a separate upstream venv/conda env. -- A scaffold is not evidence of correctness. It becomes evidence only after a - local non-skip PASS. -- Prefer env-var path overrides with repo-relative defaults. -- Keep tests local-only under `tests/local_tests/`; package/CI quality tests are - added later. -- Update shared state files as described in - `../add-model/shared/common_rules.md` whenever adding or activating parity - tests. - -## Component Template - -Copy `templates/component_parity_test.py` and fill every `TODO` marker. The -template is distilled from: - -- `tests/local_tests/transformers/test_ltx2.py` -- `tests/local_tests/transformers/test_gamecraft_parity.py` -- `tests/local_tests/encoders/test_ltx2_gemma_parity.py` -- `tests/local_tests/vaes/test_oobleck_vae_parity.py` -- `tests/local_tests/sd35/test_sd35_component_parity.py` - -The template supports three states: - -| State | Meaning | -|---|---| -| Scaffold skip | Test is committed early, but official import, FastVideo class, or weights are not available yet. | -| Debug red | Both sides load and the test fails numerically. This is useful: porting can chase the first drift. | -| Non-skip pass | Required before `/add-model` handoff. | - -## Subagent Dispatch Pattern - -After Phase 1, dispatch one parity subagent per component before or alongside -component implementation: - -```text -Create a local parity test scaffold for . - -Use the prep handoff: -- official_ref_dir/import: <...> -- local_weights_dir: <...> -- source_layout: <...> -- needs_conversion: -- official_env_status: -- local_tests_readme: tests/local_tests//README.md -- port_state_file: tests/local_tests//PORT_STATUS.md -- official_definition_files: -- official_instantiation_files: -- concerns_or_unknowns: - -The complete per-component packet must match -`../add-model/contracts/component_context.md`. - -Read the official component call path and the planned FastVideo component API. -Add tests/local_tests//test___parity.py based on -add-model-02-parity/templates/component_parity_test.py. - -The scaffold must load the official model with real weights when available, -load the FastVideo model through the standardized config/class/loader path when -available, create deterministic inputs, compare concrete outputs, and skip only -when a dependency is genuinely missing. Do not make an unconditional skip or a -shape-only test. -``` - -## FastVideo Load Patterns - -Pick the narrowest load path that matches the component: - -| Component | Preferred FastVideo load path | -|---|---| -| DiT / transformer | Bucket config + model class, or `TransformerLoader` when testing converted Diffusers component dirs. | -| VAE | VAE class `from_pretrained(...)` when implemented, or bucket config + class for local converted dirs. | -| Text/image encoder | Bucket config + model class; pass HF subpaths from `local_weights_dir` or converted component dirs. | -| Scheduler/conditioner | Native class/config plus exact official kwargs. | - -For early scaffolds, an import of the planned FastVideo class may be inside a -helper that calls `pytest.skip` if the class does not exist yet. Replace that -skip with a real import once the component PR adds the class. - -Direct class/config construction is allowed for implementation or subcomponent -parity, such as connector-only encoder checks or official monolithic-checkpoint -mapping tests. Label that scope explicitly and add separate production-loader -coverage when converted component dirs are available. - -## Official Load Patterns - -- Clone/reference repo path: add its source dir to `sys.path` before imports. -- HF/Diffusers reference: import only inside the test, not production code. -- Private deps: add a helper under `tests/local_tests/helpers/` to install - stubs before importing upstream modules; do not rely on an external upstream - environment. -- Gated HF repos: resolve `HF_TOKEN`, `HUGGINGFACE_HUB_TOKEN`, or `HF_API_KEY` - under the token rules in `../add-model/shared/common_rules.md`. - -## Non-Skip Activation Checklist - -Before `/add-model` handoff, each scaffolded test must be activated: - -```text -[ ] Official side imports and loads real weights. -[ ] FastVideo side imports and loads the converted or original weights. -[ ] Test executes at least one real forward call on both sides. -[ ] Test compares output tensors, not only shapes or state-dict keys. -[ ] Local pytest output contains PASSED, not SKIPPED or XFAIL. -[ ] Tolerance is justified for the component scope and kernel alignment. -``` - -## Component Parity Details - -Reference imports: - -- Import from `official_ref_dir` or the recorded package/import path. -- If upstream has private deps, add a helper under - `tests/local_tests/helpers/_upstream.py` that installs minimal stubs - before importing upstream modules. -- Common stubs: identity compile/op-registration decorators, CP world size set to - 1, identity scatter/gather, and test-friendly custom-op kernels. -- Stub decorators that register `torch.ops..` must preserve the - `torch.library` registration side-effect. Identity decorators alone are not - enough. -- Delete stub helpers and every `install_stubs()` call as soon as the real deps - become required installs. No-op shims are dead code. - -Kernel and wrapper pitfalls: - -- If parity routes flash-attn GQA through SDPA, expand KV heads manually on the - SDPA side with `repeat_interleave` along the head axis. -- If upstream VAE `decode()` denormalizes internally but FastVideo/Diffusers - expects pre-denormalized latents, apply `z = z * std + mean` only on the - FastVideo side in the parity test. -- Per-channel VAE `latents_mean` / `latents_std` must be reshaped explicitly, - e.g. `.view(1, z_dim, 1, 1, 1)` for 5D video latents. - -Tolerance guide: - -| Scope | Start `atol` / `rtol` | Notes | -|---|---|---| -| Single block, same kernel | `1e-4` / `1e-4` | Tight default. | -| Full DiT, aligned kernels | `1e-2` / `1e-2` | Cross-layer accumulation. | -| Full DiT, cross-kernel bf16 | `0.1` / `0.1` | Also require abs-mean drift below 5% and per-modality diagnostics. | -| VAE decode fp32 | `5e-2` / `5e-2` | After normalization alignment. | -| Encoder wrapper around same HF class | `1e-3` / `1e-3` | Should be near-zero. | - -Element-wise `assert_close` alone is not enough for deep full-DiT parity. Also -log global abs-mean drift and per-modality summaries. - -When a non-skip component parity run is numerically red after weight/input -checks, invoke `../add-model-08-trace/SKILL.md` before adding bespoke forward -hooks. Use `docs/contributing/activation_trace.md` to keep -`FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and `FASTVIDEO_TRACE_STEPS` -identical across FastVideo and upstream traces. - -Useful local commands: - -```bash -pytest tests/local_tests//test__*parity*.py -v -s -pytest tests/local_tests -k " and parity" -v -s -``` - -## Escape Hatches - -Follow `../add-model/shared/common_rules.md`. Parity-specific ask cases include -private dependency approval, choosing between incompatible official references, -accepting a shape-only substitute, or loosening required tolerances. - -## Pipeline Parity - -Pipeline parity is later than component parity because it needs stages, presets, -registry wiring, converted weights, and green component parity. Record official -pipeline call notes in `local_tests_readme`, but do not treat pipeline parity as -owned by this skill. - -Use `../add-model-09-pipeline/SKILL.md` and its -`templates/pipeline_parity_test.py` for pipeline parity scaffolding and -debugging. Compare denoised latents or decoded media, not just successful -generation. - -## Handoff Status Block - -Return `../add-model/contracts/parity_status.md` to `/add-model` and update the -shared state files before handoff. diff --git a/.agents/skills/add-model-02-parity/templates/component_parity_test.py b/.agents/skills/add-model-02-parity/templates/component_parity_test.py deleted file mode 100644 index 8d24e292a..000000000 --- a/.agents/skills/add-model-02-parity/templates/component_parity_test.py +++ /dev/null @@ -1,188 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -"""Component parity scaffold for . - -This file is intended to be created early in a port. It may skip until the -official reference, FastVideo class, and real weights are available, but it must -never become an unconditional skip or shape-only test. - -Fill every TODO before considering this test active. -""" -from __future__ import annotations - -import importlib -import os -from pathlib import Path -import sys - -import pytest -import torch -from torch.testing import assert_close - -os.environ.setdefault("MASTER_ADDR", "localhost") -os.environ.setdefault("MASTER_PORT", "29519") -os.environ.setdefault("DISABLE_SP", "1") -os.environ.setdefault("FASTVIDEO_ATTENTION_BACKEND", "TORCH_SDPA") - -REPO_ROOT = Path(__file__).resolve().parents[3] -FAMILY = "" # TODO: snake_case family name. -COMPONENT = "" # TODO: transformer | vae | encoder | conditioner | ... -PARITY_SCOPE = "implementation_subcomponent" # TODO: production_loader | implementation_subcomponent | both -OFFICIAL_MODULE = "" # TODO: e.g. "ltx_core.model.transformer". -OFFICIAL_CLASS = "" # TODO: official class/factory name. -FASTVIDEO_CONFIG_MODULE = "fastvideo.configs.models." # TODO. -FASTVIDEO_CONFIG_CLASS = "" # TODO. -FASTVIDEO_MODEL_MODULE = "fastvideo.models.." # TODO. -FASTVIDEO_MODEL_CLASS = "" # TODO. - -OFFICIAL_REF_DIR = Path(os.getenv("_OFFICIAL_REF_DIR", REPO_ROOT / "")) -LOCAL_WEIGHTS_DIR = Path(os.getenv("_LOCAL_WEIGHTS_DIR", REPO_ROOT / "official_weights" / FAMILY)) -CONVERTED_WEIGHTS_DIR = Path(os.getenv("_CONVERTED_WEIGHTS_DIR", - REPO_ROOT / "converted_weights" / FAMILY)) - - -def _resolve_hf_token() -> str | None: - for key in ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY"): - value = os.environ.get(key) - if value: - return value - return None - - -def _add_official_to_path() -> None: - """Add the official source path before importing upstream modules.""" - # TODO: adjust for the official repo layout. Common examples: - # OFFICIAL_REF_DIR / "src" - # OFFICIAL_REF_DIR / "packages" / "" / "src" - # OFFICIAL_REF_DIR - official_src = OFFICIAL_REF_DIR / "src" - if not official_src.exists(): - official_src = OFFICIAL_REF_DIR - if official_src.exists() and str(official_src) not in sys.path: - sys.path.insert(0, str(official_src)) - - -def _import_or_skip(module_name: str, attr_name: str | None = None): - if "<" in module_name or (attr_name is not None and "<" in attr_name): - pytest.skip(f"Template import placeholder not filled: {module_name}.{attr_name}") - try: - module = importlib.import_module(module_name) - except Exception as exc: # noqa: BLE001 - local parity should skip missing refs. - pytest.skip(f"Cannot import {module_name}: {exc}") - if attr_name is None: - return module - try: - return getattr(module, attr_name) - except AttributeError: - pytest.skip(f"{module_name} has no attribute {attr_name}") - - -def _load_official_model(device: torch.device, dtype: torch.dtype) -> torch.nn.Module: - """Load the official component with real weights.""" - _add_official_to_path() - if not OFFICIAL_REF_DIR.exists(): - pytest.skip(f"Official reference missing: {OFFICIAL_REF_DIR}") - if not LOCAL_WEIGHTS_DIR.exists(): - pytest.skip(f"Local weights missing: {LOCAL_WEIGHTS_DIR}") - - # TODO: import official class/factory and load real weights strictly. - # Examples in-tree: - # - LTX2: SingleGPUModelBuilder(...).build(device=device, dtype=dtype) - # - GameCraft: torch.load(...)["module"] -> official_model.load_state_dict(...) - # - Oobleck: create_model_from_config(config) + ckpt state_dict - OfficialClass = _import_or_skip(OFFICIAL_MODULE, OFFICIAL_CLASS) - model = OfficialClass() # TODO: pass official config kwargs. - state_dict = {} # TODO: load official state dict from LOCAL_WEIGHTS_DIR. - missing, unexpected = model.load_state_dict(state_dict, strict=True) - assert not missing and not unexpected, (f"official load mismatch missing={missing[:5]} unexpected={unexpected[:5]}") - return model.to(device=device, dtype=dtype).eval() - - -def _load_fastvideo_model(device: torch.device, dtype: torch.dtype) -> torch.nn.Module: - """Load the FastVideo component with the same tensor content.""" - if not CONVERTED_WEIGHTS_DIR.exists() and not LOCAL_WEIGHTS_DIR.exists(): - pytest.skip(f"No FastVideo loadable weights: {CONVERTED_WEIGHTS_DIR} or {LOCAL_WEIGHTS_DIR}") - - # TODO: replace with the bucket-specific FastVideo config/class/loader. - # DiT examples: - # from fastvideo.configs.models.dits import - # from fastvideo.models.dits. import - # VAE examples: - # from fastvideo.models.vaes. import - # model = .from_pretrained(...) - FastVideoConfig = _import_or_skip(FASTVIDEO_CONFIG_MODULE, FASTVIDEO_CONFIG_CLASS) - FastVideoModel = _import_or_skip(FASTVIDEO_MODEL_MODULE, FASTVIDEO_MODEL_CLASS) - - config = FastVideoConfig() - model = FastVideoModel(config=config) - state_dict = {} # TODO: load converted or directly mapped state dict. - missing, unexpected = model.load_state_dict(state_dict, strict=True) - assert not missing and not unexpected, ( - f"FastVideo load mismatch missing={missing[:5]} unexpected={unexpected[:5]}") - return model.to(device=device, dtype=dtype).eval() - - -def _make_inputs(device: torch.device, dtype: torch.dtype) -> dict[str, torch.Tensor]: - """Create deterministic inputs matching the official component call.""" - torch.manual_seed(0) - # TODO: replace with component-specific tensors and metadata. - return { - "hidden_states": torch.randn(1, 4, 16, device=device, dtype=dtype), - "timestep": torch.tensor([10], device=device), - } - - -def _run_official(model: torch.nn.Module, inputs: dict[str, torch.Tensor]) -> torch.Tensor: - """Run official component and return the tensor to compare.""" - with torch.inference_mode(): - output = model(**inputs) # TODO: adapt official call signature. - if isinstance(output, dict): - sample = output.get("sample") - output = sample if sample is not None else output.get("x") - elif hasattr(output, "sample"): - output = output.sample - elif isinstance(output, tuple): - output = output[0] - assert torch.is_tensor(output), f"official output is not tensor: {type(output)}" - return output.detach().float().cpu() - - -def _run_fastvideo(model: torch.nn.Module, inputs: dict[str, torch.Tensor]) -> torch.Tensor: - """Run FastVideo component and return the tensor to compare.""" - with torch.inference_mode(): - output = model(**inputs) # TODO: adapt FastVideo call signature. - if isinstance(output, dict): - sample = output.get("sample") - output = sample if sample is not None else output.get("x") - elif hasattr(output, "sample"): - output = output.sample - elif isinstance(output, tuple): - output = output[0] - assert torch.is_tensor(output), f"FastVideo output is not tensor: {type(output)}" - return output.detach().float().cpu() - - -@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required for this parity test.") -def test_component_parity(): - """Compare official and FastVideo outputs on identical inputs.""" - device = torch.device("cuda:0") - dtype = torch.bfloat16 - - official = _load_official_model(device, dtype) - fastvideo = _load_fastvideo_model(device, dtype) - inputs = _make_inputs(device, dtype) - - official_out = _run_official(official, inputs) - fastvideo_out = _run_fastvideo(fastvideo, inputs) - - assert official_out.shape == fastvideo_out.shape - diff = (official_out - fastvideo_out).abs() - print(f"official abs_mean={official_out.abs().mean().item():.6f} " - f"fastvideo abs_mean={fastvideo_out.abs().mean().item():.6f} " - f"diff_max={diff.max().item():.6f} diff_mean={diff.mean().item():.6f}") - - # TODO: pick tolerance by scope: - # - single block / same kernel: 1e-4 - # - full DiT aligned kernels: 1e-2 - # - full DiT cross-kernel bf16: 1e-1 + abs_mean drift check - # - VAE decode fp32: 5e-2 after normalization alignment - assert_close(fastvideo_out, official_out, atol=1e-4, rtol=1e-4) diff --git a/.agents/skills/add-model-03-port-dit/SKILL.md b/.agents/skills/add-model-03-port-dit/SKILL.md deleted file mode 100644 index 028f6b346..000000000 --- a/.agents/skills/add-model-03-port-dit/SKILL.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -name: add-model-03-port-dit -description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native DiT/transformer component. ---- - -# Add Model Port DiT - -## Goal - -Prototype or parity-debug one diffusion transformer in FastVideo-native code. -This skill is for one component only; do not work on the VAE, encoders, -pipeline, or unrelated conversion code unless the current component cannot load -without a minimal fix there. - -## Inputs - -Follow `../add-model/shared/component_skill_common.md` and require the complete -packet from `../add-model/contracts/component_context.md`. - -DiT-specific packet fields: - -- `component`: transformer or DiT name. -- `parity_test`: `tests/local_tests//test___parity.py`. -- `weights`: converted transformer dir or local official path. -- `target_files`: `fastvideo/models/dits/.py` and - `fastvideo/configs/models/dits/.py`. - -## Modes - -Use the common prototype and parity-debug modes from -`../add-model/shared/component_skill_common.md`. - -DiT-specific prototype concerns include ambiguous official flags, shape -mismatches, missing FastVideo layer equivalents, and dedicated output heads. - -## Reuse Proof - -Apply the shared reuse proof. DiT-specific comparison must include attention -algorithm, positional embeddings, RoPE/patching, timestep/guidance embeddings, -scaling constants, dtype casts, state-dict names, and every output head. - -## Existing FastVideo Patterns - -- Base class: `fastvideo/models/dits/base.py::BaseDiT`. -- Config bases: `DiTConfig` and `DiTArchConfig` in - `fastvideo/configs/models/dits/base.py`. -- Use the matching DiT config bucket. Wrong bucket inheritance can typecheck but - fail during pipeline wiring. -- Config export: add the config to - `fastvideo/configs/models/dits/__init__.py`. -- Registry discovery: set `EntryClass = ` in the model file. -- Loader path: `TransformerLoader` reads `transformer/config.json`, calls - `dit_config.update_model_arch(config)`, resolves `_class_name` through - `ModelRegistry`, and constructs the class with `config` and `hf_config`. -- Reference examples: `stable_audio.py`, `wanvideo.py`, `sd3.py`, `longcat.py`, - and `ltx2.py`. -- Layer guidance: `fastvideo/layers/AGENTS.md`. - -## Implementation Rules - -- Use FastVideo-native layers by default: `ReplicatedLinear` for DiT hot-path - linears, `DistributedAttention` for standard full-sequence attention, and - `LocalAttention` for local/window attention or simple single-GPU parity paths. -- Raw SDPA is acceptable for cross-modality flat streams when no FastVideo - distributed equivalent exists; document the SP gap in the module docstring. -- Mirror official tensor contracts exactly: latent packing, patch ordering, - timestep embedding scale, RoPE/positional embedding, guidance embedding, - cross-attention context order, output head order, and dtype casts. -- Preserve all output heads that the official DiT emits. Do not silently drop - audio, depth, pose, mask, or auxiliary heads. -- Put architecture fields on `DiTArchConfig`; keep inference steps, CFG scales, - FPS, flow shift, and sampling defaults out of the arch config. -- Define `_fsdp_shard_conditions`, `_compile_conditions`, - `param_names_mapping`, and `reverse_param_names_mapping` where needed. -- Follow the production import boundary in - `../add-model/shared/common_rules.md`. - -## Prototype Checks - -Follow the shared prototype success criteria. A useful one-off check is: - -```bash -python - <<'PY' -# Import the target config/class, instantiate with random weights, and print -# state_dict names/shapes for the conversion mapping. -PY -``` - -## Parity-Debug Loop - -Run the shared parity-debug loop. The component test command is: - -```bash -pytest -v -s -``` - -For numerical drift, use `../add-model-08-trace/SKILL.md` before writing bespoke -hooks. Start with FastVideo's activation trace (`fastvideo/hooks/activation_trace.py`; -`docs/contributing/activation_trace.md`) and a block-level regex such as -`FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$"`. Only fall back to custom -per-block hooks if the needed boundary or statistic is not exposed by -`FASTVIDEO_TRACE_STATS`. - -## Escape Hatches - -Follow `../add-model/shared/common_rules.md` and the component-specific guidance -in `../add-model/shared/component_skill_common.md`. DiT-specific ask cases include -dropping an output head/modality, accepting an unsupported kernel/private op, or -choosing between incompatible official transformer definitions. - -## Handoff - -Return `../add-model/contracts/component_skill_handoff.md` following the common -handoff rules in `../add-model/shared/component_skill_common.md`. diff --git a/.agents/skills/add-model-04-port-vae/SKILL.md b/.agents/skills/add-model-04-port-vae/SKILL.md deleted file mode 100644 index 16c50bc3f..000000000 --- a/.agents/skills/add-model-04-port-vae/SKILL.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -name: add-model-04-port-vae -description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native VAE component. ---- - -# Add Model Port VAE - -## Goal - -Prototype or parity-debug one VAE or autoencoder in FastVideo-native code. This -skill covers video, image, and audio VAEs. - -## Inputs - -Follow `../add-model/shared/component_skill_common.md` and require the complete -packet from `../add-model/contracts/component_context.md`. - -VAE-specific packet fields: - -- `component`: VAE or autoencoder name. -- `parity_test`: `tests/local_tests/vaes/test___parity.py`. -- `weights`: converted VAE dir, HF subfolder, or local official path. -- `target_files`: `fastvideo/models/vaes/.py` and - `fastvideo/configs/models/vaes/.py`. - -## Modes - -Use the common prototype and parity-debug modes from -`../add-model/shared/component_skill_common.md`. - -VAE-specific prototype concerns include latent normalization, stochastic -posterior behavior, tiling incompatibility, temporal/spatial/audio layout, and -decode output containers. - -## Reuse Proof - -Apply the shared reuse proof. VAE-specific comparison must include latent layout, -temporal/spatial/audio compression, scaling factor, mean/std normalization, -posterior behavior, encode/decode output objects, tiling flags, and cropping. - -## Existing FastVideo Patterns - -- Shared tiling wrapper: `fastvideo/models/vaes/common.py::ParallelTiledVAE`. -- Config bases: `VAEConfig` and `VAEArchConfig` in - `fastvideo/configs/models/vaes/base.py`. -- Use the matching VAE config bucket. Wrong bucket inheritance can typecheck but - fail during pipeline wiring. -- Config export: add the config to - `fastvideo/configs/models/vaes/__init__.py`. -- Registry discovery: set `EntryClass = ` in the model file. -- Loader path: VAE loaders resolve `_class_name` through `ModelRegistry` and - load converted component weights from the VAE subdir. -- Reference examples: `oobleck.py`, `autoencoder_kl.py`, `wanvae.py`, - `ltx2vae.py`, and `gamecraftvae.py`. -- Layer guidance: `fastvideo/layers/AGENTS.md`. - -## Implementation Rules - -- Name reusable VAE architectures by architecture (`oobleck.py`, - `autoencoder_kl.py`); name family-specific VAEs by family. -- Match official encode/decode contracts exactly: input layout, latent layout, - temporal/spatial/audio compression, scaling factor, mean/std normalization, - posterior sampling behavior, decode output object, and frame/sample cropping. -- Compare deterministic outputs in parity: decode outputs, encode mean/mode, or - round-trip tensors. Do not compare stochastic samples unless the RNG path is - explicitly controlled. -- Use FastVideo tiling only when it preserves official numerics for the tested - shape; disable it in config for audio or unsupported dimensions. -- Put architecture constants on `VAEArchConfig`; put `load_encoder`, - `load_decoder`, tiling, dtype, and pretrained path fields on `VAEConfig`. -- Follow the production import boundary in - `../add-model/shared/common_rules.md`. - -## Prototype Checks - -Follow the shared prototype success criteria. - -## Parity-Debug Loop - -Run the shared parity-debug loop. The component test command is: - -```bash -pytest -v -s -``` - -For numerical drift, check normalization, latent scaling, posterior mode vs -sample, channel order, and temporal/spatial/audio cropping before changing -layers. - -## Escape Hatches - -Follow `../add-model/shared/common_rules.md` and the component-specific guidance -in `../add-model/shared/component_skill_common.md`. VAE-specific ask cases include -dropping an encode/decode path, accepting an unsupported private op, or choosing -between incompatible official VAE definitions. - -## Handoff - -Return `../add-model/contracts/component_skill_handoff.md` following the common -handoff rules in `../add-model/shared/component_skill_common.md`. diff --git a/.agents/skills/add-model-05-port-encoder/SKILL.md b/.agents/skills/add-model-05-port-encoder/SKILL.md deleted file mode 100644 index 42760ad6a..000000000 --- a/.agents/skills/add-model-05-port-encoder/SKILL.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -name: add-model-05-port-encoder -description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native text, image, audio, or compound encoder component. ---- - -# Add Model Port Encoder - -## Goal - -Prototype or parity-debug one encoder or encoder-like conditioner in -FastVideo-native code. Use this for text encoders, image encoders, audio -encoders, and compound conditioners that fit the encoder config/loader bucket. - -## Inputs - -Follow `../add-model/shared/component_skill_common.md` and require the complete -packet from `../add-model/contracts/component_context.md`. - -Encoder-specific packet fields: - -- `component`: encoder or encoder-like conditioner name. -- `parity_test`: `tests/local_tests/encoders/test___parity.py`. -- `weights`: converted encoder dir, HF subfolder, or external HF id. -- `target_files`: `fastvideo/models/encoders/.py` and - `fastvideo/configs/models/encoders/.py`. - -## Modes - -Use the common prototype and parity-debug modes from -`../add-model/shared/component_skill_common.md`. - -Encoder-specific prototype concerns include tokenizer kwargs, hidden-state -extraction, output packing, connector order, and external/passthrough weight -needs. - -## Reuse Proof - -Apply the shared reuse proof. Encoder-specific comparison must include tokenizer -contracts, hidden-state extraction, masks, positional IDs, output packing, -connector/projection ordering, passthrough paths, and returned dataclass shape. - -## Existing FastVideo Patterns - -- Base classes: `TextEncoder` and `ImageEncoder` in - `fastvideo/models/encoders/base.py`. -- Output type: `BaseEncoderOutput`. -- Config bases: `TextEncoderConfig`, `ImageEncoderConfig`, - `TextEncoderArchConfig`, and `ImageEncoderArchConfig` in - `fastvideo/configs/models/encoders/base.py`. -- Use the matching encoder config bucket. Wrong bucket inheritance can typecheck - but fail during pipeline wiring. -- Config export: add the config to - `fastvideo/configs/models/encoders/__init__.py`. -- Registry discovery: set `EntryClass = ` or a list of class names in - the model file. -- Reference examples: native `t5.py`, `clip.py`, `siglip.py`, `llama.py`, - `qwen2_5.py`, `gemma.py`, and compound `stable_audio_conditioner.py`. -- Layer guidance: `fastvideo/layers/AGENTS.md`. - -## Implementation Rules - -- Reuse tokenizers and pure data utilities when needed, but do not add runtime - third-party model-class imports as a placeholder for a component that owns - weights or numerical behavior. -- For LLM-style encoders, follow existing tensor-parallel patterns such as - `QKVParallelLinear`, `MergedColumnParallelLinear`, `RowParallelLinear`, - `VocabParallelEmbedding`, and `RMSNorm` when matching native examples. -- Match official hidden-state extraction exactly: layer index, pooled output, - attention mask dtype, padding side, truncation, special tokens, final norm, - output_hidden_states, and returned tuple/dataclass shape. -- For connector or conditioner modules, preserve sub-conditioner order and the - exact packing of cross-attention tokens, masks, and global conditioning. -- Put tokenizer kwargs and architecture constants on the arch config when they - affect numerical behavior. -- If an external HF encoder is explicitly accepted as a lazy wrapper, keep it - isolated, document why it is not a native port, and still require parity for - the wrapper's output contract. - -Hybrid external-HF encoder checklist: - -- Put external model folders in passthrough subfolders such as - `text_encoder//`, or record a root `model_index.json` path field - that the loader resolves to a local directory. -- Keep external model parameters out of the FastVideo-owned state-dict surface - when the external model is loaded lazily from its own HF files. -- Convert and strict-check only the FastVideo-owned connector/projection weights; - document external model weights as passthrough. -- Add parity for the wrapper's final output contract and, when useful, a narrower - connector-only parity test that labels its scope as - `implementation_subcomponent`. -- Verify the production loader resolves the same external path used by the - pipeline, not just the direct class used in the parity test. - -## Prototype Checks - -Follow the shared prototype success criteria. - -## Parity-Debug Loop - -Run the shared parity-debug loop. The component test command is: - -```bash -pytest -v -s -``` - -For numerical drift, check tokenization, masks, hidden-state selection, -positional IDs, dtype/autocast, and output packing before changing layers. - -## Escape Hatches - -Follow `../add-model/shared/common_rules.md` and the component-specific guidance -in `../add-model/shared/component_skill_common.md`. Encoder-specific ask cases -include accepting private model-code execution, choosing between incompatible -tokenizer/encoder references, or dropping a required conditioning stream. - -## Handoff - -Return `../add-model/contracts/component_skill_handoff.md` following the common -handoff rules in `../add-model/shared/component_skill_common.md`. diff --git a/.agents/skills/add-model-06-port-generic/SKILL.md b/.agents/skills/add-model-06-port-generic/SKILL.md deleted file mode 100644 index 3ba8fb864..000000000 --- a/.agents/skills/add-model-06-port-generic/SKILL.md +++ /dev/null @@ -1,111 +0,0 @@ ---- -name: add-model-06-port-generic -description: Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one non-DiT, non-VAE, non-encoder FastVideo component. ---- - -# Add Model Port Generic - -## Goal - -Prototype or parity-debug one scheduler, conditioner, upsampler, vocoder, -adapter, preprocessor, or unknown component in FastVideo-native code. - -## Inputs - -Follow `../add-model/shared/component_skill_common.md` and require the complete -packet from `../add-model/contracts/component_context.md`. - -Generic-component packet fields: - -- `component`: component name. -- `component_type`: scheduler, conditioner, upsampler, vocoder, adapter, - preprocessor, or unknown. -- `parity_test`: `tests/local_tests//test___parity.py`. -- `weights`: converted component dir, HF subfolder, or none. -- `target_files`: matching `fastvideo/models/` and `fastvideo/configs/models/` - bucket files when applicable. - -## Modes - -Use the common prototype and parity-debug modes from -`../add-model/shared/component_skill_common.md`. - -Generic-component prototype concerns include stateless/stateful ambiguity, -missing loader buckets, source prefixes, mutable scheduler state, and output -container shape. - -## Reuse Proof - -Apply the shared reuse proof. Generic-component comparison must include mutable -state, scaling constants, scheduler/conditioner semantics, output containers, and -whether the component owns state or is stateless. - -## Existing FastVideo Patterns - -- Schedulers live under `fastvideo/models/schedulers/` and expose `EntryClass`. -- Upsamplers use `fastvideo/models/upsamplers/` plus configs under - `fastvideo/configs/models/upsamplers/`; see `hunyuan15.py`. -- Vocoders and audio-specific modules can live under `fastvideo/models/audio/` - with configs under `fastvideo/configs/models/audio/`; see `ltx2_audio_vae.py`. -- Compound conditioners may fit the encoder bucket when the pipeline loader uses - `ConditionerLoader`; see `stable_audio_conditioner.py`. -- Registry discovery uses `EntryClass`; config bucket exports are required when - pipeline configs import them by bucket. -- Use the narrowest matching config bucket. Wrong bucket inheritance can typecheck - but fail during pipeline wiring. -- Layer guidance: `fastvideo/layers/AGENTS.md`. - -## Bucket Decision - -- If the component is a transformer/DiT, stop and use `add-model-03-port-dit`. -- If the component is a VAE/autoencoder, stop and use `add-model-04-port-vae`. -- If the component is a text/image/audio encoder or encoder-like conditioner, - stop and use `add-model-05-port-encoder` unless the loader requires a different - bucket. -- Otherwise choose the narrowest existing bucket. Add a new bucket only when no - existing loader/config shape can represent the component without misleading - names or unsafe runtime behavior. - -## Implementation Rules - -- Match official behavior, not just shapes: constructor args, default values, - runtime flags, RNG use, dtype/autocast, scaling constants, masks, and output - containers all matter. -- Keep the implementation minimal and native. Do not keep a runtime import of - the official implementation as the production component. -- For schedulers, compare timesteps, sigmas/noise levels, step outputs, shift - handling, prediction type, and any mutable internal state. -- For upsamplers, compare resize mode, align_corners, residual branches, - causal padding, normalization, and exact target-shape behavior. -- For vocoders/audio components, compare waveform shape, sample-rate contract, - channel order, hop length, normalization, and dtype. -- If private upstream deps are required only for tests, keep stubs under - `tests/local_tests/helpers/` and do not import them from production code. - -## Prototype Checks - -Follow the shared prototype success criteria. - -## Parity-Debug Loop - -Run the shared parity-debug loop. The component test command is: - -```bash -pytest -v -s -``` - -For numerical drift, add targeted intermediate comparisons in the test to -identify the first divergent operation. - -## Escape Hatches - -Follow `../add-model/shared/common_rules.md` and the component-specific guidance -in `../add-model/shared/component_skill_common.md`. Generic-component ask cases -include creating a new loader bucket, accepting an unsupported private op, -choosing between incompatible official definitions, or dropping a required -component. - -## Handoff - -Return `../add-model/contracts/component_skill_handoff.md` following the common -handoff rules in `../add-model/shared/component_skill_common.md`. diff --git a/.agents/skills/add-model-07-conversion/SKILL.md b/.agents/skills/add-model-07-conversion/SKILL.md deleted file mode 100644 index 727b80f06..000000000 --- a/.agents/skills/add-model-07-conversion/SKILL.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -name: add-model-07-conversion -description: Use during /add-model Phase 5 to write and verify a FastVideo checkpoint conversion script after native component prototypes expose FastVideo state-dict keys/shapes. ---- - -# Add Model Conversion - -## Goal - -Convert official weights into a FastVideo-loadable component layout after Phase 4 -native prototypes exist. The conversion script owns parameter mapping, component -splitting, passthrough assets, config emission, and strict-load verification. - -## Inputs - -Follow `../add-model/shared/common_rules.md` for token/auth safety, state files, -escape hatches, production boundaries, and skip/pass semantics. - -Require the initial request from -`../add-model/contracts/conversion_request.md`. - -If the FastVideo key/shape dump is missing, return to `/add-model` Phase 4. Do -not write a final mapping against an unimplemented component. - -For Phase 6 retry requests from component skills, also require the retry shape -from `../add-model/contracts/conversion_request.md`. - -## Output - -- `scripts/checkpoint_conversion/_to_diffusers.py`. -- `converted_weights//` with `model_index.json` and per-component - subfolders. -- Updated `tests/local_tests//README.md` with conversion command, - source layout, output path, and strict-load status. -- Updated `tests/local_tests//PORT_STATUS.md` with conversion - state, retry history, open questions, and issues/blockers. - -## Reference Scripts - -- `scripts/checkpoint_conversion/convert_ltx2_weights.py`: component prefix - splitting, metadata config extraction, passthrough Gemma/tokenizer assets, and - optional component-only output. -- `scripts/checkpoint_conversion/stable_audio_to_diffusers.py`: monolithic - `model.safetensors` split into transformer/VAE/conditioner, plus copied - passthrough subfolders. Use this shape for single-checkpoint official repos. -- `scripts/checkpoint_conversion/convert_gamecraft_full.py`: separate official - sources for transformer, VAE, text encoders, tokenizers, scheduler, and root - `model_index.json`. -- `scripts/checkpoint_conversion/longcat_to_fastvideo.py`: fused QKV/KV split, - renamed native transformer weights, and copied existing Diffusers components. -- `scripts/checkpoint_conversion/pt_to_safetensors.py`: simple `.pt` extraction - helper for nested checkpoint dictionaries. - -## Source Layout Decision - -Choose exactly one primary layout: - -| Layout | Conversion behavior | -|---|---| -| `diffusers` | Usually no tensor remap; verify configs/classes and copy or update `_class_name` only when needed. | -| `raw_official` | Convert a raw official checkpoint file or directory. Choose explicit component ownership before writing output. | -| `separate_components` | Convert/copy each component from its own file or directory. | -| `monolithic` | Load one model checkpoint and split state dict by authoritative prefixes into component buckets. | -| `mixed` | Convert some components and copy passthrough components such as tokenizers, text encoders, schedulers, or already-Diffusers VAE dirs. | -| `custom` | Document why none of the above fits before writing conversion code. | - -Monolithic checkpoints need explicit prefix ownership. For example, Stable Audio -uses one `model.safetensors` with DiT, pretransform/VAE, and conditioner keys; -the converter splits those keys into FastVideo component subfolders and writes -per-component configs. - -## Script Shape - -Start from `templates/family_to_diffusers.py` or the closest reference script. -Keep the script explicit and reviewable: - -- `COMPONENT_SPECS` or `COMPONENT_PREFIXES` declares component ownership. -- `PARAM_NAME_MAP` declares key renames. -- `SKIP_PATTERNS` declares intentionally dropped training-only keys. -- tensor split/fuse helpers are named by operation, e.g. `split_qkv`. -- `build_component_configs(...)` writes loader-compatible config files. Most - model components use `config.json`; schedulers use `scheduler_config.json`. -- `build_model_index(...)` writes a root `model_index.json` matching the target - FastVideo pipeline and component classes. -- verification reports missing, unexpected, skipped, unchanged, renamed, and - shape-mismatched keys. - -`model_index.json` library tokens must match FastVideo loaders: - -- standard native DiT/VAE/audio/vocoder/upsampler components loaded by existing - Diffusers-style loaders usually use `"diffusers"` with a FastVideo - `_class_name` in the component `config.json`; -- text encoders, tokenizers, image encoders, processors, and feature extractors - usually use `"transformers"`; -- `conditioner` currently expects `"fastvideo"`; -- use fully qualified `"fastvideo."` only when intentionally relying on - the custom fastvideo-library escape path; -- do not write bare `"fastvideo"` for transformer, VAE, or other loaders that - expect `"diffusers"` unless the loader explicitly expects it. - -## Mapping Rules - -- Use Phase 4 key/shape dumps to derive mappings. Do not guess from official key - names alone. -- Preserve each component's official file paths, parity test path, and prototype - concerns in comments or structured constants near the mapping that uses them. -- Every official inference parameter should be mapped, copied through, or listed - as intentionally skipped with a reason. -- Every FastVideo prototype parameter should receive a tensor or be listed as an - intentional external/passthrough parameter. -- Shape matches are necessary but not sufficient; check semantic pairing for - Q/K/V, gate/up/down, norm scale/bias, LoRA/base, and modality-specific heads. -- If official and FastVideo fuse or split tensors differently, convert tensors in - the script rather than changing production code to match checkpoint quirks. - -## Verification - -Run conversion locally, then verify before returning to Phase 6. For retry -requests, update the mapping, rerun conversion, and refresh only the implicated -converted component when safe; otherwise rerun the full conversion. - -```bash -python scripts/checkpoint_conversion/_to_diffusers.py \ - --src \ - --revision \ - --dst converted_weights/ -``` - -Omit `--revision` for local sources or when prep recorded `default` / `none`. - -Minimum output layout: - -```text -converted_weights// - model_index.json - transformer/config.json - transformer/*.safetensors - vae/config.json - vae/*.safetensors - scheduler/scheduler_config.json as needed - text_encoder/... as needed -``` - -Required checks: - -- `model_index.json` exists and lists every required component. -- Each converted component has the config filename its loader expects and - safetensors weights when it owns weights. Scheduler dirs require - `scheduler_config.json`; most other native model dirs use `config.json`. -- Weight filenames may vary by loader: transformer and VAE loaders glob all - `*.safetensors`; text encoders may load `*.safetensors`, `*.bin`, and - sometimes `*.pt`; `conditioner` currently expects - `diffusion_pytorch_model.safetensors`. Use the loader's actual accepted layout - rather than assuming one global filename. -- Passthrough components are copied or referenced deliberately. -- Each emitted component config validates through the same path production - loaders use. Instantiate the relevant config and call `update_model_arch(...)` - or `update_model_config(...)` with the emitted JSON so unknown keys fail during - conversion, not at pipeline load time. -- Record production loader strictness for every stateful component. If the loader - intentionally uses non-strict loading, add explicit missing/unexpected-key - assertions in the parity test and document exactly which keys are allowed. -- Each new FastVideo component strict-loads converted weights where its production - loader is strict. If strict loading is impossible, record the exact allowed - missing/unexpected keys and why they are not inference weights. -- Retry fixes include the original component parity evidence and the new - strict-load result in `local_tests_readme` so the component subagent can resume - without rediscovering context. -- `local_tests_readme` records the command, output directory, and strict-load - result. - -Do not chase numerical parity in this skill except to identify a conversion -mapping bug. Long parity-debug loops belong to `/add-model` Phase 6. - -## Escape Hatches - -Follow `../add-model/shared/common_rules.md`. Conversion-specific ask cases -include selecting between incompatible official checkpoints, publishing/uploading -weights, overwriting an existing converted repo not created by this run, -accepting non-strict missing inference weights, or dropping a component/output -from scope. - -## Handoff - -Return `../add-model/contracts/conversion_handoff.md` and update the shared state -files before handoff. diff --git a/.agents/skills/add-model-07-conversion/templates/family_to_diffusers.py b/.agents/skills/add-model-07-conversion/templates/family_to_diffusers.py deleted file mode 100644 index 0d12589bb..000000000 --- a/.agents/skills/add-model-07-conversion/templates/family_to_diffusers.py +++ /dev/null @@ -1,293 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: Apache-2.0 -"""Convert official weights to a FastVideo Diffusers-style tree. - -This template supports both separate component sources and a monolithic pipeline -checkpoint that must be split by component prefix. Replace every TODO before -using it for a real port. -""" - -from __future__ import annotations - -import argparse -import json -import os -import re -import shutil -from collections import OrderedDict -from pathlib import Path -from typing import Any - -import torch -from safetensors import safe_open -from safetensors.torch import load_file, save_file - -try: - from huggingface_hub import snapshot_download -except ImportError: # pragma: no cover - optional local conversion dependency - snapshot_download = None - -# TODO: fill with authoritative component prefixes for monolithic checkpoints. -# Example: {"model.model.": "transformer", "pretransform.model.": "vae"} -COMPONENT_PREFIXES: dict[str, str] = {} - -# TODO: fill with component-specific source paths for separate-component repos. -# Example: {"transformer": "transformer/model.safetensors", "vae": "vae/"} -SEPARATE_COMPONENT_PATHS: dict[str, str] = {} - -# TODO: copy passthrough dirs that are already loadable by FastVideo/Diffusers. -PASSTHROUGH_SUBFOLDERS: tuple[str, ...] = ("tokenizer", "scheduler") - -# TODO: add regex renames derived from Phase 4 key/shape dumps. -PARAM_NAME_MAP: dict[str, str] = {} - -# TODO: include training-only or dynamically-computed keys that must not load. -SKIP_PATTERNS: tuple[str, ...] = () - - -def _hf_token() -> str | None: - return (os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACE_HUB_TOKEN") or os.environ.get("HF_API_KEY")) - - -def resolve_src(src: str, revision: str | None) -> Path: - if os.path.exists(src): - return Path(src) - if snapshot_download is None: - raise RuntimeError("huggingface_hub is required when --src is a repo id") - return Path(snapshot_download(repo_id=src, revision=revision, token=_hf_token())) - - -def load_checkpoint(path: Path) -> dict[str, torch.Tensor]: - if path.is_dir(): - weights: dict[str, torch.Tensor] = {} - for shard in sorted(path.glob("*.safetensors")): - weights.update(load_file(str(shard))) - if weights: - return weights - raise FileNotFoundError(f"No safetensors found in {path}") - - if path.suffix == ".safetensors": - return load_file(str(path)) - - checkpoint = torch.load(path, map_location="cpu", weights_only=True) - if isinstance(checkpoint, dict): - for key in ("state_dict", "model_state_dict", "model", "module", "ema"): - if key in checkpoint and isinstance(checkpoint[key], dict): - return checkpoint[key] - return checkpoint - raise TypeError(f"Unsupported checkpoint type: {type(checkpoint)!r}") - - -def should_skip_key(key: str) -> bool: - return any(re.search(pattern, key) for pattern in SKIP_PATTERNS) - - -def apply_mapping(key: str) -> str | None: - if should_skip_key(key): - return None - for pattern, replacement in PARAM_NAME_MAP.items(): - if re.match(pattern, key): - return re.sub(pattern, replacement, key) - return key - - -def split_monolithic(state: dict[str, torch.Tensor], ) -> dict[str, OrderedDict[str, torch.Tensor]]: - components: dict[str, OrderedDict[str, torch.Tensor]] = { - name: OrderedDict() - for name in set(COMPONENT_PREFIXES.values()) - } - intentionally_skipped: list[str] = [] - unowned: list[str] = [] - for key, value in state.items(): - if should_skip_key(key): - intentionally_skipped.append(key) - continue - for prefix, component in COMPONENT_PREFIXES.items(): - if key.startswith(prefix): - mapped = apply_mapping(key[len(prefix):]) - if mapped is not None: - components[component][mapped] = value - break - else: - unowned.append(key) - if unowned: - sample = ", ".join(unowned[:10]) - raise ValueError(f"Unowned monolithic keys: {len(unowned)}. " - f"Add COMPONENT_PREFIXES or SKIP_PATTERNS entries. Sample: {sample}") - if intentionally_skipped: - print(f"Intentionally skipped {len(intentionally_skipped)} keys") - return {name: weights for name, weights in components.items() if weights} - - -def load_separate_components(src_dir: Path) -> dict[str, OrderedDict[str, torch.Tensor]]: - components: dict[str, OrderedDict[str, torch.Tensor]] = {} - for component, rel_path in SEPARATE_COMPONENT_PATHS.items(): - state = load_checkpoint(src_dir / rel_path) - converted: OrderedDict[str, torch.Tensor] = OrderedDict() - for key, value in state.items(): - mapped = apply_mapping(key) - if mapped is not None: - converted[mapped] = value - components[component] = converted - return components - - -def build_component_configs(_src_dir: Path) -> dict[str, dict[str, Any]]: - # TODO: emit config content accepted by FastVideo loaders. Most components use - # config.json; schedulers use scheduler_config.json. - return { - "transformer": { - "_class_name": "" - }, - "vae": { - "_class_name": "" - }, - } - - -def config_filename(component: str) -> str: - if component == "scheduler": - return "scheduler_config.json" - return "config.json" - - -def source_label(src: str) -> str: - if os.path.exists(src): - return Path(src).name - return src - - -def build_model_index( - src: str, - revision: str | None, - available_components: set[str], -) -> dict[str, Any]: - # TODO: match the target pipeline and every required component. - index: dict[str, Any] = { - "_class_name": "", - "_diffusers_version": "0.30.0", - "_fastvideo_converted_from": source_label(src), - # Existing transformer/VAE loaders expect "diffusers" even when - # _class_name names a FastVideo-native class registered in FastVideo. - "transformer": ["diffusers", ""], - "vae": ["diffusers", ""], - } - if revision: - index["_fastvideo_converted_revision"] = revision - return {key: value for key, value in index.items() if key.startswith("_") or key in available_components} - - -def validate_component_configs(configs: dict[str, dict[str, Any]]) -> None: - # TODO: instantiate each FastVideo config and call update_model_arch(...) or - # update_model_config(...) with this JSON so unknown emitted keys fail here. - placeholder_configs = [name for name, config in configs.items() if "<" in json.dumps(config)] - if placeholder_configs: - raise ValueError(f"Replace config placeholders for: {placeholder_configs}") - - -def verify_conversion( - dst_dir: Path, - components: dict[str, OrderedDict[str, torch.Tensor]], -) -> None: - del dst_dir, components - # TODO: load each emitted stateful component through its production loader and - # assert strict load, or document exact allowed missing/unexpected keys. - raise NotImplementedError("Implement production config validation and strict-load checks") - - -def write_component( - dst_dir: Path, - name: str, - state: dict[str, torch.Tensor], - config: dict[str, Any] | None, -) -> None: - component_dir = dst_dir / name - if component_dir.exists() and any(component_dir.iterdir()): - shutil.rmtree(component_dir) - component_dir.mkdir(parents=True, exist_ok=True) - save_file(dict(state), str(component_dir / "diffusion_pytorch_model.safetensors")) - if config is not None: - config_path = component_dir / config_filename(name) - with config_path.open("w", encoding="utf-8") as f: - json.dump(config, f, indent=2) - f.write("\n") - print(f"Wrote {name}: {len(state)} tensors") - - -def copy_passthrough(src_dir: Path, dst_dir: Path) -> list[str]: - copied: list[str] = [] - for subfolder in PASSTHROUGH_SUBFOLDERS: - src = src_dir / subfolder - if not src.is_dir(): - continue - dst = dst_dir / subfolder - if dst.exists(): - shutil.rmtree(dst) - shutil.copytree(src, dst) - copied.append(subfolder) - print(f"Copied {subfolder}/") - return copied - - -def default_monolithic_checkpoint(src_path: Path) -> Path: - if src_path.is_file(): - return src_path - return src_path / "model.safetensors" - - -def convert( - src: str, - dst: str, - layout: str, - revision: str | None, -) -> None: - src_path = resolve_src(src, revision) - dst_dir = Path(dst) - dst_dir.mkdir(parents=True, exist_ok=True) - model_index_path = dst_dir / "model_index.json" - - if layout in {"monolithic", "raw_official"}: - # TODO: replace model.safetensors with the official monolithic file name. - components = split_monolithic(load_checkpoint(default_monolithic_checkpoint(src_path))) - elif layout in {"separate_components", "mixed"}: - if not src_path.is_dir(): - raise ValueError(f"{layout} layout requires a source directory: {src_path}") - components = load_separate_components(src_path) - else: - raise ValueError(f"Unsupported template layout: {layout}") - - copied = (copy_passthrough(src_path, dst_dir) if src_path.is_dir() else []) - configs = build_component_configs(src_path if src_path.is_dir() else src_path.parent) - validate_component_configs(configs) - for name, state in components.items(): - write_component(dst_dir, name, state, configs.get(name)) - - available = set(components) | set(copied) - with model_index_path.open("w", encoding="utf-8") as f: - json.dump(build_model_index(src, revision, available), f, indent=2) - f.write("\n") - print(f"Wrote {dst_dir / 'model_index.json'}") - verify_conversion(dst_dir, components) - - -def main() -> None: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--src", required=True, help="HF repo id, local dir, or checkpoint path") - parser.add_argument("--revision", help="HF branch, tag, or commit for repo sources") - parser.add_argument( - "--dst", - required=True, - help="Output converted_weights/ directory", - ) - parser.add_argument( - "--layout", - choices=("raw_official", "monolithic", "separate_components", "mixed"), - required=True, - help="Official source layout", - ) - args = parser.parse_args() - convert(args.src, args.dst, args.layout, args.revision) - - -if __name__ == "__main__": - main() diff --git a/.agents/skills/add-model-08-trace/SKILL.md b/.agents/skills/add-model-08-trace/SKILL.md deleted file mode 100644 index fb3363b77..000000000 --- a/.agents/skills/add-model-08-trace/SKILL.md +++ /dev/null @@ -1,214 +0,0 @@ ---- -name: add-model-08-trace -description: Use during /add-model Phase 6 when component parity has failed and root cause requires layer-by-layer divergence analysis. Uses FastVideo activation trace first, falling back to custom hooks only for boundaries or stats the utility cannot observe. ---- - -# Add-Model Trace - -## Manual Invocation - -Load this skill when `/add-model` Phase 6 component parity has failed and the -root cause requires layer-by-layer divergence analysis. This skill is not -auto-fired. The calling subagent (DiT, VAE, encoder, or generic port skill) -loads it when its standard parity-debug loop hits a wall and cannot isolate -the divergence from end-to-end tensor comparisons alone. - -Do not load this skill for first-pass parity failures. Try weight-diff and -end-to-end tensor comparison first. Load this skill only when those do not -isolate the cause. - -## Goal - -Find the first numerical divergence point between FastVideo's port and the -official reference, layer by layer, by instrumenting both sides at matching -tensor boundaries. The investigation must leave zero source residue in -production code when it closes. - -## When To Run - -After a component parity test FAILS at a bf16-noise-realistic tolerance AND -the calling subagent's first-pass debug (weight-diff, end-to-end tensor -compare) does not isolate the cause. - -Required inputs before starting: - -- A working FastVideo loader for the component under investigation. -- A working official loader, typically via - `tests/local_tests/helpers/_upstream.py::load_upstream_`. -- Shared deterministic test inputs (same tensors on both sides). -- The component parity test file path and its current failure output. - -## Primary Path: FastVideo Activation Trace - -Use FastVideo's first-class activation trace before writing custom hooks: -`fastvideo/hooks/activation_trace.py`, documented in -`docs/contributing/activation_trace.md`. - -Pipeline runs attach trace to the transformer during pipeline initialization. -Component-only parity harnesses may call `attach_activation_trace(model)` from -local test/debug code; do not add trace calls to production model code. - -Prefix the failing parity command with a tight layer regex: - -```bash -FASTVIDEO_TRACE_ACTIVATIONS=1 \ -FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$" \ -FASTVIDEO_TRACE_STATS="abs_mean,sum,max,shape" \ -FASTVIDEO_TRACE_STEPS="0" \ -FASTVIDEO_TRACE_OUTPUT="/tmp/opencode/fv_trace.jsonl" \ -pytest tests/local_tests -k "parity" -v -s -``` - -Match the layer regex to the actual `model.named_modules()` names. Empty or -broad regexes are expensive; prefer block-level names first, then narrow to -submodules after the first divergent block is known. - -## Trace Compare Contract - -One JSONL file per side. FastVideo output should use `FASTVIDEO_TRACE_OUTPUT`; -the upstream harness should emit the same JSONL shape: - -```json -{"module":"block.layers.0","tensor":"out","step":0,"abs_mean":0.0123,"sum":1.0,"max":0.5,"shape":[1,16,32]} -``` - -Compare rows by `(module, step, tensor)`. The first row whose `shape`, -`abs_mean`, or `max` diverges beyond the component tolerance is the first broken -boundary. Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and -`FASTVIDEO_TRACE_STEPS` identical between sides; if row order differs, sort or -normalize before diffing. - -## Drill-Down Loop - -**Initial run:** trace every top-level block (`^block\.layers\.[0-9]+$` or the -family's equivalent). Identify the first block index where `abs_mean` or `max` -drifts beyond tolerance while earlier blocks match. - -**Drill run:** tighten `FASTVIDEO_TRACE_LAYERS` to submodules inside the first -divergent block: attention output, MLP projections, norm outputs, modality -adapters, or other named boundaries exposed by `named_modules()`. - -**Iterate:** if the first divergent operation is a free function or tensor op not -visible as an `nn.Module`, use the fallback instrumentation hierarchy below. - -The loop ends when the first divergent submodule or operation is identified with -a file:line citation in the official source. - -## Fallback Instrumentation Hierarchy - -Use these only when activation trace cannot observe the needed boundary or -statistic. - -### (1) Custom forward hooks - -`module.register_forward_hook(...)` and `register_forward_pre_hook(...)`. -Always within `try/finally` with `handle.remove()`. Zero source residue. - -### (2) Runtime monkey-patch - -`module.attr = wrapped_func` or `cls.method = wrapped_method`, restored via -`try/finally` (save original first). Use for free functions and non-Module sites -such as activation functions (`swiglu`, `apply_rotary_emb`). - -### (3) Source edits in FastVideo's own code - -Only when (1) and (2) are insufficient. Track all edits within a single named -`git stash` boundary OR a temporary branch. Run `git diff` before closing the -investigation to confirm cleanup. - -### (4) Source edits in official repo source - -Allowed only when hook and monkey-patch approaches cannot capture the site. -For git-tracked or editable official clones, use `git diff` in the clone path to -verify cleanup. For non-editable site-packages, back up the target file before -editing and restore it before handoff. - -## Hypothesis Toggles - -Use env-var-gated monkey-patches to A/B test suspect implementations without -source edits. Pattern: `_DEBUG_PATCH_=1`. - -Example from the magi-human investigation: - -``` -MAGI_DEBUG_PATCH_LINEAR=1 -``` - -This patched `PackedExpertLinear.forward` to mirror upstream's -`_BF16ComputeLinear` explicit-cast pattern, isolating a dtype-cast difference -as the root cause. - -Document all toggles in the script docstring. Each toggle must: - -- save the original before patching; -- restore the original in a `try/finally` block; -- print a `[debug] Patched .` line to stdout when active. - -## Cleanup Gate - -The calling agent MUST report `[cleanup-gate] PASS` on all five items before -handoff. Do not hand off with any item unresolved. - -1. `git diff` in the FastVideo repo: empty. No stray prints, hooks, or - monkey-patches in production code. -2. `git diff` in the official-repo clone (if used): empty. For non-editable - site-packages installs: `diff original.py original.py.trace-backup` is - empty OR `pip install --force-reinstall ` succeeded and the installed - file matches the original. -3. `git stash list`: only the named investigation stash (or empty). No - unnamed stashes left from this session. -4. No new untracked files outside `/tmp/opencode/` (logs) and the existing - debug script directory (`tests/local_tests/transformers/` or equivalent). -5. `mypy` clean on any production files touched during the investigation. - -## Escape Hatches - -Escalate to the calling bucket skill when: - -- A forward hook on an official module raises because of a custom `forward` - signature or varlen handler args that the hook closure cannot satisfy. The - bucket skill has component-specific knowledge to work around this. -- The first divergent layer is `block[0]`, meaning the divergence is in the - adapter, modality dispatcher, coordinate embedding, or packing step before - any block runs. Check those sites first; the bug is not in attention or MLP. -- Per-block drift is never zero anywhere across all blocks. This usually means - the inputs are not bit-identical between sides. Verify with a state-dict - compare (weight-diff script) AND confirm the input tensors are the same - object or have identical values before the forward call. - -## Handoff - -Return to the calling subagent with: - -- FastVideo trace JSONL path and upstream trace JSONL path. -- Trace settings used: `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and - `FASTVIDEO_TRACE_STEPS`. -- The first divergent `(module, step, tensor)` row and observed drift. -- The upstream file:line citation where the divergence originates. -- Fallback hook/patch verdict if activation trace could not observe the boundary. -- Hypothesis verdict if an A/B toggle was used, for example `PATCH_LINEAR=1`. -- Cleanup-gate status: `[cleanup-gate] PASS` or a list of unresolved items. - -The calling agent uses this to scope the production fix in the FastVideo -component file. - -## References - -- `docs/contributing/activation_trace.md` for canonical activation-trace env vars, - JSONL output, cost model, and troubleshooting. -- `fastvideo/hooks/activation_trace.py` for the implementation and - `attach_activation_trace(model)` entry point. -- `templates/block_trace_debug.py` in this skill directory: fallback custom-hook - template when activation trace cannot observe the needed boundary or stat. -- `tests/local_tests/transformers/_debug_magi_human_block_parity.py` in the - FastVideo3 repo: historical worked example for custom hook/patch debugging. -- `add-model/SKILL.md` Phase 6: the calling context for this skill. -- `add-model-03-port-dit/SKILL.md`, `add-model-04-port-vae/SKILL.md`, - `add-model-05-port-encoder/SKILL.md`, `add-model-06-port-generic/SKILL.md`: - bucket-specific debug language and component-specific escape-hatch knowledge. - -## Changelog - -| Date | Change | -|---|---| -| 2026-05-01 | Initial skill extracted from `_debug_magi_human_block_parity.py` pattern. | diff --git a/.agents/skills/add-model-08-trace/templates/block_trace_debug.py b/.agents/skills/add-model-08-trace/templates/block_trace_debug.py deleted file mode 100644 index 45e113953..000000000 --- a/.agents/skills/add-model-08-trace/templates/block_trace_debug.py +++ /dev/null @@ -1,334 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -"""Per-block divergence debugger template for FastVideo model ports. - -Run directly (not a pytest test): - python tests/local_tests/transformers/_debug___parity.py - -Generalizes: tests/local_tests/transformers/_debug_magi_human_block_parity.py - -Fill FAMILY, COMPONENT, and the two loader functions. Run once for the initial -drift table, then set _DEBUG_DRILL_LAYER=NN to drill into submodules. -Add _DEBUG_PATCH_=1 to A/B test a suspect implementation. - -CLEANUP: all hooks removed in try/finally; monkey-patches restored in -try/finally; source edits tracked in a named git stash. Zero source residue. -See add-model-08-trace/SKILL.md for the full cleanup gate checklist. -""" -from __future__ import annotations - -import gc -import os -import sys -from pathlib import Path -from typing import Any - -import torch - -FAMILY: str = "" # e.g. "magi_human", "ltx2", "wan" -COMPONENT: str = "" # e.g. "dit", "vae", "encoder" -DRILL_LAYER_ENV: str = "_DEBUG_DRILL_LAYER" -HYPOTHESIS_ENV: str = "_DEBUG_PATCH_" -REL_THRESHOLD: float = 0.005 # 0.5% abs_mean drift flags a block as divergent -LOG_DIR: Path = Path("/tmp/opencode") - -REPO_ROOT = Path(__file__).resolve().parents[3] -sys.path.insert(0, str(REPO_ROOT)) - - -def load_official(device: torch.device) -> torch.nn.Module: - """Load the official upstream model. TODO: implement for your family. - - Example (magi-human): - from tests.local_tests.helpers.magi_human_upstream import install_stubs, load_upstream_dit - install_stubs() - return load_upstream_dit(base_shard_dir, device=device, dtype=None) - """ - raise NotImplementedError(f"Fill load_official() for {FAMILY}/{COMPONENT}.") - - -def load_fastvideo(device: torch.device) -> torch.nn.Module: - """Load the FastVideo-native model. TODO: implement for your family. - - Example (magi-human): - from fastvideo.configs.models.dits.magi_human import MagiHumanVideoConfig - from fastvideo.models.dits.magi_human import MagiHumanDiT - from safetensors.torch import load_file; import glob - fv = MagiHumanDiT(MagiHumanVideoConfig()) - state = {} - for shard in sorted(glob.glob(str(transformer_dir / "*.safetensors"))): state.update(load_file(shard)) - fv.load_state_dict(state, strict=False); return fv.to(device).eval() - """ - raise NotImplementedError(f"Fill load_fastvideo() for {FAMILY}/{COMPONENT}.") - - -def build_inputs(device: torch.device) -> dict[str, Any]: - """Return deterministic inputs shared by both sides. TODO: replace. - - Both sides must receive the SAME tensors (clone before each forward call). - Non-identical inputs cause non-zero drift everywhere. - """ - torch.manual_seed(0) - return {"x": torch.randn(64, 1024, dtype=torch.bfloat16, device=device)} - - -def _stat(name: str, t: torch.Tensor) -> dict: - f = t.detach().float() - return { - "name": name, - "shape": tuple(t.shape), - "abs_mean": f.abs().mean().item(), - "sum": f.sum().item(), - "min": f.min().item(), - "max": f.max().item(), - } - - -def _attach_block_hooks( - model: torch.nn.Module, - label: str, - log: list[dict], - tensors: dict[str, torch.Tensor] | None = None, - drill_layer: int | None = None, -) -> list[Any]: - """Return hook handles. Caller MUST remove them in try/finally.""" - handles: list[Any] = [] - - def _hook(name: str): - - def fn(_module, _inputs, outputs): - t = outputs[0] if isinstance(outputs, tuple) else outputs - if not torch.is_tensor(t): - return - log.append({"side": label, **_stat(name, t)}) - if tensors is not None: - tensors[name] = t.detach().float().cpu() - - return fn - - def _pre_hook(name: str): - # Pre-hooks observe a free function's output by intercepting the next - # module's input (useful when the activation is not an nn.Module). - def fn(_module, inputs): - t = inputs[0] if isinstance(inputs, tuple) else inputs - if not torch.is_tensor(t): - return - key = f"{name}" - log.append({"side": label, **_stat(key, t)}) - if tensors is not None: - tensors[key] = t.detach().float().cpu() - - return fn - - # TODO: adapt attribute paths to your model. Remove adapter block if absent. - if hasattr(model, "adapter"): - handles.append(model.adapter.register_forward_hook(_hook("adapter"))) - - # TODO: adapt model.block.layers to your block container. - # Alternatives: model.transformer.layers, model.blocks, model.layers - block_layers = model.block.layers # type: ignore[attr-defined] - for i, layer in enumerate(block_layers): - handles.append(layer.register_forward_hook(_hook(f"block[{i:02d}]"))) - if drill_layer is not None and i == drill_layer: - tag = f"L{i:02d}" - # TODO: adapt submodule names to your layer's attributes. - # magi-human uses: attention, mlp.pre_norm, mlp.up_gate_proj, - # mlp.down_proj (pre+post), mlp, attn_post_norm, mlp_post_norm. - if hasattr(layer, "attention"): - handles.append(layer.attention.register_forward_hook(_hook(f"{tag}.attention"))) - if hasattr(layer, "mlp"): - mlp = layer.mlp - if hasattr(mlp, "pre_norm"): - handles.append(mlp.pre_norm.register_forward_hook(_hook(f"{tag}.mlp.pre_norm"))) - if hasattr(mlp, "up_gate_proj"): - handles.append(mlp.up_gate_proj.register_forward_hook(_hook(f"{tag}.mlp.up_gate_proj"))) - if hasattr(mlp, "down_proj"): - handles.append(mlp.down_proj.register_forward_pre_hook(_pre_hook(f"{tag}.mlp.down_proj"))) - handles.append(mlp.down_proj.register_forward_hook(_hook(f"{tag}.mlp.down_proj"))) - handles.append(mlp.register_forward_hook(_hook(f"{tag}.mlp"))) - if hasattr(layer, "attn_post_norm"): - handles.append(layer.attn_post_norm.register_forward_hook(_hook(f"{tag}.attn_post_norm"))) - if hasattr(layer, "mlp_post_norm"): - handles.append(layer.mlp_post_norm.register_forward_hook(_hook(f"{tag}.mlp_post_norm"))) - return handles - - -def _apply_hypothesis_patch() -> bool: - """Apply an optional monkey-patch gated by HYPOTHESIS_ENV. TODO: implement. - - Pattern: save original on the class, patch, restore in _restore_hypothesis_patch(). - """ - if os.getenv(HYPOTHESIS_ENV) != "1": - return False - # TODO: import FastVideo class, save original, apply patch. - print(f"[debug] Hypothesis patch {HYPOTHESIS_ENV}=1 applied.") - return True - - -def _restore_hypothesis_patch() -> None: - if os.getenv(HYPOTHESIS_ENV) != "1": - return - # TODO: restore original, e.g.: _mod.TargetClass.method = _mod._ORIGINAL_METHOD - - -def _write_log(entries: list[dict], path: Path) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - with open(path, "w") as f: - for e in entries: - f.write(f"{e['name']} {e['shape']} " - f"{e['abs_mean']:.8f} {e['sum']:.4f} " - f"{e['min']:.6f} {e['max']:.6f}\n") - - -def _sort_key(name: str, drill_layer: int) -> tuple: - if name == "adapter": - return (0, "") - if name.startswith(f"L{drill_layer:02d}."): - sub_order = { - "attention": 0, - "attn_post_norm": 1, - "mlp.pre_norm": 2, - "mlp.up_gate_proj": 3, - "mlp.down_proj": 4, - "mlp.down_proj": 5, - "mlp": 6, - "mlp_post_norm": 7, - }.get(name.split(".", 1)[1], 9) - return (1, f"block[{drill_layer:02d}]", sub_order) - if name.startswith("block["): - return (1, name, 99) - return (2, name, 0) - - -def _print_table(by_name: dict[str, dict], drill_layer: int) -> int | None: - hdr = (f"{'name':<18} {'up_shape':<22} {'up_absmean':>12} {'fv_absmean':>12} " - f"{'absmean_diff':>14} {'rel%':>8} {'up_sum':>14} {'fv_sum':>14} {'sum_diff':>12}") - print(f"\n{hdr}\n{'-' * len(hdr)}") - first_div: int | None = None - for name in sorted(by_name.keys(), key=lambda n: _sort_key(n, drill_layer)): - d = by_name[name] - up, fv = d.get("up"), d.get("fv") - if up is None or fv is None: - continue - am_diff = abs(up["abs_mean"] - fv["abs_mean"]) - am_rel = am_diff / max(up["abs_mean"], 1e-9) - sum_diff = abs(up["sum"] - fv["sum"]) - flag = "" - if name.startswith("block[") and am_rel > REL_THRESHOLD: - flag = " <<< DIVERGE" - if first_div is None: - first_div = int(name[len("block["):-1]) - print(f"{name:<18} {str(up['shape']):<22} {up['abs_mean']:>12.6f} " - f"{fv['abs_mean']:>12.6f} {am_diff:>14.6f} {am_rel * 100:>7.3f}% " - f"{up['sum']:>14.4f} {fv['sum']:>14.4f} {sum_diff:>12.4f}{flag}") - return first_div - - -def _print_elementwise(up_t: dict[str, torch.Tensor], fv_t: dict[str, torch.Tensor], drill_layer: int) -> None: - common = set(up_t.keys()) & set(fv_t.keys()) - if not common: - return - hdr = f"{'name':<30} {'shape':<22} {'diff_max':>12} {'diff_mean':>12} {'diff_rel%':>10}" - print(f"\nElement-wise diffs for drilled L{drill_layer:02d} submodules:\n{hdr}\n{'-' * len(hdr)}") - for name in sorted(common): - a, b = up_t[name], fv_t[name] - if a.shape != b.shape: - continue - diff = (a - b).abs() - rel = (diff.mean().item() / max(a.abs().mean().item(), 1e-9)) * 100 - print(f"{name:<30} {str(tuple(a.shape)):<22} " - f"{diff.max().item():>12.6f} {diff.mean().item():>12.6f} {rel:>9.4f}%") - - -def main() -> None: - if not torch.cuda.is_available(): - print("Need CUDA. Skipping.") - return - - # TODO: add precondition checks (official clone present, weights available). - - drill_layer = int(os.getenv(DRILL_LAYER_ENV, "0")) - device = torch.device("cuda:0") - patched = _apply_hypothesis_patch() - try: - inputs = build_inputs(device) - - print("Loading official model...") - official = load_official(device) - up_log: list[dict] = [] - up_t: dict[str, torch.Tensor] = {} - up_handles = _attach_block_hooks(official, "up", up_log, up_t, drill_layer) - print("Running official forward (with hooks)...") - try: - with torch.inference_mode(): - # TODO: adapt forward call signature to your component. - ref_out = official(**{k: v.clone() for k, v in inputs.items()}) - if isinstance(ref_out, dict): - sample = ref_out.get("sample") - ref_out = sample if sample is not None else ref_out.get("x") - elif hasattr(ref_out, "sample"): - ref_out = ref_out.sample - elif isinstance(ref_out, tuple): - ref_out = ref_out[0] - assert torch.is_tensor(ref_out), f"official output is not tensor: {type(ref_out)}" - ref_out = ref_out.detach().float().cpu() - finally: - for h in up_handles: - h.remove() - del official - gc.collect() - torch.cuda.empty_cache() - - print("Loading FastVideo model...") - fv = load_fastvideo(device) - fv_log: list[dict] = [] - fv_t: dict[str, torch.Tensor] = {} - fv_handles = _attach_block_hooks(fv, "fv", fv_log, fv_t, drill_layer) - print("Running FastVideo forward (with hooks)...") - try: - with torch.inference_mode(): - # TODO: adapt forward call signature to your component. - fv_out = fv(**{k: v.clone() for k, v in inputs.items()}) - if isinstance(fv_out, dict): - sample = fv_out.get("sample") - fv_out = sample if sample is not None else fv_out.get("x") - elif hasattr(fv_out, "sample"): - fv_out = fv_out.sample - elif isinstance(fv_out, tuple): - fv_out = fv_out[0] - assert torch.is_tensor(fv_out), f"FastVideo output is not tensor: {type(fv_out)}" - fv_out = fv_out.detach().float().cpu() - finally: - for h in fv_handles: - h.remove() - finally: - _restore_hypothesis_patch() - - LOG_DIR.mkdir(parents=True, exist_ok=True) - up_path = LOG_DIR / f"{FAMILY}_{COMPONENT}_up_layers.log" - fv_path = LOG_DIR / f"{FAMILY}_{COMPONENT}_fv_layers.log" - _write_log(up_log, up_path) - _write_log(fv_log, fv_path) - print(f"\nLogs: {up_path} {fv_path}\nDiff: diff {up_path} {fv_path}") - - by_name: dict[str, dict] = {} - for entry in up_log + fv_log: - by_name.setdefault(entry["name"], {})[entry["side"]] = entry - first_div = _print_table(by_name, drill_layer) - - print() - if first_div is not None: - print(f"First block exceeding {REL_THRESHOLD * 100:.2f}% drift: block[{first_div:02d}]") - print(f"Re-run with {DRILL_LAYER_ENV}={first_div} to drill submodules.") - else: - print(f"No block exceeded {REL_THRESHOLD * 100:.2f}% -- divergence is amortized or pre-block.") - - diff = (ref_out - fv_out).abs() - print(f"\nFinal ref_abs={ref_out.abs().mean():.6f} fv_abs={fv_out.abs().mean():.6f} " - f"diff_max={diff.max():.6f} diff_mean={diff.mean():.6f}") - _print_elementwise(up_t, fv_t, drill_layer) - if patched: - print(f"\n[debug] Hypothesis {HYPOTHESIS_ENV}=1 was active this run.") - - -if __name__ == "__main__": - main() diff --git a/.agents/skills/add-model-09-pipeline/SKILL.md b/.agents/skills/add-model-09-pipeline/SKILL.md deleted file mode 100644 index e91257b0a..000000000 --- a/.agents/skills/add-model-09-pipeline/SKILL.md +++ /dev/null @@ -1,255 +0,0 @@ ---- -name: add-model-09-pipeline -description: Use during /add-model Phase 7 after all required component parity tests pass to define FastVideo pipeline wiring, configs, presets, registry entries, examples, smoke tests, and pipeline parity tests. ---- - -# Add Model Pipeline - -## Goal - -Implement and verify the end-to-end FastVideo pipeline after the native -components and converted weights have passed non-skip component parity. This -skill owns pipeline class/stage wiring, pipeline configs, presets, registry -entries, examples, smoke tests, and pipeline parity-debug. - -FastVideo has one pipeline architecture: stage-based composition through -`ComposedPipelineBase`. Add or specialize stages only when existing stages cannot -represent the official behavior safely. - -## Hard Gate - -Do not start pipeline work until every required component, including reused -components, has a non-skip local parity PASS. - -If any component row is missing, skipped, red, or blocked, return to `/add-model` -Phase 6. Pipeline parity cannot distinguish stage wiring mistakes from broken -component numerics when component parity is still unresolved. - -## Inputs - -Follow `../add-model/shared/common_rules.md` for token/auth safety, state files, -escape hatches, production boundaries, and skip/pass semantics. - -Require a complete packet matching -`../add-model/contracts/pipeline_context.md`. - -The packet must include: - -- official pipeline files and official call/default sources; -- workload types, input/output modalities, and output contract; -- converted or source `model_index.json` path; -- component parity rows, all `non_skip_pass`; -- target FastVideo pipeline/config/preset/registry/example/test paths; -- `local_tests_readme` and `port_state_file` paths. - -## Outputs - -- Pipeline package under `fastvideo/pipelines/basic//`. -- Pipeline config under `fastvideo/configs/pipelines/.py` or a documented - family-local config file when that matches existing project style. -- Presets under `fastvideo/pipelines/basic//presets.py`. -- Registry updates in `fastvideo/registry.py`. -- Basic example under `examples/inference/basic/basic_*.py`. -- Local smoke and parity tests under `tests/local_tests/pipelines/`. -- Updated `tests/local_tests//README.md`. -- Updated `tests/local_tests//PORT_STATUS.md`. -- Handoff matching `../add-model/contracts/pipeline_handoff.md`. - -## Mode: Pipeline Definition - -Use this mode first. - -1. Read the official pipeline call path before editing FastVideo code. -2. Compare official defaults against the planned FastVideo config and presets: - steps, CFG scales, secondary CFG, flow shift, schedulers, sigmas, seed/RNG, - resolution, frames, FPS, duration, VAE scaling, decode slicing, negative - prompt defaults, and output heads. -3. Create or update the pipeline class with `_required_config_modules` matching - the emitted `model_index.json` and `ComposedPipelineBase.load_modules`. - Runtime pipeline resolution is exact: `model_index.json["_class_name"]` must - match a registered `EntryClass.__name__`, or a wrapper/alias class in - `EntryClass`. Registry detectors do not select the executable pipeline class. -4. Add new public generation kwargs to `fastvideo/api/sampling_param.py` before - examples or presets use them. `SamplingParam.update()` ignores unknown keys - except for logging, and preset defaults apply only to declared fields. Add CLI - args when the option should be available from command-line entrypoints. -5. Put loader-time changes in `load_modules()` or earlier, not - `initialize_pipeline()`. `ComposedPipelineBase.__init__` loads modules before - `post_init()` calls `initialize_pipeline()`, so process-global flags, loader - path rewrites, dtype overrides, and tokenizer path changes needed for loading - cannot be introduced there. -6. Use `self.get_module("transformer_2", None)` and similar optional accessors - for truly optional modules. Do not hard-require optional modules by accident. -7. Avoid mutating class-level `_required_config_modules` in custom code. If a - pipeline needs dynamic modules, copy the list to an instance-owned value or - pass `required_config_modules` explicitly so one pipeline instance cannot leak - module requirements into another. -8. Create the stage chain in official execution order. Prefer existing shared - stages for standard text encoding, timestep preparation, latent preparation, - denoising, and decoding. -9. Add model-specific stages only for family-specific behavior that does not fit - the shared stage contracts. -10. Add pipeline config classes for wiring and runtime defaults. Do not duplicate - component architecture fields unless a loader requires them in the subconfig. - Family-local config files such as - `fastvideo/pipelines/basic//pipeline_configs.py` are valid only when - `fastvideo/registry.py` imports and registers the classes explicitly. -11. Add `InferencePreset` objects with `model_family`, `name`, `version`, - `defaults`, optional validation-only `stage_schemas`, and an `ALL_PRESETS` - tuple. `stage_schemas` validates user-facing `stage_overrides` names; it does - not drive `create_pipeline_stages()` execution. -12. Register config classes and presets in `fastvideo/registry.py`: add - `register_configs(...)`, import the family's `ALL_PRESETS`, and append it to - `_register_presets()`. Detectors should cover HF paths and `_class_name` - strings for config/preset lookup, but not as a replacement for exact pipeline - class-name resolution. -13. Add a basic example with a user-story docstring and normal file-path inputs - for image, audio, or video references. Keep orchestration glue in the - pipeline or a helper, not in the example. -14. Add a separate smoke test - `tests/local_tests/pipelines/test__pipeline_smoke.py` that proves - imports, `EntryClass`, registry, presets, config defaults, and at least one - real load/generate path when weights are local. Older local tests sometimes - colocate smoke checks in parity files; new ports should use the separate file - convention. -15. Add or update pipeline parity test scaffolding with - `templates/pipeline_parity_test.py`. -16. Update `local_tests_readme` and `port_state_file` with commands, statuses, - default sources, decisions, and blockers. - -Production import boundaries are defined in -`../add-model/shared/common_rules.md`. - -## Mode: Pipeline Parity Debug - -Run after pipeline definition and after smoke can execute far enough to load the -pipeline. Loop until pipeline parity is a non-skip PASS or a precise blocker is -returned. - -Mandatory order: - -```bash -pytest tests/local_tests/pipelines/test__pipeline_smoke.py -v -s -DISABLE_SP=1 pytest tests/local_tests/pipelines/test__pipeline_parity.py -v -s -python examples/inference/basic/basic_.py -``` - -Pipeline parity must compare real outputs, not only successful generation: - -- denoised latents when decode parity is expensive or nondeterministic; -- decoded videos/images when visual output should be deterministic enough; -- decoded waveform or audio features for audio pipelines; -- separate video and audio targets for joint AV pipelines unless a validated - joint metric exists. - -Debug pipeline drift in this order: - -1. Confirm both sides use the same component weights and component parity PASS - results are still valid. -2. Align official and FastVideo call arguments, presets, and default values. -3. Align scheduler timesteps, sigmas/noise levels, prediction type, flow shift, - guidance math, and secondary-guidance branches. -4. Align RNG: initial latents/noise, generator device, seed, per-step noise, VAE - sampling, and any official `+1 frame` or crop/slice behavior. -5. Align conditioning: prompt templates, negative prompts, masks, image/audio - preprocessing, modality packing, text truncation, and dtype/autocast. -6. Align decode: latent scaling, per-channel mean/std, tiling flags, output - channel order, sample rate, FPS, and final slicing. -7. Add targeted stage-level diagnostics to identify the first divergent stage. - -If stage diagnostics show the first bad stage is transformer/denoising or a -mid-DiT block, enable activation trace before adding ad hoc pipeline prints; see -`docs/contributing/activation_trace.md` and `../add-model-08-trace/SKILL.md`. -Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and -`FASTVIDEO_TRACE_STEPS` identical across reruns so pipeline parity traces diff -one-to-one. - -If the first divergence belongs to component implementation, strict loading, or -conversion mapping, stop pipeline edits and return `next_step=return_to_phase_6` -with the exact failing evidence. Do not patch conversion from this skill. - -## Stage And Variant Rules - -- Canonical video T2V order: `InputValidationStage`, `TextEncodingStage`, - `ConditioningStage`, `TimestepPreparationStage`, `LatentPreparationStage`, - `DenoisingStage`, `DecodingStage`. -- Canonical video I2V delta adds image loading/encoding and image VAE encoding in - the official order, commonly: `TextEncodingStage`, `ImageEncodingStage`, - `ConditioningStage`, `TimestepPreparationStage`, `LatentPreparationStage`, - `ImageVAEEncodingStage`, `DenoisingStage`, `DecodingStage`. -- Treat `ConditioningStage` as default-present for Wan-style pipelines, but still - follow the reference if another family truly skips or replaces it. -- T2V video pipelines usually use validation, text encoding, conditioning, - timestep preparation, latent preparation, denoising, and decoding. -- I2V adds image loading/encoding and image-latent preparation according to the - official pipeline, not by assuming CLIP or Wan-specific branches. -- Pick image, audio, and video encoders from the reference. Do not assume CLIP or - any other common encoder unless the reference uses it. -- Cross-attention class names are not prescribed; match the family style and - preserve the official tensor contract. -- `WorkloadType` currently has no `T2A`, `A2A`, or `AV` values. Until that enum - is extended, audio-only pipelines may register with `WorkloadType.T2V` and - preset `workload_type="t2v"` as a compatibility shim, but must document the - rationale in code and `PORT_STATUS.md`. -- Audio-only pipelines should not force real video semantics into presets. Use - minimal video-shaped placeholders such as small `height`/`width` and - `num_frames=1` only when shared `VideoGenerator`/validation paths require them, - and document that the real output is audio. -- Record modality-specific shape knobs and output contract in the pipeline - handoff: video uses `height`, `width`, `num_frames`, and `fps`; audio uses - `audio_seconds` and `sampling_rate`; joint AV records both plus whether output - is muxed or paired files. -- Use sibling pipeline classes/configs when required modules, HF repo layout, - stage chains, or inputs differ materially. -- Use one kwargs-driven pipeline class only when variants share weights, - modules, stage chain, and safe call semantics. -- Split later if components diverge, workload tags require separate discovery, - signatures become unsafe, or stage branches become substantial. -- If the DiT branches on `added_kv_proj_dim`, document the T2V/I2V split. -- If the reference uses `transformer_2`, `boundary_ratio`, `guidance_scale_2`, or - DMD step lists, keep those on config, presets, or stages deliberately. -- Support every official output head in scope. If a head is out of scope, record - explicit user approval in `PORT_STATUS.md`. - -Pipeline verification order: - -```bash -pytest tests/local_tests/pipelines/test__pipeline_smoke.py -v -s -DISABLE_SP=1 pytest -v -s tests/local_tests/pipelines/test__pipeline_parity.py -python examples/inference/basic/basic_.py -``` - -Smoke tests prove loadability only. They are not a substitute for numerical -component or pipeline parity. - -## Escape Hatches - -Follow `../add-model/shared/common_rules.md`. Pipeline-specific ask cases include -dropping a public mode, modality, or output head; adding a new workload enum; -changing official defaults for user-facing behavior; accepting a known pipeline -parity blocker; running GPU-heavy quality work outside the agreed scope; or -publishing/uploading generated references or converted weights. - -## Handoff - -Return `../add-model/contracts/pipeline_handoff.md` and update the shared state -files before handoff. - -Do not hand back a green pipeline if smoke or parity skipped locally. A skip is a -setup gap, not a pass. - -## References - -- `fastvideo/pipelines/composed_pipeline_base.py` for module loading and stage - execution. -- `fastvideo/pipelines/basic/wan/` for standard video T2V/I2V/DMD variants. -- `fastvideo/pipelines/basic/stable_audio/` for audio-specific stage composition. -- `fastvideo/configs/pipelines/stable_audio.py` and - `fastvideo/pipelines/basic/stable_audio/presets.py` for config/preset shape. -- `fastvideo/registry.py` for `register_configs(...)` and preset registration. -- `tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py` for latent - parity structure. -- `tests/local_tests/pipelines/test_stable_audio_pipeline_parity.py` for audio - parity structure. -- `tests/local_tests/pipelines/test_stable_audio_pipeline_smoke.py` for no-GPU - import/registry/preset preflight shape. diff --git a/.agents/skills/add-model-09-pipeline/templates/pipeline_parity_test.py b/.agents/skills/add-model-09-pipeline/templates/pipeline_parity_test.py deleted file mode 100644 index 07c29c0da..000000000 --- a/.agents/skills/add-model-09-pipeline/templates/pipeline_parity_test.py +++ /dev/null @@ -1,147 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -"""Pipeline parity scaffold for TODO_MODEL_FAMILY. - -Copy this file to -`tests/local_tests/pipelines/test__pipeline_parity.py` and replace every -TODO before treating it as an executable scaffold. - -The filled test should compare denoised latents, decoded media, audio waveform, -or another concrete output from the official pipeline against FastVideo. A -successful generation without tensor/media comparison is not parity. -""" -from __future__ import annotations - -import os -import sys -from pathlib import Path -from typing import Any - -import pytest -import torch -from torch.testing import assert_close - -_REPO_ROOT = Path(__file__).resolve().parents[3] -_MODEL_FAMILY = "TODO_MODEL_FAMILY" -_OFFICIAL_REF_ENV = "TODO_OFFICIAL_REF_PATH" -_OFFICIAL_REF_DEFAULT = _REPO_ROOT / "TODO_OFFICIAL_REF_DIR" -_FASTVIDEO_MODEL_ENV = "TODO_FASTVIDEO_MODEL_PATH" -_FASTVIDEO_MODEL_DEFAULT = _REPO_ROOT / "converted_weights" / _MODEL_FAMILY - - -def _path_from_env(env_name: str, default: Path) -> Path: - return Path(os.getenv(env_name, str(default))).expanduser() - - -def _add_official_to_path() -> Path: - official_path = _path_from_env(_OFFICIAL_REF_ENV, _OFFICIAL_REF_DEFAULT) - if not official_path.exists(): - pytest.skip(f"Official reference not found at {official_path}") - if str(official_path) not in sys.path: - sys.path.insert(0, str(official_path)) - return official_path - - -def _log_tensor_stats(label: str, tensor: torch.Tensor) -> None: - value = tensor.detach().float() - print(f"[{_MODEL_FAMILY} PIPELINE] {label}: shape={tuple(tensor.shape)} " - f"dtype={tensor.dtype} device={tensor.device} " - f"min={value.min().item():.6f} max={value.max().item():.6f} " - f"mean={value.mean().item():.6f} std={value.std().item():.6f}") - - -def _extract_tensor(output: Any, key: str) -> torch.Tensor: - if isinstance(output, dict): - value = output.get(key) - else: - value = getattr(output, key, None) - if value is None: - raise AssertionError(f"Pipeline output did not contain {key!r}") - if not torch.is_tensor(value): - try: - import numpy as np - value = torch.from_numpy(np.asarray(value)) - except Exception as exc: # pragma: no cover - scaffold guard - raise AssertionError(f"Could not convert {key!r} to tensor") from exc - return value.detach().float().cpu() - - -def _run_official_pipeline( - official_path: Path, - params: dict[str, Any], - device: torch.device, -) -> Any: - del official_path, params, device - pytest.skip("TODO: import the official pipeline/factory, load official weights, " - "run with params, and return the comparison target.") - - -def _run_fastvideo_pipeline(model_path: Path, params: dict[str, Any]) -> Any: - from fastvideo import VideoGenerator - - generator = VideoGenerator.from_pretrained( - str(model_path), - num_gpus=1, - use_fsdp_inference=False, - dit_cpu_offload=False, - vae_cpu_offload=False, - text_encoder_cpu_offload=False, - ) - try: - return generator.generate_video( - prompt=params["prompt"], - negative_prompt=params.get("negative_prompt"), - output_path=f"outputs_{_MODEL_FAMILY}/pipeline_parity", - save_video=False, - height=params.get("height"), - width=params.get("width"), - num_frames=params.get("num_frames"), - fps=params.get("fps"), - num_inference_steps=params["num_inference_steps"], - guidance_scale=params.get("guidance_scale"), - seed=params["seed"], - ) - finally: - generator.shutdown() - - -@pytest.mark.skipif( - not torch.cuda.is_available(), - reason="TODO_MODEL_FAMILY pipeline parity requires CUDA.", -) -def test_todo_model_family_pipeline_official_parity() -> None: - official_path = _add_official_to_path() - fastvideo_model_path = _path_from_env( - _FASTVIDEO_MODEL_ENV, - _FASTVIDEO_MODEL_DEFAULT, - ) - if not fastvideo_model_path.exists(): - pytest.skip(f"FastVideo model path not found at {fastvideo_model_path}") - - device = torch.device("cuda:0") - params = { - "prompt": "TODO: stable parity prompt", - "negative_prompt": "", - "height": 64, - "width": 64, - "num_frames": 9, - "fps": 8, - "num_inference_steps": 4, - "guidance_scale": 1.0, - "seed": 0, - } - - official_output = _run_official_pipeline(official_path, params, device) - fastvideo_output = _run_fastvideo_pipeline(fastvideo_model_path, params) - - comparison_key = "TODO_COMPARISON_KEY" - official_tensor = _extract_tensor(official_output, comparison_key) - fastvideo_tensor = _extract_tensor(fastvideo_output, comparison_key) - - _log_tensor_stats("official", official_tensor) - _log_tensor_stats("fastvideo", fastvideo_tensor) - assert official_tensor.shape == fastvideo_tensor.shape - - diff = (official_tensor - fastvideo_tensor).abs() - print(f"diff max={diff.max().item():.6f} " - f"mean={diff.mean().item():.6f} median={diff.median().item():.6f}") - assert_close(fastvideo_tensor, official_tensor, atol=1e-2, rtol=1e-2) diff --git a/.agents/skills/add-model-10-pr-review/SKILL.md b/.agents/skills/add-model-10-pr-review/SKILL.md deleted file mode 100644 index 8e6d7a1eb..000000000 --- a/.agents/skills/add-model-10-pr-review/SKILL.md +++ /dev/null @@ -1,141 +0,0 @@ ---- -name: add-model-10-pr-review -description: Review rubric for FastVideo PRs that add or modify model families, variants, first-class components, checkpoint conversion, pipelines, parity coverage, or generated-media quality baselines. Use when reviewing a PR whose diff touches fastvideo/models/, fastvideo/pipelines/basic/, fastvideo/registry.py, scripts/checkpoint_conversion/, fastvideo/tests/ssim/, or related model-port surfaces. Pairs with review-pr-link as a project-scoped review pass; produces findings, not fixes. ---- - -# Add-Model PR Review - -Use this skill when a reviewed PR appears to add, port, or substantially modify -a FastVideo model family, model variant, first-class model component, -checkpoint conversion, model pipeline, or local parity coverage. - -This is a review skill, not an implementation workflow. Do not run `/add-model` -or start writing missing port code during review. Use the add-model skill stack -as a rubric for findings. - -## Trigger Paths - -Trigger this skill if `git diff --name-only ...HEAD` includes any of: - -- `fastvideo/models/dits/`, `fastvideo/configs/models/dits/` -- `fastvideo/models/vaes/`, `fastvideo/configs/models/vaes/` -- `fastvideo/models/encoders/`, `fastvideo/configs/models/encoders/` -- `fastvideo/models/schedulers/`, `fastvideo/configs/models/schedulers/` -- `fastvideo/models/upsamplers/`, `fastvideo/configs/models/upsamplers/` -- `fastvideo/models/audio/`, `fastvideo/configs/models/audio/` -- `fastvideo/pipelines/basic/`, `fastvideo/configs/pipelines/` -- `fastvideo/registry.py`, `fastvideo/api/sampling_param.py` -- `scripts/checkpoint_conversion/` -- `examples/inference/basic/` -- `tests/local_tests/`, especially component or pipeline parity tests -- `fastvideo/tests/ssim/` or other quality-regression tests for generated media - -Also trigger when the PR title/body claims a new model, model variant, VAE, -encoder, scheduler, conditioner, pipeline, conversion script, or generated-media -quality baseline even if the path list is incomplete. - -## Review Inputs - -Read these add-model references as review checklists: - -- `../add-model/SKILL.md`: phase gates and final handoff requirements. -- `../add-model/shared/common_rules.md`: token/auth safety, production import - boundaries, state files, and skip/pass semantics. -- `../add-model/contracts/final_handoff.md`: final evidence expected from a - complete port. -- `../add-model/contracts/component_context.md` and - `../add-model/contracts/component_skill_handoff.md`: component evidence and - parity-debug expectations. -- `../add-model/contracts/conversion_request.md` and - `../add-model/contracts/conversion_handoff.md`: conversion evidence, - strict-load status, config validation, and retry context. -- `../add-model/contracts/pipeline_context.md` and - `../add-model/contracts/pipeline_handoff.md`: pipeline class/stage/config/ - preset/registry/example evidence. - -Then read only the satellite skill(s) that match touched areas: - -- DiT/transformer changes: `../add-model-03-port-dit/SKILL.md`. -- VAE changes: `../add-model-04-port-vae/SKILL.md`. -- Encoder/conditioner changes: `../add-model-05-port-encoder/SKILL.md`. -- Scheduler/upsampler/vocoder/other components: - `../add-model-06-port-generic/SKILL.md`. -- Component parity tests: `../add-model-02-parity/SKILL.md`. -- Checkpoint conversion: `../add-model-07-conversion/SKILL.md`. -- Pipeline/config/presets/registry/examples: - `../add-model-09-pipeline/SKILL.md`. -- Prep/state docs: `../add-model-01-prep/SKILL.md`. - -## Required Review Lanes - -For a full model-family or model-variant PR, cover all lanes. For a -component-only PR, cover the component, conversion/parity as applicable, and the -documented downstream consumer. - -1. Scope and source-of-truth lane: - Verify the PR clearly identifies the official reference, weights/revision, - supported variants, modalities, output heads, and any approved scope cuts. - -2. Component lane: - Verify each required component is FastVideo-native or has a documented and - accepted lazy-wrapper exception. Check bucket/config inheritance, `EntryClass`, - state-dict surface, reused-component evidence, and output heads. - -3. Conversion lane: - Verify mappings are derived from prototype key/shape dumps, source layout is - supported, skipped keys are intentional, emitted configs validate through - production paths, component strict-load status is recorded, `model_index.json` - library tokens match loaders, and revisions are pinned when converting from - HF. - -4. Component parity lane: - Verify local parity tests exist for every required component, including reused - components. Scaffolds may skip in CI, but the PR must provide local non-skip - PASS evidence or an explicit accepted blocker. - -5. Pipeline lane: - Verify stage order, required modules, `_class_name` / `EntryClass.__name__` - resolution, config defaults, presets, `SamplingParam` fields, registry - registration, examples, smoke tests, and pipeline parity. - -6. Quality and evidence lane: - Verify media quality regression is added or explicitly deferred, examples run, - generated outputs are non-corrupt, `tests/local_tests//README.md` and - `PORT_STATUS.md` are current, and final blockers are surfaced in the review. - -## Findings To Prioritize - -Prioritize review findings in this order: - -- Missing or skipped required component parity without accepted blocker. -- Pipeline parity/smoke/example missing or skipped for a pipeline PR. -- Conversion emits unloadable or unvalidated configs/weights. -- Wrong `model_index.json` `_class_name`, component library token, or registry - class resolution. -- Runtime diffusers/transformers model-class imports for components that own - weights or numerical behavior. -- Dropped modalities, output heads, variants, or conditioning streams without - explicit approval. -- Reused FastVideo component lacks exact definition/instantiation proof or - non-skip parity. -- Public generation kwargs/preset defaults missing from `SamplingParam`. -- Tests only check shapes, importability, or successful generation without - numerical/media comparison. -- Tokens, credentials, reference clones, staged weights, or generated bulk assets - committed to the PR. - -## Output Format - -Write normal code-review findings first, ordered by severity. Include file and -line references from the PR diff when possible. - -Use this phrasing for missing add-model evidence: - -```text -This PR does not satisfy the add-model -gate because is missing. The risk is . -``` - -Keep the summary short. Mention which lanes were reviewed and which could not be -verified because assets, GPU time, or external credentials were unavailable. diff --git a/.agents/skills/add-model/SKILL.md b/.agents/skills/add-model/SKILL.md deleted file mode 100644 index d7f83de23..000000000 --- a/.agents/skills/add-model/SKILL.md +++ /dev/null @@ -1,438 +0,0 @@ ---- -name: add-model -description: Manual /add-model workflow for implementing a FastVideo model or first-class component port after add-model-01-prep has staged reference code and weights. Organizes the port into numbered phases with conversion rules, component policies, parity gates, and handoff checks. ---- - -# Add Model - -## Manual Invocation - -This skill is for explicit `/add-model` use only. Do not auto-start it from a -casual model-port mention. The setup-only workflow is -`../add-model-01-prep/SKILL.md`. - -## Goal - -Port a new FastVideo model family, model variant, or first-class reusable -component so it can be loaded through FastVideo's native model, config, stage, -registry, preset, and test infrastructure. - -FastVideo has one pipeline architecture: stage-based composition via -`ComposedPipelineBase`. Vary the stages and modules, not the architecture. - -## Scope Shapes - -Use this skill for either shape: - -| Shape | Required output | -|---|---| -| Full model family or variant | Native components, conversion if needed, pipeline config/class, presets, registry, smoke test, local parity tests, example, quality regression. | -| First-class component contribution | Native component class/config, bucket export, component parity test, and a documented downstream pipeline that will consume it. Skip pipeline/preset/registry rows only when the contribution is intentionally component-only. | - -If upstream ships many variants, lock scope before coding. "Base model" means -checkpoint variant, not a modality subset. If the base checkpoint produces -audio, pose, depth, masks, or other output heads, either support those outputs -or get explicit user agreement to drop them. - -## Required Input - -Start from an `add-model-01-prep` handoff, or equivalent fields matching -`contracts/prep_handoff.md`. - -Before Phase 0, read the shared rules and all relevant schemas: - -- `shared/common_rules.md` -- `contracts/prep_handoff.md` -- `contracts/port_state.md` -- `contracts/escape_hatch.md` -- `contracts/component_context.md` -- `contracts/parity_status.md` -- `contracts/conversion_request.md` -- `contracts/conversion_handoff.md` -- `contracts/component_skill_handoff.md` -- `contracts/pipeline_context.md` -- `contracts/pipeline_handoff.md` -- `contracts/final_handoff.md` - -## Hard Rules - -- Follow `shared/common_rules.md` for token/auth safety, state files, escape - hatches, production import boundaries, and skip/pass semantics. -- If the prep handoff is missing or ambiguous, stop and run - `../add-model-01-prep/SKILL.md`. -- If a needed component is not ported, do not ship the pipeline that needs it. -- Wan is grandfathered for missing local parity; do not copy its missing-test - precedent for new work. - -## Escape Hatches - -Follow `shared/common_rules.md` and `contracts/escape_hatch.md`. The main -orchestrator should ask only when no phase skill can safely continue under the -shared rules. - -## Files Map - -| Area | Paths | -|---|---| -| DiT | `fastvideo/models/dits/.py`, `fastvideo/configs/models/dits/.py`, bucket `__init__.py`. | -| VAE | `fastvideo/models/vaes/.py`, `fastvideo/configs/models/vaes/.py`, bucket `__init__.py`. Name by shared arch when reusable (`oobleck.py`, `autoencoder_kl.py`), otherwise by family (`wanvae.py`). | -| Encoder / conditioner / scheduler / upsampler | Native class/config in the matching `fastvideo/models//` and `fastvideo/configs/models//` bucket. | -| Lazy loader wrapper | Optional `fastvideo/models//_loader.py` or similar thin `nn.Module` wrapper when a component is fetched from an external HF repo and should be hidden from host-pipeline state-dict matching. | -| Conversion | `scripts/checkpoint_conversion/_to_diffusers.py` only when `needs_conversion=yes`. | -| Pipeline | `fastvideo/pipelines/basic//_pipeline.py` plus sibling files for variants whose components or required modules differ. | -| Pipeline config | `fastvideo/configs/pipelines/.py` or `fastvideo/pipelines/basic//pipeline_configs.py`. | -| Stages | `fastvideo/pipelines/basic//stages/` only for model-specific stage subclasses. | -| Presets / registry | `fastvideo/pipelines/basic//presets.py`, `fastvideo/registry.py`. | -| Tests | Component parity under `tests/local_tests//`; pipeline smoke/parity under `tests/local_tests/pipelines/`; CI-backed quality tests under `fastvideo/tests/`. | -| Example | `examples/inference/basic/basic_*.py`, one per public mode/variant. | - -## Phase 0: Scope And Handoff Gate - -1. Validate every required handoff field. -2. Resolve `needs_conversion=unknown` before component work: - -```bash -python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \ - "" \ - --json -``` - -3. List first-PR scope across both axes: - - Variant axis: base, distill, SR/refine, causal, DMD, I2V, V2V, etc. - - Modality axis: video, image, audio, pose, depth, masks, text, etc. -4. For component-only work, explicitly name the downstream full-pipeline PR or - planned consumer. -5. Confirm `official_env_status` is `imports_ok` or - `private_deps_need_stubs`. If it is `blocked`, return to - `../add-model-01-prep/SKILL.md` before parity scaffolding. -6. Confirm `local_tests_readme` exists and records official setup, HF weights, - dependency changes, and planned parity commands for reviewers. -7. Confirm `port_state_file` exists, follows `contracts/port_state.md`, and has - rows for open questions/issues found during prep. -8. If there are multiple official implementations, choose the one whose - architecture matches the published weights. A blessed library port can be a - better parity reference than a highly configurable research repo; document - the choice in tests. - -## Phase 1: Reference And Architecture Study - -Read the official pipeline call path before writing code. - -Record: - -- Required modules from `model_index.json` or equivalent: transformer, VAE, - text encoders, tokenizers, scheduler, image encoders, audio VAE, vocoder, - conditioners, upsamplers. -- Input/output modalities and every dedicated DiT output head. -- Text/image/audio encoding flow, latent shape, dtype, scaling, packing, - scheduler/timestep math, guidance math, VAE normalization, and decode flow. -- Whether the official code relies on private deps, custom ops, or special - kernels that parity tests must stub. - -Arch config rule: - -- `ArchConfig` fields must match the emitted per-component config, especially - `transformer/config.json`, one-to-one. -- Pipeline knobs do not belong on the DiT arch config: inference steps, CFG - scales, flow shift, FPS, VAE stride, text target length, data-proxy knobs, - eval defaults, and sampling defaults go on `PipelineConfig`, presets, or - stages. -- If the HF repo is raw or has empty configs, synthesize - `transformer/config.json` from the official Python model-config class, not - from data/eval config classes. - -## Phase 2: Early Parity Scaffolding - -Create component parity tests before or alongside implementation. Use -`../add-model-02-parity/SKILL.md` and its `templates/component_parity_test.py`. -The official reference must import in the current FastVideo environment, or the -prep handoff must identify private deps that will be stubbed locally for tests. -Use `local_tests_readme` as the reviewer-facing source for setup commands and -update its planned test table as parity scaffolds are added. - -This phase is early by design: - -- Official loading can be implemented from the reference study. -- FastVideo loading can target planned standardized class/config/loader paths. -- Tests may initially skip because the FastVideo class or converted weights do - not exist yet. -- The scaffold must still contain real official loading, deterministic inputs, - output extraction, and concrete tensor comparisons. No unconditional skips, - no shape-only tests. - -Use subagents here: dispatch one parity-test subagent per required component, -including components that may be reused. Their output becomes the red/skip -target that porting or reuse-verification subagents make pass later. - -## Phase 3: Reuse Gate And Component Dispatch - -Build a component inventory before implementation: - -| Field | Meaning | -|---|---| -| Component | transformer, VAE, text encoder, image encoder, scheduler, conditioner, upsampler, vocoder, etc. | -| Official definition | Repo-relative source file, class/function name, and relevant line/range if known. | -| Official instantiation | Repo-relative pipeline/config/factory call site plus constructor args and runtime flags. | -| FastVideo target | Existing class to reuse or new bucket/file/config to add. | -| Parity test | Required local test path, including reused components. | -| Status | `reuse_pending`, `reuse_proven`, `port_pending`, `non_skip_pass`, or `blocked`. | - -Reuse is allowed only from the checked-out FastVideo tree. Do not wait for or -depend on an open PR adding a native class; add the native port directly in this -PR if the current tree cannot be reused. - -Reuse decision: - -1. Record exact official definition and instantiation evidence for every - component. -2. If an existing FastVideo class and config match both definition and - instantiation, pass that reused target to the bucket-specific skill in - `mode=prototype` and require reuse evidence plus key/shape dumps. -3. If either definition or instantiation differs, port the component directly as - FastVideo-native code through the bucket-specific skill. -4. Reused components still require non-skip component parity against the exact - official instantiation used by the target pipeline. - -Porting subagent dispatch: - -- Dispatch one subagent per component after Phase 2 parity scaffolds exist. -- Use `../add-model-03-port-dit/SKILL.md` for DiTs/transformers. -- Use `../add-model-04-port-vae/SKILL.md` for VAEs. -- Use `../add-model-05-port-encoder/SKILL.md` for text, image, audio, or compound - encoders/conditioners that fit the encoder config bucket. -- Use `../add-model-06-port-generic/SKILL.md` for schedulers, upsamplers, - vocoders, adapters, preprocessors, or unknown components. -- Each subagent owns one component only and must loop on that component's local - parity test until it produces a non-skip PASS or returns a precise blocker. - -Every component subagent must receive a complete packet matching -`contracts/component_context.md`. If any required path is unknown, pass `unknown` -plus the exact search already performed. Do not silently omit ambiguous official -files or prototype concerns. - -Bucket, layer, and attention rules live in the bucket-specific skills and -`fastvideo/layers/AGENTS.md`. - -## Phase 4: Native Component Prototype - -Conversion needs a FastVideo state-dict surface. Use the Phase 3 -bucket-specific skill in `mode=prototype` for every required component, including -reused components. - -Prototype success criteria: - -- the FastVideo-native or reused class/config can import and instantiate with the - exact official architecture args; -- official and FastVideo key/shape dumps exist for every stateful component; -- `local_tests_readme` and `port_state_file` record prototype status and concerns; -- the returned handoff matches `contracts/component_skill_handoff.md`. - -Do not chase numerical parity in Phase 4. Prototype mode ends when conversion has -the key/shape surface it needs, or when the component skill returns a precise -blocker or escape hatch. - -## Phase 5: Param Mapping And Weight Conversion - -Use `../add-model-07-conversion/SKILL.md` after Phase 4 prototypes exist. -Send a request matching `contracts/conversion_request.md`; consume the returned -`contracts/conversion_handoff.md` update before Phase 6. - -Use the prep handoff's `needs_conversion` value: - -- `no`: verify the source already has the component layout FastVideo loaders can - consume, then record any passthrough components. -- `yes`: write `scripts/checkpoint_conversion/_to_diffusers.py` and - output `converted_weights//`. -- `unknown`: return to Phase 0. - -The conversion skill owns source-layout handling, mapping derivation, config and -`model_index.json` emission, passthrough assets, strict-load verification, and -Phase 6 retry requests. Component skills must not patch conversion scripts or -converted weights ad hoc. - -## Phase 6: Component Parity Debug - -This is the expected expensive loop. Dispatch one subagent per required -component, including reused components, using the bucket-specific skill in -`mode=parity-debug`. - -Each subagent gets: - -- the complete component context packet from Phase 3/4; -- updated conversion mapping notes and strict-load result from Phase 5; -- any prototype concerns or unknowns that were not resolved before conversion. - -The bucket-specific skills own parity-debug tactics. If a failure belongs to -conversion, route it through `../add-model-07-conversion/SKILL.md` with a retry -request matching `contracts/conversion_request.md`, then resume the component -skill with the updated conversion handoff. - -When a component failure narrows to layer-by-layer numerical drift, load -`../add-model-08-trace/SKILL.md` before writing custom hooks. It uses -`fastvideo/hooks/activation_trace.py`; canonical env vars and JSONL format are -documented in `docs/contributing/activation_trace.md`. - -Phase 6 ends only when every required component handoff reports -`parity_status=non_skip_pass`, or when a precise blocker or escape hatch is -recorded in `port_state_file`. - -## Phase 7: Pipeline, Stages, And Variants - -Do not start Phase 7 until every required component, reused or ported, has a -non-skip local parity PASS from Phase 6. If any component parity test is still -`scaffold_skip`, `debug_red`, `blocked`, or missing, resume Phase 6 first. - -Use `../add-model-09-pipeline/SKILL.md` for pipeline definition and parity-debug. -Send a complete packet matching `contracts/pipeline_context.md`; consume the -returned `contracts/pipeline_handoff.md` before moving to quality regression or -final handoff. - -The pipeline skill owns: - -- pipeline class, stage chain, and optional model-specific stages; -- pipeline config, presets, registry updates, and examples; -- official args/defaults/presets comparison before setting FastVideo defaults; -- pipeline smoke and parity tests; -- continuous pipeline parity-debug until non-skip PASS or precise blocker; -- updates to `local_tests_readme` and `port_state_file`. - -The pipeline handoff must explicitly cover stage order, variants, modality and -output-head handling, config/preset/registry/example status, smoke/parity tests, -and any return-to-Phase-6 evidence. - -## Phase 8: PipelineConfig, Presets, Registry, Examples - -This phase is implemented through `../add-model-09-pipeline/SKILL.md` after the -Phase 7 component-parity gate passes. Accept the pipeline handoff only if it -covers configs, presets, registry detection/exact class resolution, examples, -new `SamplingParam` fields for public kwargs/defaults, and local smoke/parity -status. Detailed rules live in `../add-model-09-pipeline/SKILL.md`. - -## Phase 9: Parity Activation And Local Verification - -Local parity is author-run, not CI-enforced. CI may only run package-level -quality tests later. Before handoff, Phase 2 scaffolds must be activated into -non-skip PASS results. - -Order is mandatory: - -1. Run conversion if needed. -2. Run component parity for every required component, including reused ones. -3. Run pipeline smoke. -4. Run pipeline parity. -5. Run the basic example. - -If pipeline smoke or parity points back to component implementation, -strict-load, or conversion mapping, return to Phase 6 or Phase 5 rather than -patching around the issue in the pipeline. - -Skip policy: - -- Follow `shared/common_rules.md`: a committed local test may skip for absent - clones/weights, but a local skip is not a verified pass. - -Use the commands and tolerance guidance from `../add-model-02-parity/SKILL.md` for -component checks and from `../add-model-09-pipeline/SKILL.md` for pipeline smoke, -pipeline parity, and examples. Record exact commands, status, and blockers in -`local_tests_readme` and `port_state_file`. - -## Phase 10: Quality Regression - -Video outputs: - -- Add `fastvideo/tests/ssim/test__similarity.py` when output video - quality must be preserved. -- Seed references through `seed-ssim-references` after the test exists. - -Audio outputs: - -- SSIM does not apply. Use an audio-specific regression metric such as - mel-spectrogram L1, multi-resolution STFT, CLAP cosine, or a project-approved - learned metric. -- Document the metric and hardware/runtime assumptions in the test. - -Joint AV outputs: - -- Keep video and audio regression checks separate unless there is a validated - joint metric. - -## Phase 11: Post-Parity Review And Handoff - -After parity is green, run a hot-path review before handoff: - -- Hoist constant tensor allocations out of sampler/denoising loops. -- Replace per-step `randn_like` churn with preallocated buffers plus - `.normal_()` when safe. -- Move `torch.backends.*` flag changes to one-shot setup/load paths. -- Delete `batch.extra` writes that nothing reads. -- Derive magic constants from configs when possible. - -Pre-handoff checklist: - -```text -[ ] Prep handoff is complete and committed nowhere with token values. -[ ] Conversion was run if needed and output loads with real weights. -[ ] Every required component, reused or newly ported, has a non-skip local parity PASS. -[ ] `local_tests_readme` lists every component parity test, command, status, and blocker if any. -[ ] `port_state_file` has every open question/issue either resolved or listed as an explicit blocker. -[ ] Any `next_step=ask_user` has a matching `escape_hatch` block and `E###` row. -[ ] Pipeline smoke has a non-skip local PASS. -[ ] Pipeline parity has a non-skip local PASS against the official reference. -[ ] Basic example runs and writes a non-corrupt output. -[ ] Video SSIM or audio-specific quality regression is added or explicitly deferred. -[ ] Runtime production code has no diffusers/transformers model-class imports. -[ ] Production comments are WHY-focused; examples have user-story docstrings. -[ ] Post-parity hot-path pass is complete. -``` - -Ask before deleting any reference clone or staged weights created by -`add-model-01-prep`. Leave `.gitignore` entries so future parity assets stay -untracked. Never commit the clone, weights, `.env`, credentials, or anything -matching `*secret*`. - -## References - -- `../add-model-01-prep/SKILL.md` for user-input collection, HF inspection, - weight staging, reference cloning, and setup handoff. -- `contracts/` for canonical handoff schemas used by prep, parity, conversion, - component porting, escape hatches, and final handoff. -- `../add-model-02-parity/SKILL.md` for early component parity scaffolds and - activation templates. -- `../add-model-07-conversion/SKILL.md` for Phase 5 mapping, conversion scripts, - monolithic checkpoint splitting, and strict-load checks. -- `../add-model-03-port-dit/SKILL.md`, `../add-model-04-port-vae/SKILL.md`, - `../add-model-05-port-encoder/SKILL.md`, and - `../add-model-06-port-generic/SKILL.md` for component subagent implementation - and parity-debug loops. -- `../add-model-09-pipeline/SKILL.md` for pipeline definition, config/preset/ - registry/example wiring, smoke tests, and pipeline parity-debug. -- `fastvideo/layers/AGENTS.md` for native layer selection and state-dict surface - guidance. -- `docs/contributing/coding_agents.md` for narrative context. -- `docs/design/overview.md` for pipeline/config/registry architecture. -- `fastvideo/pipelines/basic/wan/` for standard T2V/I2V/DMD/Causal variants. -- `fastvideo/pipelines/basic/ltx2/` for non-standard stages and audio/video - patterns. -- `tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py` for pipeline - parity shape. -- `tests/local_tests/transformers/test_ltx2.py`, - `tests/local_tests/vaes/test_ltx2_vae.py`, and - `tests/local_tests/encoders/test_ltx2_gemma_parity.py` for component parity. -- `scripts/checkpoint_conversion/convert_ltx2_weights.py` for modern conversion - script shape. -- `scripts/checkpoint_conversion/wan_to_diffusers.py` for legacy regex mapping - reference only. - -## Changelog - -| Date | Change | -|---|---| -| 2026-04-24 | Initial FastVideo add-model workflow. | -| 2026-04-30 | Split external setup into `add-model-01-prep`. | -| 2026-04-30 | Rewrote as manual `/add-model` phase workflow and incorporated prior review decisions. | -| 2026-04-30 | Extracted early parity scaffolding into `add-model-02-parity` and moved it before conversion/component implementation. | -| 2026-04-30 | Added component reuse proof gate, bucket-specific porting skills, and parity PASS requirement for reused components. | -| 2026-04-30 | Split prototype, conversion, and parity-debug phases; added conversion skill for monolithic and separate checkpoint layouts. | -| 2026-04-30 | Extracted handoff schemas into `contracts/` for shared use across skills. | -| 2026-04-30 | Added pipeline skill contract and Phase 7 component-parity gate. | -| 2026-04-30 | Added escape-hatch contract for user decisions and `ask_user` handoffs. | diff --git a/.agents/skills/add-model/contracts/README.md b/.agents/skills/add-model/contracts/README.md deleted file mode 100644 index 5eaa766dd..000000000 --- a/.agents/skills/add-model/contracts/README.md +++ /dev/null @@ -1,29 +0,0 @@ -# Add Model Contracts - -Canonical handoff schemas for the `/add-model` workflow. When a skill needs to -send or receive structured context, use these files instead of inventing a local -schema. - -| Contract | Use | -|---|---| -| `prep_handoff.md` | `add-model-01-prep` output and `/add-model` Phase 0 input. | -| `port_state.md` | Per-port `PORT_STATUS.md` file tracking progress, open questions, and issues. | -| `escape_hatch.md` | Shared pause-and-ask schema for user decisions the workflow cannot safely choose. | -| `component_context.md` | Per-component packet passed to parity, prototype, conversion, and parity-debug subagents. | -| `parity_status.md` | `add-model-02-parity` scaffold/activation status returned to `/add-model`. | -| `conversion_request.md` | Phase 5 conversion input and Phase 6 conversion retry request. | -| `conversion_handoff.md` | `add-model-07-conversion` output back to `/add-model` and component subagents. | -| `component_skill_handoff.md` | Component porting skill output in prototype or parity-debug mode. | -| `pipeline_context.md` | Phase 7 packet passed to `add-model-09-pipeline` after component parity is green. | -| `pipeline_handoff.md` | `add-model-09-pipeline` output back to `/add-model` after pipeline definition or parity-debug. | -| `final_handoff.md` | Final `/add-model` pre-handoff checklist summary. | - -Rules: - -- Do not omit required fields. Use `unknown` plus the search already performed - when the value is not known yet. -- Do not include raw token values. Use env var names only. -- Keep model-specific mapping details in conversion scripts and the local tests - README/status notes, not in generic skill docs. -- Use `next_step=ask_user` only with an `escape_hatch` block matching - `escape_hatch.md`. diff --git a/.agents/skills/add-model/contracts/component_context.md b/.agents/skills/add-model/contracts/component_context.md deleted file mode 100644 index 5343cb10e..000000000 --- a/.agents/skills/add-model/contracts/component_context.md +++ /dev/null @@ -1,55 +0,0 @@ -# Component Context Contract - -Canonical per-component packet passed from `/add-model` to parity, prototype, -conversion, and parity-debug subagents. - -```text -component_context: - model_family: - component: - component_type: - mode: parity-scaffold | prototype | parity-debug - official_ref_dir: - official_definition_files: - - path: - symbols: - notes: - official_instantiation_files: - - path: - symbols: - args: - official_weight_source: - fastvideo_target_files: - - fastvideo/models//.py - - fastvideo/configs/models//.py - local_tests_readme: tests/local_tests//README.md - port_state_file: tests/local_tests//PORT_STATUS.md - parity_test: tests/local_tests//test___parity.py - prototype_key_dumps: - official: converted_weights//_mapping/_official_keys.json | planned | unknown - fastvideo: converted_weights//_mapping/_fastvideo_keys.json | planned | unknown - conversion: - script: scripts/checkpoint_conversion/_to_diffusers.py | not_created | not_needed | unknown - converted_component_dir: converted_weights// | not_created | not_needed | unknown - model_index_library: - config_file: - mapping_notes: - production_loader_strictness: - strict_load: - concerns_or_unknowns: - - -``` - -Rules: - -- If any required path is unknown, pass `unknown` plus the exact search already - performed. -- Do not silently omit ambiguous official files, instantiation args, or prototype - concerns. -- For reused components, still fill every field and set `fastvideo_target_files` - to the reused class/config. -- In `mode=parity-scaffold`, prototype and conversion fields may be `planned`, - `not_created`, `not_needed`, or `unknown`; do not invent paths or statuses that - do not exist yet. -- Update `port_state_file` when concerns, issues, conversion status, or parity - status change. diff --git a/.agents/skills/add-model/contracts/component_skill_handoff.md b/.agents/skills/add-model/contracts/component_skill_handoff.md deleted file mode 100644 index a12583eb0..000000000 --- a/.agents/skills/add-model/contracts/component_skill_handoff.md +++ /dev/null @@ -1,41 +0,0 @@ -# Component Skill Handoff Contract - -Returned by `add-model-03-port-dit`, `add-model-04-port-vae`, -`add-model-05-port-encoder`, and `add-model-06-port-generic`. - -```text -component: -mode: prototype | parity-debug -files_changed: -official_files_used: -prototype_key_dumps: -port_state_file: tests/local_tests//PORT_STATUS.md -concerns_or_unknowns: -parity_test: -parity_status: scaffold_skip | debug_red | non_skip_pass | blocked -production_loader_strictness: strict | non_strict_with_allowed_keys | stateless -strict_load: pass | pass_with_documented_exclusions | blocked | not_run -pytest_output: -blocker: -conversion_retry_request: -readme_updated: yes | no -next_step: phase_5_conversion | phase_5_conversion_retry | phase_6_continue | ask_user | blocked -escape_hatch: -``` - -Rules: - -- In `mode=prototype`, parity may be `scaffold_skip` or `blocked`; key dumps are - the required artifact. Successful prototype handoff should use - `next_step=phase_5_conversion`. -- In `mode=parity-debug`, final success requires `parity_status=non_skip_pass`. -- If conversion is implicated, return `conversion_retry_request` and do not edit - conversion scripts or converted weights directly. -- If production loading is non-strict, list allowed missing/unexpected keys in - the parity test or handoff and mark `strict_load=pass_with_documented_exclusions`. -- Update `port_state_file` before returning: component row, open questions, - issues/blockers, decisions, and handoff notes. -- Return an `escape_hatch` only for user decisions, not for normal component - implementation or parity-debug failures. -- Use `next_step=ask_user` only with an `escape_hatch` block and a matching - `PORT_STATUS.md` row. diff --git a/.agents/skills/add-model/contracts/conversion_handoff.md b/.agents/skills/add-model/contracts/conversion_handoff.md deleted file mode 100644 index af9e778ec..000000000 --- a/.agents/skills/add-model/contracts/conversion_handoff.md +++ /dev/null @@ -1,41 +0,0 @@ -# Conversion Handoff Contract - -Returned by `../add-model-07-conversion/SKILL.md` to `/add-model` and component -parity-debug subagents. - -```text -conversion_script: scripts/checkpoint_conversion/_to_diffusers.py -source_layout: -converted_weights_dir: converted_weights/ -port_state_file: tests/local_tests//PORT_STATUS.md -components_written: -passthrough_components: -strict_load: pass | pass_with_documented_exclusions | blocked -component_context_updates: - - component: - converted_component_dir: - model_index_library: - config_file: - config_validation: pass | blocked | not_applicable - mapping_notes: - production_loader_strictness: strict | non_strict_with_allowed_keys | stateless - strict_load: pass | pass_with_documented_exclusions | blocked | not_run - retry_resolved: - concerns_or_unknowns: -blocked_on: -next_step: phase_6_component_parity_debug | ask_user -escape_hatch: -``` - -Rules: - -- Include strict-load evidence for every stateful converted component. -- If a component intentionally loads non-strictly, list the exact missing or - unexpected keys and why they are safe. -- Include the actual `model_index.json` library token, config filename, and config - validation result for every emitted component. -- Preserve retry evidence so the requesting component subagent can resume with - updated context. -- Keep `port_state_file` synchronized with `component_context_updates`. -- Use `next_step=ask_user` only with an `escape_hatch` block and a matching - `PORT_STATUS.md` row. diff --git a/.agents/skills/add-model/contracts/conversion_request.md b/.agents/skills/add-model/contracts/conversion_request.md deleted file mode 100644 index 18dd99d7f..000000000 --- a/.agents/skills/add-model/contracts/conversion_request.md +++ /dev/null @@ -1,54 +0,0 @@ -# Conversion Request Contract - -Consumed by `../add-model-07-conversion/SKILL.md` in Phase 5 and during Phase 6 -conversion retries. - -Initial conversion request: - -```text -model_family: -source_layout: diffusers | raw_official | monolithic | separate_components | mixed | custom -official_weights: -hf_revision: -converted_weights_dir: converted_weights/ -local_tests_readme: tests/local_tests//README.md -port_state_file: tests/local_tests//PORT_STATUS.md -components: - - name: - component_type: - official_definition_files: - official_instantiation_files: - official_weight_source: - official_keys: - fastvideo_keys: - fastvideo_class: - model_index_library: - config_filename: - production_loader_strictness: - source_prefix_or_path: - parity_test: - prototype_concerns_or_unknowns: -``` - -Retry request from a component skill: - -```text -conversion_retry_request: - component: - parity_test: - failing_keys: - expected_actual_shapes: - source_prefix_or_path: - evidence: - suspected_fix: -``` - -Rules: - -- Phase 5 conversion requires Phase 4 official/FastVideo key dumps. -- Component skills must use the retry request instead of editing conversion - scripts or converted weights directly. -- Conversion must update `port_state_file` with conversion status, retry history, - strict-load status, new issues, and resolved issues. -- Conversion must validate emitted config keys through the production config - update path and record the config filename expected by each loader. diff --git a/.agents/skills/add-model/contracts/escape_hatch.md b/.agents/skills/add-model/contracts/escape_hatch.md deleted file mode 100644 index 1fa3bd935..000000000 --- a/.agents/skills/add-model/contracts/escape_hatch.md +++ /dev/null @@ -1,51 +0,0 @@ -# Escape Hatch Contract - -Canonical pause-and-ask schema for `/add-model` skills. Use this when the next -action requires user input instead of autonomous debugging. - -```text -escape_hatch: - needs_user_input: yes | no - decision_type: scope | dependency | auth | cost | destructive | ambiguity | blocker - question: - recommended_option: - options: - -