[feat] v2: port Wan2.1-T2V-14B (bucket B) + document the all-models backlog
First bucket-B port toward 'support every fastvideo model': Wan2.1-T2V-14B reuses the Wan recipe + torch adapter unchanged (same WanTransformer3DModel/AutoencoderKLWan/UMT5) — only a registry entry + build_wan_t2v_14b_card (720p, flow_shift 5.0) + SamplingDefaults differ. Without the entry the arch fallback would give it the 1.3B 480p defaults; the explicit entry gives 50 steps / 720x1280. Also recorded the full backlog in V2_PORTING_STATUS.md: 63 fastvideo models = 8 ported / 21 bucket-B (reuse Wan/Causal/LTX2 arch — registry+recipe+defaults, no new adapter) / 34 bucket-C (13 new architectures needing a TorchComponent adapter). Updated the stale 'how to add a model' steps to the post-redesign structure (v2/recipes/, torch_backend.py, SamplingDefaults). CPU mini 211 pass, 2 skip.
This commit is contained in:
@@ -54,8 +54,16 @@ Wan2.1 model — covered by wan21; those runtime modes are out of scope.
|
||||
| `basic_lucy_edit.py` | `decart-ai/Lucy-Edit-Dev` (Wan v2v) | **V2V mechanism (identified):** `WanVideoToVideoPipeline` + `VideoVAEEncodingStage` — VAE-encode the input video to a conditioning latent. v2 needs input-video loading + a VAE-encode-video node + the v2v conditioning (no image_encoder). |
|
||||
| `basic_ltx2_3_distilled_i2v*.py` | LTX-2.3-Distilled I2V | Image conditioning on the ltx2 single-stage program (encode the input image + condition the denoise). |
|
||||
|
||||
## Out of current scope — new families (kept as an upstream reference map only)
|
||||
Each would need a new card + adapters (DiT/VAE/encoder forwards) + maybe a sampler:
|
||||
## Backlog — goal: port ALL fastvideo models (63 total: 8 ported · 21 reuse-arch · 34 new-arch)
|
||||
Ported (8): the 7 above + **Wan2.1-T2V-14B** (bucket B — reuses the Wan recipe/adapter; 720p defaults).
|
||||
|
||||
**Bucket B (21 left) — reuse a v2 architecture** (`WanTransformer3DModel` / `CausalWanTransformer3DModel` /
|
||||
`LTX2Transformer3DModel`): a `ModelEntry` in `v2/registry.py` + a recipe (parameterize the wan21/wan_causal/
|
||||
ltx2 builders) + `SamplingDefaults`, NO new adapter. Includes other Wan T2V sizes; i2v variants (need an
|
||||
image-encoder component + i2v program); FastWan/Turbo (DMD/RCM schedules → a loop variant); Wan2.2-I2V-A14B;
|
||||
SF-Wan2.2; LTX-2 base/2.3 repo aliases (already arch-fallback-resolvable, so they work without an entry).
|
||||
|
||||
**Bucket C (34 models, ~13 new architectures) — need a new `TorchComponent` adapter + recipe per arch:**
|
||||
`basic_cosmos2_5_*` (Cosmos-Predict2.5), `basic_gen3c.py` (GEN3C — **local `converted_weights/` only**),
|
||||
`basic_hy15*.py` / `basic_gamecraft.py` / `basic_hyworld.py` (Hunyuan family),
|
||||
`basic_longcat_*.py` (LongCat + LoRA), `basic_lingbotworld_base_cam.py`,
|
||||
@@ -66,11 +74,13 @@ Each would need a new card + adapters (DiT/VAE/encoder forwards) + maybe a sampl
|
||||
(interactive mouse/keyboard-conditioned world models).
|
||||
|
||||
## How to add a model to the v2 VideoGenerator
|
||||
1. Ensure a v2 card exists (`v2/models/<family>/`) whose `load_id`s resolve to real `fastvideo.models.*`.
|
||||
2. Register it in the shared `v2/registry.py` (used by every entrypoint — VideoGenerator, CLI, server):
|
||||
add a `ModelEntry` mapping its HF id(s) → builders (the explicit PRIMARY path — the only way to split
|
||||
same-architecture capability variants, e.g. T2V vs the i2v "InP"), and/or a branch in
|
||||
`select_by_architecture` for the arch-inference FALLBACK (local paths / unregistered repos).
|
||||
3. If a DiT/VAE/encoder/upsampler forward differs, add an adapter + class-detection in `torch_adapters.py`
|
||||
and register the component kind in `torch_cuda.py`.
|
||||
1. Add a v2 recipe under `v2/recipes/<family>/` (card/loop/program) whose `load_id`s name the real models;
|
||||
set per-model `SamplingDefaults` on the card (`v2/card/specs.py`) — see `wan21`/`ltx2` for the pattern.
|
||||
2. Register it in the shared `v2/registry.py` (every entrypoint resolves through it): add a `ModelEntry`
|
||||
mapping its HF id(s) → builders (the PRIMARY path — the only way to split same-arch capability variants,
|
||||
e.g. T2V vs the i2v "InP"), and/or a branch in `select_by_architecture` for the arch-inference FALLBACK.
|
||||
3. **Bucket B (reuse an existing arch):** nothing more — the redesigned `torch_backend.py` already builds
|
||||
Wan/LTX components. **Bucket C (new arch):** add a `TorchComponent` subclass + a `_MAKERS`/class-detect
|
||||
branch in `torch_backend.py`, vendor the new model code as marked re-export stubs under `v2/models/...`
|
||||
(see `v2-vendoring-approach`), and register any new component kind in `torch_cuda.py`.
|
||||
4. Add `examples/inference/basic/v2_<name>.py` mirroring the official example; generate to confirm.
|
||||
|
||||
@@ -5,10 +5,11 @@ from v2.recipes.wan21.card import (
|
||||
build_wan21_card,
|
||||
build_wan22_a14b_card,
|
||||
build_wan22_ti2v_card,
|
||||
build_wan_t2v_14b_card,
|
||||
stamp_wan21_checkpoints,
|
||||
)
|
||||
from v2.recipes.wan21.loop import WanDenoiseLoop, latent_shape
|
||||
from v2.recipes.wan21.program import build_wan_t2v_program
|
||||
|
||||
__all__ = ["build_wan21_card", "build_wan22_ti2v_card", "build_wan22_a14b_card",
|
||||
__all__ = ["build_wan21_card", "build_wan22_ti2v_card", "build_wan22_a14b_card", "build_wan_t2v_14b_card",
|
||||
"stamp_wan21_checkpoints", "build_wan_t2v_program", "WanDenoiseLoop", "latent_shape"]
|
||||
|
||||
@@ -118,6 +118,16 @@ def build_wan22_ti2v_card(model_id: str = "wan2.2-ti2v-5b", *,
|
||||
num_frames=121, fps=24, negative_prompt=WAN_NEG_CN))
|
||||
|
||||
|
||||
def build_wan_t2v_14b_card(model_id: str = "wan2.1-t2v-14b", *, checkpoint_root: str | None = None) -> ModelCard:
|
||||
"""Wan2.1-T2V-14B (720p). Same WanTransformer3DModel / AutoencoderKLWan / UMT5 + 16/8/4 VAE geometry
|
||||
as the 1.3B — reuses the Wan recipe + torch adapter unchanged; only the size (resolved from the
|
||||
checkpoint by the loader), the 720p flow shift 5.0, and the per-model defaults differ."""
|
||||
return build_wan21_card(model_id=model_id, flow_shift=5.0, checkpoint_root=checkpoint_root,
|
||||
sampling_defaults=SamplingDefaults(
|
||||
num_steps=50, guidance_scale=5.0, height=720, width=1280,
|
||||
num_frames=81, fps=16, negative_prompt=WAN_NEG_EN))
|
||||
|
||||
|
||||
def build_wan22_a14b_card(model_id: str = "wan2.2-t2v-a14b", *, checkpoint_root: str | None = None,
|
||||
boundary: float = 0.875) -> ModelCard:
|
||||
"""Wan2.2-T2V-A14B card — MoE: two WanTransformer3DModel experts (in_ch=16, Wan2.1 16/8/4 geometry)
|
||||
|
||||
@@ -48,10 +48,12 @@ def _entries() -> list[ModelEntry]:
|
||||
build_wan21_card,
|
||||
build_wan22_a14b_card,
|
||||
build_wan22_ti2v_card,
|
||||
build_wan_t2v_14b_card,
|
||||
build_wan_t2v_program,
|
||||
)
|
||||
return [
|
||||
ModelEntry(("Wan-AI/Wan2.1-T2V-1.3B-Diffusers",), build_wan21_card, build_wan_t2v_program),
|
||||
ModelEntry(("Wan-AI/Wan2.1-T2V-14B-Diffusers",), build_wan_t2v_14b_card, build_wan_t2v_program),
|
||||
ModelEntry(("Wan-AI/Wan2.2-TI2V-5B-Diffusers",), build_wan22_ti2v_card, build_wan_t2v_program),
|
||||
ModelEntry(("Wan-AI/Wan2.2-T2V-A14B-Diffusers",), build_wan22_a14b_card, build_wan_t2v_program),
|
||||
ModelEntry(("wlsaidhi/SFWan2.1-T2V-1.3B-Diffusers",), build_wan_causal_card, build_wan_causal_program),
|
||||
|
||||
@@ -3,11 +3,26 @@ from types import SimpleNamespace
|
||||
|
||||
from v2.card import SamplingDefaults
|
||||
from v2.recipes.ltx2 import build_ltx2_3_card, build_ltx2_base_card, build_ltx2_card
|
||||
from v2.recipes.wan21 import build_wan21_card, build_wan22_a14b_card, build_wan22_ti2v_card
|
||||
from v2.recipes.wan21 import (
|
||||
build_wan21_card,
|
||||
build_wan22_a14b_card,
|
||||
build_wan22_ti2v_card,
|
||||
build_wan_t2v_14b_card,
|
||||
)
|
||||
from v2.recipes.wan_causal import build_wan_causal_card
|
||||
from v2.registry import resolve
|
||||
from v2.video_generator import _resolve_default
|
||||
|
||||
|
||||
def test_wan_t2v_14b_defaults_and_registry():
|
||||
# Bucket-B port: reuses the Wan recipe/adapter, only defaults differ (720p, flow_shift 5.0).
|
||||
sd = build_wan_t2v_14b_card().sampling_defaults
|
||||
assert (sd.num_steps, sd.guidance_scale, sd.height, sd.width, sd.fps) == (50, 5.0, 720, 1280, 16)
|
||||
# the HF id resolves to the 14B card via the shared registry (not the 1.3B fallback defaults)
|
||||
build_card, _ = resolve("Wan-AI/Wan2.1-T2V-14B-Diffusers")
|
||||
assert build_card().sampling_defaults.height == 720
|
||||
|
||||
|
||||
def test_wan21_defaults():
|
||||
sd = build_wan21_card().sampling_defaults
|
||||
assert (sd.num_steps, sd.guidance_scale, sd.height, sd.width, sd.num_frames, sd.fps) == (50, 3.0, 480, 832, 81, 16)
|
||||
|
||||
Reference in New Issue
Block a user