ConditioningInjector (ABC) + PassthroughConditioning were defined and exported
but never instantiated or called — zero call sites anywhere. Unlike the policies
the §5 thesis actually names (CFG / flow-shift / precision / expert-routing),
which every recipe wires (cfg=ClassicCFG(), expert=NoRouting(), ...), conditioning
is injected INLINE by the loops: `st.cond["prompt_embeds"] = ctx.slots.get(...)`
(wan21/loop.py, ltx2/loop.py); the qwen_omni cascade conditions via loop wiring.
PassthroughConditioning even described a dataflow (state.scratch["cond"]) that is
not how conditioning actually flows (loops read ctx.slots). So it was a designed-
but-bypassed seam, not forward-design — and conditioning isn't in the README §5
policy list. Removed the ABC + impl + exports; kept the live `cond` field (the
loops fill it directly) and fixed its comment.
Tests: 143 passed, 2 skipped.
DataRef (dataset_id/revision/description — "what a recipe trained on") was only
the type of RecipeSpec.data_contract, which no recipe authored and no code read
(0/0). It is absent from the README RecipeSpec contract (§2.1: method, parents,
assumes_loop, assumes_precision, consistency_required) and not among the §18
"wire the inert metadata" roadmap items — i.e. unwired governance metadata, not
deliberate forward-design. RecipeSpec now matches §2.1 exactly.
Kept CheckpointManifest: unlike DataRef it IS wired (ModelCard.checkpoint), the
declarative "explicit components + key maps, no name-detector guessing" load
contract — declared intent, not dead.
Tests: 143 passed, 2 skipped.
state_schema / step_schema / result_schema (LoopSpec) and config_schema /
io_schema (ComponentSpec), plus valid_parallel_plans and parallel_constraints,
were authored by no recipe and read by no executor (audited: 0 reads / 0 sets
across native + tests, no dynamic dataclasses.fields/asdict/__dict__ access, no
consumer in scripts/_vendor/examples). They duplicated mechanisms that already
exist: the concrete LoopState/WorkPlan/StepResult classes the driver uses
directly, and the card-level ParallelismContract. Removing them slims the core
spec surface with zero behavior change.
Thesis untouched: cards still own components/loops/recipe/parity; kept the
parity, precision/placement, behavior-capture (behavior_schema), wired
extension_schema, and roadmap required_for/optional_for/resident_for fields.
Also drop the stale README step_cost_model mention (that field went with the
cost mechanism in 352c1b28).
Tests: 143 passed, 2 skipped (toy backend).
ToyLoRA / ToyControlNet / ToyTargetModel / ToyDraftModel / ToyRewardModel (+ the
_spec_target_next helper) in the toy backend, and build_karras_sigmas in the
sampler, were defined but referenced nowhere — no recipe, card, loop, test, or
example used them. They were toy stand-ins for capabilities (adapter plane,
speculative decode, served reward) and an EDM/Karras noise schedule that were
written ahead of being wired. Remove them (-134 LOC); a toy can be re-added when
the capability is actually wired. No thesis impact, no behavior change.
Suite green: 143 passed, 2 skipped (toy backend); import v2 stays torch-free.
Second step of the runtime simplification (after cost removal). Drop the
coordinated step-interleave scheduler + the interleave-parity gate; serving is
now pooled run-to-completion.
- Engine: remove run_interleaved + the WorkUnit/BatchScheduler imports; run /
run_serial drive each request to completion (tick/run_to_completion kept as the
per-request stepper).
- scheduler.py: remove BatchScheduler + WorkUnit + the batches metric; the
AdmissionController is now a pure refundable memory/OOM guard.
- AsyncEngine: bound concurrency with a serving pool (asyncio.Semaphore,
max_concurrent) — each request waits for a slot, then runs to completion.
- parity: remove assert_interleave_parity (the run_serial==run_interleaved gate);
rename interleave_gate.py -> compare.py (compare_outputs stays — bit-parity
between execution paths, e.g. disaggregated==inline).
- card specs: drop ParitySpec.interleave_required + LoopSpec.allows_interleaving;
stripped interleave_required from all cards.
- tests: delete the interleave-gate/parity tests; refocus the ones that exercised
real behavior (residual-skip, compare_outputs symmetric-empty).
- README: removed the design-doc references at the top; simplified the thesis /
scheduler (§6) / parity (§9) / package-layout / comparison sections to pooled
run-to-completion (no cost model, no interleave gate).
CPU mini: 143 passed / 2 skipped. Native omni port (P3) still to come. (pyproject
kernel hack excluded.)
Replace the re-export stubs with real vendored copies of the fastvideo modeling
+ layer + supporting infra, for the kept diffusion models (wan21, wan_causal,
ltx2, flux2, matrixgame2). v2 now imports ZERO `fastvideo.*` — it is
self-contained. (bagel/qwen_omni load from vllm_omni, an external pkg, not
fastvideo; cosmos3's load_id was already dangling — both out of scope here.)
Vendored (cp + `sed fastvideo. -> v2.`):
- models/ the 5 models' nn.Module dits/vaes/encoders/audio/upsamplers + the
component loader/ + the lazy class registry (other families' rows are
dormant/lazy — only the 5 resolve).
- layers/ attention/ platforms/ distributed/ configs/ logging_utils/ hooks/
third_party/pynvml + top-level forward_context/fastvideo_args/envs/logger/
utils/version — copied verbatim (layers et al. 'as is').
- api/ slimmed to schema + results (the VideoGenerator's config dataclasses);
the fastvideo parser/presets/overrides (which pull the pipeline runtime) are
intentionally NOT vendored — v2 has its own runtime/loop.
Decoupling surgery (cut the loader's coupling to the fastvideo runtime):
- configs/pipeline_registry.py (vendored from fastvideo/registry.py, renamed to
avoid colliding with v2/registry.py): dropped the _register_presets() auto-call
and matrixgame3 (removed model); config-class resolution preserved.
- configs/pipelines/__init__.py: dropped the registry back-edge (fixes an import
cycle) — base.py imports the registry lazily where used.
- torch_backend: load_component now uses v2.models.loader.
The only remaining external 'fastvideo*' refs are `fastvideo_kernel` (the
separate optional CUDA-kernel pkg for sparse/MoBA attention) — guarded; the dense
TORCH_SDPA path v2 uses never imports it.
Vendored subtrees added to the pre-commit exclude (faithful copies, mirroring the
existing fastvideo/models exclusion — not re-linted, to stay re-syncable).
Verified: grep finds zero fastvideo-package imports in v2/; Wan2.1 T2V on H100 is
BIT-IDENTICAL to the fastvideo-backed path (same .npy SHA256, byte-for-byte); CPU
mini 156 (154 passed + 2 env-skipped on x86, torch present). Backup: branch v2_backup.
Keep: bagel, cosmos3, flux2, matrixgame2, ltx2, qwen_omni, wan21, wan_causal
(plus the shared omni/ package that bagel/cosmos3/qwen_omni depend on). Remove
the other 26 recipe packages.
- Delete 26 recipe dirs (adapters, adaptive, cosmos2, cosmos25, fastwan, gen3c,
hunyuangamecraft, hunyuan_video(15), hyworld, image_video, kandinsky5,
lingbotworld, longcat, lucy_edit, matrixgame3, multi_expert, reward, sd35,
sfwan22, speculative, stable_audio, tiled, turbowan, unified, wan_fun_control).
- recipes/__init__.py: keep-closure builders + build_default_engine /
build_omni_engine (dropped the workflow/tiled/unified/image_video engine helpers).
- registry.py: _BUCKET_C pruned to flux2 + matrixgame2; removed the cosmos2
ModelEntry + the CosmosTransformer3DModel arch branch + the TurboWan-14B entry.
- Delete 11 tests for removed recipes/features; patch test_bucket_c_ports to drop
the cosmos2 reference (it now auto-derives from the pruned _BUCKET_C).
- README: correct the recipes/ roster to the kept families.
Backup of the full pre-prune tree is on branch v2_backup. CPU mini 156 passed / 0
failed (the prior 5 torch-absent bucket_c failures are gone with sd35/stable_audio);
no dangling references to any removed recipe; all kept model ids still resolve.
The torch adapter boundary marshalled the latent host<->device on EVERY denoise
step: _t uploaded the latent (and re-uploaded the text embeds) and _n downloaded
the velocity with a forced CUDA sync — 2*N PCIe copies + N syncs per generation,
buying nothing, since the latent could stay resident on the GPU the whole loop.
Root cause was a numpy loop surface. But the loop MATH is already array-agnostic
(CFG combine + flow-match Euler are pure arithmetic; the solver kernel already
passes torch through). So introduce a per-platform array namespace (v2/platform/
array_ns): numpy on CPU (torch-free — the parity mini is unchanged), torch-on-
device on cuda. The latent is seeded with numpy and uploaded ONCE; it then stays
resident through forward -> CFG combine -> solver -> next step; a single host
marshal happens at the request/output boundary (engine._to_artifact).
Opt-in per recipe via ModelCard.device_io (set on the Wan cards). When set on a
GPU box, build_component flips the components' TorchComponent.device_io so _out
keeps tensors on-device (in fp32, matching the old _to_numpy cast so the combine
dtype is unchanged). Un-migrated families and the CPU toy keep numpy in/out.
Also: PrecisionPolicy.cast is array-preserving; _t accepts resident tensors.
Verified BIT-IDENTICAL on real Wan2.1-1.3B / H100: the on-device latent equals
the pre-change numpy-path latent exactly (max_abs_diff 0.0, np.array_equal True).
CPU mini holds 237 passed / 5 pre-existing; pre-commit clean.
Other WanDenoiseLoop families can flip device_io next (per-family GPU re-verify);
non-Wan loops migrate to the xp namespace later.
platform/backends/ had become a flat dump of 15 per-model torch_<model>.py
adapters next to the genuinely-shared infra. Each adapter is referenced from
its card by a plain 'module:Class' string loaded via importlib, so there was
no real coupling forcing it into platform/ — the Cosmos/Flux/etc adapter
belongs WITH its recipe (card/loop/program).
Move each torch_<model>.py -> v2/recipes/<model>/adapter.py and flip the card
strings to v2.recipes.<model>.adapter:<Class>. backends/ now holds only the
shared substrate (torch_backend base, torch_cuda registration, torch_kernels,
toy/cpu/accel). Each recipe is now a self-contained package.
Cross-refs updated: gen3c/adapter imports CosmosT5Encoder from cosmos2/adapter;
sd35/program + stable_audio/card import from their own package. Recipes still
import torch-free (adapters pulled only via the string on a GPU box) — CPU mini
holds 237 passed / 5 pre-existing (bucket_c torch-absent). pre-commit clean.
Sweep all of v2/ through pre-commit (was previously only run on changed
files). Fixes surfaced across untouched modules:
- ruff: isinstance-tuple -> X | Y (UP038), try/except/pass ->
contextlib.suppress (SIM105), negated-return (SIM103), %-format ->
f-string (UP031).
- mypy: add annotations for no-untyped-call + var-annotated across recipes,
training methods, torch/toy backends, and serving.
- yapf reflow of the SF-Wan KV-cache call sites (semantics unchanged).
yapf/ruff/codespell/mypy all pass; v2 tests 237 passed / 5 pre-existing
(bucket_c torch-absent on CPU venv).
The causal adapter never passed a kv_cache, so CausalWanTransformer3DModel.forward routed to
_forward_train (no cross-chunk KV) on every chunk instead of _forward_inference (the CausVid
Alg-2 KV-cache path). Each chunk denoised blind to the previous ones; the loop's cross-chunk
"context" was a toy mean(prior_latents) the adapter ignored. Result: hard discontinuities at
every chunk boundary (frame-to-frame absdiff spikes 43-55 every ~12 frames).
Fix (cuda path only; toy/CPU path and the 237-test suite untouched):
- WanDiT.alloc_causal_caches(): allocate the persistent per-block KV + cross-attn caches sized
from the model config (mirrors CausalDenoisingStage._initialize_kv_cache).
- WanDiT.__call__: thread kv_cache/crossattn_cache/current_start/cache_start/start_frame/
frame_seqlen so the model runs _forward_inference.
- wan_causal/loop.py: own the caches in LoopState (per-request -> interleave-safe); pass
current_start = chunk_idx*chunk_size*frame_seqlen per chunk; do the clean-KV write
(timestep ~0) after each chunk so the next attends to it.
Verified on H100: frame-to-frame absdiff mean 12.8->4.4, max 55.4->8.3; chunk-boundary spikes
eliminated; coherent across all 7 chunks. CPU causal toy tests unchanged (24 passed).
Sweep all 296 v2 modules: simplify verbose docstrings/comments and remove 376 dangling
"(design_vN §X)" citations to the now-deleted design docs (v2/README.md is the source of
truth). Comment/docstring-only — AST-verified code-identical; the CPU suite holds at 237
passed / 5 pre-existing. Also applies yapf + ruff --fix auto-fixes (import ordering,
forward-ref annotation de-quoting under `from __future__ import annotations`; behavior-
neutral, suite-confirmed) and adds the legitimate domain terms mot/clen/te to the codespell
ignore-list. Remaining ruff (24) + mypy (68 no-untyped-call) findings are pre-existing v2
debt, untouched here.
Unify the four design docs (design.md, designv2.md, design_v3.md, designv4.md) into a single
authoritative v2/README.md: the (recipe, runtime) thesis, driven loops, planes, one-WorkUnit
scheduler, the parity ladder + interleave gate, training-on-shared-loops, the weight-sharing
topology catalog, the current GPU status (20+ models + the BAGEL/Qwen-Omni/Cosmos3 trio
verified), and a prioritized roadmap. Recast design_summary.md as a one-page design philosophy
pointing to it. Add .agents/exploration/mstar-v2-roadmap.md (the adversarially-verified M*
Walk-Graph gap analysis driving the roadmap). Delete the four superseded design docs.
MatrixGame2 (causal DMD loop): bump st.step_idx on every executed work unit (each
DMD step and each clean-context pass). The loop drives its own control flow off
block_idx/dmd_idx/phase, but the runtime's no-progress watchdog keys on
st.step_idx, so a multi-block causal rollout was seen as stalled. Mirrors what
every other recipe loop does.
MatrixGame3 (5B WanModel): patch-align the latent H/W (patch_size (1,2,2)) before
denoise. The model folds (H/2, W/2) tokens, so an odd latent dim made the
unpatchified velocity come back one row/col short of the noise latent. Crop to
(latent // patch) * patch, faithful to MatrixGame3DenoisingStage.
v2 mini: 240 passed.
Two distilled few-step video models rendered incorrectly on GPU; root-caused via
dense frame sampling (contact sheets) and fixed in the recipe cards.
SF-Wan2.1 (self-forcing causal, wan_causal card): was oversaturated/overcooked.
The distilled student is CFG-FREE (guidance 1.0, single forward/step) and denoises
with the 4-step DMD schedule [1000,750,500,250] (warped by FlowShiftPolicy(5.0)),
at a native causal block of 3 latent frames. Defaults were ClassicCFG@6.0 + 2 steps
+ block 2 -> overcooked AND under-denoised. Fixed: num_chunks=7, chunk_size=3,
steps_per_chunk=4; SamplingDefaults num_steps=4, guidance_scale=1.0 (7x3=21 latent
-> 81 frames). Renders a clean raccoon-in-sunflowers across all 81 frames.
LTX2-Distilled 2-stage SR (ltx2 card + LTX2VAE): was temporally blocky. Root cause
was an OOM-forced 57-frame reduction (only 8 latent temporal frames); the model is
designed for 121 (16 latent frames). Enable VAE tiling in LTX2VAE so the 121-frame
full-res decode fits the 80 GiB GPU; keep base cfg_scale=3.0 (drives brightness;
cfg=1 washed out) with stg_scale=0.0 (v2's drop-text perturbation is not real
skip-layer STG). Renders the on-prompt backyard shot, bright + temporally coherent.
torch_backend.py: enable LTX2VAE tiling; clear pre-existing mypy no-untyped-call /
yapf debt on the file (surfaced once per-file linting bypassed the duplicate-module
flakiness) by annotating the helper/constructor/maker signatures.
Tests: update the 3 affected CPU defaults/chunk-count tests. v2 mini: 240 passed.
V2_PORTING_STATUS.md now records the GPU bring-up outcome: 20 models generate
real video/audio on H100 (the 7 prior + 13 newly-ported), with the remaining
split into fastvideo/env-blocked (SLA/VSA kernels, transformers incompat, fastvideo
registry/flash_attn gaps) and HF-access-blocked (gated cosmos2/flux2/sd35) — none
a v2 recipe bug.
Second GPU pass (distilled + huge dense, 2-wide). 5 verified end-to-end with real
weights; CPU toy path kept green (240 passed, 2 skipped).
VERIFIED:
* matrixgame3 — mp4 (9,256,256,3), zero fixes. 6.47B, standard Wan attn (NOT
sparse-attn-blocked, like its mg2 sibling); degenerate single-clip.
* fastwan — FastWan2.2-TI2V-5B-FullAttn DMD 3-step, mp4 (17,256,256,3), zero
fixes. The FULL-ATTENTION variant has no VSA params -> the generic
Wan loader maps it cleanly (reuses WanDiT via load_id, no adapter).
* longcat — LongCat-Video-T2V 13.58B, mp4 (17,256,256,3), zero fixes, CPU
offload (~40GB peak).
* sfwan22 — Self-Forcing Wan2.2-A14B causal+MoE (2x14B), mp4 (29,288,288,3),
CPU expert offload (~80GB peak).
* lingbotworld — Wan2.2-class 2x14B camera world model, mp4 (9,256,256,3), offload
(~98GB peak transient).
BLOCKED: turbowan-i2v-a14b — SLA sparse-attn params (attn1.attn_impl.proj_l on all
40 layers of both experts) cannot load into the dense Wan build + needs the
fastvideo-kernel SLA Triton kernels (no nvcc here). Confirmed via the safetensors
header (no 60GB download). Same SLA family as turbowan-1.3b.
Fixes (own-port only): sfwan22/loop.py, lingbotworld/{card,program}.py +
torch_lingbotworld.py. pre-commit clean per-package.
VideoGenerator._result hardcoded out.artifacts['video'].frames, so an audio-only
(Stable Audio) or image-only (SD3.5 / FLUX.2 T2I) generation crashed with
KeyError 'video' even though the engine had correctly produced the AudioArtifact /
image TensorArtifact (surfaced during stable_audio GPU bring-up). _result now
guards on the artifact present: video -> mp4 (unchanged), else image -> png
([C,H,W]/[B,..] normalized to [H,W,C]), plus the existing audio -> sibling .wav;
image_path recorded in result.extra. The video path is byte-for-byte unchanged.
v2 mini 240 passed, 2 skipped; pre-commit clean.
Ran each dense public port through the real VideoGenerator on H100 (2-wide across
both GPUs). 8 produce real finite output end-to-end; per-model adapter/loop fixes
landed in each port's OWN files (no shared/fastvideo edits). CPU toy path kept
green (240 passed, 2 skipped) — GPU-only conditioning gated to the cuda backend.
VERIFIED (real GPU output):
* stable_audio — stereo audio (2, 441000) @44.1kHz. Fixes: dedicated
'conditioner' component kind (SA owns its T5, empty
text_encoder_configs) + ConditionerLoader from conditioner/
+ VDenoiser c_noise = atan(sigma)/(pi/2).
* matrixgame2 — mp4 (9,256,256,3). Loads CLEAN (not sparse-attn-blocked).
Fixes: 20ch cond_concat (4ch mask + 16ch img), mandatory i2v
(synth blank first frame), pre-sized kv_cache/crossattn_cache
(SDPA inference path, avoids flex_attention compile), bf16
autocast, per-request reset_caches.
* gen3c — video (9,256,256,3), zero fixes (worked first try).
* wan_fun_control — video (9,256,256,3).
* lucy_edit — video (17,256,256,3).
* hunyuangamecraft— video (9,256,256,3).
* hunyuan_video — video (3,9,256,256), dual LLaMA+CLIP + Hunyuan VAE.
* hunyuan_video15 — video (9,256,256,3), dual Qwen+ByT5 (gated to cuda; CPU passes
single embed).
BLOCKED (not v2 recipe bugs — load/run reached, then a fastvideo/env wall):
* cosmos25 — DiT + VAE ran finite on GPU; the Qwen2.5-VL Reason1 encoder hits a
transformers 5.12.1 incompat in fastvideo shared code
(Qwen2_5_VLConfig.pad_token_id).
* kandinsky5 — fastvideo registry.py registers it with a bare PipelineConfig (no
Kandinsky5 config) -> load fails fastvideo-side. (latent z=16 +
visual_cond adapter corrected; toy decoupled to its own channels.)
* hyworld — fastvideo's hyworld DiT hardcodes flash_attn (no SDPA fallback);
flash_attn kernel not built here.
CPU-safety fixes (mine): kandinsky5 toy ToyDiT/ToyVAE use the toy LATENT_CHANNELS
(not the real z=16); hunyuan_video15 dual-encoder packing gated to cuda.
Sparse-attn distilled (turbowan/SLA, fastwan/VSA) + gated (cosmos2/flux2/sd35) +
huge (>80GB) handled separately. pre-commit clean per-package.
Rewrites V2_PORTING_STATUS.md to reflect completion: the scope is now ALL
fastvideo models (not Wan+LTX-2 only). Documents the self-contained recipe-package
porting mechanism (ComponentSpec.adapter), the 15 net-new architectures + 5
Wan-family variants newly ported (CPU-verified end-to-end; GPU=BRINGUP), the 7
GPU-verified models, and the single env-blocked id (VSA-14B, needs nvcc).
Adds explicit ModelEntry aliases for the LTX-2 (FastVideo/LTX2-Diffusers,
LTX2-base, Lightricks/LTX-2 -> single-stage base) and LTX-2.3 (LTX2.3-Diffusers,
LTX2.3-Distilled-Diffusers, LTX2.3-base, Lightricks/LTX-2.3, lightricks/ltx-2.3
-> the distilled joint-A/V card) naming variants of already-ported LTX
checkpoints (the arch fallback also resolves LTX2Transformer3DModel from a root).
v2 now resolves 63/64 fastvideo registry ids by exact id; the only remaining id,
FastVideo/Wan2.1-VSA-T2V-14B-720P-Diffusers, is ENV-BLOCKED (VSA Sparse-Linear
Attention kernels require nvcc, not built in this bring-up; it arch-resolves to
the base Wan card but needs the VSA kernel build to run faithfully).
v2 mini 240 passed, 2 skipped; pre-commit clean.
Closes the bucket-B sampler/conditioning gap — each reuses the Wan/Causal ARCH
(no new torch adapter) with a new in-package loop/sampler/conditioning, declared
in _BUCKET_C as explicit-HF-id-only (transformer_cls="" so the generic Wan/Causal
arch fallback is NOT hijacked — only the exact id distinguishes the capability
variant from a base Wan of the same class).
* turbowan — TurboWan rCM (Reparameterized Consistency Model) few-step: a
faithful in-package RCMScheduler port (TrigFlow->RectifiedFlow
schedule + stochastic consistency SDE step), 1.3B/14B T2V +
TurboWan2.2-I2V-A14B (MoE i2v, boundary 0.9 in raw-sigma space)
* lucy_edit — Lucy-Edit v2v editor: a video_vae_encode node (the input video
-> 48ch cond latent) threaded via the shared i2v_cond hook ->
96ch Lucy DiT input (faithful to denoising.py is_lucy_edit)
* wan_fun_control — Wan2.1-Fun-Control: control-video conditioning (reuses the
i2v [mask|cond] concat pattern)
* sfwan22 — Self-Forcing Wan2.2-A14B: causal chunk_rollout + Wan2.2 MoE
boundary routing, i2v (boundary 0.9) + t2v (boundary 0.875)
* fastwan — FastWan DMD 3-step: TI2V-5B-FullAttn loadable; the VSA-trained
variants + non-strict to_gate_compress load are BRINGUP
All 12 residual ids resolve+build; base Wan/Causal resolution unchanged (arch
fallback not hijacked). The _BUCKET_C regression test auto-extended -> v2 mini
240 passed, 2 skipped; pre-commit clean (per-package + registry).
Data-driven from registry._BUCKET_C (+ cosmos2): each net-new ported arch
resolves through the registry (exact id + arch fallback) AND runs end-to-end on
the CPU toy backend via the public Engine path (resolve -> build card+program ->
load_card -> Engine.run), emitting exactly one modality-correct artifact
(video / image / audio) + latents. Auto-covers future _BUCKET_C rows.
21 tests pass; full v2 mini 232 passed, 2 skipped.
First net-new architecture ported via the self-contained recipe-package pattern
(card-declared adapter + new loop, no shared-dispatch edit):
* CosmosDenoiseLoop (v2/recipes/cosmos2/loop.py): EDM preconditioning folded into
a flow-match Euler integrator. Faithful port of CosmosDenoisingStage — Karras
sigma schedule (rho=7, sigma_max=80 -> sigma_min=0.002, terminal clamp), latent
init randn*sigma_max, per-step c_in/c_skip/c_out (sigma_data=1) -> x0, CFG in x0
space, x0 -> velocity (x-x0)/sigma, FLOW_MATCH_STEP. video2world frame-replace
conditioning threaded but inert for the t2v preset.
* build_karras_sigmas helper added to v2/loop/sampler.py.
* CosmosDiT + CosmosT5Encoder adapters (v2/platform/backends/torch_cosmos.py),
declared on the card via ComponentSpec.adapter (the ac29750b enabler) — DiT
returns raw EDM output + builds the mandatory zero condition/padding masks + fps;
T5 uses the raw last_hidden_state (no Wan zero-pad). Reuses the WanVAE adapter.
* card/program/registry (HF id nvidia/Cosmos-Predict2-2B-Video2World + arch
fallback on CosmosTransformer3DModel) + COSMOS_NEG prompt + SamplingDefaults
(35 steps, gs 7, 704x1280, 93f, 16fps).
CPU-verified: Karras schedule, card/program build, registry resolve (id + arch),
EDM loop runs end-to-end on the CPU toy backend. GPU load/run is BRINGUP.
v2 mini green (211 passed, 2 skipped); pre-commit clean.
A new architecture can declare its own torch adapter on the card
(ComponentSpec.adapter="module:Class") instead of editing the shared _make_dit/
_make_vae/_make_text_encoder dispatch. _explicit_adapter() constructs it as
cls(module, *extra, device=, dtype=) and short-circuits the built-in Wan/LTX2
class-name dispatch when set. This makes each bucket-C port a self-contained
recipe package (card + adapter module + loop + program) with no shared-file edit
-> conflict-free parallel porting. Unset -> unchanged built-in dispatch.
CPU mini green (211 passed, 2 skipped).
Reuses everything: 2 WanTransformer3DModel experts + BoundaryTimestepRouting (from the A14B MoE pattern),
the CLIP image encoder + first-frame [mask|cond] conditioning + the i2v program (from the Fun-InP i2v
port), and the shared WanDenoiseLoop (i2v hooks + the boundary expert). No new adapter. CPU-verified: the
toy MoE i2v runs end-to-end (2 experts + boundary + conditioning -> finite video); resolves with i2v caps.
Structural (GPU-pending: 2x14B, like the A14B T2V). Wan family now largely covered (T2V 1.3B/14B/TI2V-5B/
A14B, causal SF, i2v 1.3B/14B/A14B). CPU mini 211/2.
Real image-to-video, unlocking the i2v cluster. v2/recipes/wan21/i2v.py: CLIP image-encode -> the DiT's
encoder_hidden_states_image; first-frame VAE conditioning + a 4-channel mask -> the 20ch [mask|cond] that
the Wan adapter concatenates with the 16ch noise -> the 36ch i2v DiT input (mirrors fastvideo's
ImageEncodingStage + ImageVAEEncodingStage; v2's WanVAE.encode already applies the matching (z-mean)/std).
Reuses the shared WanDenoiseLoop (its None-default i2v hooks) + the Wan torch adapter unchanged. Adds
ToyImageEncoder + the image_encoder checkpoint subfolder stamp. Registered Wan2.1-Fun-1.3B-InP.
GPU-verified: loads via the generic Wan loader (1.56B, no param-mapping issue), runs the full i2v
conditioning, produces real video (3,9,256,384, std 0.44, finite, motion 0.041). CPU mini 211/2 (T2V
unregressed). BRINGUP: visual confirmation that the output follows the conditioning image is
human-in-the-loop.
Threads i2v conditioning through the SHARED WanDenoiseLoop with zero T2V risk: init() reads optional
slots i2v_cond (the [mask|cond] latent) + i2v_img_embeds (CLIP) into scratch; _velocity passes them to
the dit (context=, cond=); WanDiT concats cond (16->36ch) and uses the embeds as
encoder_hidden_states_image; capture is disabled only when i2v conditioning is present. For T2V both are
None -> the dit call, CFG, and cudagraph capture are byte-identical (CPU mini 211 pass, 2 skip — no
regression). ToyDiT accepts+ignores cond (image-conditioning is a GPU-path concern). Completes the i2v
backend seam; the program's mask+cond construction + the Wan i2v card/registry + GPU verify follow.
Adds CLIPImageEncoder (encode_image -> the DiT's encoder_hidden_states_image) + the generic builder's
image_encoder maker (ImageEncoderLoader + ImageProcessorLoader), registered as the cuda 'image_encoder'
kind. Mirrors fastvideo's ImageEncodingStage. CPU-verified (component-kinds + lazy invariant, 211/2);
GPU path marked BRINGUP/written-not-run (processor subfolder + dtype to confirm on a real i2v checkpoint),
matching how the rest of the torch backend was originally landed. Reusable by the Wan i2v cluster + many
bucket-C models (Hunyuan/Cosmos i2v). Next i2v increments: the mask+cond latent construction + the
concat-into-DiT-input loop, then the card + registry + GPU-verify with Wan2.1-Fun-1.3B-InP.
GPU verification (Wan2.1-Fun... no: FastWan2.1-T2V-1.3B) failed at load: 'Parameter blocks.0.to_gate_compress.bias
not found in custom model state dict' — the FastVideo/* DMD-distilled checkpoints carry gated-attention
params (to_gate_compress) that the generic WanTransformer3DModel loader can't map. v2/registry.py's
select_by_architecture ALREADY rejects WanDMDPipeline for exactly this reason; my explicit ModelEntry
wrongly bypassed it. Reverted FastWan (1.3B + 14B-480P) and the unverified VSA-14B alias (same FastVideo/*
risk). Kept the official Wan2.1-T2V-14B (standard weights, same loader path as the GPU-verified 1.3B).
Lesson recorded in V2_PORTING_STATUS.md: FastWan/Turbo/VSA need a param-mapping fix (like LTX-2.3 did),
not just a schedule — bucket-C-effort. GPU-verify every port before claiming support. CPU mini 211/2.
Same WanTransformer3DModel arch (VSA is an attention-backend choice, not a weight/arch difference); v2
runs dense TORCH_SDPA, so it resolves to build_wan_t2v_14b_card. Registry resolution verified; the GPU
forward path is the 1.3B-proven Wan adapter (specific 14B/VSA weights not separately GPU-run).
First bucket-B port toward 'support every fastvideo model': Wan2.1-T2V-14B reuses the Wan recipe +
torch adapter unchanged (same WanTransformer3DModel/AutoencoderKLWan/UMT5) — only a registry entry +
build_wan_t2v_14b_card (720p, flow_shift 5.0) + SamplingDefaults differ. Without the entry the arch
fallback would give it the 1.3B 480p defaults; the explicit entry gives 50 steps / 720x1280.
Also recorded the full backlog in V2_PORTING_STATUS.md: 63 fastvideo models = 8 ported / 21 bucket-B
(reuse Wan/Causal/LTX2 arch — registry+recipe+defaults, no new adapter) / 34 bucket-C (13 new
architectures needing a TorchComponent adapter). Updated the stale 'how to add a model' steps to the
post-redesign structure (v2/recipes/, torch_backend.py, SamplingDefaults). CPU mini 211 pass, 2 skip.
Addresses the adapter-setup pains: collapses the torch_cuda(trampolines)/torch_adapters/torch_ltx2 split
+ 11 near-identical adapter classes + 6 build_torch_* builders into:
- v2/platform/backends/torch_backend.py: a TorchComponent base centralizing .to/.eval, the numpy<->torch
marshalling (ONE place), the set_forward_context wrap, and the weight surface; thin per-model subclasses
(WanDiT/LTX2DiT/WanVAE/LTX2VAE/T5Encoder/Gemma/LTX2Upsampler/LTX2AudioVAE/LTX2Vocoder) carrying only
forward semantics; and ONE build_component(spec) dispatching by spec.kind via _MAKERS.
- torch_cuda.py: registers that single generic builder for all 6 cuda kinds (no per-kind trampolines).
- v2 owns its namespace via re-export STUBS (facade, marked '# STUB'): v2/forward_context, v2/fastvideo_args,
v2/distributed, v2/loader (the load_component seam), v2/api, v2/models/{dits,audio,upsamplers}/*. All v2
code imports v2.*; 'from fastvideo' now lives ONLY in those 8 stub files -> a future per-module vendored
cutover swaps a stub body, no caller changes. No divergence (stubs run fastvideo's live code).
- Deleted torch_adapters.py + torch_ltx2.py.
Verified: CPU mini 210 pass/2 skip; lazy invariant (platform load imports no torch); GPU bit-parity LTX-2.3
T2VS (audio std 0.04304, identical to pre-redesign) + Wan2.1 (video std 31.94, motion 5.997).
Frees v2/models/ to become the vendored-architecture namespace that mirrors fastvideo/models
(part of making v2 self-contained / able to replace fastvideo). The v2 recipe layer (per-family
card.py/loop.py/program.py + common.py + the build_*_engine re-exports) is the recipe, not the
architectures, so it moves to v2/recipes/. The pure-numpy toy/parity implementations (ToyDiT etc.)
move from v2/models/backend.py to v2/platform/backends/toy.py (alongside cpu.py/accel.py/torch_*).
Mechanical: all imports are absolute, so v2.models.<x> -> v2.recipes.<x> and v2.models.backend ->
v2.platform.backends.toy across v2/ + examples/ (89 files, 178 refs). No behavior change.
CPU mini green (202 passed, 2 skipped); all v2 files compile.
Mechanical conversion of every relative import under v2/ to an absolute v2.* path
(from .x / ..x / ...x -> from v2.<pkg>.x) so imports are unambiguous, grep-able, and
stable when code is copied/moved between entrypoints (VideoGenerator, CLI, server).
Surgical prefix-only rewrite: only the 'from <dots><module>' prefix changed — import
names, parentheses, multi-line formatting, comments, and ordering are byte-for-byte
preserved (no collapsing, no reorder, no unrelated reformatting).
- 471 imports across 125 files; v2/tests/ was already absolute (untouched).
- Validated: all 125 files compile, every 'from v2.* import' target resolves to a real
module/package, zero relative imports remain (full sweep), CPU mini suite green
(202 passed, 2 skipped).
Makes the 'Full A/V' LTX-2.3 deliverable reachable from the user-facing entrypoint, not just the
engine. A model advertising TEXT_TO_VIDEO_SOUND (LTX-2.3) now auto-issues a T2VS request, so generate()
/ generate_video() return BOTH modalities in one joint pass:
- VideoGenerator stores the resident instance + a supports_av flag (from card.capabilities); generate()
gains want_audio (None=auto-by-capability, True/False to force) and routes T2V vs T2VS+{video,audio}.
- _result saves the stereo waveform as a sibling .wav at the vocoder's REAL rate (24000) — read off the
built audio_vae adapter (TorchLTX2AudioVAE.sample_rate = Vocoder.output_sample_rate), since the
AudioArtifact default rate is a placeholder. Populates GenerationResult.audio/.audio_sample_rate and
extra['audio_path']. scipy IEEE-float WAV; [channels,samples] auto-transposed.
- Rewrote v2_basic_ltx2_3_distilled.py: registry routes to build_ltx2_3_card (its own joint T2VS A/V
card, not the LTX-2 base/2-stage card); the example prints both the mp4 and the wav.
- GPU-verified via the convenience API: ev.mp4 + ev.wav (24000 Hz, stereo 61920x2, nonzero, std 0.043).
CPU mini green (202 passed, 2 skipped); engine/program/toy paths untouched (test_ltx2_av pins 44100).
Peiyuan Zhang