hao-ai-lab-FastVideo/v2

FastVideo v2 — A Model-Native Runtime for the (Recipe, Runtime) Era

Status: source of truth. This README is the single design document for v2 — it supersedes and absorbs the four prior design docs (design.md v19 strategic proposal, designv2.md model-plane thesis, design_v3.md unconstrained north star, designv4.md as-built + joint-RL validation), which have been removed. It carries their load-bearing ideas, reflects what is actually built and running in v2/, and states the forward roadmap (including the M* paper comparison — see §18 and .agents/exploration/mstar-v2-roadmap.md).

What v2 is. A model-native serving and training substrate for composite multimodal models — video, image, audio, omni/MoT, world models, VLAs — where the atomic unit is a typed (recipe, runtime) pair owned by a ModelCard, every iterative computation is a driven loop, and one scheduler runs the steps of all loops in one currency. The control flow, contracts, scheduling, caching, parity gates, and training math are real and CPU-testable (216 tests, 34 files, two runners); the heavy neural forwards run either on a numpy toy backend (laptop, no GPU) or the real torch backend (platform/backends/torch_backend.py), selected by the platform/ dispatch substrate with no change to loops/scheduler/caches/parity/training. As of this writing 20+ real model families are GPU-verified on H100, and the omni trio (BAGEL, Qwen2.5-Omni, Cosmos3) runs real via vllm-omni (§17).

---

Table of contents

1. The thesis · 2. Three signature ideas ·
2. Planes & dependency order · 4. Model Plane ·
3. The driven-loop contract · 6. Runtime & scheduler ·
4. Memory, cache, transport, compile · 8. Parallelism ·
5. Correctness — parity as a typed gate · 10. Training & RL ·
6. Weight-sharing topologies & stress tests ·
7. Request/session/artifact + programs/workflows ·
8. Serving & fleet · 14. Extensions ·
9. Model taxonomy ·
10. Package layout (actual) · 17. Current status & GPU bring-up ·
11. Roadmap · 19. Reference synthesis ·
12. Honest unknowns & falsifiers

---

1. The thesis

Three facts about video/omni generation dictate the architecture:

• A deployable model is a post-training artifact. Unlike an LLM (where inference optimizes frozen weights post-hoc), a usable video model is created by training: step distillation is mandatory for latency, low precision needs QAT, causal/world models are made by distillation + self-forcing. Every inference capability is therefore a (recipe, runtime) pair — the weights and the loop that produced-and-assumes them are one versioned object, a ModelCard.
• Video/omni systems are loop systems. The work is iteration — denoise timesteps, AR decode, chunked rollout, VAE tiles, encoder chunks, audio tokens, reward batches, optimizer steps, media chunks — not a single forward(). A runtime that reduces everything to forward() cannot schedule, batch, cancel, stream, reserve memory for, or capture the behavior of what actually runs.
• Omni models share weights across loop types within one request. Cosmos3's text reasoner and multimodal denoiser are the same resident weights, driven by an AR loop then a diffusion loop in one request. This cannot be a DAG of separate engines (that doubles 30B+ of weights and severs the shared KV/denoise state); it must be one resident instance running many loops. The hard part — the differentiation — is making those loops runtime-visible, step-scheduled, batchable, and cost-priced (vllm-omni's bagel_single_stage/lance prove the sharing is expressible, but bury it in one opaque DIFFUSION stage the scheduler never sees inside).

The one invariant, stated once:

Model cards own components, loops, recipes, and parity.
Programs compose loops into tasks.   Workflows compose models into pipelines.
The scheduler executes the steps of loops as WorkUnits under one budget.
Caches are correct by key, not by hope.
Training records behavior on the same loops it serves.
Deployment places and routes; products stream artifacts; neither defines the model.

Everything below is the elaboration of that invariant.

---

2. Three signature ideas

2.1 The (recipe, runtime) pair is a first-class, versioned, typed object

A model is not a checkpoint. It is a ModelCard owning, as one versioned unit: the components (weights, loaders, layouts), the loops it can run, the recipe that produced the weights (distillation/QAT/RL method, parents, assumes_loop, assumes_precision), and the parity contract binding the train-forward to the serve-forward at a declared consistency level. You cannot ship the weights without the loop they assume (assumes_loop), or change the loop without re-proving parity. This turns "we do training and inference in one repo" from an org chart into a typed guarantee.

2.2 Driven loops — the model owns control flow, the runtime owns execution

A loop is a serializable state machine init → (next → advance)* → finalize: the model describes the next step it needs (next is kernel-free — it returns a typed WorkPlan thunk), the runtime decides when and with whom that step runs (await ctx.execute(plan) — the inversion point: admission, batching, placement, streaming, behavior capture), and the model folds the result back (advance) and decides what to do next. Content-adaptive decisions (cache-dit skips, EOS, VSA tile selection) are ordinary control flow in the model; per-request state lives in a typed LoopState, never in module globals, so interleaving requests through one instance cannot smear state — the failure mode that makes naive loop-inversion dangerous is structurally excluded (and proven by the interleave gate, §9).

2.3 One vocabulary spans every weight-sharing topology

The Card/Loop/Program split is not specialized to one sharing pattern. The same primitives express the whole spread — and every one is just shared_weight_components bindings on LoopSpecs plus hand-off nodes in a Program, no new primitive:

Topology	Components	Loops	Sharing	Recipe
Single diffusion	transformer	diffusion_denoise	—	wan21
MoT omni	one transformer	ar_decode + diffusion_denoise	both loops → same component	cosmos3, bagel
Joint LM+gen RL	llm + transformer	ar_decode→llm, diffusion_denoise→transformer	disjoint experts, one request, jointly RL'd	unified
Cascade omni-speech	thinker+talker+vocoder	ar_decode→ar_decode→audio_decode	disjoint, chained	qwen_omni
N-way joint RL	N refiner_i + transformer	N×ar_decode + diffusion_denoise	disjoint, N-way jointly RL'd	multi_expert

The signature claim, validated by the §11 stress tests: the weight-sharing graph is data on the card, not structure in the runtime. (BAGEL's real MoT is partial sharing — co-resident experts sharing attention, separate FFNs — expressed via the expert-routing policy over one resident instance; a fifth point on the same axis, still no new primitive.)

---

3. Planes & dependency order

   Products: Python · CLI · OpenAI server · ComfyUI · Dreamverse · RTC · Trainer   (thin: validate, request, subscribe)
   Request / Session / Artifact / Stream                                           v2/request/, v2/runtime/session.py
   Program Plane  (typed loop programs; cross-model Workflows)                      v2/program/
   ┌──────────────── Model Plane (CENTER) ────────────────┐                         v2/card/, v2/recipes/*/card.py
   │  ModelCard: components · loops · recipe · parity ·    │
   │  capabilities · caches · parallelism · precision      │
   └───────────────────────┬───────────────────────────────┘
   ┌────────────────────────┼────────────────────────┐     same loops, different capture
   │ Runtime / Scheduler      │ Training / RL          │     v2/runtime/  ·  v2/training/
   │ WorkUnits · GPU-time budget │ rollout·reward·sync │
   └────────────────────────┼────────────────────────┘
   Memory · Cache · Transport · Compile                                             v2/memory, v2/cache, v2/transport, v2/runtime/cudagraph.py
   Parallelism (named axes → DeviceMesh, validated, part of the cache key)          v2/parallel/
   Platform dispatch (COMPONENTS/KERNELS registries → toy | torch backend)          v2/platform/
   Deployment / Fleet (DeploymentCard → LocalFleet | Dynamo; never the core)        v2/deploy/, v2/serving/

Enforced boundaries: card/ imports no product/runtime; runtime/ executes card/ loops but defines no semantics; training/ requires behavior records but forks no loop and the engine never imports training (verified — the only training mention in runtime/ is the comment documenting this rule); cross-model Workflow orchestration sits above the engine (no change to the single-instance hot path). parity/ is a first-class package, not a test folder.

---

4. Model Plane — the center

class ModelCard:
    model_id: str                       # "fastwan-1.3b-nvfp4-4step"
    family: str
    components: dict[str, ComponentSpec]
    loops: dict[str, LoopSpec]
    capabilities: CapabilityMatrix      # text_to_video, image_to_video, reasoning_text, vae_decode, ...
    recipe: RecipeSpec                  # what produced these weights (§2.1)
    parity: ParitySpec                  # train-forward ≡ serve-forward, to a declared level (§9)
    caches: dict[str, CacheContract]
    parallelism: ParallelismContract
    precision: PrecisionContract

The card is both a declarative contract (validatable before any GPU touches it) and a runtime factory (it instantiates components, binds loops, resolves caches). card.validate() checks every loop's components + cache policies exist and that recipe.assumes_loop is a declared loop.

• RecipeSpec — method (dmd2/self_forcing/diffusion_nft/unified_rl/base/…), parents (teacher/base ids), assumes_loop, assumes_precision, consistency_required. assumes_loop is the teeth: a 4-step distilled model cannot be served under a 50-step sampler without a typed mismatch error.
• ComponentSpec — component_id, kind (dit/vae/text_encoder/reasoner_tower/…), load_id (the real module to load on the torch backend), factory (the toy stand-in), checkpoint (weights path/HF id), required_for/optional_for/resident_for task sets, precision/placement policies. (Note: required_for is currently declared on every card but not yet consumed by the executor — see §18 roadmap P0.)
• LoopSpec — loop_id, kind (LoopKind.DIFFUSION_DENOISE/AR_DECODE/CHUNK_ROLLOUT/AUDIO_DECODE/…), work_unit_kind, step_cost_model (predicted GPU-time per step — §6), shared_weight_components, cache_policy, loop_factory.

A ModelInstance is a resident, loaded card: component instances, model state, caches, compiled graphs, a parallel plan. A request may run several of the card's loops against one ModelInstance — that single sentence is what makes omni native, and two loops binding the same shared_weight_components get the same live object via instance.component() (no weight duplication, no DAG split).

---

5. The driven-loop contract

class Loop(Protocol):                 # v2/loop/contracts.py — protocols, not ABCs
    def init(self, req, model, ctx) -> LoopState         # per-request state (seeded rng, latents…)
    def next(self, st)              -> WorkPlan | Done    # describe the next step (KERNEL-FREE: a run() thunk)
    def advance(self, st, result)   -> LoopState          # fold result; capture behavior under ROLLOUT
    def finalize(self, st)          -> LoopResult          # outputs + metrics + behavior

The runtime's LoopRunner is the only place iteration lives: next → await ctx.execute(plan) → advance, emitting plan.emits as stream chunks. Why this contract is right, against the failure modes:

• Content-adaptive steps are natural — next() reads state (and advance already folded in the last StepResult), so cache-dit's skip, AR's EOS, and VSA's tile selection are ordinary control flow; next() is still kernel-free (it describes work), which is all the scheduler needs.
• Cross-request state safety is structural — all per-request state is in LoopState; there are no module-level residual/KV globals, so interleaving cannot smear state (the §9.3 interleave gate proves it).
• Serializable ⇒ resumable/migratable — a LoopState is a resume point (preempt, migrate, crash-recover).

Policies decompose the step body (CFG/flow-shift/precision/expert-routing) — composed into a loop, not branched inside it. CFG is a policy over one shared denoise body (proven by vllm-omni's CFGParallelMixin unifying 2-forward / batched / cfg-parallel under one predict/combine pair), expressed as three layers: (1) CFGPolicy is in-loop — branch vocabulary, combine formula, per-request mutable state (the adaptive-gate cached delta is the canonical state case; batched-vs-2-forward is a dispatch detail inside one policy); (2) cfgp is a parallelism axis that shards branches across ranks and runs the same rank-invariant combine; (3) companions are an orchestrator pattern upstream of diffusion. Two caveats: combine runs in the step body's numeric space (Cosmos combines in x0-space post-EDM, not noise-space), and embedded-guidance (Flux) is a degenerate single-branch identity-combine policy, not "no CFG". A family whose math is genuinely braided ships a custom next/advance using samplers/CFG as a library — the runtime requires only the four methods.

---

6. Runtime & scheduler

One WorkUnit, one currency. Every ctx.execute(plan) is a WorkUnit — the smallest schedulable action with a resource reservation and a loop boundary. Kinds: AR_TOKEN, AR_PREFILL, DIFFUSION_STEP, DIFFUSION_WINDOW, CHUNK_STEP, ENCODER_CHUNK, VAE_TILE, AUDIO_CHUNK, REWARD_BATCH, LOGPROB_BATCH, TRANSFER, CACHE_IO, GRAPH_CAPTURE (a 13-kind taxonomy). Tokens are one kind, not the scheduler — the generalization of vLLM's token scheduler that diffusion forces.

The budget currency is predicted GPU-time, not counts. A bidirectional denoise step re-attends the full latent at O(L²) with zero KV amortization; an AR decode step is ~O(context) against a cache. Counting "steps" or "tokens" puts items three orders of magnitude apart in one bucket. Each WorkUnit converts to GPU-seconds via LoopSpec.step_cost_model, online-calibrated by the Profiler. The same cost model is the object published to the fleet (§13) — internal budget and Dynamo routing input are one thing.

Admission rule (the soundness condition of multiplexing): do not admit a waiting WorkUnit unless every resource it requests can be reserved — compute budget AND memory (resident + worst-case peak) AND cache blocks AND transfer bandwidth AND graph-capture shape AND output sinks. Infeasible requests fail fast (AdmissionInfeasible); budget is refunded on completion. Honesty caveats kept from contact with reality: admission uses the conservative baseline (cache-dit skips, VSA tiles, AR length are unknowable in advance — budgeted at the cap, refunded on early EOS); a denoise step is indivisible (mitigated by cost-class pools + SP-within-a-node + SLO classes).

Scheduler in layers (each testable on a fake pool, no GPU): RequestScheduler → LoopScheduler → BatchScheduler (groups compatible WorkPlans by (instance, loop_kind, shape_sig, precision, parallel_plan, graph_key)) → PlacementScheduler → TransferScheduler → AdmissionController. SPMD consistency: rank-0 decides and broadcasts (the same channel as the abort broadcast — scheduling and failure isolation share one mechanism). Cancellation is common-path (vibe-directing makes abandoning in-flight work normal): it takes effect at the next step boundary, drops queued WorkUnits, releases LoopState + cache handles.

---

7. Memory, cache, transport, compile

Cache correctness is a contract. CacheKey carries every output-semantic field — model_id, component_id, loop_id, per-component weights_version, adapter_versions, precision, parallel_plan_hash, shape_sig, layout_sig, scheduler_sig, guidance_sig, seed, input_hashes, step_index, contract_version. If a field can change output semantics, it is in the key. Incorrect reuse is worse than no reuse: the key is partitioned by adapter_versions (a te-LoRA-differing request doesn't serve stale embeddings), and a weight-sync bumps only the affected component's weights_version — so a transformer sync does not flush the frozen text-encoder's feature cache (a K-sample RL group encodes its shared prompt once).

Per-class pools (the granularity reality). No single unified block pool — cache classes differ by 150–500× in natural granularity (text-KV page ≈ 64 KB/layer; causal-video latent-chunk slab ≈ 9.6–32 MB/layer). Each class gets a statically budgeted pool: paged text-KV (ar_decode), slab chunk-KV (chunk_rollout, with a training mode that disables mid-rollout recycling), feature caches (content-hash keyed, ref-counted), residual caches (cache-dit, scoped per LoopState), weight/adapter cache. KV is the minority case — a pure bidirectional deployment allocates none of it.

Memory / transport / compile. Tagged pools with sleep/wake by tag (CuMem-style, component-granular for RL). Transport is manifest-based and pluggable: in-proc reference → SHM → CUDA IPC → NCCL/NIXL → object-store; KV-bearing edges speak a KVConnector-shaped protocol (chunk_ready readiness + credit-based flow control, sglang-omni's model). Compile: CUDA graphs + torch.compile keyed on (model, component, loop, work_kind, shape_sig, precision, parallel_plan, backend) — never full-graph across the engine; per-step piecewise capture is wired (runtime/cudagraph.py, declared on 19 cards) with a static-buffer discipline and version-eviction on weight sync.

---

8. Parallelism as a model contract

Parallelism is not a launch flag — it affects cache keys, scheduling, transport, capture, and parity, so it lives on the card. ParallelPlan axes (v2/parallel/plan.py): ("dp","tp","sp","cp","cfgp","pp_patch","vae","ep","fsdp","role","replica"). Declarative, validated (validation.py: cfgp ≤ 2; pp_patch is invalid for causal/AR because stale KV breaks causality; ownership conflicts like a BatchedCFG policy and a cfgp group are build errors), compiled to a PyTorch DeviceMesh via a ParallelDims-style builder. Pre-flight or it fails at load, never halfway. Degree-one axes exist as trivial groups so component code needs no special cases. Pools are single-node; multi-node scale is multiple pools fronted by the fleet (§13). Note: Wan/LTX shipped weights parallelize via sequence parallelism (sp), not TP (they use ReplicatedLinear); real ColumnParallelLinear lives in flux2.

---

9. Correctness — parity as a typed gate

Every card carries a ParitySpec. Parity is measured, never assumed, by a ParityAligner observer: record named taps per step/block from a reference, replay with fixed seeds, report the first divergence beyond per-tap tolerance.

The consistency ladder:

C0  component parity   — VAE/encoder/transformer-block/scheduler-step in isolation
C1  loop parity        — full denoise trajectory / AR logits, fixed seed
C2  behavioral identity — the train-forward and serve-forward agree on the quantity the RL objective uses:
       · likelihood-based (GRPO/UniRL): per-step log-prob identity ⇒ PPO ratio == 1
       · likelihood-free (DiffusionNFT): seeded final-sample + prediction-space identity — NO log-probs to match
C3  distribution parity — rollout distribution under allowed nondeterminism (defined; not yet consumed — §18)
C4  artifact quality   — SSIM/reward/human-preference (gates product claims; needs the eval system)

The C2 split is load-bearing and the lesson of the landed RL stack: the shipped DiffusionNFT is likelihood-free (no log-probs; "log-prob identity" is undefined for it), while UniRL is likelihood-based — both are demonstrated, on opposite halves of the rung.

The interleave gate (§9.3) — the bet loop-inversion lives or dies on. Loop inversion's real hazard is cross-request state smearing under interleaving, and a batch-of-1 gate is structurally blind to it. So a batch-of-N interleave parity test is a required gate: N concurrent requests interleaved at step granularity must be bit-identical to the same requests run serially — and it holds across every model, the MoT omni cards, the two-loop unified program, the three-loop cascade, and heterogeneous WorkUnit kinds. (A buggy module-global interceptor breaks the gate; the per-request one passes.) Three execution profiles, one loop definition: serve (no-grad, graphed, cached), rollout (serve + behavior capture), train (grad, checkpointed) — they differ only in grad mode and capture; the ladder measures the gap.

---

10. Training & RL on the same loops

serve   : request      → program → loop → WorkUnits → artifacts
rollout : prompt batch → program → loop → WorkUnits → BehaviorRecords → rewards → update

The loop kernel is shared; the only difference is capture and training policy. This is the moat — the one place a serving-only runtime structurally cannot follow. The rollout forward is the serve forward plus capture, so every serving optimization (distilled samplers, cache-dit skips, CFG-parallel, paged/feature caches, step batching) is automatically a rollout optimization, and there is one numerics surface (the ladder measures the gap rather than a correction layer papering over it). The industry's two-runtime tax (verl-omni re-implements Wan inside vLLM-Omni + a correction layer; miles' TIS/MIS/bitwise-logprobs/R3 are mismatch patches) is exactly what collocation deletes — viable at FastVideo's 1–30B FSDP2 scale.

• BehaviorRecord — captured at generation time: seeds, scheduler trajectory, timesteps, latents-or-refs, log-probs where applicable, sampled/action tokens, guidance, reward in/out, precision, parallel plan, weights_version. Sized honestly — an opt-in instrument for goldens, not always-on.
• WeightSyncPlan ships a role, not "the weights" (student / EMA / decay-blended old-policy / reference / teacher / critic), with a per-component scope so a sync versions and cache-invalidates one expert in isolation. Lifecycle (the RL flywheel's hardest correctness): freeze admission → drain/boundary-stop in-flight loops → transfer → bump version + invalidate that component's caches → resume.

Methods (a faithful CPU port — NFT is line-for-line vs the source — carrying none of the GPU/FSDP/checkpoint infra):

Method	Consistency	Roles	Notes
finetune	C1	student	plain flow-match regression
dmd2	C2 (free)	student + fake-score critic + teacher	distribution-matching distillation
diffusion_nft	C2 (free)	student + old (decay-blended) + reference	samples from old, not student
self_forcing	C2 (free)	student + teacher	causal/chunked student
unified_rl	C2 (based)	student (llm+transformer) + reference	§11 — joint LM+gen RL
joint_multi_rl	C2 (based)	N refiners + generator	N-way joint RL
workflow_rl	C2 (based)	two instances	end-to-end RL across a cross-model workflow

---

11. Weight-sharing topologies & the stress-test catalog

The central question — does the Card/Loop/Program split generalize beyond serving + MoT, or will joint multi-expert RL / cross-model pipelines / interactive sessions / joint A/V / content-adaptive compute / hot weight-sync force a redesign? — was answered by a battery of stress tests. Every frontier capability landed as a new card / method / loop / workflow / session-driver / controller, with NO new runtime primitive (the only real bug any test surfaced — a no-op generator gradient — was a fix in the sampler library). Condensed:

• Joint LM+generator RL (UniRL/PromptRL) — one reward → token policy-gradient on the LM and FlowGRPO PPO on the DiT; dual log-prob capture (categorical + Gaussian SDE); likelihood-based C2 (per-step identity ⇒ ratio == 1); two independently-versioned weight-sync plans; SDE rollout sampler gated behind sde_rollout so the serve path is byte-for-byte unchanged.
• Qwen-Omni cascade — three disjoint experts, three loop types (ar_decode→ar_decode→audio_decode), chained cross-stage conditioning, streaming codec→waveform.
• Cross-model Workflow (T2I→I2V) — composition across distinct model instances; each model keeps its own interleave-parity guarantee; a workflow_id is a first-class servable in the same namespace as a model_id, registered via WorkflowRegistry. Plus nested workflows (recursive) and non-linear shapes (fan-out, best-of-N feedback).
• N-way joint RL — generalizes joint RL to arbitrary N (per-component sync, dict grad-targets); surfaced a credit-assignment finding (per-expert reward clean; shared reward noisy) — a reward-shaping choice, not a substrate change.
• Interactive world-model session — persistent cross-request state, transactional step-boundary cancellation, no cross-session smearing.
• End-to-end RL over a workflow — one final-video reward trains an earlier model; proven causal by a control (constant reward ⇒ nothing moves).
• Heterogeneous WorkUnit co-scheduling — VAE_TILE interleaves bit-identically with DIFFUSION_STEP through one budget (the §20 falsifier's mechanism half).
• Joint A/V (LTX-2 T2VS, per-modality guidance), content-adaptive compute (cache-dit skip + early-exit, ragged step counts that still pass the interleave gate), hot weight-sync under in-flight serving (drain-correct), served reward model (REWARD_BATCH), speculative decoding (exact, lower-latency AR), the RL→distill flywheel (RL-improve → distill from the RL'd teacher → faster card with provenance), and the adapter plane (per-request LoRA/ControlNet over one base).

---

12. Request / session / artifact + programs/workflows

Typed runtime objects, not IDs in a batch: Request (task is declared, never inferred; inputs: list[ModalPart]; AR sampling vs diffusion params; OutputSpec), Session (long-lived interactive context: prompt memory, media streams, persistent cross-request chunk-KV), Artifact (named, typed, with provenance — VideoArtifact, AudioArtifact(sample_rate), TextArtifact, … — killing the extra["audio"] pattern), Stream (one ordered event channel), CancelScope. A Program composes one card's loops into a task DAG (ComponentNode for kernel-free seams, ModelLoopNode to drive a loop to completion); a Workflow composes across model instances (each stage a full engine.run, artifacts threaded stage→stage) — the crossing is a Workflow boundary, not a program loop step, so each model keeps its parity guarantee. Workflows compile, they are not the runtime (a ComfyUI graph maps onto a Program; unknown nodes become ExternalNodes or a coverage rejection — never silent wrongness).

---

13. Serving & fleet

Per the standing instruction "don't completely rely on Dynamo; we still need our own version," v2 ships a complete stack and treats Dynamo as one optional backend: AsyncEngine (queue, lifecycle, live SSE streaming, step-boundary cancellation) + an OpenAI-compatible server on stdlib asyncio (serving/http.py: /v1/chat SSE, /v1/images, /v1/videos job+poll, /v1/models, /health, /metrics); a DisaggregatedRunner proven bit-identical to inline + role/stage pools; connectors with chunk_ready + credit-based flow control; our own LocalFleet (cost/affinity/least-loaded routing, health/drain) and a DynamoWorkerAdapter exporting the same DeploymentCard + cost model (one object, two consumers) so Dynamo can front us but never defines the core. The clean line: the fleet owns global routing / cold start / role-pool scaling / SLO placement / failover; the engine owns model load / loop execution / local scheduling / local memory+cache / parity / WorkUnit batching.

---

14. Extensions

Versioned hook points assembled at loop build (an unused hook is literally absent from the hot path), wrapping ctx.execute(plan). Observers (read-only): ParityAligner, Profiler (calibrates the cost model), NaNWatch, ActivationTrace. Interceptors (compute-altering): StepInterceptor (step-skip / cached prediction) and BlockInterceptor (cache-dit DBCache/FBCache/TaylorSeer). State lives in LoopState.plugin_state[id], keyed per request and per CFG branch — the structural fix for module-global residual state that silently corrupts cache-dit/TeaCache forks under concurrency. Capability negotiation: a 4-step distilled card rejects a residual-skip interceptor rather than producing garbage. Trust boundary: plugins are enabled at deploy scope only; requests only parameterize pre-enabled plugins through validated schemas. This is the seam M* (§18) calls "extensible — integrate FastVideo-STA / xDiT / Inferix / FlashDrive"; v2 already has the mechanism.

---

15. The model taxonomy (what the runtime must serve)

Paradigm	Examples	Loop shape	State	Output
Bidirectional video diffusion	Wan2.1/2.2, Hunyuan(15), LongCat, Cosmos2/2.5, LTX-2	N denoise steps over full clip	latents, CFG branches, block caches	video
Few-step distilled	DMD/FastWan, TurboWan, rCM	1–4 denoise steps	latents	video
Causal/AR video	Wan-Causal(-DMD), MatrixGame2/3, LongCat-VC, SF-Wan	outer chunk loop × inner denoise	DiT KV (slab chunk-KV)	video (streamable)
Interactive world models	MatrixGame, GameCraft, HYWorld, Gen3C, LingBotWorld	chunk loop driven by live actions	KV + action/camera cond	video stream
Image	Flux2, SD3.5, Qwen-Image, Kandinsky5	N denoise steps	latents	image (batchable)
Audio / Joint A/V	Stable Audio; LTX-2 (video+audio), Cosmos3 t2vs	denoise over (joint) latents	per-modality CFG	audio / video+audio
Multi-stage refinement	LTX-2 (base→upsample→refine), Hunyuan15-SR	pipeline of loops	inter-stage latents	video
AR token decode	Cosmos3 reasoner; omni thinkers/talkers	token loop until EOS	paged text-KV	text / codec tokens
Vocoder / one-shot	LTX-2 vocoder, audio codec → wav	single forward / chunked	none	audio
Hybrid MoT (omni)	Cosmos3, BAGEL	AR loop then/while denoise, shared weights	text-KV + denoise state + packed seq	text+video+audio+action

A runtime that serves the MoT row serves everything above it; a DAG-of-engines cannot (the reasoner and denoiser share weights).

---

16. Package layout (actual)

v2/
  card/        ModelCard, ComponentSpec, LoopSpec, RecipeSpec, ParitySpec, instance, load_card
  loop/        contracts (LoopState/WorkPlan/StepResult/Done), driver (LoopRunner), policies (cfg/flowshift/
               precision/routing), sampler (flow-match + FlowGRPO SDE)
  program/     ComponentNode/ModelLoopNode/Program; Workflow + WorkflowRegistry (cross-model; §12)
  request/     requests, params (DiffusionParams + sde_rollout), tasks (TaskType), streams, artifacts, cancel
  runtime/     engine (run/run_serial/run_interleaved, workflow-aware), async_engine, scheduler, admission,
               cudagraph (piecewise capture), disaggregated (DisaggregatedRunner), pools, session (WorldModelSession), context
  cache/       keys (CacheKey, content_hash), classes (feature/residual/slab_kv/paged_kv), manager
  memory/      allocator, reservations, refundable budget
  transport/   manifests + connectors (in-proc; chunk_ready + credit flow)
  parallel/    plan (axis vocab), mesh, validation
  parity/      aligner, ladder, interleave_gate, compare_outputs
  extend/      observers (Profiler/NaNWatch), interceptors (cache-dit), registry, base
  platform/    Platform.detect(); COMPONENTS(kind,device,variant) + KERNELS(op,device,arch,variant) registries;
               backends/{toy.py (numpy reference + parity oracle), torch_backend.py (real GPU)}
  loader/      v2-owned component-loader seam (currently delegates to fastvideo; vendored cutover later)
  models/      v2-namespaced model code (re-export stubs over fastvideo until vendored; see vendoring memory)
  recipes/     the concrete cards/programs/loops — 35+ families: wan21, ltx2, wan_causal, sfwan22, fastwan,
               turbowan, flux2, sd35, kandinsky5, hunyuan_video(15), longcat, cosmos2/25/3, gen3c, hyworld,
               hunyuangamecraft, matrixgame2/3, lingbotworld, lucy_edit, stable_audio, wan_fun_control,
               bagel, qwen_omni, omni, unified, multi_expert, image_video, tiled, adaptive, adapters,
               speculative, reward
  training/    rollout, behavior, rewards (+ServedRewardScorer), weight_sync (+WeightSyncController),
               flywheel, methods/{finetune,dmd2,diffusion_nft,self_forcing,unified_rl,joint_multi_rl,workflow_rl}
  serving/     AsyncEngine glue, OpenAI server (http.py)
  deploy/      DeploymentCard (card.py), LocalFleet (fleet.py), DynamoWorkerAdapter (dynamo.py)
  distributed/ single-process dist-init seam (1×1 device mesh)
  tests/       34 files, 216 tests — `pytest v2/tests/` OR `python3 v2/run_tests.py` (zero deps)

---

17. Current status & GPU bring-up

The kernels are no longer toys-only. The platform/ substrate selects backends through two tuple-keyed registries (COMPONENTS(kind, device, variant) + KERNELS(op, device, arch, variant)) that a detected Platform resolves, with the numpy toy backend as the terminal fallback rung and parity oracle. On a GPU box, platform/backends/torch_backend.py provides real TorchComponent adapters wrapping the real model code (resolved from each card's load_id, weights from ComponentSpec.checkpoint) — and the loops, scheduler, caches, parity, training, and workflows are unchanged, exactly as the (recipe, runtime) separation promised.

Verified on H100 this session:

• 20+ real model families GPU-verified end-to-end via the torch backend (VideoGenerator.from_pretrained + generate_video), including SF-Wan (self-forcing causal, CFG-free 4-step DMD) and LTX-2 two-stage SR (frame-verified) — see ../v2_debug_videos/vlm.md.
• The omni trio runs real via vllm-omni (in an isolated venv; FastVideo's env untouched): BAGEL-7B-MoT (two-stage MoT, on-prompt 1024² image), Qwen2.5-Omni-7B (thinker→talker→code2wav, coherent text + 24kHz speech, 2-GPU), Cosmos3-Nano (Cosmos3OmniDiffusersPipeline T2V, 720p, frame-verified). Recipe + box-specific flags (VLLM_USE_FLASHINFER_SAMPLER=0, VLLM_USE_DEEP_GEMM=0, guardrails-off) are saved in the vllm-omni-bringup memory; outputs in ../v2_debug_videos/omni/.

What is genuinely not built yet (named, not hidden): real distributed parallelism inside one pool (collectives are stubbed; multi-node is multiple pools fronted by the fleet); the ComfyUI workflow compiler; WebRTC realtime wire (the interactive session logic is built); full LLM-grade AR serving (radix trees, chunked-prefill sophistication); C3 (batch-invariance) and C4 (quality/preference + the eval system); a torch-native loop surface (loops still marshal numpy↔torch at the boundary — a perf follow-up).

cd /home/scratch.willlin_ent/FastVideo
python3 -m pytest v2/tests/ -q      # 216 tests
python3 v2/run_tests.py             # same suite, zero deps

---

18. Roadmap

The forward plan has two tracks: (A) adopt the valuable ideas from the M* paper (the closest external work to v2's thesis), and (B) finish the GPU port. The full M* gap-analysis (28-agent workflow, adversarially verified against the code) is in .agents/exploration/mstar-v2-roadmap.md.

M* in one line. M*: A Modular, Extensible, Serving System for Multimodal Models (arXiv 2606.12688) is essentially v2's thesis one step more mature: "every composite model is a dataflow graph; every request is a Walk over it." It beats vLLM-Omni/SGLang-Omni on exactly the models v2 now runs (BAGEL, Qwen3-Omni) and explicitly names FastVideo as an integratable technique. v2 already implements the harder half (its Program is M*'s graph; shared_weight_components is cross-Walk node sharing) and exceeds M* on a validated cost model, the interleave bit-parity gate, integrated training, and the extend/ plugin seam. The insight: v2's substrate is ~80% built but parts are inert (authored metadata never wired to an executor).

Prioritized (P0 = highest leverage / lowest risk; parity-safe, CPU-toy-testable):

Pri	Item	Action
P0	Min-components per request	Consume required_for in Program.active_nodes; fix the real bug: runtime/engine.py:88 uses self.program.nodes while runtime/disaggregated.py:96 uses active_nodes(request) — the two runners disagree. Deliver via the registry/card builder so all cards inherit it. → M*'s "execute the minimum components per request," cutting wasted reasoner/AR steps on single-modality BAGEL/Cosmos3 requests.
P0	Real EOS + declarative DynamicLoop	recipes/omni/ar_loop.py docstring claims EOS-stop but next() only checks max_tokens. Honor eos_id + req.sampling.stop; add LoopSpec.dynamic_stop + register_loop_stop. Also training-enabling (world-model rollout horizon).
P1	CFG/branch as a label over one paged KV pool	Make PagedKVCache a real (namespace,label) store over one budget; reuse CacheKey.guidance_sig for the hash (NOT partition_field). → the measured BAGEL win (AR path only; diffusion has no KV).
P1	extend/ plugin: FastVideo-STA / Inferix	Expose this repo's sparse/sliding-tile attention + block-diffusion as Interceptor/EngineKind plugins — the paper's named integration, highest paper-alignment, low risk (seam exists).
P1	ParitySpec.output_determinism	Close the dormant C3 rung so SDE/stochastic RL can declare its parity contract honestly.
P2	Pluggable data plane; per-(node,Walk) placement; declarative TP/SP degrees; loop-spanning CUDA graphs	Gated on the multi-GPU runtime; the live 2-GPU Qwen-Omni bring-up is the natural first test. Keep the cheap declarative halves now (EngineKind tag, placement key, populate parallel_plan_hash on the serving cache path).

Do NOT regress (where v2 already meets or beats M*): the validated per-loop cost model, the interleave bit-parity gate, the C0–C4 consistency ladder, the integrated training plane (RL→distill flywheel, drain-correct hot weight-sync), CPU-toy parity for the whole stack, the extend/ plugin seam, Dynamo citizenship.

GPU-port track: wire real distributed collectives (one-pool TP/SP); torch-native loop surface (drop the numpy↔torch boundary marshalling); admission budgeting of capture cost; per-stream workspace pool for a concurrent executor; the AR/vocoder GPU op families; the GPU training surface (mse_grad_step).

---

19. Reference synthesis

What v2 takes and what it constrains, per surveyed system (the design is a synthesis, never a copy):

Source	Take	Constrain / reject
Cosmos3 (official + port)	Shared instance across reason/diffusion/action/sound; packed multimodal sequences; component+scheduler parity matrices	A strong ModelCard, not the framework; no Cosmos branching in the global runtime
vLLM core	Running-first scheduling, reservation-before-admission, model-owned state, KV/encoder cache managers, CuMem sleep/wake, CUDA-graph dispatch, KV-connector split	Token scheduling is one WorkUnit kind; never full-graph compile
sglang multimodal_gen	Role pools, request lifecycle, capacity dispatch, transfer manifests, disagg state machine, cache-dit integration	No giant mutable Req/ForwardBatch as the API; not single-item diffusion scheduling
vLLM-Omni	Frozen pipeline-spec ⟂ deploy-YAML split; OmniConnectorBase + chunk_ready; SupportsStepExecution as loop-inversion prior art (we generalize to always-on); CFGParallelMixin proves CFG-as-policy; 3 cache subsystems confirm per-class pools	Expresses MoT only as one opaque request-scheduled stage (no step visibility, no cross-request batching); cross-stage KV is a copy; no cost model
sglang-omni	The next/wait_for/merge_fn/stream_to edge vocabulary; Relay + credit-based flow control	Stages own disjoint weights; hybrid only as AR-stage→DiT-stage
Dynamo	Fleet routing, disagg role pools, KV-aware routing, KVBM, SLA planner, cold-start weight streaming	Orchestrates engines; never the core. Export a DeploymentCard + cost model to it
diffusers Modular	ComponentSpec/modular_model_index.json interchange; Guiders ≈ CFG policies	A Python pipeline interpreter is not the perf boundary; loop blocks own their iteration (not inversion)
xDiT / PipeFusion / USP	DiT parallelism catalog (USP, ring/ulysses, PipeFusion, CFG-parallel, DistVAE) + world-size validation	Parallelism lives in the runtime + card, not a wrapper-per-model; pp_patch invalid for causal
TorchTitan	Named mesh axes, ParallelDims validation, ModelSpec discipline, batch-invariance utils	Adopt the discipline, not the stack; WeightSyncPlan owns layout (DCP/TorchStore don't reshard)
verl-omni / miles / cosmos-rl	Rollout adapters, per-step capture, async rewards, group-relative advantage, TIS/MIS, deterministic modes, per-payload weight-version, AIPO off-policy masking	The two-runtime tax is the thing to delete; capture behavior in the serving loop
UniRL-Zero / PromptRL	Joint LM-refiner + flow-generator RL under one reward; FlowGRPO SDE/ODE with per-step log-probs; group advantage → token-PG + PPO; prompt-only vs joint ablation	A card + a method, not a bespoke trainer; SDE sampler gated behind sde_rollout; PPO ratio rests on the likelihood-based C2 gate
ComfyUI	Workflow graph, node-signature cache, model memory management	Compile to Program; dynamic node execution is not the core; GPL hygiene
Dreamverse / LiveKit	Sessions, prompt memory, typed media IPC, cancellation, duty-cycle capacity, preference-data flywheel; realtime frame/PTS streaming	Product/session behavior is first-class in the request plane, never merged into the model core; RTC only when triggers fire (<100ms interactive)
M* (2606.12688)	The Walk-Graph framing (named Walks + state machine), per-(node,Walk) placement, CFG-as-cache-label over one paged pool, the "extensible: integrate FastVideo/xDiT/Inferix" call-out	v2 already has the harder half + exceeds on cost/parity/training; adopt the declarative authoring layer where it's inert (§18)

---

20. Honest unknowns & falsifiers

An ambitious design is not an unfalsifiable one. The bets, with the experiment that kills each:

• Step-level scheduling must pay for video. Runtime-owned diffusion iteration has narrow precedent (vllm-omni's opt-in SupportsStepExecution); v2 makes it the always-on universal contract. Falsifier: on a real duty-cycle trace, if step-level scheduling does not beat a request-level baseline (≥2 concurrent sessions/GPU, p95 within SLO), degrade to request-level dispatch and keep only the loop contract's streaming/cancellation/behavior seams (which still justify it). The contract is safe even if the scheduling bet loses — that is the insurance.
• The general WorkUnit scheduler may be over-general. The mechanism half is validated (VAE_TILE and REWARD_BATCH interleave/schedule through one budget); the economic half (does it pay vs an in-loop call) is a GPU-port measurement.
• Cost-model admission is a modeling bet — argued on its narrow window (many small concurrent jobs), measured on the port.
• Quality is unmeasured. C4 (artifact quality / preference) and the eval system it needs do not exist yet, and they gate every product claim ("fast mode is equivalent", RL reward validity, distillation comparisons).

Final position. A model card is a (recipe, runtime) pair with a parity obligation; the model owns loop semantics, the runtime owns loop lifecycle; one resident instance runs many loops, one scheduler runs their steps in one currency; caches are correct by key, parity by test, and the interleave gate is non-negotiable; training records behavior on the same loops it serves. The weight-sharing topology, the composition graph, the training recipe, the reward, and the session/sync lifecycle are all data over cards, loops, workflows, and controllers — so a new frontier capability is a card or a driver, not a rewrite.