hao-ai-lab-FastVideo/design.md

FastVideo Next-Generation Inference Runtime — Design Proposal

Status: Draft for discussion (v19 — v18 reconciled against the landed RL reference: PR #1450 Wan DiffusionNFT training and PR #1396 per-method grad-norm regression CI; §8 claims converted from prediction to landed evidence) Date: 2026-06-12 Scope: The inference runtime and the shared substrate beneath it. The legacy fastvideo/training/ stack is fully out of scope (frozen, per repo policy). The modular fastvideo/train/ stack is in scope as a consumer, not as a redesign target: its trainer machinery (Trainer, method loss logic, callbacks, FSDP/optimizer/checkpoint/ data) is untouched, but its embedded sampling/rollout paths migrate onto the shared loop substrate, its config imports the shared layers, and the RL layer is built under it against the engine's rollout surface — §8 defines the exact boundary.

---

Table of contents

1. Executive summary
2. Goals and non-goals
3. Where we are today
4. What we must support: the model taxonomy
5. Survey of reference systems
6. Proposed architecture
7. Worked examples: seven workloads on the new runtime
8. Training, post-training, and RL on the same substrate
9. The application pull: Dreamverse, ComfyUI, and the workflow layer
10. Migration plan
11. Trade-offs and alternatives considered
12. Risks, concerns, and open questions

---

1. Executive summary

FastVideo today is a pipeline library that runs one request at a time: a linear list of stages, a 111-field ForwardBatch blackboard (~153 counting the training/preprocess batches in the same file), a 1,381-line denoising.py whose ~670-line DenoisingStage owns the sampling loop, and an SPMD executor with no request queue. This served bidirectional video diffusion well, but the model landscape has moved:

• Autoregressive/causal video (Wan-Causal, MatrixGame2/3, LongCat, Self-Forcing lineage) each re-implement ad-hoc KV-cache dicts inside their DiT files.
• Audio arrived twice — as a joint output modality (LTX-2 video+audio) and as a standalone diffusion model (Stable Audio) — and both had to be threaded through video-shaped abstractions (batch.extra["audio"]).
• Image models (Flux2, SD3.5) inherit a video denoising loop with prefix == "Flux" special-casing.
• Omni models are the forcing function. The Cosmos3 port — the full-omni branch feat/cosmos3-reasoning, top of the stacked chain covering T2V/I2V/T2I, joint video+audio (t2vs), action, and text reasoning, all bit-exact against the official framework (worktree /home/william5lin/FastVideo_cosmos3_port) — proved FastVideo can run a mixture-of-transformers world model — but only by writing one monolithic stage that owns tokenize→encode→denoise→decode, returning audio via batch.extra, and running text reasoning with no KV cache (full re-prefill per token).

The proposal is to evolve FastVideo into an engine with three cleanly separated planes:

1. Request plane — typed multimodal requests/outputs (OmniRequest/OmniOutput), an async engine with a real queue, streaming, and an OpenAI-compatible server.
2. Pipeline plane — pipelines become declarative graphs of stages; the two iterative computations (denoise loops and AR decode loops) are inverted so the runtime owns iteration and stages expose init/step/finalize. Model-specific behavior moves from isinstance forests into policy objects resolved at pipeline-build time.
3. Execution plane — the existing SPMD worker pool, plus a step-level scheduler that multiplexes loop steps across concurrent requests — with first-class cancellation, failure isolation, and memory admission designed alongside it, because multiplexing creates the blast radius it must then contain (§6.3.1) — a cache manager with per-class budgeted pools (paged text KV, slab chunk-KV, feature caches), and an optional placement layer (role-disaggregated pools, per-stage devices, connectors) borrowed from sglang multimodal_gen and vllm-omni. Pools are single-node in v1; fleet scale is Dynamo's plane.

Two systems cut across all three planes. An extension system (§6.4) provides versioned hook points: read-only observers for debugging, profiling, and numerical-parity alignment (generalizing the existing fastvideo/hooks/activation_trace.py), and compute-altering interceptors for cache accelerations — cache-dit (DBCache/TaylorSeer, the library sglang's multimodal serving already integrates) as the flagship — with per-request, per-CFG-branch state so they stay correct under concurrency. And declarative stacked parallelism (§6.3.4, adapted from vLLM-Omni RFC #4084) makes pool meshes and engine topology one validated spec instead of scattered flags. Above the engine, fleet scale is explicitly delegated: NVIDIA Dynamo is the first-class partner for discovery, cross-node routing, SLA autoscaling, and cold starts — FastVideo already runs as a minimal Dynamo video backend, and the design hardens that contract (§6.3.5) rather than rebuilding any of it. The partnership is two-way and Dynamo is not treated as frozen: §6.3.6 states seven concrete upstream asks (generic affinity key spaces, a request-cost interface, chunked media streaming, role-graph disagg, an RL weight plane with staleness-aware routing, cache-object tiering, sessions), each with a fallback. The asks are unfiled proposals on an NVIDIA-governed roadmap — fallbacks are first-class plan elements, not afterthoughts, and the goal is Dynamo fronting both production serving and RL rollout fleets.

The same substrate answers the training question (§8). Integrated training/post-training is FastVideo's advantage — structurally, not incidentally: unlike LLMs, where inference optimization is post-hoc on frozen weights, a usable video model is itself a post-training artifact (step distillation is mandatory; low precision needs QAT — our NVFP4 line; AR/world models need distillation plus self-forcing), so every inference capability is a (recipe, runtime) pair and the training loop embeds the inference loop (§8.3). But today the denoise/sampling loop exists in four copies (inference stages, train/ distillation methods, legacy training/ pipelines — and the RL work, landed as PR #1450, vendored the fourth and documented why in its own docstring: there was no shared loop it could call without binding to pipeline classes, §8.1). Loop inversion makes the step functions the single shared implementation for inference, train/, and RL where they fit — self-forcing ships as a custom training loop consuming substrate pieces (§8.2), and the legacy training/ copy stays frozen and retires per-family on the Phase-5 schedule, not by in-place unification; the engine doubles as the RL rollout engine (trajectory/log-prob capture, weight sync, sleep/wake) under a strict engine never imports train rule. Two production RL systems sharpen the rule (§8.4): the industry's pain is two runtimes with different kernels (Megatron/FSDP training vs vLLM/sglang rollout) — verl-omni re-implements Wan2.2 inside vLLM-Omni and corrects the numerics afterward; miles' headline features (TIS/MIS, bitwise log-probs, R3 routing replay, unified FP8) are all mismatch patches. We answer with one model definition, one kernel set, and an explicit consistency ladder (C0 corrected / C1 kernel-pinned / C2 bitwise, §8.5) — measured by ParityAligner, never assumed. The fragmented config story — FastVideoArgs/TrainingArgs/TrainingConfig/SamplingParam plus the in-flight fastvideo/api/ schema and its 651-line compat.py — collapses into four owned layers with generated views (§6.6). And the design has a shipping product pulling on it (§9): Dreamverse, the vibe-directing app in apps/dreamverse/, today hand-rolls its own GPU pool, queue, warmup, and streaming relay at a cost of one B200 per user session — the engine's first production customer, and (via preference data from directing sessions) the eventual data engine for the RL layer. The same section sets the ComfyUI strategy (§9.4): embed (custom nodes, shipping), compile (a clean-room workflow→PipelineSpec compiler + weight-fleet cache powering an accelerated alternative to running workflows on stock runtimes), and productize (Dreamverse Studio) — sequenced, with the custom-node long tail explicitly out of scope.

The single most important architectural claim: multi-engine DAG composition (vllm-omni / sglang-omni style) is necessary but not sufficient. It handles separable omni models (thinker → talker → vocoder, distinct weights per stage). vllm-omni's bagel_single_stage/lance prove one resident instance can run both loops on shared weights in one request — AR generate_text and diffusion generate_image over co-resident *_moe_gen experts (bagel_transformer.py:718,726; pipeline_bagel.py:677,808). What a DAG of separate engines cannot do is share the weights; what vllm-omni does not do is make those interleaved loops runtime-visible — it buries them in one opaque request-scheduled DIFFUSION stage (RequestScheduler, max_num_running_reqs=1, diffusion_engine.py:166-169), so the scheduler never sees the AR+denoise interleaving. FastVideo's differentiation is therefore not expressing MoT (vllm-omni already does) but making the loops first-class: a runtime where one model instance serves interleaved AR steps and denoise steps that are runtime-visible, step-scheduled, and batchable — and that requirement, not serving features, is what dictates the stage/loop abstraction below. Stated with open eyes: no surveyed system makes runtime-owned diffusion iteration the always-on default at scheduler granularity — there is only narrow, opt-in precedent: vllm-omni's SupportsStepExecution (prepare_encode/denoise_step/step_scheduler/post_decode, diffusion/models/interface.py:44-67) is runtime-owned diffusion iteration at step granularity, but Qwen-Image-only and off in every deploy; this design generalizes it to an always-on universal contract (diffusers' loop blocks, by contrast, own their own iteration; §6.2.3). The risk is retired by Phase-1 bit-identical parity gates and a measured falsifier (§10), not by borrowed validation. The contract is also kept deliberately small: families may ship custom step bodies — the runtime owns iteration, never the factoring of step internals.

Migration is incremental: seven phases (−1 to 5), each independently shippable, with SSIM/parity regression gates and a compatibility adapter so the existing 20+ model families never break. The deprecation commitment is backed by enforcement the prior freeze lacked — the training/-vs-train/ "lesson" is not invoked as a moral but as data: that freeze was broken 19 times including a new legacy pipeline nine days after it was forbidden — so this plan carries CI path gates, CODEOWNERS, named owners, a calendar target for Phase 5, and an explicit inflow rule for the ~1–2 new model families arriving every month (§10).

---

2. Goals and non-goals

Goals

• G1 — One runtime for all generation paradigms: bidirectional diffusion (video/image/audio), causal/AR video with KV cache, few-step distilled, multi-stage refinement, control/world models, AR token decoding (text, audio codecs), and hybrid AR+diffusion (Cosmos3 MoT, BAGEL/Transfusion-style).
• G2 — Omni-native I/O: requests and outputs are typed sets of modality parts (text, image, video, audio, action, latents). Video+audio+text outputs are first-class, not extra-dict passengers.
• G3 — A real serving runtime: async engine, request queue, concurrency, cross-request batching where shapes permit, streaming (tokens, audio chunks, frame chunks), warmup, metrics — at parity with sglang multimodal_gen, which is FastVideo's own descendant.
• G4 — Cosmos3-class MoT support as a first-class citizen: packed factored sequences, dual-pathway attention, per-request hybrid KV+denoise state, reasoner KV cache, chunked world-model rollout.
• G5 — Zero regression for existing models: every shipped pipeline keeps working (SSIM-gated — coverage is completed for all families in Phase −1, since ~7 currently have no SSIM test and would otherwise pass vacuously; substrate-wide PRs in Phases 0–2 run the full GPU matrix on Modal L40S as a budgeted cost, because for exactly those PRs "touched families" means all of them) throughout the migration; the offline VideoGenerator API survives.
• G6 — Keep the hot loop fast: abstractions must resolve at pipeline-build time; the per-step path stays compile/cudagraph-friendly; batch-of-1 video on N GPUs (SP/TP today, CFG-parallel once added) must not get slower.
• G7 — One substrate for inference, training, and RL: models, loaders, configs, schedulers, parallel state, and the generative loop bodies are single-source; train/ and RL consume them in-process or via the engine as a rollout service; the engine never depends on training code (§8).

Non-goals

• N1 — Datacenter orchestration. Multi-node routing, SLA planners/autoscaling, fleet health, and cold-start orchestration are not built here — NVIDIA Dynamo is the named first-class partner for that layer (dynamo/ checkout in repo root; FastVideo already runs as a minimal Dynamo video backend). The engine's job is to be an excellent Dynamo citizen: registration, health/drain, cost metrics, affinity events, NIXL-compatible disagg — the full contract and separation-of-concerns table is §6.3.5. We never schedule across nodes ourselves.
• N2 — Trainer internals. Legacy fastvideo/training/ is frozen and untouched. In fastvideo/train/, everything that makes it a trainer — Trainer, method loss logic, callbacks, FSDP wrapping, optimizers, checkpointing, data loading — is not redesigned. §8 changes only what train/ consumes: its embedded sampling/rollout loops (today private copies) move onto the shared substrate, its distributed:/model config blocks import the shared layers (§8.8), and the RL layer lands under it. Dependency rule: train/ imports the substrate/engine; never the reverse.
• N3 — Replacing the model porting methodology. Bit-exact per-component parity vs official frameworks (the Cosmos3 rules) remains the porting bar; this design changes where ported components plug in, not how they are validated.
• N4 — Migrating all 20+ model families at once. The plan defines a per-family migration checklist; legacy pipelines run unmodified behind an adapter until migrated.
• N5 — Standalone LLM-serving excellence. We host AR computations because omni and interactive models require them — Cosmos3's reasoner, thinkers/talkers, audio-codec decode, causal-video chunk loops — not to compete with vLLM/sglang at LLM serving. The priority order is: omni and diffusion done well — bidirectional, causal, and interactive — first; AR machinery enters only at the sophistication those workloads pull. Concretely: paged KV and continuous batching for AR pathways, yes; radix prefix trees, speculative decoding, and chunked-prefill sophistication, only when an omni workload demonstrates the need (they are flagged as open questions, not commitments); a big separable thinker delegates to an external engine (§6.3.4) rather than pulling its serving problems into ours. This is also the standing guard against over-indexing on LLM-engine borrowings (§5): every vLLM/sglang adoption in this document is admitted under a diffusion/omni justification, not on its LLM merits.

---

3. Where we are today

3.1 Current architecture

VideoGenerator / StreamingVideoGenerator / CLI / OpenAI entrypoints   (fastvideo/entrypoints/)
        │  one ForwardBatch per call (synchronous)
        ▼
Executor ABC ── MultiprocExecutor (1 proc/GPU, SPMD) ── RayDistributedExecutor   (fastvideo/worker/)
        │  collective_rpc / execute_forward
        ▼
ComposedPipelineBase  (fastvideo/pipelines/composed_pipeline_base.py)
   loads modules from model_index.json via ComponentLoaders
   runs a LINEAR list of stages:
        for stage in self._stages: batch = stage(batch, fastvideo_args)
        ▼
PipelineStage ABC  (fastvideo/pipelines/stages/base.py:29)
   __call__ → verify_input → forward(batch: ForwardBatch, args) → verify_output
        ▼
ForwardBatch  (fastvideo/pipelines/pipeline_batch_info.py, 335 lines)
   111 fields: prompts, embeds, latents, timesteps, camera_states, mouse_cond,
   ltx2_*, trajectory…, plus `extra: dict[str, Any]` (line 238) as escape hatch

Parallelism today: TP and SP (Ulysses all-to-all only — ring-attention SP is not wired on main) — CFG-parallel exists in the sglang multimodal_gen fork and in cosmos-framework (CFGP) but not on main; this design adds it as an axis (§6.3.4). Plus per-component CPU/layerwise offload, per-component torch.compile, and a strong attention-backend registry (FA, SDPA, Sage, VSA, V-MoBA, BSA, SLA) selected via fastvideo/attention/selector.py.

3.2 What strains (evidence-backed)

#	Pain point	Evidence
P1	Monolithic denoising loop. A 1,381-line denoising.py (the base DenoisingStage is ~670 lines; 35 isinstance/hasattr/getattr model probes file-wide); 22 model/variant denoising-stage classes across files, 23 with the base (Causal, CausalDMD, LongCat ×3, LTX2, GameCraft, HYWorld, MatrixGame2/3, Gen3C, Cosmos ×5, DMD, SR, SD35, Magi-class, +2 more). Flux gating via getattr(config, "prefix", "") == "Flux"; the Wan2.2 expert switch and TI2V inline VAE encode are hardcoded in the shared loop; Cosmos2.5's per-step clamping lives in its in-file subclass.	fastvideo/pipelines/stages/denoising.py:129,237,352-376,441-475
P2	The loop is owned by the stage, so nothing outside it can schedule, preempt, interleave, stream, or batch steps. Causal/AR models bolt token/chunk loops inside stage subclasses; Cosmos3 had to bypass stages entirely with a free-function denoise() engine.	causal_denoising.py:44,427; worktree cosmos3_stages.py
P3	Blackboard state. ForwardBatch mixes user intent, derived state, and per-model fields (mouse_cond, camera_states, ltx2_*, pose…); multi-modal outputs leave via batch.extra["audio"]. No stage declares what it reads/writes.	pipeline_batch_info.py:238
P4	No runtime. One request per call; assert latent_model_input.shape[0] == 1, "only support batch size 1"; no queue, no async engine, no concurrency, no cross-request batching, no step interleaving. Streaming exists only as a bespoke StreamingTask path for game models.	denoising.py:237; worker/multiproc_executor.py
P5	AR support is per-model folklore. causal_wanvideo.py threads a raw kv_cache: dict through forward (:80,100,143); LongCat has its own LongCatKVCacheInitStage; MatrixGame2 its own; Cosmos3 text reasoning re-prefills the full sequence per token (no cache at all). No allocator, no reuse, no eviction, no paging.	models/dits/causal_wanvideo.py:80-143
P6	No graph, no placement. Multi-stage models (LTX-2 stage-1→upsample→stage-2 refine→audio decode; Hunyuan15+SR) are hand-wired linear lists; all stages run on all ranks; encoders cannot live on different GPUs; no fan-out for multi-modality outputs.	pipelines/basic/ltx2/ltx2_pipeline.py
P7	Task modes are inferred by heuristics. Cosmos3 infers T2I from num_frames==1 and image is None; TI2V/V2V/LucyEdit ride boolean flags through the shared loop.	worktree cosmos3_stages.py:101
P8	Component set is static. Loaders are keyed by fixed module names; Cosmos3's sound VAE had to be lazy-loaded inside forward() behind an env var.	worktree cosmos3_stages.py:262-277

The Cosmos3 port is the best stress test we have: it succeeded (150/150 parity tests, every modality bit-exact), but its friction list (monolithic stage, extra["audio"], order-sensitive flat-latent slicing, no reasoning KV cache, lazy sound VAE, mode heuristics) reads as a requirements document for this design.

---

4. What we must support: the model taxonomy

Every paradigm below is either shipped, in-flight, or explicitly requested (Cosmos 3, omni):

Paradigm	Examples in/around tree	Loop shape	State needed	Output
Bidirectional video diffusion	Wan 2.1/2.2, Hunyuan(15), LongCat, Cosmos 2/2.5, LTX-2, TurboDiffusion	N denoise steps over full clip	latents, CFG branches, step/block caches	video
Few-step distilled	DMD/FastWan, TurboDiffusion, rCM	1–4 denoise steps	latents	video
Causal/AR video	Wan-Causal(-DMD), MatrixGame2-Causal, MatrixGame3, LTX-2 causal, LongCat-VC	outer loop over chunks × inner denoise per chunk	DiT KV cache (per-chunk K/V, sink/local window), rolling latents	video (streamable by chunk)
Interactive world models	MatrixGame, GameCraft, HYWorld, Gen3C, LingBotWorld, Waypoint	chunk loop driven by live user actions	KV cache + action/camera conditioning per chunk	video stream
Image	Flux2, SD3.5, Qwen-Image-class	N denoise steps, small	latents	image (batchable!)
Audio diffusion	Stable Audio	N denoise steps over audio latents	audio latents	audio
Joint A/V diffusion	LTX-2 (video+audio in one denoise), Cosmos3 t2vs	one joint denoise over concatenated modality latents	multi-modality latents, per-modality CFG scales	video + audio
Multi-stage refinement	LTX-2 (base→upsample→refine), Hunyuan15-SR	pipeline of loops	inter-stage latents	video
AR token decode	Cosmos3 reasoner (Qwen-lineage und pathway); omni thinkers/talkers (Qwen2.5/3-Omni-style)	token loop until EOS	paged text KV cache, sampling state	text / codec tokens
Vocoder / one-shot heads	LTX-2 vocoder, audio codec → wav	single forward	none	audio
Hybrid MoT (omni)	Cosmos3 (und/gen dual-pathway, shared attention, joint vision+action+sound denoise, AR reasoner on same weights), BAGEL/Transfusion lineage	AR loop then/while denoise loop, same resident weights	text KV + multimodal denoise state + packed sequence layout, in one request	text + video + audio + action

4.1 The Cosmos3 forcing function (what "support omni" concretely means)

From the official cosmos-framework (cosmos_framework/model/vfm/):

• MoT layer = per-modality expert weights — separate QKV/O projections, norms, and MLPs for the understanding (text, causal) and generation (vision/action/sound, full or NATTEN-sparse) pathways — with shared attention over one packed sequence (unified_mot.py: PackedAttentionMoT, MoTDecoderLayer; gen MLP optionally MoE).
• Packed factored sequence: per request, [text (causal) | vision patches | action | sound] with per-segment attention modes, sample offsets, and 3D-MRoPE position ids (text 1D; vision T/H/W with FPS modulation; sound at fixed 25 latent fps; temporal_modality_margin at boundaries).
• One velocity_fn denoises all generated modalities jointly; CFG = two packed forwards (cond/uncond) with interval gating; sampler pluggable (UniPC/EDM/fixed-step).
• AR reasoner runs on the und pathway of the same weights with a per-layer ReasonerKVCache (prefill + 1-token decode), used standalone (text out) and as prompt upsampling before diffusion in the same request.
• World-model rollout: video_temporal_causal masks + chunked continuation (generate chunk, re-encode, condition, continue).
• Inference batching in the framework is greedy token-budget packing of heterogeneous samples (cosmos_framework/inference/inference.py:66, _iter_packed_batches) — diffusion requests batched by total token count, not by fixed shapes.

A runtime that can serve this can serve everything else in the table. A runtime that composes separate engines per stage cannot serve this (the reasoner and denoiser share weights; splitting them doubles 30B+ of weights and breaks the interleaved KV/denoise state).

---

5. Survey of reference systems

What we take and what we reject, per system. (Full notes from the code review are in the per-repo paths cited.)

System	What it is	Take	Reject / not enough
sglang multimodal_gen (~/sglang/python/sglang/multimodal_gen/) — FastVideo's serving-hardened descendant	HTTP/OpenAI server (images/videos/meshes), rank-0 Scheduler event loop + GPUWorkers over ZMQ, Req (= ForwardBatch + SamplingParams delegation), N:M:K role disaggregation (disaggregation/orchestrator.py: encoder/denoiser/decoder pools, capacity-aware dispatch, request state machine, tensor-transfer with P2P/RDMA), warmup requests, RL weight updates	Server/API shape; scheduler-process topology; role affinity on stages + disagg state machine; warmup; Req w/ sampling-param delegation	Request-level scheduling only (one request per step, no batching/interleave); diffusion-only — no KV cache, no AR loop, no token scheduling; stage/batch abstractions inherited our P1–P3 problems
sglang srt (~/sglang/python/sglang/srt/)	LLM serving: continuous batching (Req/ScheduleBatch, EXTEND/DECODE), radix prefix cache, paged KV (ReqToTokenPool + allocators), CUDA-graph decode, overlap scheduling, PD disaggregation with bootstrap/transfer queues	Concepts: paged KV pool + allocator design, scheduler event loop w/ overlap, chunked prefill, PD-disagg queue pattern, multimodal-embedding cache keyed by content hash	Wholesale embedding: srt is token-centric; its scheduler, radix tree and attention backends assume LLM decode. We borrow designs, not the stack
vllm-omni (~/vllm-omni/, HEAD #4168)	Multi-stage omni serving: frozen declarative PipelineConfig/StagePipelineConfig (stage_id, execution_type ∈ {LLM_AR, LLM_GENERATION, DIFFUSION}, input_sources DAG, final_output_type), YAML deploy (devices/TP/memory/connectors per stage), per-stage engines in own processes (stock vLLM core for AR; DiffusionEngine for DiT, inline-or-remote), OmniConnectorBase put/get + chunk_ready readiness signals (SHM + RDMA via Mooncake/Mori/Yuanrong — not NIXL/NCCL), readiness-set parking + bounded-window-K + preemption for flow control (no credit-based flow control anywhere), global orchestrator + per-stage schedulers, async_chunk streaming	Declarative pipeline spec separate from deploy YAML (the frozen PipelineConfig + editable deploy/<model>.yaml DeployConfig fused by merge_pipeline_deploy — adopted here verified, §6.1); connector interface with readiness signals; inline-vs-remote stage execution; per-stage independent schedulers; streaming-input (resumable) requests; SupportsStepExecution as loop-inversion prior art (opt-in, Qwen-Image-only — we generalize it, §6.2.2); its 3 separate cache subsystems (OmniTensorPrefixCache, OmniKVTransferManager, diffusion/cache TeaCache/MagCache/cache-dit) confirm our per-class-pools thesis (§6.3.2)	vllm-omni does express shared-weight MoT (bagel_single_stage/lance: co-resident *_moe_gen experts, one resident instance running AR generate_text + diffusion generate_image in one request) — but only as one opaque DIFFUSION stage the scheduler never sees inside (RequestScheduler, max_num_running_reqs=1), so the AR+denoise interleaving is invisible to scheduling/batching; cross-stage KV is a TP-rank/CFG-branch-aware copy (OmniKVTransferManager), not a shared live cache; separate-stage weight sharing is absent (correctly); diffusion batching exists but is opt-in, homogeneous-SamplingParamsKey-gated, Qwen-Image-only, and off in every deploy; streaming is linear-chain in practice (routing hardwired src+1, orchestrator.py:1122) though the schema permits a DAG
sglang-omni (~/sglang-omni/)	Extension over stock sglang: StageConfig DAG (factory, gpu, tp_size, process colocation, next/route_fn/wait_for/merge_fn/stream_to), per-stage schedulers (OmniScheduler wrapping sglang by composition, SimpleScheduler, DllmScheduler for the LLaDA2-Uni diffusion-LLM thinker — Ming-Omni's DiT talker runs on a different scheduler — plus streaming variants), Relay transport (NIXL/SHM/NCCL/Mooncake, credit-based flow control), hidden-state capture thinker→talker, segmenter-based streaming	Relay abstraction + credit flow control; fan-in merge_fn / fan-out / stream_to edge semantics; scheduler-composition pattern; hidden-state capture as a declared stage output	Same gap: stages own disjoint weights/caches; hybrid AR+diffusion only as AR stage → DiT stage; per-model bootstrap duplication
vLLM-Omni RFC #4084 — "Composable Parallel Strategies" (issue, open, 2026-06)	Proposal: every parallelism scheme (TP/DP/SP/CP/EP/PP/CFG/VAE-PP/stage replicas) becomes a declarative StrategySpec — mesh_axis (name+degree), L1 routing/aggregation pattern, optional L2 per-layer hooks and L3 kernel factories — stacked into a mesh tree with generic collective-group construction; thin adapter translates onto omni's existing runtime (l1_owner marks who routes each axis, preventing double-routing)	Stacked-axis spec as single source of truth + generic group construction; the L1/L2/L3 intervention vocabulary (we reuse it for both parallelism and the extension system); pre-flight topology validation; CPU-testable composition via a fake process backend; explicit per-axis ownership; fail-loud on unsupported combos instead of silent fallback	Adapter/translation indirection is their constraint (retrofitting vLLM), not ours — we implement the spec natively in fastvideo/distributed/; parallelism-only scope (no loops, caches, or hybrid-model answers); the schemes hardest for us (CP variants, affinity routing) are exactly the parts deferred to follow-up RFCs — design proven, implementation not
verl-omni (~/verl-omni/)	RL post-training for diffusion/omni models (verl lineage): Ray single-controller, vLLM-Omni rollout servers + FSDP2/VeOmni trainers, DanceGRPO/FlowGRPO/DiffusionNFT/Diffusion-DPO recipes (Qwen-Image, SD3.5, Wan2.2), async multi-reward serving, "rollout correction"	Reward-loop architecture (weighted multi-reward, HTTP scorers, async overlap with rollout); SDE-with-logprob scheduler formulation (FlowMatchSDEDiscreteScheduler); component-granular sleep levels; group-relative advantage utils; bypass-vs-decoupled log-prob modes	Pays the full two-runtime tax (§8.4): Wan2.2 re-implemented inside vLLM-Omni for rollout and in diffusers/FSDP2 for training, stitched by rollout_correction.py (IS + rejection masks) — the machinery a single-runtime design exists to delete
miles (~/miles/)	Production LM RL (slime fork, SGLang RL team): SGLang rollout + Megatron training; the deepest extant treatment of train-inference mismatch (bitwise log-probs, TIS/MIS, R3 routing replay, unified FP8, INT4 QAT)	The mismatch toolkit → our consistency ladder C0–C2 + Behavior Record (§8.5); three weight transports behind one interface + per-sample weight_versions staleness (§8.6); the custom-rollout-function contract; health-monitored engines; log-prob-diff debug tooling	Dual model definitions + Megatron↔HF weight-name bridges by construction; Megatron is required at its scale (≥70B MoE) — which confirms our FSDP2 single-runtime bet only holds at FastVideo's 1–30B scale (§8.4 boundary condition)
cosmos-framework (./cosmos-framework/)	Official Cosmos3 (§4.1)	The model-side requirements; token-budget packed batching for diffusion; sampler pluggability; CFG interval gating; CP/CFG-parallel patterns	It is a research framework: no queue, no serving, FSDP-centric; not a runtime to copy
NVIDIA Dynamo (./dynamo/)	Rust/Python fleet orchestration above engines: OpenAI frontend, KV/affinity-aware router (NATS cache events → radix index), prefill/decode disagg coordination with NIXL transfer, SLA Planner (replica autoscaling via K8s CRDs), KVBM multi-tier KV offload, ModelExpress cold-start weight streaming. Already fronts our ecosystem: a FastVideo video worker exists (examples/diffusers/worker.py), plus sglang-diffusion and vllm-omni backends	The first-class partner for everything fleet (§6.3.5): registration/health/drain contract, cost metrics for the Planner, affinity events, NIXL-aligned disagg — production scaling we never build	Rebuilding any of it in-engine; its router keys on token prefixes today and media streaming/cost-models/role-graphs have gaps — addressed as concrete upstream asks A1–A7 (§6.3.6), not workarounds
xDiT / xfuser (~/xDiT)	The de-facto diffusion-parallelism library: USP (sp = ulysses × ring), PipeFusion (displaced patch pipeline-parallelism that reuses stale KV from the previous denoise step — mandatory warmup steps), CFG-parallel (cfg ≤ 2), DistVAE (VAE on its own process group), TeaCache/FBCache adapters; world = dp×cfg×sp×pp×tp with nested group init	Two axis-vocabulary updates (§6.3.4): pp_patch with declared applicability conditions (non-causal only; wins at high resolution / weak interconnect; tolerable for few-step distilled), and the recognition that DistVAE ≡ our decode role pool — no new axis needed; their PipeFusion×SP hybrid rule (KV replication required) as a composition constraint	Wrapper-per-model integration (pipeline + transformer + attention-processor wrappers per family) is exactly the per-model maintenance shape we're eliminating; no serving/scheduler; PipeFusion is architecturally incompatible with causal/AR video (stale KV breaks causality) — a pre-flight validation rule, not a footnote
cosmos-rl (~/cosmos-rl)	NVIDIA's RL framework for physical AI — now deprecated as Cosmos3 unifies the model families: single controller + Redis command queues, POLICY/ROLLOUT replicas (colocated or disagg), pluggable rollouts (vLLM, TRT-LLM, diffusers NFTRollout, in-process WFMRollout), GRPO with AIPO one-sided clipping + dual-clip + KL-threshold off-policy masking, buffer-model async weight sync on a side CUDA stream, elastic HighAvailabilitylNccl mesh rebuild, weight_version in every payload	AIPO + off-policy masking into the §8.7 objectives (C0-mode stability kit); the buffer-model async-sync pattern (§8.6); per-payload weight versioning (independently validates our staleness design); elastic-mesh fault tolerance as the recovery reference	Dual model definitions per role (weight mappers, per-role parallelize) — the two-runtime tax again; and no production video rollout engine (NFTRollout is generic diffusers) — confirmation that the §8 rollout engine is the missing piece in this space, not a duplicate of an existing one (our own landed DiffusionNFT, PR #1450, confirms it from the other side: trainer-side library code sampling through a vendored bare-model loop, §8.1 — still no serving-grade video rollout engine anywhere)
diffusers Modular Pipelines + Guiders (~/diffusers, pulled to head)	HF's blessed re-architecture of pipelines: block composition (Sequential/Conditional/Auto), LoopSequentialPipelineBlocks (marked experimental), untyped PipelineState/BlockState, ComponentSpec + ComponentsManager (auto-offload), modular_model_index.json hub format, and Guiders (CFG/PAG/APG/SkipLayer… with start/stop step scheduling); 12+ families incl. Wan, LTX, HunyuanVideo1.5, Flux2	Convergent validation of the policy/components pillars only: Guiders are exactly pluggable CFG policies, and ComponentSpec matches our ModelSpec components — adopt compatible serialization and a BaseGuidance adapter (§6.2.3, §6.6). It is not loop inversion: LoopSequentialPipelineBlocks.__call__ raises NotImplementedError; every concrete family hand-writes for i, t in enumerate(timesteps) inside its own wrapper (wan/denoise.py:434, sdxl/denoise.py:701 — SDXL ships four such wrappers, the subclass forest again). Iteration is block-owned and invisible to any runtime — equally consistent with the alternative this design rejects ("keep loops in stages, make bodies pluggable")	No serving, batching, scheduler, AR loops, or parallelism; untyped blackboard state (their version of our P3); class-defined blocks vs our serializable graph IR — export is clean, import is lossy
torchtitan (~/torchtitan)	PyTorch-native pretraining reference: ParallelDims building multi-axis DeviceMesh (pp/dp_replicate/dp_shard/cp/tp/ep) with product-of-degrees pre-flight validation and cached submeshes; FSDP2 per-block + MoE prefetch; selective activation checkpointing by FQN; async DCP checkpointing; per-block compile; per-mesh-dim deterministic seeding; set_batch_invariance() utilities; an RL experiment (PolicyTrainer → vLLM generator) syncing weights via TorchStore push/pull with direct RDMA	The substrate answer for §6.3.4: compile the stacked-axis spec to DeviceMesh via a ParallelDims-style builder; TorchStore/DCP as the PyTorch-ecosystem weight-sync precedent for §8.6 (caveat: structure-matched fill, not automatic cross-mesh resharding — WeightSyncPlan still owns layout math); trainer patterns for train/ (selective AC, async ckpt, determinism utils); upstream batch-invariance feeding C2 (§8.5)	LLM-pretraining-shaped; no real inference engine (Forge is minimal); full-DTensor everywhere is overkill for engine hot paths — adopt the pattern, not the stack
LiveKit agents / WebRTC (~/agents)	Realtime AI agents over a WebRTC SFU: rtc.VideoSource.capture_frame() raw-frame publishing (RGBA/I420; encoding in a native thread — non-blocking to asyncio), AVSynchronizer, data channels + named RPCs for low-latency control (interrupts, turn-taking), avatar VideoGenerator protocol	The §9 transport trigger matrix: fMP4/WS suffices for segment streaming; WebRTC is mandatory for interactive world models (<100 ms motion-to-photon ≈ 20–40 ms action RTT + ≤40 ms/frame inference + ~10 ms encode/decode), with data channels as the action path; engine contract: OmniEvent optionally carries raw frames + PTS, sessions accept an async action stream + flush RPC	Real operational weight (SFU, TURN, tokens) — adopt only when triggers fire; frames must be CPU-side buffers (GPU→CPU copy on the hot path)
vLLM core — RFC #42770 "Changes in vLLM Model Development" (issue, Woosuk Kwon, 2026-05, open; ~/vllm pulled to head)	A philosophical reversal by the largest engine: drop the full-graph torch.compile requirement (manual "big fused ops" with hardware dispatch written directly in model code; local compile kept for adjacent-op fusion; breakable CUDA graphs — the SGLang technique, prototyped in PR #42304 — replacing compiler-managed piecewise capture; forward_context named as a compiler-compat hack to remove); per-vendor model code for head models; an explicit Model/ModelConfig interface where models own their state and prepare_inputs (model runner V2's ModelState) and config decouples from HF (#24384). After community pushback (the compile lead's detailed dissent; OOT vendors' CustomOp concerns), the RFC settled on a two-tier resolution: hand-fused vendor-specific head models + shared portable definitions for the long tail	Validations and one technique: model-owned state + model-agnostic runner converges with this design (diffusers Modular converges on the components/policy half — not the runner, §6.2.3); HF-decoupled ModelConfig ≈ our ModelSpec manifest; the two-tier head/tail resolution ≈ our flagship-vs-family split (§6.3.3); and breakable CUDA graphs as the capture mechanism for loop step bodies with interceptor branches. Plus a direct work item: FastVideo's fastvideo/forward_context.py is inherited from exactly the hack vLLM is deleting — staged retirement starting Phase 1 (shim until Phase 5, §6.3.3). From the v1 code itself (beyond the RFC): the unified-progress scheduling structure — adapted to a cost-model currency, since bidirectional denoise steps and causal AR tokens are incommensurable work units (§6.3.1) — plus in-step encoder budgeting, the KVCacheGroups hybrid allocator studied and not adopted (its single BlockPool requires uniform page bytes across groups — get_uniform_page_size — which our 150–500× KV-granularity spread violates; we keep per-class budgeted pools, §6.3.2), EncoderCacheManager semantics for feature caches, KVConnectorBase_V1 as the connector protocol shape (§6.3.4), Punica batched multi-LoRA evaluated for the workflow cloud (adopted only for fixed-adapter fast modes — its one-id/baked-scale semantics cannot express ComfyUI's stacked-LoRAs-with-strengths, §9.4), CuMemAllocator tag-based sleep (§8.6), and the persistent-batch/staged-write/delta-output CPU discipline behind the ≤2% step budget	The dissent is worth keeping in view: manual fusions don't scale across quant schemes/hardware, and per-fused-op kernel selection pressure re-creates IR-like autotuning ("starts to look very similar to IROps") — arguments for keeping our policy/registry layer rather than going fully manual; vLLM core remains LLM-scoped (diffusion/omni stays in vllm-omni) — and per N5, every borrowing from this repo is admitted on a diffusion/omni justification, never on LLM-serving merits

---

6. Proposed architecture

6.0 Overview

┌────────────────────────────── Request plane ───────────────────────────────┐
│  OpenAI-compatible server (videos/images/audio/chat, SSE/WebSocket)        │
│  VideoGenerator (offline sync API, unchanged)   AsyncEngine (new)          │
│            OmniRequest ──► admission ──► request queue ──► OmniOutput      │
└─────────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────── Pipeline plane ──────────────────────────────┐
│  PipelineSpec (declarative graph, per model family)                        │
│    nodes: Stage (one-shot) | LoopStage (DenoiseLoop, ARDecodeLoop)         │
│    edges: typed Artifacts (Embeddings, Latents[modality], Tokens, KVRef…)  │
│    policies: CFGPolicy, ExpertRouting, AttnMetadata, Precision, FlowShift  │
│    components: shared module registry (one resident copy per worker pool)  │
└─────────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────── Execution plane ─────────────────────────────┐
│  Engine scheduler (per-request, graph-node dispatch, role pools)           │
│  StepScheduler (per worker pool: multiplex denoise steps + AR steps)       │
│  Worker pool(s): SPMD procs, TP/SP groups (as today) + CFG-parallel (new)  │
│  CacheManager: paged text-KV ▪ chunked causal-video KV ▪ feature caches    │
│  Connectors (in-proc → SHM → NCCL/NIXL) for cross-pool edges               │
│  Attention backends, torch.compile / cudagraph capture per (node, bucket)  │
└─────────────────────────────────────────────────────────────────────────────┘

Default deployment is exactly today's: one SPMD pool, all graph nodes co-located, synchronous offline call. Every serving feature is additive configuration, not a different code path for models. Two cross-cutting surfaces thread through all planes: the hook/plugin extension system (§6.4) and the declarative parallelism spec (§6.3.4).

6.1 Request plane: typed omni I/O

@dataclass
class OmniRequest:
    request_id: str
    task: TaskType                    # explicit: T2V, I2V, TI2V, V2V, V2W, T2I, T2A, T2VS, A2W, REASON, ...
    inputs: list[ModalPart]           # TextPart | ImagePart | VideoPart | AudioPart | ActionPart | LatentPart
    sampling: SamplingParams          # AR knobs: max_tokens, temperature, top_p, stop, seed
    diffusion: DiffusionParams        # steps, guidance (per-modality scales), sigmas/shift, resolution, frames
    outputs: OutputSpec               # requested modalities + streaming flags + return_latents/trajectory
    node_params: dict[str, NodeOverrides] = field(default_factory=dict)
                                      # per-graph-node overrides, validated against the PipelineSpec
    priority: int = 0

@dataclass
class OmniOutput:
    request_id: str
    artifacts: dict[str, Artifact]    # "video": VideoArtifact, "audio": AudioArtifact(sample_rate=…),
                                      # "text": TextArtifact, "action": TensorArtifact, "latents": …
    metrics: RequestMetrics

Decisions this encodes:

• Task is declared, never inferred (kills P7). Heuristics may suggest a default at the API boundary, but the pipeline graph branches on request.task.
• Outputs are named artifacts (kills the extra["audio"] pattern, P3). Audio carries its sample rate; text carries token ids + decoded string; everything carries provenance (which node produced it).
• Streaming is part of the request: outputs.stream = {"text": True, "audio": chunk_ms=200, "video": per_chunk}. The engine returns an AsyncIterator[OmniEvent] when any stream flag is set.
• Multi-loop graphs are parameterized per node — decided here, in Phase 0, because external consumers build against this schema first: sampling/diffusion are defaults; node_params["refine"].steps or node_params["talker"].max_tokens override per graph node, validated against the node-parameter schema each PipelineSpec declares. This is what LTX-2's refine loop needs (its own step count and guidance scale are first-class fields today, fastvideo_args.py:204-205) and what any thinker/talker pair needs; the rejected alternative — per-model fields on the universal schema — is exactly the ltx2_* leakage P3 indicts.
• VideoGenerator.generate_video(prompt=…) remains and constructs an OmniRequest internally — the offline API is a thin shim, preserved verbatim for G5.

6.2 Pipeline plane

6.2.1 State: replace the god-batch with Request / RequestState / typed edges

ForwardBatch conflated three things. We split them:

• OmniRequest — immutable user intent (above).
• RequestState — per-request mutable execution state, owned by the runtime: artifact slots keyed by graph edge, loop states, cache handles, RNG state, per-request metrics. A small typed core (latents: dict[Modality, Tensor], cond: ConditioningSet, timesteps, step_idx) plus a per-model extension dataclass registered by the pipeline (e.g. Cosmos3State(packed: PackedSeq, reasoner_kv: KVHandle, …), MatrixGameState(mouse, keyboard, kv: KVHandle)).
• Edges carry typed Artifacts (Embeddings, ImageLatents, VideoLatents, AudioLatents, TokenIds, HiddenStates, KVRef, ActionTensor). Stages declare input/output artifact types; the graph validates at build time — which fields a stage reads/writes stops being archaeology (P3, P6).

Trade-off (discussed in §11.5): full typed dataflow everywhere is verbose; a pure blackboard is unverifiable. The hybrid — typed edges between nodes + a typed per-model state extension inside nodes — is the deliberate middle. ForwardBatch survives during migration as an adapter view over RequestState for unmigrated stages.

6.2.2 Stages and loop inversion (the core change)

class Stage(Protocol):                 # one-shot transforms: encoders, VAE en/decode, packers, vocoders
    def forward(self, ctx: StageContext, state: RequestState) -> None: ...

class LoopStage(Protocol):             # iterative computations: the runtime owns the loop
    def init(self, ctx, state) -> LoopState: ...
    def step(self, ctx, loop: LoopState) -> StepResult: ...   # ONE step: one solver step / one token / one chunk
    def finalize(self, ctx, loop: LoopState) -> None: ...

@dataclass
class StepResult:
    done: bool
    emit: list[Artifact] | None = None   # streamed chunks: tokens, audio chunks, decoded frame chunks

Two LoopStage families cover every paradigm in §4:

• DenoiseLoop — init builds timesteps/sigmas, allocates latents, resolves policies; step performs one solver step (model forward(s) + CFG combine + scheduler step). Sub-variants by composition, not subclassing: joint multi-modality latents (LTX-2, Cosmos3 t2vs), chunked-causal (step = one denoise step of the current chunk; chunk advance handled by an outer ChunkRollout loop node holding the KV handle), few-step distilled (timesteps from config).
• ARDecodeLoop — init runs prefill into a KVHandle; step decodes one token (or one speculative bundle), samples, appends, optionally emits; done on EOS/max_tokens. Used by Cosmos3 reasoner, omni thinkers/talkers, audio-codec generators.

Why inversion is the keystone:

1. Scheduling: only the runtime sees step boundaries, so only inversion enables continuous batching of AR steps, interleaving denoise steps of concurrent requests, preemption between steps, and fair sharing between a 50-step video job and a 4-step image job (P2, P4).
2. Streaming falls out: StepResult.emit is the universal chunk channel (text tokens, talker codec chunks, causal-video decoded chunks) instead of three bespoke mechanisms (P4).
3. Hybrid models become a graph of loops over shared components: Cosmos3 = ARDecodeLoop(reasoner) → Pack → DenoiseLoop(joint) → decoders — same transformer module bound to both loop nodes (G4).
4. Checkpointing/elasticity: a LoopState is a serializable resume point (future: migration between workers, crash recovery for 1000-step jobs).

Cost: the loop body must be re-entrant and the per-step dispatch must stay cheap (§12, risk 2). DenoiseLoop.step for batch-of-1 video compiles/captures exactly like today's inner loop body; nothing tensor-level changes.

Note the contract's deliberate smallness — this is the design's main simplicity lever: the runtime requires only init/step/finalize. How a family implements step is free — composed from policies by default, or a custom body when the math is genuinely entangled (§6.2.3). Loop inversion is about who owns iteration, not about how step bodies are factored; everything the scheduler, streamer, and parity harness need comes from the small contract, never from the factoring.

6.2.3 Policies: dissolving the isinstance forest

By default, a model-conditional currently probed inside DenoisingStage becomes a small strategy object resolved once at pipeline build and injected into the loop (kills P1 without 22 subclasses) — default, not admission requirement: families whose step bodies braid across slot boundaries keep custom steps under the escape hatch below:

Policy	Replaces	Examples
CFGPolicy	chunked cond/uncond logic, adaptive-gate delta reuse, LongCat batched-CFG, per-modality scales, Cosmos interval gating, embedded-guidance (Flux ×1000)	ClassicCFG, BatchedCFG, AdaptiveGateCFG, PerModalityCFG, EmbeddedGuidance (cfg-parallel is a parallelism axis, not a policy — see below)
ExpertRouting	Wan2.2 boundary_ratio transformer switch	BoundaryTimestepRouting(transformer, transformer_2)
AttnMetadataProvider	VSA / V-MoBA / STA builder isinstance chain in the loop	registered per backend; built in init, refreshed per step only if needed
PrecisionPolicy	prefix == "Flux" autocast hacks, scheduler_step_in_fp32, cast flags	declared in model config
FlowShiftPolicy	Cosmos3 resolution-bucket shift, per-task sigmas	config-driven lookup table
ConditioningInjector	camera/action/pose/audio conditioning plumbed through batch fields	per-model injectors that write into RequestState.cond; the loop is agnostic

The step skeleton and the policy contract. A three-phase step (forward → CFG → scheduler step) does not factor the shipped loops; they need more slots, with dependencies that cross policy boundaries. The default step body is a fixed skeleton of ordered, typed slots:

prepare_inputs → [per guidance branch: pre_forward → forward → post_forward] → combine
              → scheduler_step → post_step → inter_step

with three rules. (1) Binding vs state. Policy bindings resolve at pipeline build (which classes, which slots); policy state is allocated per-request in LoopState at init — the same scoping rule as plugins, because the same failure applies: AdaptiveGateCFG's cached delta (denoising.py:338-343, 507-551) is per-request mutable state, and a build-time singleton would smear request A's CFG delta into request B the moment interleaving lands. (2) Observation. Policies read a typed per-step StepContext (timestep, branch, active expert/model id, sampler coefficients) — the channel by which AdaptiveGateCFG observes ExpertRouting's switch to invalidate its delta (today an inline id(model) check), and by which the guidance branch vocabulary is policy-declared per request at init (plugin per-branch state scopes over the declared set — LTX2's 1–4 runtime-decided passes declare theirs up front, not a fixed cond/uncond pair). (3) Slots cover the common shapes, not all shapes — verified against shipped code: TI2V's post-scheduler.step latent clamp is a post_step policy (denoising.py:570-573); CausalDMD's between-step renoising with expert-boundary-dependent noise selection is inter_step × ExpertRouting (causal_denoising.py:268-301); Cosmos2.5's per-frame timestep vectors and per-step GT re-clamp are prepare_inputs/pre_forward. But Cosmos's conditioning-frame injection consumes the sampler's EDM coefficients inside each CFG branch with an x0-space combine (denoising.py:845-933), and LTX2's guidance passes alter the network via forward kwargs (ltx2_denoising.py:503-605) — braids no slot factoring owns cleanly.

CFG-as-policy is proven by precedent, not speculative. vllm-omni's CFGParallelMixin already unifies sequential-2-forward, batched, and cfg-parallel under one pair — predict_noise_maybe_with_cfg + combine_cfg_noise (vllm_omni/diffusion/.../cfg_parallel.py:76-212) — so one shared denoise-step body plus a swappable CFGPolicy expressing classic (2-forward), batched (1-forward stack+chunk), adaptive-gate (cached-delta reuse with model-id invalidation), per-modality, and embedded-guidance (single-branch) is an existence proof, not a hope. The right reading of vllm-omni's "three CFG mechanisms" is one unified in-loop pair + one parallelism axis + one orchestrator layer, not three independent CFG systems. We make that boundary explicit in three layers: (1) CFGPolicy is in-loop — it owns the branch vocabulary, the combine formula, and per-request mutable state (the adaptive-gate cached delta is the canonical state case); batched-vs-2-forward is a dispatch detail inside one policy, not a separate mechanism — which is why there is no standalone BatchedCFG axis to collide with parallelism. (2) cfg-parallel is a parallelism axis — it shards policy.predict's branches across ranks and runs the same rank-invariant combine on every rank, and composes under any policy; a request never owns both a batched-CFG policy variant and a cfg parallel group (declaring both is rejected, §6.3.4). (3) companions are an orchestrator pattern — a request is split into companion sub-requests upstream of diffusion, and the loop receives already-computed conditioning unchanged. Two caveats the slots must respect: the combine runs in the step body's numeric space (Cosmos combines in x0-space post-EDM, not noise-space — denoising.py:845-933); and embedded-guidance/Flux is a degenerate single-branch identity-combine policy (guidance rides inside the forward kwarg), not "no CFG".

Which is why the scheduler's contract is deliberately smaller than the policy system: a family may ship a custom step body — a hand-written function using samplers, CFG math, and conditioning utilities as a library (the Cosmos3 port's denoise-engine pattern, legitimized). A custom step is still a LoopStage: scheduled, budgeted, preempted, streamed, observed, and parity-gated exactly like a composed one, and the skeleton-owned plumbing (offload dance, attention metadata, autocast, trajectory capture) is provided either way. Policy factoring is the default because it deletes duplication; it is not an admission requirement. So the honest form of the claim: a new model contributes at minimum a graph spec and a step body — policies where they fit — and never edits shared loop code.

External validation, stated at its true scope: diffusers' Modular Pipelines independently converged on the policy pillar — Guiders (CFG, PAG, APG, SkipLayer…, each with start/stop step scheduling) are exactly pluggable CFG policies, and CFGPolicy stays adapter-compatible with their BaseGuidance interface so model ports and guidance research transfer in both directions (§6.6). It does not validate loop inversion: their loop blocks own their own iteration, invisible to any runtime (see §5). The honest position: there is only narrow, opt-in precedent — vllm-omni's SupportsStepExecution (prepare_encode/denoise_step/step_scheduler/ post_decode, diffusion/models/interface.py:44-67) is runtime-owned diffusion iteration at step granularity, but Qwen-Image-only and off in every deploy; no surveyed system makes it the always-on default. This design generalizes that contract to a universal, always-on one (diffusers' loop blocks, by contrast, own their own iteration), and the Phase-1 bit-identical parity gates are what carry that risk (risk 3).

6.2.4 PipelineSpec: declarative graphs

Frozen, validated, registered per model family (the vllm-omni PipelineConfig shape, extended with loop nodes, branches, and fan-out):

COSMOS3_OMNI = PipelineSpec(
    family="cosmos3",
    components={"transformer": …, "vae": …, "sound_vae": Component(optional_for={…}, required_for={"t2vs"}),
                "vision_encoder": …, "tokenizer": …},          # declared, eagerly loadable per task (kills P8)
    nodes=[
        Node("validate",  InputValidation()),
        Node("tokenize",  Cosmos3Tokenize()),                                    # text → TokenIds
        Node("vis_cond",  VAEEncodeConditioning(), when=task_in(I2V, V2W)),      # image/video → clean latents
        Node("reason",    ARDecodeLoop(model="transformer", pathway="und"),      # prompt upsampling / REASON
                          when=task_in(REASON) | opt("upsample_prompt")),
        Node("pack",      Cosmos3Pack()),                                        # → PackedSeq artifact
        Node("denoise",   DenoiseLoop(model="transformer",                       # SAME module as "reason"
                          modalities=[VISION, ACTION?, SOUND?],
                          cfg=PerModalityCFG(interval_gated=True), sampler=FromConfig())),
        Node("vdec",      VAEDecode("vae")),                                     # fan-out ↓
        Node("adec",      AudioVAEDecode("sound_vae"), when=task_in(T2VS)),
        Node("assemble",  AssembleOutput()),                                     # → OmniOutput artifacts
    ],
    edges=[…typed…],
)

Properties:

• Branches on request.task and request options replace geometry heuristics (P7).
• Fan-out/fan-in are real: video and audio decode in parallel after a joint denoise; thinker hidden-states can feed both a text detokenizer and a talker (sglang-omni wait_for/merge_fn semantics).
• Components are shared by reference. Two loop nodes binding "transformer" get the same resident module — this is the MoT requirement, impossible in engine-per-stage designs.
• Linear pipelines are the degenerate case; the existing stage lists translate mechanically.
• A separate deploy config (YAML) maps nodes → pools/devices/parallelism (vllm-omni's split of frozen pipeline vs deployment), defaulting to "one pool, everything colocated".

6.3 Execution plane

6.3.1 Engine and two-level scheduling

Level 1 — engine scheduler (per request, per graph node). Async intake → admission (validate, resolve PipelineSpec, allocate request state) → dispatch nodes to worker pools as their inputs become ready; a request state machine generalizing multimodal_gen's RequestTracker (waiting → node-running → node-done → …), with role-pool capacity dispatch (N:M:K encode/denoise/decode) available when configured.

Level 2 — StepScheduler (per worker pool). The new capability. Each pool runs an event loop over ready loop steps:

while True:
    work = select(ready_loops, policy)        # rank0 decides, broadcast to pool (SPMD-consistent, as today)
    ├─ AR group:       batch decode steps of all active ARDecodeLoops (continuous batching;
    │                  paged KV; chunked prefill for long prompts/vision contexts)
    ├─ Denoise group:  pick one (or a shape-compatible batch of) DenoiseLoop step(s)
    └─ interleave by priority/deadline; preemption only at step boundaries

Batching semantics are paradigm-aware (this is where "one scheduler" earns its keep):

• AR steps: classic continuous batching (sglang-style), batched across requests every iteration. CUDA-graph decode per batch-size bucket.
• Image diffusion: cross-request batching with resolution buckets — the workload where batching pays most (Flux2/SD3.5 serving).
• Video diffusion: batch-of-1 with TP/SP (plus the new CFG-parallel axis) stays the default (memory-bound); concurrency comes from interleaving steps of multiple requests (a 4-step distilled job no longer queues behind a 50-step job) and, later, Cosmos-framework-style token-budget packing of heterogeneous samples as an opt-in.
• Hybrid (Cosmos3): reasoner decode steps join the AR group; the joint denoise joins the denoise group; both hit the same resident weights, so the StepScheduler is also the mode multiplexer the MoT model needs. The multiplexer needs a parallelism answer too, because the two loop types want disjoint layouts — denoise wants SP(+CFG); AR decode is sequence-length-1, where SP has nothing to shard and CFG doesn't exist. The stated answer, rather than a deferred contradiction: on a [cfg × sp] pool, AR decode runs data-parallel across the cfg×sp weight-replica axes (every replica already holds full weights; the decode batch shards across them, paged text-KV partitioned per replica — not duplicated), optionally over per-pathway TP subgroups (§6.3.3); and pure-REASON traffic is routable to differently-shaped pools via cost classes and affinity routing, so reasoner token latency is not gated by indivisible denoise steps except when a single request genuinely interleaves both on the same weights. The Phase-4 gate measures pool efficiency (reasoner tokens/s/GPU at target denoise throughput), not just the re-prefill strawman.

Mechanics adopted from the vLLM v1 scheduler (vllm/v1/core/sched/scheduler.py — production-proven, and a cleaner formulation than our group split above):

• Unified progress accounting — but the budget currency is cost, not counts. v1 has no prefill/decode phases: every request is num_computed_tokens catching up to a target under a per-iteration token budget, and that works because a token maps near-linearly to work in the LLM regime. A step does not. A bidirectional denoise step re-attends over the entire latent sequence at O(L²) with zero KV amortization — every step pays full price — while AR prefill is causal-masked incremental work against a cache (O(n·C) for n new tokens on context C), a decode step is ~O(C) per layer, and a chunked-causal video step sits between (one chunk vs cached context). Counting "steps" would put items three orders of magnitude apart in one bucket. So we keep v1's accounting structure — steps_done/steps_target progress counters, one admission/budget loop — but the budget is predicted GPU-time from a per-(model, phase, shape) cost model: each schedulable item (a denoise step at given L and CFG mode, an AR prefill chunk, a decode token-batch, an encoder forward) converts to the common currency before admission. The cost model is policy-aware (batched-CFG ≈ 2× batch vs 2 forwards; VSA/ NATTEN sparsity changes the attention term; cache-dit changes the block count) and it is the same cost model we publish to Dynamo's Planner and router (ask A2, §6.3.6) — one model, two consumers, calibrated online by the Profiler observer.
• Encoder work budgeted inside the same step — scoped to what diffusion actually needs. v1 interleaves encoder forwards with decode because VLM prefill demands it. Our shape is different: for diffusion, encoding is a one-shot prologue per request, so the value here is narrower and specific — waiting requests' encode work fills budget headroom alongside running requests' loop steps on the same pool (hiding encode latency under denoise), with the full interleaved pattern reserved for omni workloads that genuinely stream multimodal context mid-request. We take the budgeting mechanism, not the VLM workload assumption.
• CPU discipline, applied where it bites. v1's machinery (persistent InputBatch state allocated once with sliced H2D copies, staged writes flushed once per step, delta-only scheduler outputs, async dispatch via Futures) exists because LLM decode steps are ~10 ms and CPU time is the bottleneck. Our pressure is uneven: it bites for the AR decode group and batched image steps (short, frequent) — adopt fully there — while video denoise steps are 50–500 ms+ and tolerate a simpler dispatch path; we take the async-Future overlap everywhere but do not pre-build LLM-grade batch plumbing for loops that don't need it.
• Same-step speculative contract. Draft tokens ride the scheduler output and are verified by rejection sampling within the same step — the proven shape for ARDecodeLoop multi-token bundles (§12, open question 10).

Two properties make cost-currency scheduling easier here than vLLM's problem, and one makes it harder. Easier: diffusion baseline costs are static and known at admission — (resolution, frames, steps, CFG mode, model) determine per-step cost, flat across steps (no growing context) — with the honesty caveat the doc owes itself: admission always uses the conservative baseline, because the design's own flagship features make realized cost runtime-dependent (cache-dit skip decisions are residual comparisons unknowable in advance; VSA tile selection is content-dependent; AR decode lengths are unbounded — budgeted at the max_tokens cap with refund on early EOS). Runtime telemetry refines calibration and utilization accounting; it never licenses admission optimism. Harder: a denoise step is indivisible, so the largest step bounds iteration latency — one 30 s-1080p step can block an interactive Dreamverse segment regardless of budgets. Mitigations, in order: don't co-schedule jumbo steps with latency-class work (cost classes map to pools — which model-affinity pooling §9.4 wants anyway); shrink jumbo step wall-time with SP within a node (pools are single-node in v1, §6.3.4 — SP degree caps at intra-node GPU count, stated rather than implied); and admission-time SLO classes so the Planner (§6.3.5) scales pools per class instead of one pool absorbing both.

Failure isolation, cancellation, and memory admission — the soundness conditions of multiplexing. Step-multiplexing changes the failure class categorically: today, one request per pool means one request's CUDA error is its own problem; on a shared pool, a mid-step illegal access poisons the CUDA context and any in-flight collectives for every co-scheduled tenant, including resident session caches. This is designed with Phase 2, not after it:

• Cancellation is first-class and common-path. Vibe directing makes abandoning in-flight generation the normal user action, not an edge case. Cancel takes effect at the next step boundary: queued steps drop, LoopState and cache handles release, partial artifacts report with explicit cancelled status. The flush/interrupt RPC (§9.1) is this mechanism, not a separate one.
• Failure classification. Request-fatal (admission/step allocation failure, NaN flagged by NaNWatch, errors attributable to one request's step): abort that request via an SPMD-consistent abort broadcast — the exact dual of the rank-0-decides scheduling broadcast — release its state; fan-out graphs deliver the named artifacts completed so far plus a structured error (partial-artifact semantics). Pool-fatal (illegal memory access, NCCL desync/timeout, context corruption): pool re-init — kill and restart workers (the sentinel-fd worker-death watch is inherited from Dreamverse's gpu_pool.py:542-586 and explicitly preserved, so Phase 2 is not a reliability regression for the flagship customer), re-establish collectives, invalidate all pool-resident caches (sessions re-prefill from durable inputs; requests resume from serialized LoopState where one exists, else requeue), and report through the Dynamo health surface (§6.3.5). vLLM v1 needed exactly this machinery (EngineDeadError, abort_requests); we build it at the same layer.
• Memory is the second budget axis. vLLM's token budget is half its admission story; the other half is allocate_slots in the same loop. We keep both halves: every schedulable item carries (predicted time, resident bytes, peak-activation bytes) — latents, conditioning sets, CFG duplicates, and activation peaks live in RequestState outside the CacheManager, so an admission planner bounds Σ resident + worst-case peak (profiled per model × shape, vLLM-style) against the pool budget. Two requests that fit individually but jointly OOM are rejected at admission, not discovered at step 37.
• Preemption semantics, by state size. "Preemption only at step boundaries" now says what happens to the loser's multi-GB state: small resident states stay on-GPU; medium offload to pinned host memory; large or idle drop and resume from serialized LoopState — with the re-encode/re-prefill economics priced by the cost model (different economics from token-KV recompute, stated rather than assumed).

The non-serving path bypasses level 1 entirely: VideoGenerator submits one request and drives the pool to completion synchronously — preserving today's latency profile and debuggability (G5, G6).

6.3.2 CacheManager (unifying P5)

One manager, handle-based. First, the proportionality note (N5): KV is the minority case here — bidirectional diffusion, the majority of today's traffic, allocates no KV at all, so the KV machinery below is lazy: pools materialize only when a PipelineSpec declares KV-bearing loops (AR pathways, causal video, omni), and a pure bidirectional deployment carries none of it.

For that KV-bearing minority, adversarial review caught this design importing a property vLLM does not have: the single-BlockPool/no-fragmentation guarantee exists because physical bytes-per-block are uniform across all groups — kv_cache_utils.py asserts a single page size (get_uniform_page_size), its docstring calls breaking this "non-trivial due to memory fragmentation concerns", and groups differ only in tokens-per-block at equal byte size. Our classes differ by 150–500× in natural granularity (a text-KV page ≈ 64 KB/layer; a causal-Wan latent-chunk slab is 9.6–32 MB/layer), and the demand classes are workload-decoupled (text-KV and chunk-KV vary independently with request mix), so one unified pool either strands slab-sized memory under token traffic or degenerates into static partitioning anyway — which is exactly what vLLM's one multi-page-size path (DeepseekV4) does at startup. Decision — simpler and honest: separate per-class pools with static budgets. Each cache class gets a budget carved at pool init from the deploy config (cost-model defaults, driven by what the PipelineSpec declares): the text-KV class runs a vLLM-style paged block allocator; the chunk-KV class runs a slab allocator (one latent-frame chunk per slab). Both sit behind the same KVHandle interface. The fragmentation/deadlock argument: classes are statically partitioned, so cross-class fragmentation is impossible by construction; within a class, granularity is uniform, so the standard arguments apply; rebalancing budgets is a pool-drain operation — rare (class demand follows deployment workload mix, not per-request noise) but not free: a drain is priced by the cost model like any quiescing barrier (§6.3.2's weight-transition pricing), and a slab budget shrink must wait out or evict session-resident slab caches, so rebalancing is a scheduled operation, not a hot-path one. (This is the original two-pool sketch returning with the justification it lacked — the single-pool detour was built on a misread of vLLM.) Our cache classes:

• Paged text KV (paged class): used by ARDecodeLoop (Cosmos3 reasoner — replacing the re-prefill-per-token loop; omni thinkers/talkers). No radix prefix tree in v1 of our build; block-hash prefix reuse later (the allocator design accommodates it).
• Chunked causal-video KV (slab class): per-DiT-layer K/V for latent-frame chunks with sink/local-window semantics — the formalization of causal_wanvideo.py's ad-hoc dicts (Wan-Causal, MatrixGame2/3, LongCat-VC, Cosmos3 world-model rollout). vLLM's ChunkedLocalAttentionSpec / SlidingWindowSpec managers (out-of-window recycling, per-request admission caps) are the reference semantics, re-expressed over slabs. A declared training mode disables mid-rollout recycling and keeps grad-aware index snapshots (the wan_causal.py:119-120,405-431 behavior, §8.2) — the inference recycling rules are not silently applied under autograd. MoT models fall out naturally: the und pathway draws from the paged class, the gen pathway from the slab class or nothing — independent budgets, no interference.
• Feature caches: text-embedding cache (content-hash keyed; reused across CFG branches, retries, and requests), vision-encoder/deepstack cache (Cosmos3 image-conditioned reasoning), cache-dit (DBCache/FBCache) residual caches scoped per LoopState. The embedding/encoder caches adopt vLLM's EncoderCacheManager design (mm_hash-keyed, reference-counted with a freeable FIFO, budget-aware allocation). Two invalidation regimes, not one: RL update_weights bumps a weight epoch that invalidates wholesale; but workflow-cloud traffic patches text encoders per request (ComfyUI LoRAs carry lora_te* keys and a separate strength_clip), so the cache key is partitioned, not flushed: content-hash × encoder identity × adapter-set hash (ids + strengths) × weight epoch. Without the adapter component, two workflows sharing a prompt but differing in te-LoRA stacks would silently serve each other stale embeddings — in the exact product whose trust claim is reproducibility (§9.4).
• Weight/adapter cache (Phase 3+, pulled by multi-tenant workflow execution, §9.4): disk→CPU→GPU checkpoint and LoRA tiers, LRU, content-hash-keyed ModelSpec resolution, fast patch/unpatch with stacked-LoRA semantics — and model-affinity scheduling in the engine (route requests to pools where the checkpoint is already resident, the diffusion analog of prefix-affinity routing). Crucially, weight state is schedulable state: components are one resident copy per pool, so a patch/unpatch transition is a pool-quiescing barrier — drain in-flight steps, apply/undo W += scale·BA shard-consistently across ranks (undo via CPU master-copy restore or 2× backup, chosen per deploy memory budget), re-admit. Two interleaved loops requiring different patch states cannot coexist on one pool; the StepScheduler therefore treats transitions as scheduled, costed operations — the cost model carries a weight-transition term, and affinity scheduling exists precisely to minimize how often it is paid (a quiesce-vs-queue policy decides whether a lone divergent request waits or forces the barrier). Until this lands, the engine serves homogeneous model families per pool.

Handles (KVHandle, FeatureRef) live in RequestState; lifetimes are tied to the request (or to an explicit session for interactive world models, where the chunk KV must persist across many requests of one game session).

6.3.3 Model layer: MoT and packed sequences as first-class citizens

To make Cosmos3-class ports native rather than heroic (per §4.1):

• PackedSeq core type: modality segments, per-segment attention mode (causal / full / sparse-NATTEN / temporal-causal), sample offsets, position-id planes (3D-MRoPE with FPS modulation), cond masks. Pack/unpack utilities replace order-sensitive flat-latent slicing (one of the documented Cosmos3 port frictions, §3.2/P3).
• Dual/multi-pathway layer conventions in fastvideo/layers/: MultiPathwayAttention (per-pathway QKV/O + shared attention over PackedSeq), PathwayMLP (dense or MoE per pathway), modality-routing by segment — TP sharding defined per pathway. The Cosmos3 DiT port refactors onto these; future MoT/Transfusion-lineage models reuse them.
• Attention backend contract extended to accept per-segment modes/masks, so backends (FA varlen, flex, NATTEN) plug into MoT models the same way they plug into plain DiTs.
• Sampler registry (UniPC, EDM, Euler-FM, fixed-step distilled) with strict config validation that rejects or normalizes incompatible checkpoint scheduler configs (the karras-sigmas NaN incident becomes a load-time error, not a black video).

Compile & graph position (informed by vLLM RFC #42770 and its discussion). The largest engine just reversed its compiler-first philosophy: full-graph torch.compile is being dropped in favor of manual fused ops in model code, local compile for adjacent-op fusion, and breakable CUDA graphs (the SGLang technique, vLLM PR #42304 — piecewise capture with no Dynamo/Inductor involvement). Our position, stated explicitly:

• Never full-graph across the engine. Per-component/per-block compile where it pays (today's per-component flags, kept); manual fused ops are permitted in model code — the RFC's lesson — rather than forced through shared abstractions.
• Breakable CUDA graphs for loop step bodies — as an optimization tier, not a foundation. The technique lets cache-dit skip decisions and policy branches become break points between captured segments instead of graph-invalidating recompiles, and video's batch-1 fixed shapes are the friendly case for capture. But it is marked experimental in vLLM's own tree (vllm/envs.py:724), and interceptors × CFG branches × expert routing × observers-needing-eager multiply graph variants. The contract therefore is: the eager path is the always-correct baseline; graphs are captured for a deliberately small matrix (per model, shape-bucket, branch-profile), and interceptor-heavy configurations may simply run eager with the cost surfaced in metrics. The ≤2% Phase-2 latency budget is gated on the Phase-1 loop-inversion overhead measurement, not on breakable graphs maturing upstream.
• Two tiers, like vLLM's post-discussion resolution: flagship (recipe, runtime) pairs (a FastWan/LTX-2 NVFP4 build on B200) may carry hand-fused, hardware-specific module variants, swapped in per DeployConfig without touching the PipelineSpec; the 20+-family tail stays on shared layers + policies. The dissent in that RFC is our guardrail for the tail: manual fusion doesn't scale across quant schemes and hardware, which is exactly what the policy/registry layer is for.
• Retire fastvideo/forward_context.py — staged, not deleted in place. It is inherited from precisely the global the RFC names as a compiler-compat hack, but today it is also the live attention-metadata transport for fastvideo/attention/layer.py:93 and the frozen legacy training/ stack (training_pipeline.py:405, distillation_pipeline.py:672) — deleting it in Phase 1 would violate N2's frozen boundary. Staging: Phase 1 removes it from the migrated inference path (new loops carry StageContext + AttnMetadataProvider + model-owned RequestState extensions — the same "model owns its state" conclusion vLLM's Model/ModelState interface reaches), while the module itself survives as a compatibility shim fed by the new context, so attention layers and every unmigrated/legacy caller are untouched; the file is deleted in Phase 5 with the legacy paths.

What a media engine would feed back to RFC #42770's Model interface (we have standing to comment — our experience is the evidence): (1) prepare_inputs/forward assume one phase per request; diffusion needs multi-phase requests (encode once, then N denoise forwards) with memoized cross-step state (text embeddings reused across all steps) — a small contract change that would let vllm-omni's diffusion stages implement the same interface; (2) formalize encoder-output caching in the interface (their EncoderCacheManager exists but sits outside the Model contract); (3) keep the interface tensor-type-agnostic (the vllm-metal request) so non-torch runners can implement it.

6.3.4 Placement, connectors, and external engines

• Default: one SPMD pool; node→pool mapping trivial; "connector" is an in-proc handoff (zero overhead). Pools are single-node in v1 — a load-bearing decision made explicitly: the engine never owns a cross-node NCCL mesh inside one pool (no cross-node collective bring-up, no NCCL-timeout fault domains, no multi-node drain protocol); multi-node scale is multiple pools fronted by Dynamo (§6.3.5). Consequences accepted: SP degree ≤ intra-node GPUs, which bounds the jumbo-step mitigation (§6.3.1) and caps single-pool model scale — the revisit trigger is a model or resolution that needs more. RayDistributedExecutor, today's multi-node path, is frozen with the legacy adapter and retires at Phase 5; its use case is superseded by Dynamo-fronted pools.
• Role pools (opt-in): encode / denoise / decode pools with capacity dispatch — port of multimodal_gen's disaggregation, now expressible per graph node rather than per hardcoded role.
• Per-node placement (opt-in): deploy YAML pins nodes to devices/pools (vision encoder on its own GPU; vocoder on CPU); cross-pool edges select a Connector (transport: SHM/Mooncake/Mori/Yuanrong today, with NIXL as a target — not "vllm-omni's NIXL/NCCL", which it does not use). The chunk_ready readiness signal is adopted from vllm-omni's OmniConnectorBase; credit-based flow control is sglang-omni's Relay design, not vllm-omni's (vllm-omni uses readiness-set parking, no credits) — so credit-flow here is a FastVideo addition over vllm-omni's readiness model, composed with sglang-omni's proven design. For KV/cache-bearing edges specifically, the connector speaks a protocol shaped like vLLM's KVConnectorBase_V1 (scheduler-side query/alloc/finish roles + worker-side async start_load/wait_for_layer/save with layer-pipelined loading and invalid_block_ids fault reporting) — the same interface NIXL/LMCache/Mooncake already implement, which is also what makes Dynamo ask A4 (§6.3.6) an extension of an existing contract rather than a new one. Adopt vllm-omni's TP-rank-aware + CFG-branch-aware KV payload slicing (OmniKVTransferManager build_rank_aware_send_keys/recv_keys): a shared-weight MoT split across parallel groups must slice KV by rank and by CFG branch on the wire, not ship a monolithic blob.
• External-engine node (opt-in, later): an ARDecodeLoop-shaped adapter that delegates a separable AR stage (a 235B thinker) to a colocated sglang/vLLM instance over the connector interface. Useful for separable omni models; never required for MoT models (which must stay native).
• Declarative parallelism (adapted from vLLM-Omni RFC #4084). Pool meshes and engine topology are described by one stacked axis spec — e.g. parallel: [dp(2), cfg(2), sp(4), tp(2)] per pool — with collective-group membership computed generically (the RFC's mesh-tree construction) instead of hand-wired per combination in fastvideo/distributed/parallel_state.py. We adopt three properties outright: (1) pre-flight validation — rank counts, group membership, and ownership conflicts are build errors; each axis records who realizes it (e.g. CFG branches can be realized by a batching CFGPolicy (1-forward stack+chunk) or a cfg parallel group — declaring both is rejected, the RFC's l1_owner / no-double-routing rule applied to our policy-vs-axis split); (2) fail-loud on unsupported combinations rather than silent fallback; (3) CPU-testability — engine scheduler, graph dispatch, and placement planning run against a fake worker pool so topology logic is unit-tested without GPUs (FastVideo CI is GPU-starved; this is how the composition layer stays tested). The RFC's L1/L2/L3 intervention vocabulary — batch routing / per-layer behavior / kernels — maps cleanly onto this design (L1 = engine + StepScheduler, L2 = TP/SP/pathway-sharded layers, L3 = the attention-backend registry) and is reused for the extension system below. We implement the spec natively rather than depending on composable_parallel (§11.11).
• Axis vocabulary v1 + substrate (informed by xDiT and torchtitan). Axes: dp, cfg (≤2), sp = ulysses × ring, tp, and pp_patch — xDiT's PipeFusion displaced-patch pipelining, which exploits diffusion's step-to-step redundancy by reusing stale KV from the previous denoise step (mandatory warmup steps; wins at high resolution and on weak interconnects; tolerable for few-step distilled models) and is invalid for causal/AR models — an applicability condition the pre-flight validator enforces per PipelineSpec, since stale KV breaks causality. xDiT's DistVAE needs no new axis: it is our decode role pool. Composition constraints travel with the axes (e.g., pp_patch × sp requires KV replication inside attention). Implementation substrate: the spec compiles to PyTorch DeviceMesh via a ParallelDims-style builder (torchtitan's pattern — product-of-degrees validation, cached submeshes); the trainer consumes the mesh natively (FSDP2/DTensor), while engine hot paths keep raw process groups where DTensor indirection isn't earned.

6.3.5 Production scale: Dynamo as the first-class fleet partner

Everything above this line is a single-deployment engine: one pool (or a few colocated role pools) on one node-group. Truly scaling — multi-node fleets, SLA-driven autoscaling, cross-node routing, cold starts, fleet fault-recovery — is a different discipline, and the design's position is that FastVideo never builds it: NVIDIA Dynamo (./dynamo/) is the named first-class partner for that layer. This is not aspirational — the integration already exists in embryo: dynamo/examples/diffusers/worker.py registers FastVideo with register_llm(ModelInput.Text, ModelType.Videos, …) and serves VideoGenerator through @dynamo_endpoint (today behind an asyncio.Lock, one request at a time — precisely the limitation Phase 2 removes), and Dynamo already fronts our sibling stacks (sglang's diffusion worker via components/src/dynamo/sglang/init_diffusion.py, vllm-omni under components/src/dynamo/vllm/omni/). First-class means hardening that contract, not inventing one.

Separation of concerns — the hard line:

Dynamo owns (FastVideo must not build)	FastVideo engine owns (Dynamo does not reach into)
Service discovery + model registration (etcd/NATS, K8s)	model execution, loops, policies, samplers
Cross-node request routing (KV/affinity/load-aware)	StepScheduler: step interleaving, AR batching, in-pool placement
SLA Planner: replica autoscaling (per-pool, prefill/decode ratios, K8s CRDs)	caches (paged KV, chunk KV, feature, weight fleet) and their in-node lifecycle
Fleet health, canaries, request migration, graceful-drain orchestration	consistency contract (C0–C2), parity gating, plugins
OpenAI-compatible fleet frontend + preprocessing	the per-pool API surface and OmniEvent streaming semantics
KVBM multi-tier token-KV offload; ModelExpress weight streaming (cold starts)	which weights/caches are worth offloading, and when

The contract (what "first-class backend" concretely requires of us — Phase 2/3 work items):

1. Worker surface (Phase 2): the Dynamo worker wraps AsyncEngine instead of a locked VideoGenerator — registration with ModelType.Videos | Images (+ Chat once omni lands), worker_type mapping our roles onto theirs (Aggregated; Prefill/Decode for disagg), streaming handler (our OmniEvent chunks → their annotated stream), health-check endpoint, graceful drain (finish in-flight loops, release pools) wired to DistributedRuntime's signal handling.
2. Metrics for routing + Planner (Phase 2): queue depth, pool utilization, and — important for video — per-request cost estimates (a 5 s 480p distilled job and a 30 s 1080p job differ by orders of magnitude; the Planner's SLA math needs the cost model we can derive from PipelineSpec × resolution × steps; LLM backends get this implicitly from token counts, we must publish it). This is the same cost model the StepScheduler uses as its internal budget currency (§6.3.1) — built once, consumed in-pool and at fleet level.
3. Affinity events (Phase 3, with the weight-fleet cache): Dynamo's KV-aware routing rests on engines publishing cache events (lib/kv-router/src/protocols.rs:627-646: KvCacheEventData::Stored{parent_hash, blocks[{block_hash, tokens_hash}]} / Removed / Cleared) into a radix index. The index is hash-generic, but today's router derives keys from token prefixes — our two affinity signals (checkpoint residency for the workflow cloud §9.4, session residency for Dreamverse §9.1) need either a custom router policy (Dynamo exposes the router entrypoint for this) or an upstream extension — formalized as ask A1 (§6.3.6).
4. Disagg alignment (Phase 3): our encode/denoise/decode role pools map onto Dynamo's prefill/decode coordination — the worker returns disaggregated_params in its terminal chunk and transfers tensors over NIXL, which is already in our connector menu (§6.3.4); cross-node disagg then comes from contract compatibility, not new infrastructure.

What this buys, concretely: multi-node serving without writing a router; SLA autoscaling that makes the $5-subscription workflow cloud's unit economics work (engine-level multiplexing × fleet-level bin-packing); ModelExpress weight streaming for fast cold-starts — which composes directly with the weight-fleet cache (§9.4) when checkpoints migrate across the fleet; KVBM's GPU→CPU→SSD tiering pattern as the template for offloading idle Dreamverse session caches (sessions are idle-heavy between user directions); and production ops (operator, observability, fault tolerance) we never staff. Dreamverse's planned local-first controller/ (Runpod/Modal provisioning) remains the single-box story; at fleet scale it delegates to Dynamo rather than growing its own orchestration.

6.3.6 Asks of Dynamo (the partnership is two-way)

We have a direct working relationship with the Dynamo team, so Dynamo is not treated as frozen: where the ecosystem design improves by changing Dynamo rather than working around it, we say so. Each ask below states the gap, the proposed upstream change, who benefits beyond FastVideo (these are ecosystem gaps — vllm-omni, sglang-diffusion, and every media/RL backend hit the same walls), and our fallback if it doesn't land. Roughly in priority order:

• A1 — Generic affinity routing (key spaces beyond token prefixes). Gap: the KV router's radix index is hash-generic, but keys are derived from request token prefixes — useless for our affinity signals. Ask: generalize the event protocol (lib/kv-router/src/protocols.rs) to namespaced affinity events — {key_space: "checkpoint" | "session" | "lora_set" | "weight_version" | "kv_prefix", key_hash, resident, capacity_hint} — with router policies pluggable per key space. Beneficiaries: any diffusion/workflow backend (checkpoint/LoRA residency — the §9.4 fleet cache), any session workload (world models, omni chat, Dreamverse), and LLM serving itself (LoRA-set affinity). Fallback: custom router policy via Dynamo's router entrypoint; worst case, least-loaded routing + engine-side redirects.
• A2 — Heterogeneous request cost interface. Gap: router load-accounting and Planner profiles infer cost from token counts; a 5 s 480p distilled clip and a 30 s 1080p clip differ by orders of magnitude with identical "prompt lengths." Ask: a standard per-request cost annotation (predicted GPU-seconds / memory class) that engines publish and the router + Planner consume, with Planner profiling keyed on engine-declared cost features (model, resolution, steps) instead of tokens. Beneficiaries: every media backend Dynamo fronts. Fallback: encode cost classes as separate registered model names (crude bucketing).
• A3 — Streaming media through the fleet frontend. Gap: ModelType.Videos today returns one base64 MP4 blob (the examples/diffusers/worker.py pattern); Dreamverse-class UX needs chunked fMP4/audio segments and progress events end-to-end. Ask: first-class chunked-binary streaming (segments + progress + keep-alive) through the frontend for media model types, mapping 1:1 onto our OmniEvent channel. Fallback: bypass the frontend for streams (direct WebSocket to pools) and use Dynamo for routing/placement only — workable but forfeits unified auth/observability.
• A4 — Role-graph disaggregation (N roles, not two). Gap: disagg coordination is hardwired to the prefill→decode pair; our role pools are encode→denoise→decode (and omni graphs fan out further). The disaggregated_params payload is already opaque, so the protocol nearly exists. Ask: generalize PrefillRouter into a role router over engine-declared role graphs with NIXL hand-offs per edge. Beneficiaries: vllm-omni and sglang-omni each hand-built exactly this orchestration (§5) — Dynamo could own the pattern once for the whole ecosystem. Fallback: keep multi-role inside one Dynamo-visible worker (our engine-internal pools, §6.3.4) and expose only aggregated workers to the fleet.
• A5 — The RL weight plane: versioned update broadcast + staleness-aware routing. Gap: ModelExpress streams weights for cold starts; RL needs hot incremental updates — trainer publishes version N+1, the rollout fleet converges, and the router knows who's on what. Ask: (a) ModelExpress incremental in-place weight-update broadcast to a live worker set with per-worker weight_version tracking; (b) version/staleness-aware routing (route rollouts to ≥version-K workers; drain stale ones) — A1's weight_version key space; (c) a Planner objective plugin for train↔rollout GPU elasticity (scale the rollout pool against trajectory-queue depth and the staleness bound rather than serving SLAs). Beneficiaries: every RL stack that fronts rollout with Dynamo — this is miles' weight-transport + weight_versions machinery (§8.6) offered as fleet infrastructure. Fallback: our own transports (§8.6) with Dynamo unaware of versions; works colocated, weakens fleet-scale async RL.
• A6 — KVBM generalization to engine-defined cache objects. Gap: KVBM tiers token-KV blocks; our expensive idle state is latent session caches and chunk-KV (Dreamverse sessions between directions, world-model games). Ask: a generic cache-object API (register/pin/unpin, tier hints, eviction callbacks) so engine caches ride KVBM's GPU→CPU→SSD machinery under fleet policy. Fallback: engine-side offload within the CacheManager (single-node only).
• A7 — Session lifecycle as a frontend/router primitive. Gap: no session concept at the fleet layer; ours (Dreamverse scenes, interactive world models, multi-turn omni) need create/attach/expire semantics pinned to session-resident workers. Ask: session IDs as a first-class routing primitive (pairs with A1's session key space + A6 for idle offload). Fallback: sticky routing by consistent hash at our API gateway (sglang-omni's pattern), losing fleet-level visibility.

Sequencing with our phases: A2 + A3 matter by Phase 2 (Dynamo-grade worker + Dreamverse streaming); A1 + A4 by Phase 3 (fleet cache + disagg); A5–A7 by Phase 4 (RL hardening + session serving). Each ask is independently useful to Dynamo's existing backends — that's the test we apply before asking.

6.4 Extension system: hooks and plugins (cross-cutting)

Requirement: drop-in optimization plugins — cache-dit (DBCache, FBCache, TaylorSeer calibration) and successors — and drop-in instrumentation for debugging, profiling, and numerical-precision alignment, without forking loop or model code per technique. Seeds already in-tree: fastvideo/hooks/ (ModuleHookManager/ForwardHook, plus activation_trace.py — zero-overhead-when-off, regex-selected module taps with a trace_step() step-index context, built for port-parity debugging), and a cautionary tale: enable_teacache is plumbed through the entire API surface (pipeline_batch_info.py:182, OpenAI protocol, sampling params) yet has no consumer in the shared denoise loop — bolting optimizations into a monolithic loop is exactly what didn't ship. The extension system makes such techniques additive.

Two tiers with different contracts:

Tier 1 — Observers (read-only). An event bus with stable, named hook points:

Plane	Hook points
Request	request_start/end, artifact_emitted, schedule_decision
Pipeline	node_start/end, loop_init/finalize, step_start/end (step idx, timestep, state ref)
Execution	model_forward pre/post, module-level forward taps by name regex (existing ModuleHookManager), cache events (alloc/hit/evict), connector transfers

Contract: observers cannot mutate state; the chain is assembled at pipeline build, so an unused hook point is literally absent from the hot path (the activation-trace "zero overhead when off" rule, generalized); each observer declares needs_eager, and the runtime drops compile/cudagraph capture only for the scopes it watches, logging the cost. Built-ins shipped with the runtime:

• ActivationTrace — the existing module, promoted from env-var global to a per-request-scoped observer (env-var UX preserved).
• ParityAligner — formalizes the Cosmos3 oracle methodology (N3) as infrastructure: record mode dumps named taps per step/block from a reference run (official framework, or pre-change FastVideo); compare mode replays a request with fixed seeds and reports the first divergence beyond per-tap tolerances. This is the engine behind the Phase-1 "old loop vs new loop, bit-identical" gate, and the standing tool for every future port and precision change (FP8/FP4 work included). The training side already runs the coarse version of this pattern in CI: PR #1396 pins per-method, per-GPU-model layer-0 grad-norm references at 10% tolerance (fastvideo/tests/train/methods/grad_norm_regression.py — the causal-DFSFT refs differ ~9% between L40S and GB200 while the finetune refs differ only ~0.1%: cross-device reduction order is a real and method-dependent consistency variable, §8.5); ParityAligner is that idea taken to per-tap, per-step resolution.
• Profiler — per node/step/block wall+CUDA timings, NVTX ranges, on-demand torch.profiler windows, memory watermarks; exported through engine metrics.
• NaNWatch — first-NaN/Inf localization (module × step) for debugging.

Tier 2 — Interceptors (compute-altering). Explicit extension points — never monkeypatching (§11.10):

class StepInterceptor(Protocol):      # step-skip / cached-prediction schemes
    def before_step(self, loop: LoopState) -> StepOverride | None: ...  # may supply a predicted output, skipping the forward
    def after_step(self, loop: LoopState, out) -> None: ...             # update residual/calibration state

class BlockInterceptor(Protocol):     # cache-dit class: FBCache, DBCache, TaylorSeer
    def plan(self, loop: LoopState) -> BlockPlan: ...                   # which transformer blocks run vs skip this forward
    def on_skipped_block(self, idx, h) -> torch.Tensor: ...             # substitute cached / Taylor-extrapolated residual

• Block interception needs one model-side convention: in-tree DiTs execute their block stack through a run_blocks(self.blocks, h, ctx) helper — a plain loop when nothing is bound (compile-friendly) — instead of inlined for loops. Honest cost accounting: this is a per-family refactor, not a rename — today's DiTs inline divergent block loops with different arguments and gradient-checkpointing paths (wanvideo.py:712, flux_2.py:1057, ltx2.py:2648, causal_wanvideo.py:560). Adoption is per-family with ParityAligner gates (Phase 1 scope = the Phase-1 families, Wan + Flux2); a family that hasn't converted simply rejects block-level plugins at build time via capability negotiation — step-level interceptors still work everywhere.
• State scoping is the correctness fix: interceptor state lives in LoopState.plugin_state[plugin_id], keyed per request and per CFG branch. Reference implementations (cache-dit on diffusers, TeaCache forks) keep residual state in module-level globals, which silently corrupts under concurrent requests — here concurrency-safety is structural, which is what makes these optimizations usable in the serving engine at all.
• Build-time composition: ordered chain; conflicting interceptors (two block-skippers) rejected pre-flight — the same validation stance as the parallelism spec; per-model compatibility declared in the PipelineSpec (a 4-step distilled model rejects step-skip caches rather than producing garbage).
• Graph contract: interceptors declare graph_safe; graph-safe ones express skip decisions as captured tensor ops or bucketed graph variants; others force eager for their scope only.

cache-dit is the reference integration target. It is the library sglang's multimodal serving already uses (multimodal_gen/runtime/cache/cache_dit_integration.py: enable_cache_on_transformer / enable_cache_on_dual_transformer built on cache-dit's BlockAdapter + DBCacheConfig / TaylorSeerCalibratorConfig, configured by a cache_dit_config server arg). Two scars in that integration are instructive: sglang must monkeypatch CachedContextManager.similarity to behave correctly on sequence-parallel-sharded tensors, and must manually call refresh_context_on_transformer per request. Both needs are structural in our interceptor contract — the BlockInterceptor adapter sees the parallel groups (so residual comparison is group-aware by construction) and its state is LoopState-scoped (so per-request/per-branch lifecycle is automatic, no manual refresh). The Wan2.2 dual-expert case (sglang's dual_transformer path) composes with our ExpertRouting policy: one cache context per expert. We consume cache-dit as a dependency — its configs, calibrators, and skip math — behind our extension point, keeping algorithm parity with the sglang fork.

Packaging — and the trust boundary. A fastvideo.plugins entry-point group + registry: third-party packages (the cache-dit adapter, a vendor profiler) register without forking. Enablement is DeployConfig-scoped only — entry points resolve at deploy load, never per request. The reasons are load-bearing in a multi-tenant engine: plugins execute arbitrary code, and since the OpenAI protocol is generated from the request schema, a per-request plugins=[...] field would derive third-party-code selection straight into the public API. Requests therefore only parameterize pre-enabled plugins through per-plugin validated schemas (diffusion.plugin_params={"cache_dit": {"Fn": 8, "Bn": 8}}), with two guards: exact-mode requests reject distribution_altering parameterization outright (the §9.4 trust claim), and per-request plugin overhead — including a needs_eager observer de-capturing scopes shared with co-scheduled tenants — is attributed in that request's cost metrics, so noisy neighbors are visible and chargeable. Hook-point names are a versioned public surface.

In RFC #4084's vocabulary: observers and interceptors are the L1/L2 intervention levels applied to computation shape rather than parallelism (step/loop ≈ L1, block/layer ≈ L2), and L3 remains the attention/kernel backend registry. One taxonomy, three uses — parallelism, optimization, instrumentation.

6.5 Entrypoints and compatibility

• VideoGenerator / StreamingVideoGenerator / CLI: unchanged signatures, now building OmniRequests.
• New AsyncEngine (generate(request) -> OmniOutput | AsyncIterator[OmniEvent]).
• OpenAI-compatible server extends the existing fastvideo/entrypoints/openai/: /v1/videos/generations, /v1/images/generations, /v1/audio/speech, /v1/chat/completions (omni: text+media in, text+media out — lands with Phase 4), SSE/WebSocket streaming. API-shape parity with multimodal_gen is an explicit goal so a future upstream/downstream story (§11.1) stays open.
• Workflow frontend (Phase 3): the ComfyUI-workflow compiler (§9.4) is a peer entrypoint — workflow JSON parses into PipelineSpec + OmniRequests and enters the same admission path as every other client.
• Legacy adapter: LegacyPipelineNode wraps any existing ComposedPipelineBase as a single opaque graph node (request → ForwardBatch shim → stages → artifacts). Every unmigrated model runs day one.

6.6 Model loading, configs, and the typed API

The user is mid-migration to a typed API (fastvideo/api/) and has flagged it — and config specification and model loading generally — as open for redesign. Today there are seven overlapping config surfaces: arch configs (fastvideo/configs/models/, with param_names_mapping — good, unchanged); pipeline configs (fastvideo/configs/pipelines/); FastVideoArgs (81 fields) and its legacy subclass TrainingArgs (90 own + 81 inherited = 171); the new-trainer YAML TrainingConfig (whose distributed: block re-declares the same parallelism fields); legacy SamplingParam (75 fields, now at fastvideo/api/sampling_param.py); and the in-flight typed schema (fastvideo/api/schema.py: GenerationRequest, GeneratorConfig, ServerConfig, ParallelismConfig, OffloadConfig, CompileConfig, InputConfig, OutputConfig, …). The seams are load-bearing code: fastvideo/api/compat.py is 651 lines of bidirectional translation (request_to_sampling_param, generator_config_to_fastvideo_args, legacy_generate_call_to_request, dotted-path explicit-field tracking via request_metadata.py), and per-model knobs leak into the universal surface as side files (api/matrixgame2.py, api/matrixgame3.py).

The in-flight work points the right way — typed request, generator config, and streaming event types (api/results.py: VideoProgressEvent/VideoPartialEvent/VideoFinalEvent prefigure OmniEvent). Its problem is that it sits beside the legacy surfaces, translating, rather than replacing them. Target: four layers, one owner each, everything else derived:

Layer	Owns	Absorbs / retires
ArchConfig	model structure + param_names_mapping	(unchanged)
ModelSpec + PipelineSpec (§6.2.4)	components and weights, graph, policies, samplers	pipeline configs' behavior knobs; registry name-detectors; model_index.json implicitness
DeployConfig	placement, stacked parallelism axes (§6.3.4), memory/offload, compile, plugins	the runtime thirds of FastVideoArgs and GeneratorConfig; TrainingConfig.distributed
OmniRequest (§6.1)	everything per-call, incl. per-model ModelOptions blocks	GenerationRequest (evolves into it), SamplingParam, the request half of ForwardBatch

Derived views are generated, never hand-maintained: the OpenAI protocol models, CLI flags, and presets (api/presets.py becomes per-model default OmniRequest fragments) all derive from the request schema; per-model options are registered typed blocks rather than universal-schema fields (the api/matrixgame2.py pattern, formalized). compat.py is migration scaffolding, but its retirement must be honest about who depends on it: VideoGenerator itself currently routes through generator_config_to_fastvideo_args / legacy_generate_call_to_request / request_to_sampling_param (entrypoints/video_generator.py:31), and the doc promises those public signatures stay. So the policy is: Phase 0 freezes the file (no new entries, ever); each phase deletes the translations its migrations obsolete (the SamplingParam/ForwardBatch bridges die as families migrate); what legitimately remains until Phase 5 is the thin, generated legacy-signature shim — and at Phase 5 the file reaches zero lines. Monotonic shrink per phase is the tracked metric, not a Phase-1 cliff.

Model loading. Replace implicit model_index.json + name-detector resolution with an explicit ModelSpec manifest per checkpoint family: declared components with required_for/optional_for task sets (the Cosmos3 lazy-sound-VAE fix, P8), accepted weight layouts (diffusers-style, native, quantized variants), dtype/precision policy, and the default PipelineSpec binding. Component loaders stay as they are (TransformerLoader, VAELoader, … unchanged); resolution becomes manifest-first, with today's detectors as fallback for unmanifested HF checkpoints. The same manifest is what train/ ModelBase.init_from consumes — one loading path for inference, training roles (student/teacher/critic), and RL rollout (§8).

Hub interchange (diffusers Modular). ModelSpec component entries serialize compatibly with diffusers' ComponentSpec / modular_model_index.json (the modular-repo hub format), and guidance configs map onto their Guider classes — so a modular-diffusers port and a FastVideo PipelineSpec share vocabulary, models published either way load both ways, and HF remains the distribution rail for (recipe, runtime) artifacts (see the artifact-registry known gap, §12). The same shape is appearing in vLLM core (RFC #42770): an HF-decoupled ModelConfig plus a Model interface where the model owns its state and input preparation — the third independent arrival at manifest-first loading and model-owned state, after diffusers Modular and this design.

Schema stability tiers. By Phase 3, at least four serialized surfaces have external consumers: ModelSpec manifests (hub interchange, format co-owned with diffusers), compiled-workflow PipelineSpecs (content-hash-keyed in the weight-fleet cache — a schema change silently changes hashes and invalidates fleet affinity), the OmniEvent stream (Dreamverse's frontend; proposed as Dynamo ask A3's wire format), and per-node parameter schemas. Every one carries a schema_version; each is experimental until the phase its first external consumer ships (OmniEvent: Phase 2; ModelSpec + workflow specs: Phase 3), then stable-with-a-deprecation-window (one minor release); workflow-spec hashes include the schema version so affinity invalidation is explicit, never silent. The migration plan gates behavior at every phase — this is the corresponding gate for interfaces.

---

7. Worked examples

(a) Wan 2.2 T2V (today's bread and butter). Graph: validate → text-encode → DenoiseLoop(transformer, cfg=AdaptiveGateCFG, routing=BoundaryTimestepRouting, attn=VSAProvider) → vae-decode. Single pool, SP as today (CFG-parallel available once the axis lands). Behavior and perf identical to today; the boundary-switch and VSA hacks become declared policies.

(b) Wan-Causal streaming T2V. ChunkRollout outer loop holding a ChunkKVPool handle; inner DenoiseLoop(chunk); each finished chunk → incremental VAE decode → emit → SSE frames. The same graph with a session-scoped KV handle serves MatrixGame-style interactive world models (actions arrive as streaming-input updates, vllm-omni-style resumable requests).

(c) LTX-2 two-stage A/V. validate → text-encode → DenoiseLoop(joint video+audio latents) — likely a custom step body under the §6.2.3 escape hatch, since LTX2's 1–4 guidance passes alter the network via forward kwargs, a braid the slot skeleton doesn't own cleanly; per-modality CFG scales and STG remain policy-shaped if the factoring proves out → upsample → DenoiseLoop(refine, lora-swapped) → fan-out {vae-decode video, audio-vae+vocoder} → assemble (video+audio artifacts). Today's 11 hand-ordered stages become a graph with one fork either way — the graph and loop contract don't care which way the step body lands.

(d) Qwen-Omni-style thinker→talker→vocoder (separable omni). Three nodes: ARDecodeLoop(thinker, emit=hidden_states) →(stream)→ ARDecodeLoop(talker, emit=codec_chunks) →(stream)→ Vocoder. Single pool or per-node placement; async_chunk-style streaming via StepResult.emit + connector readiness. Optionally the thinker delegates to an external engine node.

(e) Cosmos3 full omni (the forcing function). The §6.2.4 spec: optional ARDecodeLoop(reasoner) with paged KV (replacing re-prefill-per-token) → pack → joint DenoiseLoop(vision[+action][+sound]) → fan-out decoders → assemble {video, audio, action, text}. One resident MoT transformer bound to both loops; REASON task short-circuits to text-only; world-model rollout = ChunkRollout with temporal-causal masks and clean-latent chunk KV.

(f) Flux2 / Stable Audio serving. Same DenoiseLoop with image/audio latents; StepScheduler batches shape-bucketed requests; embedded-guidance and precision quirks are policies. FastVideo becomes a credible image/audio server for free.

(g) DiffusionNFT RL (landed: PR #1450). The landed method is already library-mode shaped — rollouts run in-process under the trainer's own kernels — but against a vendored loop: DiffusionSampler walks the full 25-step ODE (or re-noising sde_reflow) schedule over the frozen old-policy copy, conditional-only, attn_kind="dense" pinned, and the objective is likelihood-free: only final clean latents survive the rollout; training re-noises them at shuffled schedule timesteps and contrasts the student against an implicit negative policy in prediction space, so per-step log-probs exist nowhere in the method. Under this design the same method drives the shared DenoiseLoop with a sampler policy and OutputSpec(return_final_latents=True) — deleting rl/common/sampling.py — and gains what the vendored loop cannot offer: CFG policies, distilled few-step samplers, caches, and batched scheduling for the rollout phase (today's rollouts run with zero serving-grade optimizations). Engine-client mode: seeded rollout OmniRequests fan out to a rollout pool running the same loop, and the trainer pushes update_weights between iterations — note the weights shipped are the old policy, a decay-blended copy, a concrete WeightSyncPlan case (§8.6) — with sleep/wake reclaiming rollout memory during the gradient phase. Consistency per §8.5's own rules: NFT is the easy rung — likelihood-free means no ratio exists for drift to poison, and the landed discipline (pinned attention kind, pinned autocast dtype, seeded generators) is C1 by construction; the full ladder is for the GRPO-class methods that come next — C1-pinned rollout (cache acceleration off) bounds per-step log-prob drift to ε, trajectory identity is C2's conditioned replay guarantee, not C1's; alternatively C0 with cache-dit on buys throughput and pays for it with TIS correction — never both at once.

---

8. Training, post-training, and RL on the same substrate

FastVideo's differentiation — and its complexity tax — is that training and post-training live in the same repo as inference, with RL now landed (PR #1450, DiffusionNFT for Wan — no longer a theoretical reference: a 3,500-line PR on main spanning the 1,075-line method, a vendored sampler, reward scorers, trainer hooks, tests, and three .agents/skills codifying its extension contract). The question: does a fully unified design make sense? Answer: unify the substrate, not the stacks — and for video models this is less a choice than a recognition: unlike LLMs, diffusion's inference optimizations are themselves post-training artifacts (§8.3), so the loop inversion of §6.2.2 is the missing shared piece. One honest qualification, learned from studying two production RL frameworks (~/verl-omni for diffusion RL, ~/miles for LM RL): the industry's pain is not really "many repos." It is two runtimes with different kernels — Megatron/FSDP-class training optimizations on one side, vLLM/sglang-class inference optimizations on the other — and an entire mismatch-correction industry has grown around that split (§8.4). That is simultaneously the strongest argument for what we are building and the sharpest warning about it: one repo does not buy numerical consistency. One model definition + one kernel set + an enforced, measured consistency contract does (§8.5).

8.1 What exists today (and what is duplicated)

Three consumers of the generative computation, with partial sharing:

Layer	Shared today	Duplicated today
Models, component loaders + FSDP load, arch configs (param_names_mapping), schedulers, fastvideo/distributed/, attention backends, offload hooks	✓ single-source across inference, train/, legacy training/	—
Validation generation	new stack's ValidationCallback instantiates the real inference pipeline and calls forward()	legacy stack rolls its own; the landed RL method adds a third path — TrainingMethod.on_validation_begin() + rl/common/validation.py re-implement prompt sharding, seeded sampling, and sample logging precisely because (per the hook's own docstring) these are "methods that intentionally avoid inference pipelines"
Denoise/sampling loop	—	4 copies: pipelines/stages/denoising.py (inference); DMD2Method._student_rollout() / SelfForcingMethod._student_rollout_streaming() (train/methods/distribution_matching/); inlined per-pipeline in legacy training/*_distillation_pipeline.py (~1200-line monoliths); DiffusionSampler in train/methods/rl/common/sampling.py (RL, landed with PR #1450)
CFG / uncond handling	—	3 copies: stages/denoising.py (cond/uncond branching + CFG gating; conditioning.py's forward is a no-op that only validates flags) vs ModelBase.predict_noise(cfg_uncond=…) vs legacy inline
Preprocessing	pipelines/preprocess/* already subclass ComposedPipelineBase and reuse TextEncodingStage → parquet latents/embeddings	trajectory extraction (preprocess_pipeline_ode_trajectory.py) is a separate path from return_trajectory_latents

The fourth copy is no longer a prediction — PR #1450 landed it. train/methods/rl/common/sampling.py vendors DiffusionSampler: an ODE walk delegating per-step math to the shared FlowMatchEulerDiscreteScheduler.step, plus a re-noising sde_reflow mode that is a near line-for-line re-derivation of DMD2's rollout loop (dmd2.py:484–520). Its own docstring states the reason: it "intentionally does not call FastVideo's full inference pipelines" because that would bind the method to model-family pipeline classes like WanDMDPipeline — and the accompanying skill (.agents/skills/rlhf-training-abstractions/SKILL.md) elevates this to standing policy ("do not bind RL methods to inference pipeline classes"), with a test asserting the RL YAML never references WanDMDPipeline. The authors are right to refuse that dependency, and that is precisely the indictment: today's only consumable units are family-bound pipeline classes, so every new post-training method must choose between a wrong dependency and a private loop. The add-rl-method skill now mandates the vendored sampler for future RL methods — which contains the duplication within train/methods/rl/ but cements it outside the substrate, where no inference optimization, CFG policy, or scheduler improvement will ever reach it.

Concretely the "fourth copy" is already two vendored RL/distill loops, not one: DiffusionSampler (rl/common/sampling.py:92-99 docstring, :128-147 hand-rolled for-loop, conditional=True, attn_kind=dense) and DMD2's _student_rollout (dmd2.py:430-550, attn_kind=vsa) — a second, independently vendored loop. What each of these vendored loops forgoes by living outside the substrate is itemizable: (1) cross-group / cross-request batching; (2) content-hash embedding/feature-cache reuse; (3) step scheduling and interleaving; (4) cost-aware admission; (5) component-granular sleep/wake (it holds student + old + reference resident simultaneously); (6) cache-dit / interceptor skip acceleration (forced dense, full 25-step ODE); (7) distilled few-step sampler selection (a 4-step card collapses 25→4). A DenoiseLoop over substrate step bodies is the third option this design exists to provide — and, per §8.4, it is the moat: rollout forward is serve forward + capture, so every serving optimization is automatically a rollout optimization.

8.2 The decision: shared substrate, layered dependencies

Full unification — one framework that both trains and serves — is the wrong target: trainers own optimizers/FSDP/grad-accumulation/checkpointing; the engine owns scheduling/caches/serving. What they genuinely share is the generative computation and the model substrate:

substrate (shared):  models/ + loaders/manifests + configs + schedulers + distributed/ + attention/
                     + PipelineSpec graphs, Stage/LoopStage step bodies, sampler & CFG policies, PackedSeq
engine  (inference): StepScheduler, queues, caches, serving, placement   — depends on substrate
train   (training):  Trainer, methods, callbacks, data                   — depends on substrate (+ engine as client)

Rules and consequences:

• engine never imports train (the reverse is allowed). A substrate change is a change to both worlds — and is gated as such (§12, risk 5).
• Distillation rollouts call the shared loop — with one honest exception. DMD2/KD methods drive DenoiseLoop.step directly (grad-enabled where needed — step bodies are plain tensor programs, so autograd composes) instead of re-implementing timestep iteration. Self-forcing does not fit the inference step contract and is not forced into it: its rollout samples per-block exit indices broadcast across ranks, runs no-grad steps to the exit, exactly one grad-enabled forward, then a separate no-grad store_kv=True context-caching pass — grad-window flags and per-step cache-write control that would be training-only surface in substrate code. It ships as a custom training LoopStage in train/ (the §6.2.3 escape hatch), consuming substrate pieces — model forwards, samplers, and ChunkKVPool in its declared training mode (no mid-rollout recycling; grad-aware index snapshots so activation-checkpoint recompute doesn't double-advance the cache, per wan_causal.py:119-120, 405-431). The dedup arithmetic stated honestly: shared step bodies where they fit; custom loops stay custom but consume the substrate instead of vendoring their own sampling stack. The trainer already concedes this shape in landed code: PR #1450 added TrainingMethod.manages_optimization() / managed_train_step(data_stream, …) so DiffusionNFT's sample→score→inner-train outer epoch could own its own optimizer cadence — method-owned loops are a supported trainer contract today, not a workaround this design must invent.
• Trajectory capture becomes one feature, not four. return_trajectory_latents, the ODE-trajectory preprocessing pipeline, RL rollout recording (the landed SamplingResult in rl/common/sampling.py — latents + timesteps + sigmas — is the third ad-hoc capture path), and future log-prob capture for GRPO-class methods all become OutputSpec options on the loop.
• Preprocessing pipelines are loop-less graphs of one-shot encode stages emitting parquet artifacts — they already use the pipeline machinery, so this is a rename, not a rewrite.
• Observers carry over for free: ParityAligner / NaNWatch / Profiler attach to training rollouts identically — first-class tooling for divergence hunts and precision (FP8/FP4, QAT) validation during training.
• The legacy training/ stack stays frozen (per repo policy) and is unaffected until its models migrate to train/ on their own schedule; nothing here forces that migration.

8.3 Why diffusion is different: inference optimization is post-training

In the LLM world, inference optimization is mostly post-hoc and training-free: a lab ships frozen weights, and the serving stack optimizes around them — PTQ quantization, KV-cache management, continuous batching, speculative decoding, kernels. The artifact crossing the trainer→engine boundary is a checkpoint. That is why the two-runtime split is tolerable there: the inference frontier can advance without touching training.

Video generation does not work that way. The deployable model is itself a post-training artifact, and nearly every inference capability FastVideo ships is really a (recipe, runtime) pair:

Inference capability	Training-side half	Runtime-side half
Few-step generation — the difference between unusable and usable	Step distillation (DMD2, consistency/rCM) — an active research frontier; recipes change quarterly	distilled samplers / timestep policies
Streaming + world models (AR video)	step distillation plus a self-forcing variant on top — the causal student is created by training	ChunkKVPool, causal rollout, streaming
Low-precision inference	QAT (our NVFP4 Attn-QAT line, upstreaming now) — PTQ alone doesn't hold quality; training must see deployment numerics	FP4/FP8 kernels + PrecisionPolicy
Sparse attention	VSA — trained sparsity (tile selection learned during training)	VSA kernel backend
Reward alignment	RL — DiffusionNFT landed (#1450, likelihood-free: needs only final-sample capture); GRPO-class next	samplers (ODE/SDE), final-latent + trajectory/log-prob capture

Two structural consequences:

1. The training loop embeds the inference loop — and that collocation is the central, durable moat. Distillation runs deployment-grade rollouts inside training: DMD needs student rollouts; self-forcing is literally "train against your own KV-cached causal rollout" — the runtime feature must exist before the model class does; QAT's fake-quant forward must match the deployed quant kernels. There is no valid sequencing of "build the inference engine first, training later" — each is a dependency of the other, which is exactly why the substrate (§8.2) has to come first, and why the denoise loop keeps getting copied (§8.1): every post-training method needs an inference loop inside it, and absent a shared one, each grows its own. The payoff of collocating inference + post-training/RL on one substrate: rollout forward is serve forward + capture — same loop, same caches, same batcher, same numerics — so every serving optimization is automatically a rollout optimization and there is exactly one numerics surface (no rollout-vs-train kernel gap to patch, §8.4).

   RL rollout is a strictly better cross-request batching case than open-world serving — the killer point. A GRPO/NFT group is K identical-config samples of one prompt: num_video_per_prompt:24, sample_train_batch_size:6, num_batches_per_epoch:48, num_steps:25, single-frame 448×832, 4 GPUs (diffusion_nft_pick_clip.yaml:33-37,52-55,72-91; distributed_k_repeat_indices with repeats_per_prompt=24, diffusion_nft.py:331-338). Same shape, same schedule, same CFG branch, K=24-wide → zero bucketing needed (serving must bucket heterogeneous resolutions/steps/CFG). The K samples also share one prompt embedding, so a content-hash feature cache encodes the text once and reuses it across all 24 — a 24× text-encode reduction the vendored loop cannot express (it carries the embedding per-sample, diffusion_nft.py:308-309). Per GPU per outer epoch = 6 prompts × 48 batches = 288 prompt-slots, each a 24-wide homogeneous denoise batch: near-ideal co-batchable work the vendored Python loop runs one sample at a time.

2. Train/serve skew generalizes the RL mismatch. Any sampling-in-the-loop post-training optimizes the student under the rollout implementation used during training. Serve it under different kernels/sampler/caching and quality silently degrades — the diffusion analog of RL's off-policy bias, except it applies to essentially all post-training and surfaces as quality regression rather than reward collapse (slower to notice, harder to attribute). The consistency ladder (§8.5) is therefore not an RL feature: C1 (kernel-pinned) is the default contract for every sampling-in-the-loop method — distill what you serve, serve what you distilled. (miles reaches the same place from the LLM side: its INT4 QAT exists because rollout efficiency at 1TB scale required training to see inference numerics.)

This reframes the unification question (§11.12): in LLMs, an inference engine can ignore training because optimization is training-free; in diffusion, an inference engine that cannot run inside training loops is cut off from the actual optimization frontier, and a training framework without a serving-grade rollout distills models toward a toy sampler. FastVideo's "advantage that is also its disadvantage" is precisely that it occupies the one position where the (recipe, runtime) pair can be co-designed and parity-gated. The RL argument of §8.4 is a special case of this.

8.4 The lesson from RL systems: the two-runtime tax

Why the split exists. Training forwards want FSDP/Megatron sharding, activation checkpointing, high-precision gradient accumulation, and sequence packing. Rollout forwards want KV caches, continuous batching, CUDA graphs, quantization, and inference kernels. Run the same weights through both and the log-probs differ — different kernels, precision paths, batching, reduction orders. The policy that generated the trajectories (π_rollout) is no longer the policy being optimized (π_train); PPO-style ratios are computed against the wrong baseline; at scale (especially MoE + FP8) this is a documented cause of training collapse. Everything below exists to patch that wound:

The tax	verl-omni (diffusion RL)	miles (LM RL)
Dual model definitions	Wan2.2 implemented twice: a rollout pipeline inside vLLM-Omni (pipelines/wan22_dance_grpo/vllm_omni_rollout_adapter.py, with trajectory/log-prob capture and dual-expert switching) and a diffusers/FSDP2 trainer engine (workers/engine/fsdp/diffusers_impl.py, VeOmni variant)	model exists in sglang and Megatron, bridged by weight-name iterators (Megatron fused params → HF names)
Mismatch corrections	trainer/diffusion/rollout_correction.py: bypass mode (trust rollout log-probs) vs decoupled mode (recompute under trainer + truncated importance sampling + rejection-sampling masks)	TIS/MIS in backends/training_utils/loss_hub/corrections.py: clamp exp(logp_train − logp_rollout) to configurable [tis_clip_low, tis_clip] bounds (CLI defaults [0, 2.0]; their MIS example config uses [0.5, 2.0]) or hard-mask out-of-range tokens (MIS); tis_clipfrac monitored as a health metric
Determinism engineering	deterministic rollout/reward/trainer modes; per-step seed derivation	a whole program for bit-wise identical log-probs: FlashAttention-3 + DeepGEMM batch-invariant kernels + deterministic sglang, both sides pinned to TransformerEngine
MoE routing drift	—	R3 rollout routing replay: capture per-token expert choices in sglang ((seq−1, layers, top_k) int32 — ~60 MB/sample at 32K×60×8) and replay them in the Megatron forward
Precision drift	—	Unified FP8: identical quantization recipe (blockwise / MXFP8) on both sides; weights dequantized to BF16 for transfer, re-quantized identically on arrival
Weight-sync machinery	FSDP2→vLLM-Omni LoRA collection + RPC; sleep levels (1 = keep VAE/text-encoder resident, 2 = full offload)	three transports behind one interface: Ray-IPC zero-copy (colocated), bucketed NCCL broadcast, P2P-RDMA with pinned-CPU staging; per-sample weight_versions staleness tags

The punchline for FastVideo: verl-omni's Wan2.2 rollout adapter is a re-implementation of code this repo already owns — external RL frameworks must rebuild our models inside other engines to RL-train them, then correct the numerics. Doing RL natively means the model exists once and the corrections become a fallback, not a foundation.

The moat made concrete. Under this design, rollout = the same shared DenoiseLoop + a CFGPolicy (for NFT: conditional-only / identity — predict_noise already carries conditional:bool + cfg_uncond as the policy knob, train/models/base.py:142-152) + an OutputSpec(capture=behavior) + the step scheduler + the caches — so rollout forward is serve forward + capture. Every serving optimization is automatically a rollout optimization, and there is one numerics surface, no rollout-vs-train kernel gap to patch. The alternative is the two-runtime tax the table above enumerates: verl-omni re-implements Wan inside vLLM-Omni and stitches it with rollout_correction.py's IS / rejection masks; miles' TIS/MIS, bitwise log-probs, R3 routing replay, and unified-FP8 are all mismatch patches. FastVideo's own landed NFT confirms it from the other side — a vendored bare-model loop with zero serving-grade optimizations (§8.1). Collocation at FastVideo's 1–30B FSDP2 scale deletes the second runtime, the correction layer, and the 3rd/4th denoise-loop copies — and it makes the (recipe, runtime) flywheel (§8.3, §9.5) honest, because preferences are then collected under the very serving profile that ships. This collocation of inference + post-training (including RL) on one substrate is the central, durable moat. Three caveats the design already names keep this from being hand-waving: old-policy weights are a WeightSyncPlan role (rollout samples from self.old, decay-blended diffusion_nft.py:858-862); the dense-attn pin is a Precision/Attn policy on the card; and likelihood-free NFT is the C2 behavioral rung (no log-probs to match — old_deviate / ref-MSE in prediction space, diffusion_nft.py:777-778).

What stays regardless (intrinsic to RL, not artifacts of the split): trajectory buffers and rollout management, group-relative advantage computation, multi-reward serving with async overlap, staleness control, evaluation-during-training, checkpointing, fault tolerance (health-monitored engines). We adopt these; we delete the rest. Two of them already exist landed in-repo as the baseline to build from: #1450's gathered per-prompt group-advantage normalization (diffusion_nft.py::_compute_advantages) and its weighted MultiRewardScorer (in-process, GPU-resident PickScore/CLIP — the async/HTTP serving half is the part still to come).

The boundary condition, stated honestly. Miles defaults to Megatron because its targets are ≥70B MoE models (1TB-scale K2-Thinking with INT4 QAT). FastVideo's training targets are 1–30B DiTs/MoTs on FSDP2/HSDP — that scale assumption is what makes the single-runtime bet viable. If a Megatron-class trainer ever becomes necessary (100B+ MoT training), those models re-enter the two-runtime world — and the toolkit below (C0 corrections, Behavior Record, weight transports) is designed to still apply as the fallback. This is a trigger to monitor, not a reason to build for it now.

8.5 The bet: one model definition, two execution profiles, a consistency ladder

Even in one runtime there are honestly two execution profiles: the training forward (grad-enabled, activation checkpointing, FSDP-gathered params, no caches) and the rollout forward (no-grad, CUDA graphs, KV/feature caches, TP/SP-sharded, possibly quantized, possibly cache-dit-accelerated). Same repo ≠ same numerics — and the precise mechanism matters. Thinking Machines' "Defeating Nondeterminism in LLM Inference" is the definitive analysis: forward passes contain no atomics and are run-to-run deterministic for a fixed batch; what breaks is batch invariance — kernels switch reduction strategies with batch/shape (split reductions in RMSNorm, split-K and tile/instruction selection in matmul, KV-split counts in attention), so a request's numerics depend on what it was batched with. Compose that with variable load (or, in our case, with rollout-vs-training profile differences) and you get nondeterminism — their headline: 1,000 temperature-0 completions of Qwen3-235B produced 80 unique outputs, first diverging at token 103; with batch-invariant kernels, 1,000/1,000 identical. Miles' bitwise program (FA3 + DeepGEMM) and sglang's deterministic mode descend from this work. One codebase makes parity tractable, not automatic — so consistency is a declared, measured contract, per run:

• C0 — corrected. Profiles free to differ: quantized rollout, cache-dit on, CUDA graphs. Trajectories store rollout log-probs; objectives apply TIS/MIS (miles' clamp/mask semantics ported to per-step diffusion log-probs, with verl-omni's rejection-sampling mask as an option); logprob_diff and clip-fraction are always-on training telemetry. This is the throughput mode and the graceful-degradation mode.
• C1 — kernel-pinned (default for RL). Rollout-for-training pins the profile to the trainer's: same attention backend, same PrecisionPolicy, distribution-altering interceptors disabled, eager or shape-matched graphs. ParityAligner gates per-step log-prob diff ≤ ε in CI for likelihood-based methods (ε set per model family from Phase-1 harness data, not guessed); the drift metric should sit at ~0 during training. "Trust the rollout log-probs" (verl-omni's bypass mode) is then actually sound instead of hopeful. The landed DiffusionNFT (#1450) is C1 by construction — in-process rollouts under the trainer's kernels, attn_kind="dense" hard-pinned on rollout and all three training forwards, FASTVIDEO_ATTENTION_BACKEND defaulted to FLASH_ATTN in the launch script — and it already paid a C1-class bug tax that proves the discipline isn't free: WanModel.predict_noise had to stop deriving autocast dtype from caller tensors because RL-created fp32 intermediates silently changed numerics between rollout and gradient forwards (the _get_training_dtype() fix + test_wan_training_dtype.py). The CI enforcement pattern also has a landed coarse precedent: #1396's per-method, device-keyed grad-norm regression refs (10% rtol) — the same pin-a-golden-and-gate shape ParityAligner applies at per-step resolution.
• C2 — bitwise. Batch-invariant kernel set + deterministic mode + Behavior Record replay → bit-identical log-probs between rollout and training forwards. The recipe is known (per the Thinking Machines post): fixed reduction strategy for norms, one pinned matmul kernel config across shapes (~20% off cuBLAS peak, acceptable), and attention with fixed split-size (not split-count) KV reductions plus a consistent KV layout (page-table update before the kernel) so numerics are independent of how many tokens are processed at once — implementable as a torch.Library op-swap (their batch_invariant_ops) behind our attention-backend registry as a deterministic backend mode — and torchtitan already carries set_batch_invariance() utilities, so the kernel set is landing in the PyTorch ecosystem proper, not just a vendor library. Measured cost in their vLLM demo: 26s → 42s (~1.6×) with the improved attention kernel. The payoff — stated with its conditions: in their RLVR experiment, no correction collapses mid-run, importance weighting trains with KL ≈ 10⁻³, and bitwise sampler-trainer identity holds KL flat at exactly 0. For diffusion, kernels alone do not confer that: C2's "true on-policy" holds only as the conjunction of batch-invariant kernels plus the full Behavior Record (sampler noise/RNG, guidance state, routing) plus C1's preconditions (matched precision, interceptors off, same weight version, identical SDE log-prob arithmetic) — and it is a per-trajectory replay guarantee, verified by ParityAligner, not a blanket property. C2 is our golden-run, divergence-debugging, and MoE-routing-parity mode; C1+TIS covers daily training.

Where batch variance actually bites FastVideo (the post's lens applied to our workloads): the video denoise path is effectively batch-1 with fixed shapes — accidentally near-batch-invariant already, our structural advantage. The exposure is elsewhere: (a) AR pathways (Cosmos3 reasoner, omni thinkers) under continuous batching, where batch composition changes every decode step — the blog's exact scenario, and why C1/C2 for AR segments needs either batch-invariant decode kernels or frozen-batch rollout; (b) image serving with bucketed cross-request batching; (c) parallelism-degree differences between rollout (SP/TP-sharded) and training (FSDP-gathered) forwards — sharding is a batch-like axis that changes reduction layout, so C1 pins matched degrees and C2 requires invariance across them; (d) chunked causal video attention (ChunkKVPool), where the fixed-split-size principle applies directly: KV-chunk reductions must be invariant to how many chunks are resident, or world-model rollouts won't replay bit-identically.

Behavior Record (R3, generalized) — budgeted at real bytes. Trajectories optionally capture behavior artifacts beyond latents/log-probs: per-step RNG state (cheap — seeded generators in one shared loop reproduce draws, so seeds suffice), sampler branch choices (cheap), and MoE routing decisions (not cheap for diffusion: miles' ~60 MB/sample is per-token routing with one forward per token; diffusion re-routes the entire packed sequence at every denoise step, per CFG branch — steps × branches × tokens × sparse-layers × top_k, which for a Cosmos3-class request (~50K packed tokens, ~24 sparse layers, top_k 4, 35–50 steps × 2 branches) is ~1.3–1.9 GB/sample int32, ~0.3–0.5 GB as uint8 — tens of GB per GRPO group). Consequently routing capture is an opt-in instrument for C2 goldens, MoE-parity debugging, and divergence hunts — not a training-scale default; routine RL runs at C1 (or C0+TIS) without it. If it is ever wanted at fleet scale, the GB-trajectory streaming/storage design it forces is acknowledged as undesigned (§12 known gaps). The plumbing is an OutputSpec feature of the loop (§6.2.2); the bytes are the cost, and they are stated.

Plugins are policy-relevant. cache-dit during rollout changes the sampled distribution — trajectories then come from a behavior policy β ≠ π_θ. Contract: interceptors declare distribution_altering: bool; at C1+ such plugins are disabled for rollout-for-training; at C0 they are permitted and stored log-probs must reflect the actual (cached) computation, with IS correction in the objective. Enforced declaratively at pipeline build, not by convention.

Precision unification is the same contract one level down (miles' FP8 lesson): rollout and trainer share one PrecisionPolicy / quantization recipe — "train BF16, roll out FP8" is only legal at C0. This dovetails with the in-repo Attn-QAT/FP4 work: QAT is exactly what makes low-precision rollout consistent with training by construction.

8.6 Weights, parallelism duality, and placement

The trainer mesh (FSDP/HSDP) and the engine mesh (TP/SP/CFG) are different layouts over the same parameters — both expressed in the stacked-axis spec (§6.3.4), e.g. [hsdp_replicate(2), hsdp_shard(4)] vs [dp(2), cfg(2), sp(2), tp(1)]. The WeightSyncPlan is computed from three inputs, not two — axis specs alone are necessary but not sufficient: (a) the pair of mesh specs; (b) per-model weight-layout adapters covering what specs cannot express — fused QKV/MLP packing, MoE expert placement, quantized storage formats, LoRA deltas, FSDP flattening, and checkpoint key translation (the param_names_mapping/state-dict-adapter lineage this repo already maintains for conversion; torchtitan reaches the same conclusion — its fused-SwiGLU override needs model-level state-dict adapters, and its own notes mark some redistributions as inexpressible, spmd_types.py:45, overrides/fused_swiglu.py:34); (c) the transport. The plan is validated pre-flight and CPU-testable with fake pools — replacing hand-written per-combination code (the bug class miles' weight-name iterators and verl's resharding exist to manage), while being explicit that the per-model adapter is authored, not derived. Phasing: Phase 2 ships the colocated cases (same-layout zero-copy; in-place reshard for the Phase-1 families' FSDP↔engine layouts, adapters hand-checked); the general cross-mesh plan over the declarative spec lands with the spec in Phase 3.

• Transports (miles' menu, one interface): colocated same-layout → zero-copy views — the unified-runtime dividend: often a literal no-op; colocated different-layout → in-place reshard; disaggregated → bucketed collective broadcast or P2P-RDMA with pinned-CPU staging. multimodal_gen's RL weights-updater is the in-family precedent for the RPC surface. At fleet scale, weight distribution and version tracking should ride Dynamo's ModelExpress rather than our own fan-out — that is ask A5 (§6.3.6): versioned incremental broadcast + staleness-aware routing, with our transports as the colocated/fallback path. Two further precedents to reuse: torchtitan's RL experiment syncs PolicyTrainer→vLLM via TorchStore push/pull with direct-RDMA state-dict fill — the PyTorch-ecosystem transport to evaluate first for the disaggregated path (with its own caveat: DCP and TorchStore do structure-matched fill, not automatic cross-mesh resharding — the WeightSyncPlan still owns the layout math); and cosmos-rl's buffer-model async sync (a background thread on a side CUDA stream writes incoming weights into a parameter-only clone; the live model swaps at safe points) is the pattern for non-blocking colocated updates.
• Sleep/wake is component-granular (verl-omni's levels, generalized by our component registry): a rollout pool can drop KV/feature caches and DiT weights while keeping VAE/text-encoder resident (level 1) or release everything (level 2). Implementation reference: vLLM's CuMemAllocator (vllm/device_allocator/cumem.py) — tag-based unmap/offload where our tags are component names (finer than vLLM's global weights/kv_cache tags), with its documented caveats inherited: CUDA graphs invalidate across sleep (recapture on wake), woken KV pools are zero-initialized (sessions re-prefill), and encoder/embedding caches must reset() on weight update.
• Staleness is first-class (miles): every trajectory carries weight_version; update_weights_interval and a max-staleness bound govern the async overlap loop (generate batch N+1 while training on N — the train_async.py pattern); over-sampling with abort/truncation statuses handles stragglers, and loss masks zero out incomplete samples. The landed NFT method (#1450) adds a wrinkle the plan must carry: its behavior policy is not the student but a decay-blended old copy (and validation optionally swaps in the EMA shadow) — so a WeightSyncPlan ships a declared role (student / EMA / blended old policy), never implicitly "the weights", with the blend applied trainer-side before push or pool-side on arrival.

8.7 The RL layer on the engine

Components (all under train/, consuming the substrate and the engine — never the reverse). These are now the migration target for landed code, not a blank-slate plan: train/methods/rl/ exists on main (#1450) as a 1,729-line package — a 1,075-line TrainingMethod with advantages, objective math, and inner optimization inlined, driving its own reusable-but-package-private sampler and reward modules — no RolloutFn, no TrajectoryBuffer, consuming neither substrate loops nor engine — and its three .agents skills (add-rl-method, add-reward-model, rlhf-training-abstractions) codify the current pre-§8.7 conventions. The list below is what that module refactors into as the substrate and engine land:

• RolloutFn — miles' single best interface lesson: users write one function. Contract: (ctx: RolloutContext) -> list[Trajectory], where ctx provides an engine client (seeded generation, with log-prob capture where the method's OutputSpec declares it — NFT-class methods don't), the data source, and tool hooks. Covers single-shot diffusion sampling, N-sample GRPO groups, multi-turn/agentic episodes, and future interleaved omni episodes — without touching the trainer.
• Trajectory (our Sample): request + seeds, per-step records as declared by the method's OutputSpec (final latents only for NFT-class; latents + timesteps + log-probs for GRPO-class), Behavior Record, rewards (scalar or dict), loss masks, weight_version, status (COMPLETED/TRUNCATED/ABORTED).
• TrajectoryBuffer: collection, filtering, group-by-prompt, conversion to training batches.
• Reward layer (verl-omni's shape): weighted multi-reward aggregation; sync in-process scorers and async HTTP scorers; reward computation overlapped with rollout; reward models loadable as engine components — and later as engine graph nodes scoring decoded artifacts. The landed train/methods/rl/rewards/ is the right seed: thin PickScore/CLIP scorers loading from HF (the #1438 draft's vendored HPSv3/VideoAlign repos were dropped before landing — good hygiene to keep, §12 open question 13), a duck-typed RewardScorer = Callable[[media, prompts], Tensor] contract, and weighted aggregation in MultiRewardScorer — what's missing is the async/HTTP/offload half, since today's scorers sit GPU-resident on every training rank for the whole run.
• Objectives library: FlowGRPO/DanceGRPO (clipped PPO over per-step SDE log-probs; SDE-with-logprob is a sampler policy, implemented once), DiffusionNFT (final latents + policy/ref forwards — extracted from the landed #1450, whose objective math today lives inlined in the monolithic method class rather than as a reusable module), Diffusion DPO (offline pairs), and AR GRPO/GSPO for token pathways — with cosmos-rl's stability kit (AIPO one-sided clipping for async off-policy rollouts, dual-clip, KL-threshold off-policy masking) as the C0-mode options. Shared group-relative advantage utils.
• Two rollout modes, one set of trajectories: library mode — the method drives substrate loops in-process (required when backpropagating through sampling; the landed #1450 is library-mode shaped — in-process, no-grad, trainer kernels — but drives its private DiffusionSampler instead of substrate loops, which is the Phase-1 migration); engine-client mode — RolloutClient submits seeded OmniRequests to a colocated or disaggregated engine, inheriting queueing, batching, step interleaving, and (at C0) cache acceleration. At fleet scale the disaggregated rollout pool is a Dynamo-fronted fleet: rollout requests route through Dynamo (staleness/version-aware once ask A5 lands, §6.3.6), the Planner shifts GPUs elastically between trainer and rollout pools, and weight updates propagate via ModelExpress. The consistency ladder (§8.5) plus seeded RNG in RequestState is what makes the modes interchangeable without retuning.
• Loops: sync (rollout → reward → advantage → update → weight-sync) and async (overlapped, staleness-bounded) — thin scripts over these components, deliberately the same shape as miles' train.py/train_async.py.

Omni-RL — expressibility is established; the algorithm is not. A hybrid episode over a MoT model (Cosmos3): one trajectory containing AR reasoner segments (token log-probs) and denoise segments (per-step SDE log-probs), rewarded jointly — one model, one trajectory, two loop types. Engine-per-stage RL frameworks cannot express this without two engines and two weight copies; that structural advantage is real. But "optimized with mixed objectives" names an unsolved algorithm design, stated as three open problems (open question 16): (1) scale mismatch — token log-probs are O(1–10) nats over 10²–10³ tokens while per-step diffusion log-probs are Gaussian densities over 10⁶–10⁷ latent dims; a joint clipped-ratio objective needs principled per-segment normalization that none of the cited recipes (FlowGRPO/DanceGRPO/NFT/AIPO/GSPO) provides, and getting it wrong lets one modality silently dominate the shared trunk; (2) credit assignment — the reasoner influences video reward only through sampled discrete tokens re-entering as conditioning, a non-differentiable boundary, so token segments get sparse trajectory-level signal while denoise segments get dense per-step ratios, both updating shared attention weights with no interference analysis; (3) reasoning regression — RL-updating the und pathway on video-correlated reward risks degrading reasoning; hybrid episodes need reference-model KL anchoring. Accordingly, the Phase-4 deliverable is hybrid trajectory capture demonstrated plus an objective design study — not a pilot that presumes the objective exists.

What we deliberately do not build: verl's general Ray hybrid-controller choreography (actor/critic/ref/reward worker groups with cross-runtime resharding). At our scale the trainer drives a simple loop; if critic/ref models arrive they are additional engine pools, not a new controller paradigm.

8.8 Config unification with training

train/utils/training_config.py keeps what is genuinely training's (optimizer, data, loop, checkpoint, tracker sections) and imports the shared layers for everything else: the deploy/parallelism spec (§6.3.4) replaces its distributed: block (today a field-for-field duplicate of FastVideoArgs.tp_size/sp_size/hsdp_*), and model/ pipeline identity comes from the same ModelSpec/PipelineSpec the engine loads (§6.6). One description of "what model, how sharded" serves inference, training roles (student/teacher/critic), and rollout.

---

9. The application pull: Dreamverse, ComfyUI, and the workflow layer

Dreamverse (apps/dreamverse/ — Next.js frontend + FastAPI runtime; open-sourced with the vibe-directing and release posts) looks out of place in an inference-runtime design at first glance. It is the opposite: it is the proof of the thesis and the generator of the requirements. "Vibe directing" — steering video through continuous natural-language revision, 30-second scenes composed of chained 5-second clips, ~4.55 s per 5 s 1080p clip on one B200 (faster than real-time playback) — is a product category that exists only because of (recipe, runtime) pairs (§8.3): faster-than-realtime clips come from NVFP4 (QAT-backed) + FA4 + torch.compile on the runtime side and from distillation recipes on the training side, streamed as fMP4 over WebSocket with frame/audio conditioning carrying continuity between segments. The blog's own one-liner — "better creative work comes from a faster feedback loop, not just a better model" — is §8.3 in product language.

9.1 What Dreamverse proves about the runtime gap

Read apps/dreamverse/arch.md and the division of labor is striking: the app server owns "GPU assignment and worker lifecycle," "actual generation queue semantics," warmup, and "stream chunk emission" (dreamverse/gpu_pool.py, main.py). The application layer is hand-building the engine plane of §6 because the runtime doesn't provide it — P4 made manifest in a product, with a price tag: one B200 per active user session. Dreamverse is simultaneously the best evidence for the engine (Phase 2 retires gpu_pool.py into an engine client at single-session parity and runs the duty-cycle study; multi-session multiplexing lands at the Phase-3 gate) and the best customer for nearly every §6 feature:

Dreamverse need (shipping today, hand-rolled)	Engine feature it pulls
GPU pool + queue + warmup in the app server	engine scheduler, warmup requests, multi-tenant session multiplexing (§6.3.1)
fMP4 segment streaming over WebSocket, ltx2_segment_start/chunk events	OmniEvent streaming contract; loop emit (§6.2.2)
frame/audio conditioning chained across 5 s segments; init-image persistence	session abstraction: session-scoped latents/caches and continuation conditioning as first-class request inputs — the same machinery as interactive world-model sessions (§12, open question 9, which now has a production owner)
prompt-rewrite LLM (external provider today, with timeout/fallback)	AR graph node or external-engine node (§6.3.4); a future omni model collapses rewrite+generation into one resident model
LoRA controls per direction (#1420)	LoRA hot-swap on the engine path
SP for serving (#1424), NVFP4/FA4/compile	DeployConfig parallelism + PrecisionPolicy + plugins — per-app deploy YAML instead of bespoke wiring
safety filtering, prompt memory, session protocol	stays in the app — the workflow layer is a client of the engine, never merged into it

That last row is the architectural rule this section adds: the workflow layer composes engine requests; it does not extend the engine. Dreamverse keeps UX, prompt memory, safety, and session semantics; the engine absorbs exactly the parts the app had to hand-roll (pool, queue, warmup, streaming transport, placement).

Where multi-session capacity actually comes from (a correction the review process forced — note that today's gpu_pool.py deliberately admits one user per GPU, is_available false while connected_users is non-empty): step interleaving improves fairness, not throughput — two always-on sessions are ~2× service demand, and no scheduler conjures that away. The capacity claim rests on three real mechanisms: (1) duty cycle — directing sessions are idle-heavy (the user types, watches, decides between segments), so time-averaged demand per session is well below one GPU; (2) admission control on the cost model — sessions admit against measured per-segment cost and a per-class latency SLO, with simultaneous segment requests queued within the SLO rather than co-batched; (3) the distillation roadmap lowering per-segment cost over time. Accordingly the multi-session gate is defined on a recorded duty-cycle trace of real session behavior — p95 segment latency within the SLO vs the single-session baseline — never on synthetic always-on load.

Transport triggers (from the LiveKit/WebRTC review). fMP4-over-WebSocket remains the right transport for segment streaming. WebRTC (LiveKit-class SFU) becomes the transport when any trigger fires: sub-second streaming latency, one-to-many fanout, or interactive world models — the <100 ms motion-to-photon budget (≈20–40 ms action round-trip + ≤40 ms/frame inference + ~10 ms encode/decode/render) is unreachable over TCP/WS and needs WebRTC data channels as the action path. The engine contract that keeps both transports thin adapters: OmniEvent optionally carries raw frames + PTS (not only encoded segments — the SFU SDK encodes in a native thread from RGBA/I420 buffers), sessions accept an async action stream, and a flush/interrupt RPC discards stale frames on a direction change. Until a trigger fires, the SFU/TURN/token operational weight is not paid.

9.2 The flywheel: where post-training, inference, and RL meet a product

Dreamverse's intended trajectory — post-training + inference + eventually RL composing one production-ready workflow — closes the loop of §8:

distillation/QAT recipes (§8.3) ──► faster-than-realtime engine ──► vibe directing UX
        ▲                                                                │
        │                                                                ▼
   RL / DPO post-training (§8.7) ◄── preference data: kept vs discarded revisions,
                                      edit chains, per-segment directions

Vibe-directing sessions emit dense, naturally-labeled preference data — every revision a user keeps over the one they discarded is a DPO pair; every edit chain is a trajectory with human reward. No video lab has this loop in one stack: labs with products lack open post-training; open frameworks lack products. The §8 RL layer is what turns Dreamverse from a demo into a data engine, and the consistency ladder (§8.5) is what guarantees the preferences collected under the serving profile actually transfer into training.

9.3 Positioning

The diffusion community is structurally more diverse than LLM serving — many base models, LoRA ecosystems, workflow culture (ComfyUI) — and rewards stacks that compose models rather than serve one. "Vibe directing" names a UX category the way "vibe coding" did, and the moat behind it is not the frontend: it is the only stack where the app, the engine, and the post-training that makes the app possible are co-designed and parity-gated. Dreamverse also functions as the runtime's production forcing function (sessions, safety, cost-per-user, consumer-GPU targets — the Attn-QAT NVFP4-attention work aims at RTX 5090/4090/3090) — keeping the engine honest the way Cosmos3 keeps the model layer honest (§4.1). Dreamverse's own planned controller/provider split (local-first compute over Runpod/Modal, per its AGENTS.md) stays the single-box story; at fleet scale it delegates to Dynamo (§6.3.5) rather than growing its own orchestration — provisioning is not our plane.

9.4 The ComfyUI question: the funnel, the compiler, and the accelerated cloud

ComfyUI (~/ComfyUI, 116K stars, GPL-3.0) is where the commodity diffusion community lives, and its tree tells us exactly what it is and isn't: a node-graph orchestrator — 64 core nodes + 117 comfy_extras packs (Wan, Cosmos, Hunyuan, Mochi video natively in-core), a runtime-patching model manager (comfy/model_patcher.py, LoRA hot-patch, smart VRAM offload, FP8), 39 paid API-node providers in-tree (comfy_api_nodes/: Kling, Runway, Veo, Sora… — its monetization is orchestrating other people's models), 80 in-tree workflow blueprints (more via the registry) plus an App Mode that wraps graphs in simplified UIs — and no serving runtime: a FIFO prompt queue, no cross-prompt batching, no multi-tenancy, no distillation-awareness (execution.py, server.py). Structurally: ComfyUI is a frontend and ecosystem; FastVideo is a (recipe, runtime) factory. Those are complements before they are competitors, which shapes the play as three sequenced motions:

• Motion A — embed (exists today, underinvested). comfyui/ in this repo already ships ComfyUI-FastVideo custom nodes (multi-GPU VideoGenerator inside their graphs). This is the funnel: meet 116K-stars'-worth of users inside their own tool and let the speed delta advertise itself. A "FastVideo Cloud" API node — slotting into the same comfy_api_nodes pattern Kling/Runway use — is the cheapest distribution wedge their own platform has standardized.
• Motion B — compile (the core bet). The decisive observation, stated at its defensible scope: the tier-1/tier-2 static sublanguage of ComfyUI workflows maps onto PipelineSpec — within that vocabulary, both are typed dataflow graphs over model components (the general claim "any workflow is a PipelineSpec" is false: arbitrary custom nodes, Python-level patching, and dynamic graphs are exactly what the tiers exclude). The popular surface is small and maps almost 1:1 onto §6: CheckpointLoaderSimple→ModelSpec, KSampler→DenoiseLoop (sampler/scheduler/CFG policies), LoraLoader→LoRA hot-swap, CLIPTextEncode→TextEncode node, VAEEncode/Decode→codec nodes, ControlNetApply→ConditioningInjector, LatentUpscale→upsample stage, WanImageToVideo/EmptyCosmosLatentVideo→ task presets. A clean-room workflow compiler (ComfyUI JSON → PipelineSpec + OmniRequest; no GPL exposure — translate the format, never embed the code) over a tiered core vocabulary — tier 1 ≈ 20 nodes (t2i/i2v + LoRA + upscale), tier 2 ≈ 40 (inpaint, ControlNet, video-conditioning variants) — covers the canonical workflows, with capability negotiation: unsupported nodes produce a coverage report and a clear rejection, not silent wrongness. The long tail of arbitrary custom nodes is explicitly out of scope for v1 (option later: partition the graph and run unsupported subgraphs in a sandboxed headless ComfyUI sidecar process — GPL-safe by process isolation — but that is a deliberate, deferred complexity).
• Motion C — productize (Dreamverse Studio). Per the thesis that workflows matter less as models improve — which ComfyUI itself corroborates by shipping blueprints and App Mode to hide its own graphs — the destination is not a node editor clone but workflows as products: the top community workflows as one-click, real-time primitives inside Dreamverse (5 s 480p in ~2 s end-to-end today; consumer-GPU NVFP4 builds targeting 5090/4090-class cards), with cloud execution for whatever the local GPU can't hold.

What this genuinely adds to the architecture (and it is bounded — no plane is redesigned):

1. The workflow compiler — a new request-plane frontend, peer to the OpenAI server: parse, validate against the supported vocabulary, emit graphs. The graph plane (§6.2.4) was built for exactly this shape.
2. Heterogeneous checkpoint serving — the one real new subsystem. Workflow users bring arbitrary checkpoints, stacked LoRAs, ControlNets; the engine has so far assumed one resident model family per pool. Required: a weight/adapter cache tier in the CacheManager (disk→CPU→GPU, LRU, content-hash keyed ModelSpec resolution), fast patch/unpatch with ComfyUI-grade LoRA-stacking semantics, and model-affinity scheduling (route requests to pools where the checkpoint is resident — the diffusion-serving analog of sglang's prefix-affinity routing). The LoRA story, honestly sized: affinity scheduling + patch/unpatch is primary — group by (checkpoint, adapter-set, strengths), with §6.3.2's weight-transition scheduling pricing the swaps. Punica-style batched multi-LoRA is a deferred optimization with a real semantic gap: vLLM's LoRARequest is one integer id with scaling baked into lora_b at registration, while ComfyUI traffic is N stacked LoRAs with continuous user-set strength_model/strength_clip tweaked per generation — pushing that through Punica means registering every (ordered-set × strengths) tuple as a synthetic adapter: near-zero cache hits across strength tweaks, registration churn in the stacked weight slots, and concatenated ranks colliding with max_lora_rank. Batched multi-LoRA earns a place only where adapter sets are small and strengths fixed — named fast-mode presets — not as the general workflow-cloud mechanism.
3. Equivalence modes as product trust — the consistency machinery (§8.5) reused commercially, with the claims scoped precisely. Exact mode: the user's base model and sampler math, deterministic seeds, reproducibility — within our runtime; bit-parity with ComfyUI's own execution is never promised (its model patching, sampler/noise details, and dtype/offload behavior differ). Fast mode: distilled weights, NVFP4/QAT, cache-dit — dramatically faster, labeled, with equivalence reports that are quality-metric comparisons against a ComfyUI reference render (SSIM-class + the §12 eval-system gap, which gates this product claim), not parity assertions. This is the moat restated: ComfyUI Cloud and GPU rental can only run stock workflows on rented silicon; substituting a faster model credibly requires owning the recipes (§8.3) — which is the one thing an orchestrator structurally cannot do.

Risks, named: the compatibility treadmill (mitigation: tiered vocabulary + coverage reports, never chase the custom-node long tail), output-mismatch expectations (mitigation: exact/fast labeling), GPL hygiene (clean-room compiler; sidecar only ever as a separate process; the in-tree custom-node suite carries ComfyUI-side licensing), and platform response (ComfyUI controls the registry and frontend; our defense is the recipe flywheel it cannot replicate, §9.2). Decision recorded in §11.13.

---

10. Migration plan

Principles: every phase ships green (SSIM + parity suites); every phase deletes or freezes the code it replaces (no third stack); models migrate family-by-family behind the legacy adapter. Three additions this plan makes because the repo's own history demands them:

• Enforcement, not intent. The last freeze failed empirically: since train/ landed (2026-03-09, #1159), 19 commits modified the "frozen" training/ stack — including a brand-new cosmos2_5_training_pipeline.py added nine days after training/AGENTS.md forbade new models there (#1227), plus world-model training (#1179) and LongCat finetuning (#1244). This plan's freezes are therefore mechanical: a CI path gate rejects new files under fastvideo/training/ and new DenoisingStage subclasses once Phase 1 lands; CODEOWNERS on legacy paths gives the freeze a human veto; every phase has a named owner.

• The inflow rule. New families land at ~1–2/month (Flux2-Klein and Lucy Edit in a single day; MatrixGame3, MagiHuman, Stable Audio, Gen3C this spring). Untreated, the migration tail grows faster than phases retire it. Rule: from Phase-1 completion, new model ports land on the new abstractions — loop inversion + policies + the custom-step hatch suffice for standard diffusion families; until then, new ports land legacy and join the migration checklist with the porting team owning the later migration. Exceptions are an explicit owner decision.

• Sizing and dates. Rough per-phase engineer-month estimates, validated at each kickoff: P−1: 2–3 · P0: 2–3 · P1: 6–9 · P2: 8–12 · P3: 8–12 · P4: 10–15 · P5: 4–6. Owners are assigned at the design review, and Phase 5 gets a calendar target there — decision deadlines phrased as "by Phase 2" bind to that calendar, not to phase labels.

• Phase −1 — Land the baselines (prerequisite, not optional). Merge the Cosmos3 five-branch chain (feat/cosmos3-tier-a-port → feat/cosmos3-reasoning) to main, bringing its 150-test parity suite into CI: the design's forcing function currently exists only as unmerged branches (pipelines/basic/cosmos3/ on main holds a stray __pycache__), and no substrate refactor should proceed against an unmerged baseline — the alternative is months of side-branch rebasing across ForwardBatch→RequestState and loop inversion, unowned and unsized. Seed SSIM references for the ~7 uncovered families (Cosmos 2/2.5, Hunyuan, Hunyuan15(+SR), HYWorld, MagiHuman, Waypoint, MatrixGame-v1) so G5's instrument has no vacuous passes. Gate: cosmos3 suite green in CI on main; SSIM coverage = all shipped families.

• Phase 0 — Typed I/O (low risk, immediate value). OmniRequest/OmniOutput/TaskType + artifact outputs; VideoGenerator shim; ForwardBatch adapter view. The in-flight fastvideo/api/ schema converges here instead of landing as another parallel surface — GenerationRequest evolves into OmniRequest, GeneratorConfig into DeployConfig, and compat.py is frozen — no new entries, monotonic shrink per phase, zero lines at Phase 5 (§6.6). Cosmos3 audio leaves batch.extra. Gate: all SSIM suites unchanged.

• Phase 1 — Loop inversion + policies + extension core. DenoiseLoop/ARDecodeLoop + policy registry; migrate Wan family (incl. causal) and Flux2 first — they jointly exercise CFG variants, expert routing, chunk-KV, and image path; DenoisingStage subclasses shrink to specs + policies. The extension system core lands here because the loop is being rebuilt anyway: observer bus, Step/Block interceptors, the run_blocks convention, and the ParityAligner — with the cache-dit adapter (DBCache + TaylorSeer behind BlockInterceptor, algorithm-parity with sglang's integration) as the first interceptor; the dormant enable_teacache flag is retired in favor of plugin config, and fastvideo/forward_context.py (the vLLM-inherited global, §6.3.3) retires in favor of StageContext + AttnMetadataProvider + state extensions. train/ distillation methods begin adopting the shared step functions, and the landed DiffusionNFT (#1450) migrates off its vendored rl/common/sampling.py DiffusionSampler onto them (§8.1) — a migration of shipped code with users, so it gets its own parity evidence: identical seeded rollout latents before/after, plus reward-metric and grad-norm regression neutrality. CacheManager v0 ships here: per-request chunk-KV slabs behind the KVHandle seam — the Phase-4 pooled slab allocator is a swap behind the same handle (stated staging, not a silent throwaway implementation). Gate: per-model SSIM + a step-level parity harness, built on ParityAligner (old loop vs new loop, bit-identical); on the training side, the landed per-method grad-norm regression refs (#1396) extended to cover the migrated methods, RL included.

• Phase 2 — Engine + StepScheduler. Async engine, queue, concurrency, image batching, streaming events; OpenAI server parity with multimodal_gen (videos/images endpoints — chat arrives with omni in Phase 4, and AR continuous batching is deliberately deferred to Phase 4 per N5: the pulling workload is the Cosmos3 reasoner; Phase 2 carries only the ARDecodeLoop contract from Phase 1). LTX-2's loops migrate here as a linear graph (the degenerate case; full fan-out waits for Phase 3) — required because Dreamverse runs LTX-2 and the dogfood path needs its loop boundaries open. Rollout-service surface lands here too: update_weights RPC with the colocated transports and hand-checked per-family adapters (§8.6 — the general cross-mesh WeightSyncPlan follows the spec in Phase 3), component-granular sleep/wake, trajectory weight_version tagging, and the RolloutClient for RL engine-client mode (§8.7). The Dynamo-grade worker surface ships here (§6.3.5), scoped to stock Dynamo — registration wrapping AsyncEngine (retiring the locked examples/diffusers/worker.py pattern), health/drain, cost metrics; streaming/cost integration beyond stock follows asks A2/A3 or their fallbacks, and the gate does not depend on upstream landing. Offline path still bypasses the queue. Dreamverse is the dogfooding customer at Phase-2 scope: single-session parity — gpu_pool.py/queue/warmup/stream-relay retire into engine-client calls — plus a recorded duty-cycle concurrency study that sets the Phase-3 multi-session gate (§9.1). Gate: serving load tests; batch-of-1 video latency regression ≤ 2% (gated on the Phase-1 overhead measurement); an RL smoke test runs the landed DiffusionNFT (#1450) in engine-client mode with C1-pinned rollouts — seeded final-latent parity against its in-process path on Wan, per-step activation drift measured by ParityAligner and within the Phase-1-calibrated ε (per-step log-prob drift becomes the gated metric when the first GRPO-class method lands); Dreamverse runs single-session on the engine at parity; the engine deploys under stock Dynamo as an aggregated ModelType.Videos worker and passes health/routing smoke tests.

• Phase 3 — Graphs + placement + workflow frontend. PipelineSpec graphs, fan-out models (LTX-2, Hunyuan15+SR), role pools + connectors (port multimodal_gen disagg), deploy YAML, and the declarative stacked-parallelism spec with pre-flight validation and fake-pool CPU tests (§6.3.4). The ComfyUI workflow compiler MVP lands here (it is a client of the graph plane): the tier-1 (~20-node) vocabulary with capability negotiation, plus the weight/adapter fleet cache and model-affinity scheduling (§9.4). Dynamo integration deepens accordingly: affinity events and NIXL-aligned disagg bootstrap (§6.3.5; asks A1/A4 in §6.3.6), and the general cross-mesh WeightSyncPlan lands with the spec (§8.6). Gate: LTX-2 A/V end-to-end (full fan-out graph); disagg throughput benchmark; topology test suite green on CPU; the top community t2i+LoRA / i2v / upscale workflows compile and run with equivalence reports (quality-metric vs a ComfyUI reference render, §9.4); Dreamverse multi-session — ≥2 concurrent sessions per GPU on the recorded duty-cycle trace with p95 segment latency within the SLO set by the Phase-2 study (§9.1). The study's published load profile and targets double as the §11.6 falsifier: if step-level scheduling does not beat a request-level baseline on those numbers, the simpler scheduler wins and the StepScheduler is descoped — the complexity must earn itself on measurements, not on this document.

• Phase 4 — Omni/MoT native + RL consistency. PackedSeq + multi-pathway layers; CacheManager (paged text KV + chunk KV + feature caches); Cosmos3 re-port onto the new abstractions (reasoner gets KV cache; t2vs joint denoise; rollout); session-scoped world-model serving. AR continuous batching and paged-KV scheduling arrive here with their pulling workload (the Cosmos3 reasoner and omni chat — N5's condition met), as does /v1/chat/completions. RL hardening lands alongside: the consistency ladder enforced end-to-end (C1 default, C2 bitwise mode for goldens and MoE routing parity), Behavior Record capture/replay (an opt-in instrument, §8.5), TIS/MIS objectives as C0 fallback, and the omni-RL deliverable scoped per §8.7: hybrid trajectory capture demonstrated + the objective design study (open question 16) — not a pilot that presumes the objective exists. Gate: Cosmos3 150-test parity suite passes on the new runtime; reasoner pool efficiency — tokens/s/GPU at target concurrent denoise throughput (§6.3.1) — alongside the ≥10×-vs-re-prefill sanity floor; C1 rollout↔training drift ≈ 0 on a Wan RL run, with the drift dashboard live — per-step log-prob drift if a GRPO-class method has landed by then, seeded final-latent parity for the likelihood-free NFT otherwise.

• Phase 5 — Deprecation, with its real precondition stated. Phase 5 is not just "remaining families migrated": the legacy training/ stack is a live consumer of exactly what Phase 5 deletes (training_pipeline.py:39 imports ComposedPipelineBase/ForwardBatch/LoRAPipeline; its validation instantiates real legacy pipelines running the legacy DenoisingStage; distillation_pipeline.py:31 likewise). So legacy training/ retirement is a Phase-5 precondition: its remaining users migrate to train/ per the checklist, then ComposedPipelineBase, the legacy DenoisingStage, forward_context.py, compat.py, and RayDistributedExecutor are deleted together. This is where 4 loop copies → 1 actually completes — not before, and the document claims it nowhere earlier. Docs + .agents/ maps updated. Gate: the deletion diff itself — a tree with one loop implementation.

---

11. Trade-offs and alternatives considered

11.1 Adopt sglang multimodal_gen upstream instead of building here

FastVideo's runtime already has a serving-hardened fork living in sglang. Option: make sglang the runtime and FastVideo a model/stage library.

• Pros: serving layer (server, scheduler process, disagg, warmup, RL weight updates) exists and is maintained; one community runtime; router/gateway integration free.
• Cons: multimodal_gen inherited our P1–P3 problems and is diffusion-only — no AR loop, no KV cache, no step-level scheduling; the hard 40% of this design (loop inversion, hybrid MoT, CacheManager) doesn't exist there either, so we'd build it anyway — in a repo we don't govern, on sglang's release cadence, while FastVideo is the research vehicle (Cosmos3, VSA, new ports land here first).
• Decision: build natively, but keep API and concept parity (request/queue/disagg shapes, server endpoints) so the work upstreams cleanly later. The runtime layering (§6.0) is deliberately compatible with theirs. Open strategic question for the team: do we plan a re-convergence (upstream Phase 2's engine into multimodal_gen), or accept a long-lived friendly fork? This needs an explicit owner-level decision by Phase 2.

11.2 Multi-engine DAG composition only (vllm-omni / sglang-omni model)

• Pros: fastest route to separable omni (thinker/talker/vocoder); engine reuse; process isolation.
• Cons: expresses single-weights hybrids (Cosmos3 MoT, BAGEL lineage) only as one opaque monolithic stage — vllm-omni's bagel/lance are real shared-weight MoT, but request-scheduled (max_num_running_reqs=1), batch-incapable, with the AR+denoise interleaving invisible to the scheduler — so the runtime can't schedule/batch/preempt the loops inside; cross-stage KV is a copy, not a shared live cache; weight duplication for backbones split across separate stages; connector latency on the token path.
• Decision: adopt the composition concepts (declarative spec, connectors, per-stage schedulers, async_chunk streaming) inside one runtime, and offer external-engine nodes as an opt-in — rather than making cross-engine composition the foundation.

11.3 Status quo plus per-model monolithic stages

The Cosmos3 port proved a skilled engineer can ship an omni model inside today's abstractions in days. Why change?

• Pros: zero migration risk; maximum per-model freedom; no runtime tax.
• Cons: it buys exactly one request at a time with no KV cache — reasoning KV, batching, streaming, and prefix reuse remain unbuilt (this document's assessment; the port's PORT_STATUS.md records 150/150 parity and makes no production-gap claims of its own); every future omni/AR model re-pays the monolith cost; serving never materializes.
• Decision: rejected as an end state; embraced as a migration mechanism (the legacy adapter is this option, scoped per-family and temporary).

11.4 Big-bang rewrite vs incremental

A clean-room runtime would be simpler to design but violates G5 and repeats the two-training-stacks situation at larger scale. Decision: incremental with phase gates and an explicit Phase-5 deletion milestone. The known risk of "incremental" — the old path never dies — is mitigated by making each phase delete/freeze something.

11.5 Typed dataflow vs blackboard state

Full typed ports everywhere (every tensor an edge) is maximal safety but punishing verbosity for 20+ families; the pure blackboard is what produced P3. Decision: typed artifacts on graph edges + typed per-model state extensions inside nodes. Cost: two places state can live; mitigated by a lint rule — anything crossing a node boundary must be an edge artifact.

11.6 Scheduler granularity

Request-level scheduling (multimodal_gen) is far simpler; step-level adds queue/dispatch complexity and an SPMD-consistency obligation (decisions made on rank 0, broadcast — same discipline sglang and multimodal_gen already use). Decision: step-level only where loops exist; one-shot nodes schedule at request granularity. Without step-level scheduling there is no AR batching, no interleaving, no preemption, no MoT mode multiplexing — it is the price of G1/G4. The falsifier: the Phase-2 duty-cycle study publishes a load profile and targets; if at the Phase-3 gate step-level scheduling does not beat a request-level baseline on those numbers (≥2 concurrent sessions per GPU, p95 segment latency within the study's SLO), the StepScheduler is descoped to request granularity and the loop contract keeps only its streaming/preemption seams — the decision is measured, not assumed.

11.7 Cross-request batching for video diffusion

Batched video denoise (shape-bucketed or Cosmos-style token-budget packing) is real throughput but significant complexity (ragged attention, per-sample steps, CFG alignment). Decision: defer; v1 video concurrency = interleaving + parallelism. Revisit with data after Phase 2 (the StepScheduler interface already permits multi-request denoise batches, so this is additive).

11.8 Build the KV manager vs embed sglang srt

srt's pools are excellent but token-centric and entangled with its scheduler/attention stack; our second cache class (chunked causal-video KV with sink/local windows, session lifetimes) has no srt analog. Decision: native implementation borrowing srt's allocator design; no radix prefix tree in v1 (content-hash embedding cache covers the common reuse), reconsider radix when omni chat traffic justifies it.

11.9 Fleet layer: Dynamo as first-class partner vs alternatives

• Build our own router/autoscaler: rejected outright — it is a separate engineering discipline (Dynamo is a Rust codebase with etcd/NATS discovery, radix-indexed KV routing, an SLA planner, KVBM tiering, ModelExpress cold-start streaming), and every line of it we wrote would be a line not spent on the §8 substrate.
• sglang-router only: lighter, and remains a fine drop-in for simple single-model deployments — but it has no planner, no disagg coordination, no multi-tier cache story.
• Kubernetes-only (HPA + service mesh): no request-content awareness — no affinity routing, no SLA planner with per-request cost models, no KV/disagg coordination.
• Decision: Dynamo first-class (contract in §6.3.5). FastVideo already runs as a Dynamo ModelType.Videos backend (dynamo/examples/diffusers/worker.py), and Dynamo already fronts sglang-diffusion and vllm-omni — the fleet layer has standardized around exactly our ecosystem. Risks, honestly: Dynamo is NVIDIA-governed and its router is token-prefix-centric today (our model/session-affinity routing is a collaboration item, open question 15). Because the relationship with the Dynamo team is direct, gaps are handled as upstream asks with stated fallbacks (§6.3.6) rather than permanent workarounds. Mitigation regardless: the contract is deliberately thin and OpenAI-shaped, so the engine stays deployable bare, behind sglang-router, or under any future orchestrator without rework.

11.10 Explicit extension points vs monkeypatching (the cache-dit question)

cache-dit earns one-line adoption on diffusers (cache_dit.enable_cache(pipe)) by patching pipeline/transformer internals at runtime. We could allow plugins the same freedom.

• Pros: zero model-side convention; instant compatibility with third-party techniques as published.
• Cons: patches break under torch.compile/cudagraph capture and TP-sharded modules; state lands in module-level globals (unsafe under concurrent requests — disqualifying for a serving engine); no pre-flight conflict detection between plugins; silent breakage whenever model internals shift.
• Decision: explicit, versioned hook points (§6.4) plus the run_blocks convention for in-tree models. Module-level forward hooks remain available — but only to read-only observers, where the patching risks don't apply. For cache-dit specifically: depend on the library, not its patching path — drive cache-dit's configs/calibrators/skip math from behind our BlockInterceptor. sglang's integration is the precedent and the cautionary tale at once: it adopts cache-dit via BlockAdapter, then has to monkeypatch CachedContextManager.similarity for sequence-parallel tensors and hand-call refresh_context_* per request — exactly the two things (group-aware comparison, request-scoped lifecycle) our interceptor contract provides structurally (§6.4).

11.11 Adopt composable_parallel (RFC #4084) as a dependency vs native implementation

• Pros of adopting: shared maintenance; framework-agnostic by design (the RFC names SGLang as a second backend); the spec/validation/fake-backend machinery arrives prebuilt.
• Cons: the RFC is deliberately Option-A descriptive — runtime routing stays with the host engine, and the schemes we need most (per-node placement meshes, MoT pathway-sharded TP, chunk-KV-aware CP) are exactly the parts deferred to follow-up RFCs; it is an open proposal, not shipped code; we already own a working parallel_state.py and would be coupling our roadmap to a younger external contract.
• Decision: borrow the design — stacked axis specs, per-axis ownership markers, pre-flight validation, CPU-testable composition — and implement natively, keeping our spec schema close enough to theirs that converging on composable_parallel later stays cheap if it matures.

11.12 RL: single runtime vs hybrid-engine (verl-omni) vs dual-stack-with-corrections (miles)

• Hybrid-engine (verl-omni): reuse best-of-breed engines per role. For FastVideo this would mean re-implementing our own models inside vLLM-Omni to roll them out — verl-omni literally carries a Wan2.2 rollout re-implementation — then importing the correction machinery to patch the numerics. Absurd given we own an engine.
• Dual-stack with corrections (miles): the right call at ≥70B MoE scale where Megatron is non-negotiable; its mismatch toolkit is world-class — but the toolkit exists to dress a wound we do not have to open at 1–30B scale.
• Decision: single runtime, two execution profiles, explicit consistency ladder (§8.5). Crucially, we adopt miles' corrections (TIS/MIS) and Behavior Record replay as built-in C0 degradation modes rather than pretending mismatch cannot occur — profiles will legitimately differ (quantized rollout, cache-dit on), and if a Megatron-class trainer ever becomes necessary, the same toolkit is the escape hatch (§8.4 boundary condition). And the diffusion-specific clincher (§8.3): inference optimization here is post-training — an engine that cannot run inside training loops is cut off from its own optimization frontier, so single-runtime is less optional for us than it would be for an LLM stack.

11.13 ComfyUI: embed, compile, or compete

• Embed only (custom nodes inside ComfyUI — exists today in comfyui/): cheapest distribution; but FastVideo stays trapped under their queue (no batching, no sessions, no multi-tenancy) and the performance story is capped by their runtime.
• Compete head-on (rebuild the node editor + ecosystem): rejected — 116K stars of network effects and a thousand-pack custom-node ecosystem are not assailable frontally, and the graph-editor UX is precisely the layer the market (including ComfyUI itself, via App Mode and blueprints) is abstracting away.
• Compile (clean-room workflow compiler onto the engine + accelerated cloud): the leverage play — it converts their ecosystem into our demand without forking their UX battle, and its differentiation (credibly substituting faster recipes, with equivalence reports) is the one move an orchestrator cannot copy (§9.4).
• Decision: all three, sequenced and bounded. A = funnel (maintain, add a FastVideo-Cloud API node); B = core bet (Phase 3 MVP over the tier-1 vocabulary + weight-fleet cache; never chase the custom-node long tail — tiered coverage with explicit rejection, sidecar only if demand proves it); C = Dreamverse Studio as the destination (workflows-as-products, real-time as the wedge) — an evolution of §9.1–9.3, not a separate product. Scope discipline: B's vocabulary is a product decision reviewed per tier, not an open-ended compatibility promise.

---

12. Risks, concerns, and open questions

1. Complexity creep. Three planes, policies, graphs, the cache manager (per-class KV pools + feature + weight-fleet), an extension system, a workflow compiler, and an RL layer — and the scope has grown with every review pass of this document, which is itself the warning. Mitigations: phase gates with deletion milestones; the additions are sequenced, not front-loaded (fleet cache + compiler in Phase 3, RL hardening in Phase 4, C2 kernels only where they pay); the offline path stays a ~hundred-line trace (request → linear graph → loops) that a new contributor can read top-to-bottom; policies are plain objects, not a plugin framework.
2. Hot-loop overhead / compile friendliness. Per-step dispatch, policy indirection, observer/interceptor chains, and emit handling must not cost milliseconds; cudagraph/compile capture needs stable step bodies. Mitigations: policy bindings resolved at build time (policy state is per-request in LoopState, §6.2.3); hook chains assembled at build and literally absent when unused (§6.4), with needs_eager/graph_safe declarations scoping any capture loss; step() bodies are the same tensor programs as today's loop bodies; capture per (node, shape-bucket, branch-profile), with breakable CUDA graphs as the optimization tier (experimental upstream — the eager path is the correctness guarantee, §6.3.3); Phase-2 gate includes a ≤2% batch-of-1 latency budget, itself gated on the Phase-1 loop-inversion overhead measurement.
3. Always-on loop inversion has only narrow precedent — not the hybrid slice. The one prior art is vllm-omni's SupportsStepExecution (runtime-owned diffusion iteration at step granularity, but opt-in, Qwen-Image-only, off in every deploy — §6.2.2); no surveyed system makes it the always-on default (vLLM/sglang own AR token loops; diffusers' loop blocks own their own iteration — §6.2.3), and the hybrid AR+denoise multiplexing on shared weights is the deepest end of a bet that is novel from its first step. Mitigations: the Phase-1 parity gates are the load-bearing control (old loop vs new loop, bit-identical, per family — this is where the novelty risk is actually retired); the Phase-1 overhead measurement gates the latency story; the Cosmos3 port's 150-test bit-exact suite is the hybrid-slice prototype harness, exercised early in Phase 4 before generalizing.
4. SPMD-consistent scheduling. All ranks of a pool must agree on every step decision; divergence deadlocks NCCL. Mitigations: rank-0-decides + broadcast (existing pattern in multiproc executor and multimodal_gen); decision inputs restricted to broadcast state; a debug mode asserts cross-rank decision hashes. The same channel carries the abort broadcast (§6.3.1) — scheduling and failure isolation share one consistency mechanism by design.
5. Two stacks during migration (the very smell we're fixing). Mitigations: legacy adapter means zero dual-maintained models — a family is either unmigrated (old code, untouched) or migrated (old code deleted); per-family checklist tracked in .agents/; Phase 5 is a scheduled deletion, not an aspiration. The substrate split (§8) raises the stakes: once train/ methods consume the shared loops, a loop change breaks both worlds — so training rollouts join the parity gates (trajectory-identical before/after) and substrate PRs run training smoke tests, not just SSIM. The training-side gate layer already exists in CI to extend, not invent: #1396's per-method, device-keyed layer-0 grad-norm regression refs (fastvideo/tests/train/methods/, Modal-wired) — coarse 10%-rtol tripwires today, the natural carrier for the per-step parity assertions as methods migrate.
6. CI/GPU cost. Step-parity harnesses, serving load tests, SSIM suites, Cosmos3 parity on B200s. Mitigations: step-level parity tests run on CPU/tiny configs (the Cosmos3 oracle methodology generalizes); GPU suites stay per-family and gated to touched families — except substrate-wide PRs in Phases 0–2, where "touched" means all of them and the full matrix runs as a budgeted cost (G5).
7. Relationship with sglang multimodal_gen (§11.1) — needs an explicit decision by Phase 2: upstream, friendly fork, or shared core package. Drift here is a strategic cost regardless of architecture.
8. Single runtime does not equal automatic numerical parity. Two execution profiles remain (grad/checkpointed/FSDP-gathered training forward vs no-grad/graphed/sharded/cached rollout forward), and the failure mode is precisely characterized: kernels are deterministic per fixed batch but not batch-invariant (Thinking Machines' analysis — reduction strategies change with batch/shape/sharding), so profile differences alone break bit-equality; miles needed FA3 + DeepGEMM batch-invariant kernels to reach bitwise parity even within matched stacks. Mitigations: the consistency ladder (§8.5) makes the level explicit per run instead of implied; C1 is gated by ParityAligner drift CI (log-prob diff for likelihood-based methods, seeded final-latent/activation parity for NFT-class); TIS/MIS and the Behavior Record ship as built-in C0 degradation; a batch-invariant deterministic kernel mode (op-swap, §8.5 C2) is scoped for goldens rather than promised everywhere; drift metrics (logprob_diff and IS clip fraction where the objective has them, prediction-space deviations like #1450's old_deviate/old_kl_div otherwise) are always-on training telemetry, not post-mortem tools.
9. Open: session semantics. KV/cache lifetimes beyond a single request — interactive world models (game sessions), multi-turn omni chat, and now Dreamverse scene continuation (chained-segment frame/audio conditioning, §9.1), which makes this a production requirement with a concrete customer rather than a research nicety — owner: Phase 4 design review, with Dreamverse's protocol as the reference workload.
10. Open: speculative decoding for AR stages (sglang-omni lacks it; vllm-omni inherits vLLM's). The ARDecodeLoop.step contract should permit multi-token bundles from day one even if drafting lands later — miles already ships speculative rollout with online-SFT'd draft models, evidence this matters for RL throughput too. The contract shape to copy is vLLM v1's: draft tokens ride the scheduler output and are rejection-sampled in the same step (§6.3.1).
11. Open: quantization/FP8/FP4 interaction with multi-pathway layers and the Attn-QAT work (linear/MLP FP4 paths recently landed) — verify policy/layer factoring doesn't block per-pathway quantization configs. The ParityAligner observer (§6.4) should become the standard harness for validating precision changes, and miles' unified-FP8 lesson applies directly: rollout and trainer share one PrecisionPolicy/quant recipe — "train BF16, roll out FP8" is only legal at consistency level C0 (§8.5).
12. Open: hook-point surface stability. Which hook points are public/versioned vs internal, and whether the cache-dit adapter lives in-tree or upstream (vipshop/cache-dit already ships adapters for other frameworks) — decide when the first external plugin lands (Phase 1–2).
13. Open (narrowed): heavyweight reward-model packaging. The original concern — the #1438 draft vendored the full HPSv3/VideoAlign repos, trainers and deepspeed configs included — was resolved before landing: #1450 ships only thin PickScore/CLIP scorers loading weights from HF, and the add-reward-model skill now governs additions. What remains open is the heavyweight case: when HPSv3/VideoAlign/VLM-judge-class rewards land, do they arrive as pip dependencies, pruned runtime-only ports, or engine components scored off-rank (§8.7)? The landed pattern (GPU-resident scorer on every training rank) does not survive reward models with their own multi-GB footprints.
14. Open: workflow-compatibility tiering. Which ComfyUI node vocabulary each tier covers, what the coverage SLA is, whether/when the headless-ComfyUI sidecar for unsupported subgraphs is worth its operational cost, and how exact-vs-fast equivalence is reported to users (§9.4) — decide with Phase 3 data from the top community workflows.
15. Open: which Dynamo asks land upstream, and when. §6.3.6 enumerates seven (A1 affinity key spaces, A2 cost interface, A3 media streaming, A4 role-graph disagg, A5 RL weight plane, A6 KVBM generalization, A7 sessions), each with a fallback. Sequence them with the Dynamo team against our phases (A2/A3 → Phase 2, A1/A4 → Phase 3, A5–A7 → Phase 4); commit fallbacks into the plan only for asks that miss their phase. Ask-sequencing needs an owner and decision dates at the design review — and the A3 fallback (direct-WebSocket bypass) must be treated as temporary by policy, or it hardens into exactly the permanent workaround §11.9 claims the relationship avoids.
16. Open: the omni-RL objective (§8.7). Three named problems before any hybrid pilot: per-segment normalization across O(1–10)-nat token log-probs vs 10⁶–10⁷-dim Gaussian step densities; credit assignment across the non-differentiable sampled-token boundary into shared trunk weights; reasoning-regression control (reference-KL anchoring for the und pathway). Owner: the RL lead, before Phase 4 scopes its deliverable.

Known gaps — acknowledged, deliberately not yet designed

Named here so their absence reads as a decision, not an oversight. Each needs an owner and a design note before the phase it gates:

• Generation-quality evaluation system (gates Phase 2/3 product claims). SSIM gates and ParityAligner cover regression and numerical parity; they do not measure quality. "Fast mode is visually equivalent" (§9.4), RL reward validation (§8.7), and distillation-recipe comparisons all need a real eval harness — VBench-class benchmarks, reward-model scoring, human-preference sampling — with tracked baselines per model family.
• The flywheel's data plane (gates §9.2 → §8 RL). Preference events from Dreamverse need schema, storage, lineage, filtering, and a consent/licensing policy before any RL method may train on them. The design covers trajectory capture, not the data engineering or governance around it.
• Tenancy, quotas, metering, and input hardening for the cloud API (gates the §9.4 business). The engine emits per-request cost; who meters, bills, and rate-limits is undesigned. Workflow JSON is untrusted input — resource ceilings (resolution/steps/duration), schema validation, and abuse controls are required at the boundary.
• Artifact/model registry for (recipe, runtime) outputs. Distilled/QAT checkpoints are products; their versioning, distribution (HF/S3/ModelExpress), provenance back to recipes, and license posture as derivatives of differently-licensed bases are unspecified — interacts with the weight-fleet cache (§9.4) and A5 (§6.3.6).
• Long-job resilience beyond the pool. In-pool failure isolation, cancellation, partial artifacts, and preemption are now designed (§6.3.1); what remains undesigned is durability across engine restarts (where serialized LoopState persists, retention policy) and cross-pool job migration.
• Behavior Record at training scale. Full MoE-routing capture for Cosmos3-class requests is GB-per-sample (§8.5); if it is ever used beyond goldens/debugging, the streaming/storage/transport design for GB-scale trajectories from disaggregated rollout fleets does not exist.
• One observability schema. Drift telemetry (§8.5), profiler output (§6.4), cost metrics (§6.3.5), and engine stats are each named where they arise; Phase 2 should consolidate them into a single telemetry spec.

---

Appendix A — Source material

• FastVideo main: fastvideo/pipelines/ (composed_pipeline_base.py, stages/base.py:29, stages/denoising.py:237, pipeline_batch_info.py:238), fastvideo/worker/, fastvideo/entrypoints/, fastvideo/models/dits/causal_wanvideo.py:80-143.
• Cosmos3 port: the full-omni branch is https://github.com/hao-ai-lab/FastVideo/tree/feat/cosmos3-reasoning — top of the stacked chain (feat/cosmos3-tier-a-port → -i2v → -audio → -action → -reasoning), covering T2V/I2V/T2I, joint video+stereo-audio (t2vs), action (domain-aware pathway, joint vision+action+sound denoise), and greedy text reasoning — 150/150 parity tests vs the official framework. Local worktree: /home/william5lin/FastVideo_cosmos3_port; status doc tests/local_tests/cosmos3/PORT_STATUS.md.
• Official Cosmos3: cosmos-framework/cosmos_framework/model/vfm/ (mot/unified_mot.py, mot/cosmos3_vfm_network.py, omni_mot_model.py, data/vfm/sequence_packing.py, inference/inference.py).
• sglang: ~/sglang/python/sglang/multimodal_gen/ (runtime, disaggregation, pipelines_core) and ~/sglang/python/sglang/srt/ (managers, mem_cache, model_executor, disaggregation).
• vllm-omni: ~/vllm-omni/vllm_omni/ (config/stage_config.py, engine/orchestrator.py, distributed/omni_connectors/, diffusion/diffusion_engine.py).
• sglang-omni: ~/sglang-omni/sglang_omni/ (config/schema.py, pipeline/, relay/, scheduling/).
• NVIDIA Dynamo: dynamo/ (examples/diffusers/worker.py — the existing FastVideo video worker; lib/bindings/python/ (register_llm, @dynamo_endpoint, DistributedRuntime); lib/kv-router/src/protocols.rs (KV/affinity event format); components/src/dynamo/sglang/init_diffusion.py and components/src/dynamo/vllm/omni/ (sibling-stack backends); components/src/dynamo/planner/ (SLA autoscaler); lib/kvbm-* (multi-tier KV offload); docs/design-docs/disagg-serving.md). Contract analysis in §6.3.5.
• vLLM-Omni RFC #4084 "Composable Parallel Strategies": https://github.com/vllm-project/vllm-omni/issues/4084 (open proposal by @kushanam, 2026-06-03; evaluated in §5, §6.3.4, §11.11).
• cache-dit (training-free DiT cache acceleration: DBCache/FBCache/TaylorSeer): https://github.com/vipshop/cache-dit (integration model evaluated in §11.10).
• Existing FastVideo hook seeds: fastvideo/hooks/ (hooks.py — ModuleHookManager/ForwardHook; activation_trace.py — env-gated parity tracing with trace_step; layerwise_offload.py), and the dormant enable_teacache flag (fastvideo/pipelines/pipeline_batch_info.py:182).
• sglang's cache-dit integration: ~/sglang/python/sglang/multimodal_gen/runtime/cache/cache_dit_integration.py (enable_cache_on_transformer/_dual_transformer, _patch_cache_dit_similarity, refresh_context_*; cache_dit_config in runtime/server_args.py:116).
• Training stacks: fastvideo/train/ (trainer.py, methods/base.py, methods/distribution_matching/{dmd2,self_forcing}.py, methods/rl/ (see below), models/base.py, callbacks/validation.py, utils/training_config.py), legacy fastvideo/training/*_pipeline.py, preprocessing fastvideo/pipelines/preprocess/ (incl. preprocess_pipeline_ode_trajectory.py).
• RL (landed): PR #1450 "[feat] Add Wan RL DiffusionNFT training" (commit 5854aec2c, 2026-06-11): https://github.com/hao-ai-lab/FastVideo/pull/1450 — fastvideo/train/methods/rl/{__init__.py, diffusion_nft.py, common/{__init__,prompt_sampling,sampling,validation}.py, rewards/{__init__ (thebuild_multi_reward_scorer factory), frame_rewards, media}.py}, trainer hooks (manages_optimization/managed_train_step/on_validation_begin), ModelBase.decode_latents, examples/train/configs/rl/wan/diffusion_nft_pick_clip.yaml, and .agents/skills/{add-rl-method,add-reward-model,rlhf-training-abstractions}/SKILL.md. (Supersedes the #1438 "Adam/dnft" draft cited in earlier revisions of this doc.)
• Training-side regression gates (landed): PR #1396 (commit 633d39356) — per-method, device-keyed layer-0 grad-norm regression refs: fastvideo/tests/train/methods/{grad_norm_regression.py, grad_norm_refs.json}, Modal wiring in fastvideo/tests/modal/pr_test.py::run_train_framework_tests.
• Typed API in flight: fastvideo/api/ (schema.py, sampling_param.py, compat.py (651 lines), presets.py, results.py, request_metadata.py, per-model matrixgame2.py/matrixgame3.py).
• verl-omni: ~/verl-omni/verl_omni/ (trainer/diffusion/{ray_diffusion_trainer,rollout_correction, diffusion_algos}.py, pipelines/wan22_dance_grpo/vllm_omni_rollout_adapter.py, pipelines/schedulers/flow_match_sde.py, workers/rollout/vllm_rollout/vllm_omni_async_server.py, workers/engine/{fsdp,veomni}/, reward_loop/reward_manager/multi.py).
• miles: ~/miles/ (train.py, train_async.py, miles/backends/training_utils/loss_hub/{losses,corrections}.py (TIS/MIS), miles/backends/megatron_utils/update_weight/ (IPC/broadcast/P2P transports), miles/backends/megatron_utils/replay_utils.py (R3), miles/utils/types.py (Sample with rollout_routed_experts, weight_versions), miles/rollout/base_types.py (rollout-fn contract), docs/en/advanced/{miles-router,fp8-low-precision}.md).
• ComfyUI: ~/ComfyUI (GPL-3.0; execution.py + comfy_execution/caching.py (graph execution, output caching), nodes.py (64 core nodes), comfy_extras/nodes_{wan,cosmos,hunyuan,mochi,video_model}.py (in-core video), comfy/{model_management,model_patcher,lora,multigpu}.py, comfy_api_nodes/ (39 paid API providers), blueprints/ (80 in-tree workflow templates; more via the registry), QUANTIZATION.md); FastVideo's existing custom-node suite: comfyui/ (this repo).
• Dreamverse: apps/dreamverse/ (arch.md, design.md, AGENTS.md; runtime dreamverse/{main,gpu_pool, prompt_enhancer,session_init_image}.py; web/ Next.js frontend; serve_configs/streaming_demo.yaml); blogs: https://haoailab.com/blogs/dreamverse/ (vibe directing) and https://haoailab.com/blogs/fastvideo-dreamverse-release/ (release; NVFP4/FA4/compile on B200, fMP4-over-WebSocket, frame/audio segment conditioning).
• xDiT: ~/xDiT/xfuser/ (config/config.py — ParallelConfig axes; core/distributed/{parallel_state, group_coordinator,runtime_state}.py; docs/methods/{pipefusion,usp,hybrid}.md; DistVAE adapters in model_executor/models/runner_models/wan.py).
• cosmos-rl: ~/cosmos-rl/cosmos_rl/ (dispatcher/{controller,command}.py, policy/trainer/llm_trainer/ grpo_trainer.py (AIPO/dual-clip/off-policy masking), rollout/{rollout_base,worker/weight_sync}.py (RolloutBase registry; buffer-model async sync), rollout/wfm_rollout/, utils/{distributed,parallelism_map}.py).
• diffusers Modular: ~/diffusers/src/diffusers/modular_pipelines/ (modular_pipeline.py — LoopSequentialPipelineBlocks line ~1294, PipelineState/BlockState; modular_pipeline_utils.py — ComponentSpec; components_manager.py) and src/diffusers/guiders/ (guider_utils.py — BaseGuidance).
• torchtitan: ~/torchtitan/torchtitan/ (distributed/parallel_dims.py — DeviceMesh builder + validation; distributed/{fsdp,activation_checkpoint}.py; components/checkpoint.py — async DCP; experiments/rl/actors/{trainer,generator}.py — TorchStore weight sync; batch-invariance utils in distributed/utils.py).
• LiveKit agents: ~/agents/livekit-agents/livekit/agents/ (voice/avatar/_runner.py — AvatarRunner, AVSynchronizer; voice/avatar/_datastream_io.py — data-stream/RPC control; examples/primitives/video-publisher.py — rtc.VideoSource.capture_frame).
• Thinking Machines, "Defeating Nondeterminism in LLM Inference" (He et al., 2025): https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ — batch-invariance analysis and the true-on-policy (KL ≡ 0) RL result underpinning consistency level C2 (§8.5); companion library thinking-machines-lab/batch_invariant_ops (torch.Library op-swap, FlexAttention-based deterministic vLLM mode).
• vLLM core: ~/vllm (pulled 2026-06-10) + RFC #42770 "Changes in vLLM Model Development" (https://github.com/vllm-project/vllm/issues/42770, Woosuk Kwon, 2026-05-15; discussion incl. the compile lead's dissent and the two-tier head/tail resolution of 2026-05-28); linked: PR #42304 (breakable CUDA graph, VLLM_USE_BREAKABLE_CUDAGRAPH), #43224 (porting compiler fusions to manual call sites), #24384 (HF-decoupled config), vllm/v1/worker/gpu/model_states/interface.py (ModelState). FastVideo's inherited fastvideo/forward_context.py is the in-repo instance of the hack this RFC deletes (staged retirement — 194 references across ~50 files; new-path StageContext in Phase 1, global frozen for legacy consumers, deletion at Phase 5; §6.3.3). v1 code studied in depth: vllm/v1/engine/{async_llm,core,core_client,coordinator}.py (process topology, DP waves), vllm/v1/core/sched/{scheduler,output}.py (step-budget scheduling, in-step encoder budget, delta outputs), vllm/v1/core/{kv_cache_manager,kv_cache_coordinator,block_pool,encoder_cache_manager}.py (hybrid KV groups over one BlockPool; refcounted encoder cache), vllm/v1/worker/gpu/{input_batch,states}.py (persistent batch, staged writes), vllm/v1/cudagraph_dispatcher.py, vllm/distributed/kv_transfer/kv_connector/v1/base.py (KVConnectorBase_V1), vllm/lora/ + punica_wrapper/ (batched multi-LoRA), vllm/device_allocator/cumem.py (tag-based sleep).