2295 lines
214 KiB
Markdown
2295 lines
214 KiB
Markdown
# FastVideo Next-Generation Inference Runtime — Design Proposal
|
||
|
||
**Status:** Draft for discussion (v19 — v18 reconciled against the landed RL reference: PR #1450 Wan DiffusionNFT
|
||
training and PR #1396 per-method grad-norm regression CI; §8 claims converted from prediction to landed evidence)
|
||
**Date:** 2026-06-12
|
||
**Scope:** The inference runtime and the shared substrate beneath it. The legacy `fastvideo/training/` stack is
|
||
fully out of scope (frozen, per repo policy). The modular `fastvideo/train/` stack is **in scope as a consumer, not
|
||
as a redesign target**: its trainer machinery (Trainer, method loss logic, callbacks, FSDP/optimizer/checkpoint/
|
||
data) is untouched, but its embedded sampling/rollout paths migrate onto the shared loop substrate, its config
|
||
imports the shared layers, and the RL layer is built under it against the engine's rollout surface — §8 defines the
|
||
exact boundary.
|
||
|
||
---
|
||
|
||
## Table of contents
|
||
|
||
1. [Executive summary](#1-executive-summary)
|
||
2. [Goals and non-goals](#2-goals-and-non-goals)
|
||
3. [Where we are today](#3-where-we-are-today)
|
||
4. [What we must support: the model taxonomy](#4-what-we-must-support-the-model-taxonomy)
|
||
5. [Survey of reference systems](#5-survey-of-reference-systems)
|
||
6. [Proposed architecture](#6-proposed-architecture)
|
||
7. [Worked examples: seven workloads on the new runtime](#7-worked-examples)
|
||
8. [Training, post-training, and RL on the same substrate](#8-training-post-training-and-rl-on-the-same-substrate)
|
||
9. [The application pull: Dreamverse, ComfyUI, and the workflow layer](#9-the-application-pull-dreamverse-comfyui-and-the-workflow-layer)
|
||
10. [Migration plan](#10-migration-plan)
|
||
11. [Trade-offs and alternatives considered](#11-trade-offs-and-alternatives-considered)
|
||
12. [Risks, concerns, and open questions](#12-risks-concerns-and-open-questions)
|
||
|
||
---
|
||
|
||
## 1. Executive summary
|
||
|
||
FastVideo today is a **pipeline library that runs one request at a time**: a linear list of stages, a 111-field
|
||
`ForwardBatch` blackboard (~153 counting the training/preprocess batches in the same file), a 1,381-line
|
||
`denoising.py` whose ~670-line `DenoisingStage` owns the sampling loop, and an SPMD executor with no request
|
||
queue. This served bidirectional video diffusion well, but the model landscape has moved:
|
||
|
||
- **Autoregressive/causal video** (Wan-Causal, MatrixGame2/3, LongCat, Self-Forcing lineage) each re-implement
|
||
ad-hoc KV-cache dicts inside their DiT files.
|
||
- **Audio** arrived twice — as a joint output modality (LTX-2 video+audio) and as a standalone diffusion model
|
||
(Stable Audio) — and both had to be threaded through video-shaped abstractions (`batch.extra["audio"]`).
|
||
- **Image** models (Flux2, SD3.5) inherit a video denoising loop with `prefix == "Flux"` special-casing.
|
||
- **Omni models** are the forcing function. The Cosmos3 port — the **full-omni** branch
|
||
[`feat/cosmos3-reasoning`](https://github.com/hao-ai-lab/FastVideo/tree/feat/cosmos3-reasoning), top of the
|
||
stacked chain covering T2V/I2V/T2I, joint video+audio (t2vs), **action**, and text reasoning, all bit-exact
|
||
against the official framework (worktree `/home/william5lin/FastVideo_cosmos3_port`) — proved FastVideo *can*
|
||
run a mixture-of-transformers world model — but only by writing one monolithic stage that owns
|
||
tokenize→encode→denoise→decode, returning audio via `batch.extra`, and running text reasoning with **no KV
|
||
cache** (full re-prefill per token).
|
||
|
||
The proposal is to evolve FastVideo into an **engine with three cleanly separated planes**:
|
||
|
||
1. **Request plane** — typed multimodal requests/outputs (`OmniRequest`/`OmniOutput`), an async engine with a real
|
||
queue, streaming, and an OpenAI-compatible server.
|
||
2. **Pipeline plane** — pipelines become **declarative graphs of stages**; the two iterative computations
|
||
(**denoise loops** and **AR decode loops**) are *inverted* so the runtime owns iteration and stages expose
|
||
`init/step/finalize`. Model-specific behavior moves from `isinstance` forests into **policy objects** resolved at
|
||
pipeline-build time.
|
||
3. **Execution plane** — the existing SPMD worker pool, plus a **step-level scheduler** that multiplexes loop steps
|
||
across concurrent requests — with first-class **cancellation, failure isolation, and memory admission**
|
||
designed alongside it, because multiplexing creates the blast radius it must then contain (§6.3.1) — a
|
||
**cache manager** with per-class budgeted pools (paged text KV, slab chunk-KV, feature caches), and an optional
|
||
**placement layer** (role-disaggregated pools, per-stage devices, connectors) borrowed from sglang
|
||
`multimodal_gen` and vllm-omni. Pools are single-node in v1; fleet scale is Dynamo's plane.
|
||
|
||
Two systems cut across all three planes. An **extension system** (§6.4) provides versioned hook points: read-only
|
||
*observers* for debugging, profiling, and numerical-parity alignment (generalizing the existing
|
||
`fastvideo/hooks/activation_trace.py`), and compute-altering *interceptors* for cache accelerations —
|
||
[cache-dit](https://github.com/vipshop/cache-dit) (DBCache/TaylorSeer, the library sglang's multimodal serving
|
||
already integrates) as the flagship — with per-request, per-CFG-branch state so they stay correct under
|
||
concurrency. And **declarative stacked parallelism** (§6.3.4, adapted from vLLM-Omni RFC #4084) makes pool meshes
|
||
and engine topology one validated spec instead of scattered flags. Above the engine, fleet scale is explicitly
|
||
delegated: **NVIDIA Dynamo is the first-class partner** for discovery, cross-node routing, SLA autoscaling, and
|
||
cold starts — FastVideo already runs as a minimal Dynamo video backend, and the design hardens that contract
|
||
(§6.3.5) rather than rebuilding any of it. The partnership is two-way and Dynamo is not treated as frozen: §6.3.6
|
||
states seven concrete upstream asks (generic affinity key spaces, a request-cost interface, chunked media
|
||
streaming, role-graph disagg, an RL weight plane with staleness-aware routing, cache-object tiering, sessions),
|
||
each with a fallback. The asks are unfiled proposals on an NVIDIA-governed roadmap — fallbacks are first-class
|
||
plan elements, not afterthoughts, and the goal is Dynamo fronting both production serving and RL rollout fleets.
|
||
|
||
The same substrate answers the training question (§8). Integrated training/post-training is FastVideo's advantage
|
||
— structurally, not incidentally: unlike LLMs, where inference optimization is post-hoc on frozen weights, a
|
||
*usable* video model is itself a post-training artifact (step distillation is mandatory; low precision needs QAT —
|
||
our NVFP4 line; AR/world models need distillation plus self-forcing), so every inference capability is a
|
||
**(recipe, runtime) pair** and the training loop embeds the inference loop (§8.3). But today the denoise/sampling
|
||
loop exists in **four copies** (inference stages, `train/` distillation methods,
|
||
legacy `training/` pipelines — and the RL work, landed as PR #1450, vendored the **fourth** and documented why in
|
||
its own docstring: there was no shared loop it could call without binding to pipeline classes, §8.1). Loop
|
||
inversion makes the step functions the single shared implementation **for inference, `train/`, and RL** where they
|
||
fit — self-forcing ships as a custom training loop consuming substrate pieces (§8.2), and the
|
||
legacy `training/` copy stays frozen and retires per-family on the Phase-5 schedule, not by in-place unification;
|
||
the engine doubles as the RL rollout engine
|
||
(trajectory/log-prob capture, weight sync, sleep/wake) under a strict `engine never imports train` rule. Two
|
||
production RL systems sharpen the rule (§8.4): the industry's pain is **two runtimes with different kernels**
|
||
(Megatron/FSDP training vs vLLM/sglang rollout) — verl-omni re-implements Wan2.2 inside vLLM-Omni and corrects the
|
||
numerics afterward; miles' headline features (TIS/MIS, bitwise log-probs, R3 routing replay, unified FP8) are all
|
||
mismatch patches. We answer with one model definition, one kernel set, and an explicit **consistency ladder**
|
||
(C0 corrected / C1 kernel-pinned / C2 bitwise, §8.5) — measured by ParityAligner, never assumed. The fragmented
|
||
config story — `FastVideoArgs`/`TrainingArgs`/`TrainingConfig`/`SamplingParam` plus the in-flight `fastvideo/api/`
|
||
schema and its 651-line `compat.py` — collapses into four owned layers with generated views (§6.6). And the design
|
||
has a shipping product pulling on it (§9): Dreamverse, the vibe-directing app in `apps/dreamverse/`, today
|
||
hand-rolls its own GPU pool, queue, warmup, and streaming relay at a cost of one B200 per user session — the
|
||
engine's first production customer, and (via preference data from directing sessions) the eventual data engine for
|
||
the RL layer. The same section sets the ComfyUI strategy (§9.4): embed (custom nodes, shipping), **compile** (a
|
||
clean-room workflow→PipelineSpec compiler + weight-fleet cache powering an accelerated alternative to running
|
||
workflows on stock runtimes), and productize (Dreamverse Studio) — sequenced, with the custom-node long tail
|
||
explicitly out of scope.
|
||
|
||
The single most important architectural claim: **multi-engine DAG composition (vllm-omni / sglang-omni style) is
|
||
necessary but not sufficient.** It handles separable omni models (thinker → talker → vocoder, distinct weights per
|
||
stage). vllm-omni's `bagel_single_stage`/`lance` prove one resident instance *can* run both loops on shared
|
||
weights in one request — AR `generate_text` and diffusion `generate_image` over co-resident `*_moe_gen` experts
|
||
(`bagel_transformer.py:718,726`; `pipeline_bagel.py:677,808`). What a DAG of **separate** engines cannot do is
|
||
share the weights; what vllm-omni does **not** do is make those interleaved loops runtime-visible — it buries them
|
||
in one opaque request-scheduled `DIFFUSION` stage (`RequestScheduler`, `max_num_running_reqs=1`,
|
||
`diffusion_engine.py:166-169`), so the scheduler never sees the AR+denoise interleaving. FastVideo's
|
||
differentiation is therefore not *expressing* MoT (vllm-omni already does) but making the loops first-class:
|
||
a runtime where *one model instance* serves interleaved AR steps and denoise steps that are runtime-visible,
|
||
step-scheduled, and batchable — and that requirement, not serving features, is what dictates the stage/loop
|
||
abstraction below. Stated with open eyes: **no surveyed system makes runtime-owned diffusion iteration the
|
||
always-on default at scheduler granularity — there is only narrow, opt-in precedent: vllm-omni's
|
||
`SupportsStepExecution` (`prepare_encode`/`denoise_step`/`step_scheduler`/`post_decode`,
|
||
`diffusion/models/interface.py:44-67`) is runtime-owned diffusion iteration at step granularity, but
|
||
Qwen-Image-only and off in every deploy; this design generalizes it to an always-on universal contract**
|
||
(diffusers' loop blocks, by contrast, own their own iteration; §6.2.3). The risk is
|
||
retired by Phase-1 bit-identical parity gates and a measured falsifier (§10), not by borrowed validation. The
|
||
contract is also kept deliberately small: families may ship custom step bodies — the runtime owns *iteration*,
|
||
never the factoring of step internals.
|
||
|
||
Migration is incremental: seven phases (−1 to 5), each independently shippable, with SSIM/parity regression gates
|
||
and a compatibility adapter so the existing 20+ model families never break. The deprecation commitment is backed
|
||
by **enforcement the prior freeze lacked** — the `training/`-vs-`train/` "lesson" is not invoked as a moral but as
|
||
data: that freeze was broken 19 times including a new legacy pipeline nine days after it was forbidden — so this
|
||
plan carries CI path gates, CODEOWNERS, named owners, a calendar target for Phase 5, and an explicit inflow rule
|
||
for the ~1–2 new model families arriving every month (§10).
|
||
|
||
---
|
||
|
||
## 2. Goals and non-goals
|
||
|
||
### Goals
|
||
|
||
- **G1 — One runtime for all generation paradigms**: bidirectional diffusion (video/image/audio), causal/AR video
|
||
with KV cache, few-step distilled, multi-stage refinement, control/world models, AR token decoding (text, audio
|
||
codecs), and hybrid AR+diffusion (Cosmos3 MoT, BAGEL/Transfusion-style).
|
||
- **G2 — Omni-native I/O**: requests and outputs are typed sets of modality parts (text, image, video, audio,
|
||
action, latents). Video+audio+text outputs are first-class, not `extra`-dict passengers.
|
||
- **G3 — A real serving runtime**: async engine, request queue, concurrency, cross-request batching where shapes
|
||
permit, streaming (tokens, audio chunks, frame chunks), warmup, metrics — at parity with sglang `multimodal_gen`,
|
||
which is FastVideo's own descendant.
|
||
- **G4 — Cosmos3-class MoT support as a first-class citizen**: packed factored sequences, dual-pathway attention,
|
||
per-request hybrid KV+denoise state, reasoner KV cache, chunked world-model rollout.
|
||
- **G5 — Zero regression for existing models**: every shipped pipeline keeps working (SSIM-gated — coverage is
|
||
completed for *all* families in Phase −1, since ~7 currently have no SSIM test and would otherwise pass
|
||
vacuously; substrate-wide PRs in Phases 0–2 run the **full** GPU matrix on Modal L40S as a budgeted cost,
|
||
because for exactly those PRs "touched families" means all of them) throughout the migration; the offline
|
||
`VideoGenerator` API survives.
|
||
- **G6 — Keep the hot loop fast**: abstractions must resolve at pipeline-build time; the per-step path stays
|
||
compile/cudagraph-friendly; batch-of-1 video on N GPUs (SP/TP today, CFG-parallel once added) must not get
|
||
slower.
|
||
- **G7 — One substrate for inference, training, and RL**: models, loaders, configs, schedulers, parallel state, and
|
||
the generative loop bodies are single-source; `train/` and RL consume them in-process or via the engine as a
|
||
rollout service; the engine never depends on training code (§8).
|
||
|
||
### Non-goals
|
||
|
||
- **N1 — Datacenter orchestration.** Multi-node routing, SLA planners/autoscaling, fleet health, and cold-start
|
||
orchestration are not built here — **NVIDIA Dynamo is the named first-class partner for that layer** (`dynamo/`
|
||
checkout in repo root; FastVideo already runs as a minimal Dynamo video backend). The engine's job is to be an
|
||
excellent Dynamo citizen: registration, health/drain, cost metrics, affinity events, NIXL-compatible disagg —
|
||
the full contract and separation-of-concerns table is §6.3.5. We never schedule across nodes ourselves.
|
||
- **N2 — Trainer internals.** Legacy `fastvideo/training/` is frozen and untouched. In `fastvideo/train/`,
|
||
everything that makes it a *trainer* — `Trainer`, method loss logic, callbacks, FSDP wrapping, optimizers,
|
||
checkpointing, data loading — is not redesigned. §8 changes only what `train/` *consumes*: its embedded
|
||
sampling/rollout loops (today private copies) move onto the shared substrate, its `distributed:`/model config
|
||
blocks import the shared layers (§8.8), and the RL layer lands under it. Dependency rule: `train/` imports the
|
||
substrate/engine; never the reverse.
|
||
- **N3 — Replacing the model porting methodology.** Bit-exact per-component parity vs official frameworks (the
|
||
Cosmos3 rules) remains the porting bar; this design changes where ported components plug in, not how they are
|
||
validated.
|
||
- **N4 — Migrating all 20+ model families at once.** The plan defines a per-family migration checklist; legacy
|
||
pipelines run unmodified behind an adapter until migrated.
|
||
- **N5 — Standalone LLM-serving excellence.** We host AR computations because **omni and interactive models
|
||
require them** — Cosmos3's reasoner, thinkers/talkers, audio-codec decode, causal-video chunk loops — not to
|
||
compete with vLLM/sglang at LLM serving. The priority order is: **omni and diffusion done well — bidirectional,
|
||
causal, and interactive — first**; AR machinery enters only at the sophistication those workloads pull. Concretely:
|
||
paged KV and continuous batching for AR *pathways*, yes; radix prefix trees, speculative decoding, and
|
||
chunked-prefill sophistication, only when an omni workload demonstrates the need (they are flagged as open
|
||
questions, not commitments); a big *separable* thinker delegates to an external engine (§6.3.4) rather than
|
||
pulling its serving problems into ours. This is also the standing guard against over-indexing on LLM-engine
|
||
borrowings (§5): every vLLM/sglang adoption in this document is admitted under a diffusion/omni justification,
|
||
not on its LLM merits.
|
||
|
||
---
|
||
|
||
## 3. Where we are today
|
||
|
||
### 3.1 Current architecture
|
||
|
||
```
|
||
VideoGenerator / StreamingVideoGenerator / CLI / OpenAI entrypoints (fastvideo/entrypoints/)
|
||
│ one ForwardBatch per call (synchronous)
|
||
▼
|
||
Executor ABC ── MultiprocExecutor (1 proc/GPU, SPMD) ── RayDistributedExecutor (fastvideo/worker/)
|
||
│ collective_rpc / execute_forward
|
||
▼
|
||
ComposedPipelineBase (fastvideo/pipelines/composed_pipeline_base.py)
|
||
loads modules from model_index.json via ComponentLoaders
|
||
runs a LINEAR list of stages:
|
||
for stage in self._stages: batch = stage(batch, fastvideo_args)
|
||
▼
|
||
PipelineStage ABC (fastvideo/pipelines/stages/base.py:29)
|
||
__call__ → verify_input → forward(batch: ForwardBatch, args) → verify_output
|
||
▼
|
||
ForwardBatch (fastvideo/pipelines/pipeline_batch_info.py, 335 lines)
|
||
111 fields: prompts, embeds, latents, timesteps, camera_states, mouse_cond,
|
||
ltx2_*, trajectory…, plus `extra: dict[str, Any]` (line 238) as escape hatch
|
||
```
|
||
|
||
Parallelism today: TP and SP (Ulysses all-to-all only — ring-attention SP is *not* wired on main) — CFG-parallel
|
||
exists in the sglang `multimodal_gen` fork and in
|
||
cosmos-framework (CFGP) but **not on main**; this design adds it as an axis (§6.3.4). Plus per-component
|
||
CPU/layerwise offload, per-component `torch.compile`, and a strong attention-backend registry (FA, SDPA, Sage,
|
||
VSA, V-MoBA, BSA, SLA) selected via `fastvideo/attention/selector.py`.
|
||
|
||
### 3.2 What strains (evidence-backed)
|
||
|
||
| # | Pain point | Evidence |
|
||
|---|-----------|----------|
|
||
| P1 | **Monolithic denoising loop.** A 1,381-line `denoising.py` (the base `DenoisingStage` is ~670 lines; 35 `isinstance/hasattr/getattr` model probes file-wide); **22** model/variant denoising-stage classes across files, 23 with the base (Causal, CausalDMD, LongCat ×3, LTX2, GameCraft, HYWorld, MatrixGame2/3, Gen3C, Cosmos ×5, DMD, SR, SD35, Magi-class, +2 more). Flux gating via `getattr(config, "prefix", "") == "Flux"`; the Wan2.2 expert switch and TI2V inline VAE encode are hardcoded in the shared loop; Cosmos2.5's per-step clamping lives in its in-file subclass. | `fastvideo/pipelines/stages/denoising.py:129,237,352-376,441-475` |
|
||
| P2 | **The loop is owned by the stage**, so nothing outside it can schedule, preempt, interleave, stream, or batch steps. Causal/AR models bolt token/chunk loops inside stage subclasses; Cosmos3 had to bypass stages entirely with a free-function `denoise()` engine. | `causal_denoising.py:44,427`; worktree `cosmos3_stages.py` |
|
||
| P3 | **Blackboard state.** `ForwardBatch` mixes user intent, derived state, and per-model fields (`mouse_cond`, `camera_states`, `ltx2_*`, `pose`…); multi-modal outputs leave via `batch.extra["audio"]`. No stage declares what it reads/writes. | `pipeline_batch_info.py:238` |
|
||
| P4 | **No runtime.** One request per call; `assert latent_model_input.shape[0] == 1, "only support batch size 1"`; no queue, no async engine, no concurrency, no cross-request batching, no step interleaving. Streaming exists only as a bespoke `StreamingTask` path for game models. | `denoising.py:237`; `worker/multiproc_executor.py` |
|
||
| P5 | **AR support is per-model folklore.** `causal_wanvideo.py` threads a raw `kv_cache: dict` through forward (`:80,100,143`); LongCat has its own `LongCatKVCacheInitStage`; MatrixGame2 its own; Cosmos3 text reasoning re-prefills the full sequence per token (no cache at all). No allocator, no reuse, no eviction, no paging. | `models/dits/causal_wanvideo.py:80-143` |
|
||
| P6 | **No graph, no placement.** Multi-stage models (LTX-2 stage-1→upsample→stage-2 refine→audio decode; Hunyuan15+SR) are hand-wired linear lists; all stages run on all ranks; encoders cannot live on different GPUs; no fan-out for multi-modality outputs. | `pipelines/basic/ltx2/ltx2_pipeline.py` |
|
||
| P7 | **Task modes are inferred by heuristics.** Cosmos3 infers T2I from `num_frames==1 and image is None`; TI2V/V2V/LucyEdit ride boolean flags through the shared loop. | worktree `cosmos3_stages.py:101` |
|
||
| P8 | **Component set is static.** Loaders are keyed by fixed module names; Cosmos3's sound VAE had to be lazy-loaded inside `forward()` behind an env var. | worktree `cosmos3_stages.py:262-277` |
|
||
|
||
The Cosmos3 port is the best stress test we have: it succeeded (150/150 parity tests, every modality bit-exact),
|
||
but its friction list (monolithic stage, `extra["audio"]`, order-sensitive flat-latent slicing, no reasoning KV
|
||
cache, lazy sound VAE, mode heuristics) reads as a requirements document for this design.
|
||
|
||
---
|
||
|
||
## 4. What we must support: the model taxonomy
|
||
|
||
Every paradigm below is either shipped, in-flight, or explicitly requested (Cosmos 3, omni):
|
||
|
||
| Paradigm | Examples in/around tree | Loop shape | State needed | Output |
|
||
|---|---|---|---|---|
|
||
| Bidirectional video diffusion | Wan 2.1/2.2, Hunyuan(15), LongCat, Cosmos 2/2.5, LTX-2, TurboDiffusion | N denoise steps over full clip | latents, CFG branches, step/block caches | video |
|
||
| Few-step distilled | DMD/FastWan, TurboDiffusion, rCM | 1–4 denoise steps | latents | video |
|
||
| **Causal/AR video** | Wan-Causal(-DMD), MatrixGame2-Causal, MatrixGame3, LTX-2 causal, LongCat-VC | outer loop over chunks × inner denoise per chunk | **DiT KV cache** (per-chunk K/V, sink/local window), rolling latents | video (streamable by chunk) |
|
||
| Interactive world models | MatrixGame, GameCraft, HYWorld, Gen3C, LingBotWorld, Waypoint | chunk loop driven by live user actions | KV cache + action/camera conditioning per chunk | video stream |
|
||
| Image | Flux2, SD3.5, Qwen-Image-class | N denoise steps, small | latents | image (batchable!) |
|
||
| Audio diffusion | Stable Audio | N denoise steps over audio latents | audio latents | audio |
|
||
| Joint A/V diffusion | LTX-2 (video+audio in one denoise), Cosmos3 t2vs | one joint denoise over concatenated modality latents | multi-modality latents, per-modality CFG scales | video + audio |
|
||
| Multi-stage refinement | LTX-2 (base→upsample→refine), Hunyuan15-SR | pipeline of loops | inter-stage latents | video |
|
||
| **AR token decode** | Cosmos3 reasoner (Qwen-lineage und pathway); omni thinkers/talkers (Qwen2.5/3-Omni-style) | token loop until EOS | **paged text KV cache**, sampling state | text / codec tokens |
|
||
| Vocoder / one-shot heads | LTX-2 vocoder, audio codec → wav | single forward | none | audio |
|
||
| **Hybrid MoT (omni)** | **Cosmos3** (und/gen dual-pathway, shared attention, joint vision+action+sound denoise, AR reasoner on same weights), BAGEL/Transfusion lineage | AR loop **then/while** denoise loop, same resident weights | text KV + multimodal denoise state + packed sequence layout, in one request | text + video + audio + action |
|
||
|
||
### 4.1 The Cosmos3 forcing function (what "support omni" concretely means)
|
||
|
||
From the official `cosmos-framework` (`cosmos_framework/model/vfm/`):
|
||
|
||
- **MoT layer** = per-modality expert weights — separate QKV/O projections, norms, and MLPs for the
|
||
*understanding* (text, causal) and *generation* (vision/action/sound, full or NATTEN-sparse) pathways — with
|
||
**shared attention over one packed sequence** (`unified_mot.py`: `PackedAttentionMoT`, `MoTDecoderLayer`; gen MLP
|
||
optionally MoE).
|
||
- **Packed factored sequence**: per request, `[text (causal) | vision patches | action | sound]` with per-segment
|
||
attention modes, sample offsets, and 3D-MRoPE position ids (text 1D; vision T/H/W with FPS modulation; sound at
|
||
fixed 25 latent fps; `temporal_modality_margin` at boundaries).
|
||
- **One `velocity_fn` denoises all generated modalities jointly**; CFG = two packed forwards (cond/uncond) with
|
||
interval gating; sampler pluggable (UniPC/EDM/fixed-step).
|
||
- **AR reasoner runs on the und pathway of the same weights** with a per-layer `ReasonerKVCache` (prefill +
|
||
1-token decode), used standalone (text out) and as **prompt upsampling before diffusion in the same request**.
|
||
- **World-model rollout**: `video_temporal_causal` masks + chunked continuation (generate chunk, re-encode,
|
||
condition, continue).
|
||
- Inference batching in the framework is **greedy token-budget packing** of heterogeneous samples
|
||
(`cosmos_framework/inference/inference.py:66`, `_iter_packed_batches`) — diffusion requests batched by total
|
||
token count, not by fixed shapes.
|
||
|
||
A runtime that can serve this can serve everything else in the table. A runtime that composes separate engines per
|
||
stage **cannot** serve this (the reasoner and denoiser share weights; splitting them doubles 30B+ of weights and
|
||
breaks the interleaved KV/denoise state).
|
||
|
||
---
|
||
|
||
## 5. Survey of reference systems
|
||
|
||
What we take and what we reject, per system. (Full notes from the code review are in the per-repo paths cited.)
|
||
|
||
| System | What it is | Take | Reject / not enough |
|
||
|---|---|---|---|
|
||
| **sglang `multimodal_gen`** (`~/sglang/python/sglang/multimodal_gen/`) — FastVideo's serving-hardened descendant | HTTP/OpenAI server (images/videos/meshes), rank-0 `Scheduler` event loop + `GPUWorker`s over ZMQ, `Req` (= ForwardBatch + SamplingParams delegation), **N:M:K role disaggregation** (`disaggregation/orchestrator.py`: encoder/denoiser/decoder pools, capacity-aware dispatch, request state machine, tensor-transfer with P2P/RDMA), warmup requests, RL weight updates | Server/API shape; scheduler-process topology; **role affinity on stages** + disagg state machine; warmup; `Req` w/ sampling-param delegation | Request-level scheduling only (one request per step, no batching/interleave); diffusion-only — no KV cache, no AR loop, no token scheduling; stage/batch abstractions inherited our P1–P3 problems |
|
||
| **sglang `srt`** (`~/sglang/python/sglang/srt/`) | LLM serving: continuous batching (`Req`/`ScheduleBatch`, EXTEND/DECODE), radix prefix cache, paged KV (`ReqToTokenPool` + allocators), CUDA-graph decode, overlap scheduling, PD disaggregation with bootstrap/transfer queues | **Concepts**: paged KV pool + allocator design, scheduler event loop w/ overlap, chunked prefill, PD-disagg queue pattern, multimodal-embedding cache keyed by content hash | Wholesale embedding: srt is token-centric; its scheduler, radix tree and attention backends assume LLM decode. We borrow designs, not the stack |
|
||
| **vllm-omni** (`~/vllm-omni/`, HEAD #4168) | Multi-stage omni serving: frozen declarative `PipelineConfig`/`StagePipelineConfig` (stage_id, `execution_type ∈ {LLM_AR, LLM_GENERATION, DIFFUSION}`, `input_sources` DAG, `final_output_type`), YAML deploy (devices/TP/memory/connectors per stage), per-stage engines in own processes (stock vLLM core for AR; `DiffusionEngine` for DiT, inline-or-remote), `OmniConnectorBase` put/get + `chunk_ready` readiness signals (SHM + RDMA via Mooncake/Mori/Yuanrong — *not* NIXL/NCCL), readiness-set parking + bounded-window-K + preemption for flow control (**no credit-based flow control anywhere**), global orchestrator + per-stage schedulers, `async_chunk` streaming | **Declarative pipeline spec separate from deploy YAML** (the frozen `PipelineConfig` + editable `deploy/<model>.yaml` `DeployConfig` fused by `merge_pipeline_deploy` — adopted here verified, §6.1); connector interface with readiness signals; inline-vs-remote stage execution; per-stage independent schedulers; streaming-input (resumable) requests; **`SupportsStepExecution` as loop-inversion prior art** (opt-in, Qwen-Image-only — we generalize it, §6.2.2); its **3 separate cache subsystems** (`OmniTensorPrefixCache`, `OmniKVTransferManager`, diffusion/cache TeaCache/MagCache/cache-dit) confirm our per-class-pools thesis (§6.3.2) | vllm-omni **does** express shared-weight MoT (`bagel_single_stage`/`lance`: co-resident `*_moe_gen` experts, one resident instance running AR `generate_text` + diffusion `generate_image` in one request) — but only as **one opaque `DIFFUSION` stage the scheduler never sees inside** (`RequestScheduler`, `max_num_running_reqs=1`), so the AR+denoise interleaving is invisible to scheduling/batching; cross-stage KV is a TP-rank/CFG-branch-aware **copy** (`OmniKVTransferManager`), not a shared live cache; **separate-stage** weight sharing is absent (correctly); diffusion batching exists but is opt-in, homogeneous-`SamplingParamsKey`-gated, Qwen-Image-only, and off in every deploy; streaming is linear-chain in practice (routing hardwired `src+1`, `orchestrator.py:1122`) though the schema permits a DAG |
|
||
| **sglang-omni** (`~/sglang-omni/`) | Extension over stock sglang: `StageConfig` DAG (factory, gpu, tp_size, process colocation, `next/route_fn/wait_for/merge_fn/stream_to`), per-stage schedulers (`OmniScheduler` wrapping sglang by composition, `SimpleScheduler`, `DllmScheduler` for the LLaDA2-Uni diffusion-LLM thinker — Ming-Omni's DiT talker runs on a different scheduler — plus streaming variants), **Relay** transport (NIXL/SHM/NCCL/Mooncake, credit-based flow control), hidden-state capture thinker→talker, segmenter-based streaming | Relay abstraction + credit flow control; fan-in `merge_fn` / fan-out / `stream_to` edge semantics; scheduler-composition pattern; hidden-state capture as a declared stage output | Same gap: stages own disjoint weights/caches; hybrid AR+diffusion only as AR stage → DiT stage; per-model bootstrap duplication |
|
||
| **vLLM-Omni RFC #4084 — "Composable Parallel Strategies"** ([issue](https://github.com/vllm-project/vllm-omni/issues/4084), open, 2026-06) | Proposal: every parallelism scheme (TP/DP/SP/CP/EP/PP/CFG/VAE-PP/stage replicas) becomes a declarative `StrategySpec` — `mesh_axis` (name+degree), L1 routing/aggregation pattern, optional L2 per-layer hooks and L3 kernel factories — stacked into a mesh tree with generic collective-group construction; thin adapter translates onto omni's existing runtime (`l1_owner` marks who routes each axis, preventing double-routing) | **Stacked-axis spec as single source of truth** + generic group construction; the **L1/L2/L3 intervention vocabulary** (we reuse it for both parallelism and the extension system); **pre-flight topology validation**; **CPU-testable composition** via a fake process backend; explicit per-axis ownership; fail-loud on unsupported combos instead of silent fallback | Adapter/translation indirection is their constraint (retrofitting vLLM), not ours — we implement the spec natively in `fastvideo/distributed/`; parallelism-only scope (no loops, caches, or hybrid-model answers); the schemes hardest for us (CP variants, affinity routing) are exactly the parts deferred to follow-up RFCs — design proven, implementation not |
|
||
| **verl-omni** (`~/verl-omni/`) | RL post-training for diffusion/omni models (verl lineage): Ray single-controller, vLLM-Omni rollout servers + FSDP2/VeOmni trainers, DanceGRPO/FlowGRPO/DiffusionNFT/Diffusion-DPO recipes (Qwen-Image, SD3.5, Wan2.2), async multi-reward serving, "rollout correction" | Reward-loop architecture (weighted multi-reward, HTTP scorers, async overlap with rollout); SDE-with-logprob scheduler formulation (`FlowMatchSDEDiscreteScheduler`); component-granular sleep levels; group-relative advantage utils; bypass-vs-decoupled log-prob modes | Pays the full two-runtime tax (§8.4): Wan2.2 re-implemented inside vLLM-Omni for rollout *and* in diffusers/FSDP2 for training, stitched by `rollout_correction.py` (IS + rejection masks) — the machinery a single-runtime design exists to delete |
|
||
| **miles** (`~/miles/`) | Production LM RL (slime fork, SGLang RL team): SGLang rollout + Megatron training; the deepest extant treatment of train-inference mismatch (bitwise log-probs, TIS/MIS, R3 routing replay, unified FP8, INT4 QAT) | The mismatch toolkit → our consistency ladder C0–C2 + Behavior Record (§8.5); three weight transports behind one interface + per-sample `weight_versions` staleness (§8.6); the custom-rollout-function contract; health-monitored engines; log-prob-diff debug tooling | Dual model definitions + Megatron↔HF weight-name bridges by construction; Megatron is required at *its* scale (≥70B MoE) — which confirms our FSDP2 single-runtime bet only holds at FastVideo's 1–30B scale (§8.4 boundary condition) |
|
||
| **cosmos-framework** (`./cosmos-framework/`) | Official Cosmos3 (§4.1) | The model-side requirements; token-budget **packed batching for diffusion**; sampler pluggability; CFG interval gating; CP/CFG-parallel patterns | It is a research framework: no queue, no serving, FSDP-centric; not a runtime to copy |
|
||
| **NVIDIA Dynamo** (`./dynamo/`) | Rust/Python fleet orchestration above engines: OpenAI frontend, KV/affinity-aware router (NATS cache events → radix index), prefill/decode disagg coordination with NIXL transfer, SLA Planner (replica autoscaling via K8s CRDs), KVBM multi-tier KV offload, ModelExpress cold-start weight streaming. **Already fronts our ecosystem**: a FastVideo video worker exists (`examples/diffusers/worker.py`), plus sglang-diffusion and vllm-omni backends | The **first-class partner** for everything fleet (§6.3.5): registration/health/drain contract, cost metrics for the Planner, affinity events, NIXL-aligned disagg — production scaling we never build | Rebuilding any of it in-engine; its router keys on token prefixes today and media streaming/cost-models/role-graphs have gaps — addressed as concrete upstream asks A1–A7 (§6.3.6), not workarounds |
|
||
| **xDiT / xfuser** (`~/xDiT`) | The de-facto diffusion-parallelism library: USP (`sp = ulysses × ring`), **PipeFusion** (displaced patch pipeline-parallelism that reuses *stale KV from the previous denoise step* — mandatory warmup steps), CFG-parallel (`cfg ≤ 2`), DistVAE (VAE on its own process group), TeaCache/FBCache adapters; world = dp×cfg×sp×pp×tp with nested group init | Two axis-vocabulary updates (§6.3.4): **`pp_patch`** with declared applicability conditions (non-causal only; wins at high resolution / weak interconnect; tolerable for few-step distilled), and the recognition that DistVAE ≡ our decode role pool — no new axis needed; their PipeFusion×SP hybrid rule (KV replication required) as a composition constraint | Wrapper-per-model integration (pipeline + transformer + attention-processor wrappers per family) is exactly the per-model maintenance shape we're eliminating; no serving/scheduler; PipeFusion is architecturally incompatible with causal/AR video (stale KV breaks causality) — a pre-flight validation rule, not a footnote |
|
||
| **cosmos-rl** (`~/cosmos-rl`) | NVIDIA's RL framework for physical AI — **now deprecated as Cosmos3 unifies the model families**: single controller + Redis command queues, POLICY/ROLLOUT replicas (colocated or disagg), pluggable rollouts (vLLM, TRT-LLM, diffusers `NFTRollout`, in-process `WFMRollout`), GRPO with **AIPO** one-sided clipping + dual-clip + KL-threshold off-policy masking, **buffer-model async weight sync** on a side CUDA stream, elastic `HighAvailabilitylNccl` mesh rebuild, `weight_version` in every payload | AIPO + off-policy masking into the §8.7 objectives (C0-mode stability kit); the buffer-model async-sync pattern (§8.6); per-payload weight versioning (independently validates our staleness design); elastic-mesh fault tolerance as the recovery reference | Dual model definitions per role (weight mappers, per-role `parallelize`) — the two-runtime tax again; and no production video rollout engine (`NFTRollout` is generic diffusers) — confirmation that the §8 rollout engine is the missing piece in this space, not a duplicate of an existing one (our own landed DiffusionNFT, PR #1450, confirms it from the other side: trainer-side library code sampling through a vendored bare-model loop, §8.1 — still no serving-grade video rollout engine anywhere) |
|
||
| **diffusers Modular Pipelines + Guiders** (`~/diffusers`, pulled to head) | HF's blessed re-architecture of pipelines: block composition (`Sequential`/`Conditional`/`Auto`), `LoopSequentialPipelineBlocks` (marked experimental), untyped `PipelineState`/`BlockState`, `ComponentSpec` + `ComponentsManager` (auto-offload), `modular_model_index.json` hub format, and **Guiders** (CFG/PAG/APG/SkipLayer… with start/stop step scheduling); 12+ families incl. Wan, LTX, HunyuanVideo1.5, Flux2 | **Convergent validation of the policy/components pillars only**: Guiders are exactly pluggable CFG policies, and `ComponentSpec` matches our `ModelSpec` components — adopt compatible serialization and a `BaseGuidance` adapter (§6.2.3, §6.6). **It is *not* loop inversion**: `LoopSequentialPipelineBlocks.__call__` raises `NotImplementedError`; every concrete family hand-writes `for i, t in enumerate(timesteps)` inside its own wrapper (`wan/denoise.py:434`, `sdxl/denoise.py:701` — SDXL ships four such wrappers, the subclass forest again). Iteration is block-owned and invisible to any runtime — equally consistent with the alternative this design rejects ("keep loops in stages, make bodies pluggable") | No serving, batching, scheduler, AR loops, or parallelism; untyped blackboard state (their version of our P3); class-defined blocks vs our serializable graph IR — export is clean, import is lossy |
|
||
| **torchtitan** (`~/torchtitan`) | PyTorch-native pretraining reference: `ParallelDims` building multi-axis **DeviceMesh** (pp/dp_replicate/dp_shard/cp/tp/ep) with product-of-degrees pre-flight validation and cached submeshes; FSDP2 per-block + MoE prefetch; selective activation checkpointing by FQN; async DCP checkpointing; per-block compile; per-mesh-dim deterministic seeding; `set_batch_invariance()` utilities; an RL experiment (PolicyTrainer → vLLM generator) syncing weights via **TorchStore push/pull with direct RDMA** | The substrate answer for §6.3.4: compile the stacked-axis spec to DeviceMesh via a `ParallelDims`-style builder; TorchStore/DCP as the PyTorch-ecosystem weight-sync precedent for §8.6 (caveat: structure-matched fill, **not** automatic cross-mesh resharding — `WeightSyncPlan` still owns layout math); trainer patterns for `train/` (selective AC, async ckpt, determinism utils); upstream batch-invariance feeding C2 (§8.5) | LLM-pretraining-shaped; no real inference engine (Forge is minimal); full-DTensor everywhere is overkill for engine hot paths — adopt the pattern, not the stack |
|
||
| **LiveKit agents / WebRTC** (`~/agents`) | Realtime AI agents over a WebRTC SFU: `rtc.VideoSource.capture_frame()` raw-frame publishing (RGBA/I420; encoding in a native thread — non-blocking to asyncio), `AVSynchronizer`, data channels + named RPCs for low-latency control (interrupts, turn-taking), avatar `VideoGenerator` protocol | The §9 transport trigger matrix: fMP4/WS suffices for segment streaming; **WebRTC is mandatory for interactive world models** (<100 ms motion-to-photon ≈ 20–40 ms action RTT + ≤40 ms/frame inference + ~10 ms encode/decode), with data channels as the action path; engine contract: `OmniEvent` optionally carries **raw frames + PTS**, sessions accept an async action stream + flush RPC | Real operational weight (SFU, TURN, tokens) — adopt only when triggers fire; frames must be CPU-side buffers (GPU→CPU copy on the hot path) |
|
||
| **vLLM core — RFC #42770 "Changes in vLLM Model Development"** ([issue](https://github.com/vllm-project/vllm/issues/42770), Woosuk Kwon, 2026-05, open; `~/vllm` pulled to head) | A philosophical reversal by the largest engine: **drop the full-graph `torch.compile` requirement** (manual "big fused ops" with hardware dispatch written directly in model code; local compile kept for adjacent-op fusion; **breakable CUDA graphs** — the SGLang technique, prototyped in PR #42304 — replacing compiler-managed piecewise capture; `forward_context` named as a compiler-compat hack to remove); per-vendor model code for head models; an explicit **`Model`/`ModelConfig` interface** where models own their state and `prepare_inputs` (model runner V2's `ModelState`) and config decouples from HF (#24384). After community pushback (the compile lead's detailed dissent; OOT vendors' CustomOp concerns), the RFC settled on a **two-tier resolution**: hand-fused vendor-specific head models + shared portable definitions for the long tail | Validations and one technique: model-owned state + model-agnostic runner converges with this design (diffusers Modular converges on the components/policy half — not the runner, §6.2.3); HF-decoupled `ModelConfig` ≈ our `ModelSpec` manifest; the two-tier head/tail resolution ≈ our flagship-vs-family split (§6.3.3); and **breakable CUDA graphs** as the capture mechanism for loop step bodies with interceptor branches. Plus a direct work item: FastVideo's `fastvideo/forward_context.py` is inherited from exactly the hack vLLM is deleting — staged retirement starting Phase 1 (shim until Phase 5, §6.3.3). **From the v1 code itself** (beyond the RFC): the unified-progress scheduling structure — adapted to a **cost-model currency**, since bidirectional denoise steps and causal AR tokens are incommensurable work units (§6.3.1) — plus in-step encoder budgeting, the KVCacheGroups hybrid allocator studied and *not* adopted (its single `BlockPool` requires uniform page bytes across groups — `get_uniform_page_size` — which our 150–500× KV-granularity spread violates; we keep per-class budgeted pools, §6.3.2), `EncoderCacheManager` semantics for feature caches, `KVConnectorBase_V1` as the connector protocol shape (§6.3.4), Punica batched multi-LoRA evaluated for the workflow cloud (adopted only for fixed-adapter fast modes — its one-id/baked-scale semantics cannot express ComfyUI's stacked-LoRAs-with-strengths, §9.4), `CuMemAllocator` tag-based sleep (§8.6), and the persistent-batch/staged-write/delta-output CPU discipline behind the ≤2% step budget | The dissent is worth keeping in view: manual fusions don't scale across quant schemes/hardware, and per-fused-op kernel selection pressure re-creates IR-like autotuning ("starts to look very similar to IROps") — arguments for keeping our policy/registry layer rather than going fully manual; vLLM core remains LLM-scoped (diffusion/omni stays in vllm-omni) — and per N5, every borrowing from this repo is admitted on a diffusion/omni justification, never on LLM-serving merits |
|
||
|
||
---
|
||
|
||
## 6. Proposed architecture
|
||
|
||
### 6.0 Overview
|
||
|
||
```
|
||
┌────────────────────────────── Request plane ───────────────────────────────┐
|
||
│ OpenAI-compatible server (videos/images/audio/chat, SSE/WebSocket) │
|
||
│ VideoGenerator (offline sync API, unchanged) AsyncEngine (new) │
|
||
│ OmniRequest ──► admission ──► request queue ──► OmniOutput │
|
||
└─────────────────────────────────────────────────────────────────────────────┘
|
||
┌────────────────────────────── Pipeline plane ──────────────────────────────┐
|
||
│ PipelineSpec (declarative graph, per model family) │
|
||
│ nodes: Stage (one-shot) | LoopStage (DenoiseLoop, ARDecodeLoop) │
|
||
│ edges: typed Artifacts (Embeddings, Latents[modality], Tokens, KVRef…) │
|
||
│ policies: CFGPolicy, ExpertRouting, AttnMetadata, Precision, FlowShift │
|
||
│ components: shared module registry (one resident copy per worker pool) │
|
||
└─────────────────────────────────────────────────────────────────────────────┘
|
||
┌────────────────────────────── Execution plane ─────────────────────────────┐
|
||
│ Engine scheduler (per-request, graph-node dispatch, role pools) │
|
||
│ StepScheduler (per worker pool: multiplex denoise steps + AR steps) │
|
||
│ Worker pool(s): SPMD procs, TP/SP groups (as today) + CFG-parallel (new) │
|
||
│ CacheManager: paged text-KV ▪ chunked causal-video KV ▪ feature caches │
|
||
│ Connectors (in-proc → SHM → NCCL/NIXL) for cross-pool edges │
|
||
│ Attention backends, torch.compile / cudagraph capture per (node, bucket) │
|
||
└─────────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
Default deployment is exactly today's: one SPMD pool, all graph nodes co-located, synchronous offline call. Every
|
||
serving feature is additive configuration, not a different code path for models. Two cross-cutting surfaces thread
|
||
through all planes: the hook/plugin extension system (§6.4) and the declarative parallelism spec (§6.3.4).
|
||
|
||
### 6.1 Request plane: typed omni I/O
|
||
|
||
```python
|
||
@dataclass
|
||
class OmniRequest:
|
||
request_id: str
|
||
task: TaskType # explicit: T2V, I2V, TI2V, V2V, V2W, T2I, T2A, T2VS, A2W, REASON, ...
|
||
inputs: list[ModalPart] # TextPart | ImagePart | VideoPart | AudioPart | ActionPart | LatentPart
|
||
sampling: SamplingParams # AR knobs: max_tokens, temperature, top_p, stop, seed
|
||
diffusion: DiffusionParams # steps, guidance (per-modality scales), sigmas/shift, resolution, frames
|
||
outputs: OutputSpec # requested modalities + streaming flags + return_latents/trajectory
|
||
node_params: dict[str, NodeOverrides] = field(default_factory=dict)
|
||
# per-graph-node overrides, validated against the PipelineSpec
|
||
priority: int = 0
|
||
|
||
@dataclass
|
||
class OmniOutput:
|
||
request_id: str
|
||
artifacts: dict[str, Artifact] # "video": VideoArtifact, "audio": AudioArtifact(sample_rate=…),
|
||
# "text": TextArtifact, "action": TensorArtifact, "latents": …
|
||
metrics: RequestMetrics
|
||
```
|
||
|
||
Decisions this encodes:
|
||
|
||
- **Task is declared, never inferred** (kills P7). Heuristics may *suggest* a default at the API boundary, but the
|
||
pipeline graph branches on `request.task`.
|
||
- **Outputs are named artifacts** (kills the `extra["audio"]` pattern, P3). Audio carries its sample rate; text
|
||
carries token ids + decoded string; everything carries provenance (which node produced it).
|
||
- **Streaming is part of the request**: `outputs.stream = {"text": True, "audio": chunk_ms=200, "video": per_chunk}`.
|
||
The engine returns an `AsyncIterator[OmniEvent]` when any stream flag is set.
|
||
- **Multi-loop graphs are parameterized per node** — decided here, in Phase 0, because external consumers build
|
||
against this schema first: `sampling`/`diffusion` are *defaults*; `node_params["refine"].steps` or
|
||
`node_params["talker"].max_tokens` override per graph node, validated against the node-parameter schema each
|
||
`PipelineSpec` declares. This is what LTX-2's refine loop needs (its own step count and guidance scale are
|
||
first-class fields *today*, `fastvideo_args.py:204-205`) and what any thinker/talker pair needs; the rejected
|
||
alternative — per-model fields on the universal schema — is exactly the `ltx2_*` leakage P3 indicts.
|
||
- `VideoGenerator.generate_video(prompt=…)` remains and constructs an `OmniRequest` internally — the offline API is
|
||
a thin shim, preserved verbatim for G5.
|
||
|
||
### 6.2 Pipeline plane
|
||
|
||
#### 6.2.1 State: replace the god-batch with Request / RequestState / typed edges
|
||
|
||
`ForwardBatch` conflated three things. We split them:
|
||
|
||
- **`OmniRequest`** — immutable user intent (above).
|
||
- **`RequestState`** — per-request mutable execution state, owned by the runtime: artifact slots keyed by graph
|
||
edge, loop states, cache handles, RNG state, per-request metrics. A small **typed core** (`latents:
|
||
dict[Modality, Tensor]`, `cond: ConditioningSet`, `timesteps`, `step_idx`) plus a **per-model extension
|
||
dataclass** registered by the pipeline (e.g. `Cosmos3State(packed: PackedSeq, reasoner_kv: KVHandle, …)`,
|
||
`MatrixGameState(mouse, keyboard, kv: KVHandle)`).
|
||
- **Edges carry typed `Artifact`s** (Embeddings, ImageLatents, VideoLatents, AudioLatents, TokenIds, HiddenStates,
|
||
KVRef, ActionTensor). Stages declare input/output artifact types; the graph validates at build time — which fields
|
||
a stage reads/writes stops being archaeology (P3, P6).
|
||
|
||
Trade-off (discussed in §11.5): full typed dataflow everywhere is verbose; a pure blackboard is unverifiable. The
|
||
hybrid — typed edges between nodes + a typed per-model state extension inside nodes — is the deliberate middle.
|
||
`ForwardBatch` survives during migration as an adapter view over `RequestState` for unmigrated stages.
|
||
|
||
#### 6.2.2 Stages and loop inversion (the core change)
|
||
|
||
```python
|
||
class Stage(Protocol): # one-shot transforms: encoders, VAE en/decode, packers, vocoders
|
||
def forward(self, ctx: StageContext, state: RequestState) -> None: ...
|
||
|
||
class LoopStage(Protocol): # iterative computations: the runtime owns the loop
|
||
def init(self, ctx, state) -> LoopState: ...
|
||
def step(self, ctx, loop: LoopState) -> StepResult: ... # ONE step: one solver step / one token / one chunk
|
||
def finalize(self, ctx, loop: LoopState) -> None: ...
|
||
|
||
@dataclass
|
||
class StepResult:
|
||
done: bool
|
||
emit: list[Artifact] | None = None # streamed chunks: tokens, audio chunks, decoded frame chunks
|
||
```
|
||
|
||
Two `LoopStage` families cover every paradigm in §4:
|
||
|
||
- **`DenoiseLoop`** — `init` builds timesteps/sigmas, allocates latents, resolves policies; `step` performs one
|
||
solver step (model forward(s) + CFG combine + scheduler step). Sub-variants by composition, not subclassing:
|
||
joint multi-modality latents (LTX-2, Cosmos3 t2vs), chunked-causal (`step` = one denoise step of the current
|
||
chunk; chunk advance handled by an outer `ChunkRollout` loop node holding the KV handle), few-step distilled
|
||
(timesteps from config).
|
||
- **`ARDecodeLoop`** — `init` runs prefill into a `KVHandle`; `step` decodes one token (or one speculative bundle),
|
||
samples, appends, optionally emits; `done` on EOS/max_tokens. Used by Cosmos3 reasoner, omni thinkers/talkers,
|
||
audio-codec generators.
|
||
|
||
Why inversion is the keystone:
|
||
|
||
1. **Scheduling**: only the runtime sees step boundaries, so only inversion enables continuous batching of AR steps,
|
||
interleaving denoise steps of concurrent requests, preemption between steps, and fair sharing between a 50-step
|
||
video job and a 4-step image job (P2, P4).
|
||
2. **Streaming** falls out: `StepResult.emit` is the universal chunk channel (text tokens, talker codec chunks,
|
||
causal-video decoded chunks) instead of three bespoke mechanisms (P4).
|
||
3. **Hybrid models** become a graph of loops over shared components: Cosmos3 = `ARDecodeLoop(reasoner)` →
|
||
`Pack` → `DenoiseLoop(joint)` → decoders — same transformer module bound to both loop nodes (G4).
|
||
4. **Checkpointing/elasticity**: a `LoopState` is a serializable resume point (future: migration between workers,
|
||
crash recovery for 1000-step jobs).
|
||
|
||
Cost: the loop body must be re-entrant and the per-step dispatch must stay cheap (§12, risk 2). `DenoiseLoop.step` for
|
||
batch-of-1 video compiles/captures exactly like today's inner loop body; nothing tensor-level changes.
|
||
|
||
Note the contract's deliberate smallness — this is the design's main simplicity lever: the runtime requires only
|
||
`init/step/finalize`. *How a family implements `step` is free* — composed from policies by default, or a custom
|
||
body when the math is genuinely entangled (§6.2.3). Loop inversion is about **who owns iteration**, not about how
|
||
step bodies are factored; everything the scheduler, streamer, and parity harness need comes from the small
|
||
contract, never from the factoring.
|
||
|
||
#### 6.2.3 Policies: dissolving the `isinstance` forest
|
||
|
||
By default, a model-conditional currently probed inside `DenoisingStage` becomes a small strategy object **resolved
|
||
once at pipeline build** and injected into the loop (kills P1 without 22 subclasses) — *default*, not admission
|
||
requirement: families whose step bodies braid across slot boundaries keep custom steps under the escape hatch below:
|
||
|
||
| Policy | Replaces | Examples |
|
||
|---|---|---|
|
||
| `CFGPolicy` | chunked cond/uncond logic, adaptive-gate delta reuse, LongCat batched-CFG, per-modality scales, Cosmos interval gating, embedded-guidance (Flux ×1000) | `ClassicCFG`, `BatchedCFG`, `AdaptiveGateCFG`, `PerModalityCFG`, `EmbeddedGuidance` (cfg-parallel is a parallelism *axis*, not a policy — see below) |
|
||
| `ExpertRouting` | Wan2.2 `boundary_ratio` transformer switch | `BoundaryTimestepRouting(transformer, transformer_2)` |
|
||
| `AttnMetadataProvider` | VSA / V-MoBA / STA builder `isinstance` chain in the loop | registered per backend; built in `init`, refreshed per step only if needed |
|
||
| `PrecisionPolicy` | `prefix == "Flux"` autocast hacks, `scheduler_step_in_fp32`, cast flags | declared in model config |
|
||
| `FlowShiftPolicy` | Cosmos3 resolution-bucket shift, per-task sigmas | config-driven lookup table |
|
||
| `ConditioningInjector` | camera/action/pose/audio conditioning plumbed through batch fields | per-model injectors that write into `RequestState.cond`; the loop is agnostic |
|
||
|
||
**The step skeleton and the policy contract.** A three-phase step (forward → CFG → scheduler step) does not
|
||
factor the shipped loops; they need more slots, with dependencies that cross policy boundaries. The default
|
||
`step` body is a fixed skeleton of ordered, typed slots:
|
||
|
||
```
|
||
prepare_inputs → [per guidance branch: pre_forward → forward → post_forward] → combine
|
||
→ scheduler_step → post_step → inter_step
|
||
```
|
||
|
||
with three rules. **(1) Binding vs state.** Policy *bindings* resolve at pipeline build (which classes, which
|
||
slots); policy *state* is allocated per-request in `LoopState` at `init` — the same scoping rule as plugins,
|
||
because the same failure applies: `AdaptiveGateCFG`'s cached delta (`denoising.py:338-343, 507-551`) is
|
||
per-request mutable state, and a build-time singleton would smear request A's CFG delta into request B the moment
|
||
interleaving lands. **(2) Observation.** Policies read a typed per-step `StepContext` (timestep, branch, active
|
||
expert/model id, sampler coefficients) — the channel by which `AdaptiveGateCFG` observes `ExpertRouting`'s switch
|
||
to invalidate its delta (today an inline `id(model)` check), and by which the guidance **branch vocabulary is
|
||
policy-declared per request at `init`** (plugin per-branch state scopes over the *declared* set — LTX2's 1–4
|
||
runtime-decided passes declare theirs up front, not a fixed cond/uncond pair). **(3) Slots cover the common
|
||
shapes, not all shapes** — verified against shipped code: TI2V's post-`scheduler.step` latent clamp is a
|
||
`post_step` policy (`denoising.py:570-573`); CausalDMD's between-step renoising with expert-boundary-dependent
|
||
noise selection is `inter_step` × `ExpertRouting` (`causal_denoising.py:268-301`); Cosmos2.5's per-frame timestep
|
||
vectors and per-step GT re-clamp are `prepare_inputs`/`pre_forward`. But Cosmos's conditioning-frame injection
|
||
consumes the *sampler's* EDM coefficients inside each CFG branch with an x0-space combine
|
||
(`denoising.py:845-933`), and LTX2's guidance passes alter the network via forward kwargs
|
||
(`ltx2_denoising.py:503-605`) — braids no slot factoring owns cleanly.
|
||
|
||
**CFG-as-policy is proven by precedent, not speculative.** vllm-omni's `CFGParallelMixin` already unifies
|
||
sequential-2-forward, batched, *and* cfg-parallel under one pair — `predict_noise_maybe_with_cfg` +
|
||
`combine_cfg_noise` (`vllm_omni/diffusion/.../cfg_parallel.py:76-212`) — so one shared denoise-step body plus a
|
||
swappable `CFGPolicy` expressing classic (2-forward), batched (1-forward stack+chunk), adaptive-gate (cached-delta
|
||
reuse with model-id invalidation), per-modality, and embedded-guidance (single-branch) is an existence proof, not
|
||
a hope. The right reading of vllm-omni's "three CFG mechanisms" is **one unified in-loop pair + one parallelism
|
||
axis + one orchestrator layer**, not three independent CFG systems. We make that boundary explicit in **three
|
||
layers**: **(1)** `CFGPolicy` is *in-loop* — it owns the branch vocabulary, the combine formula, and per-request
|
||
mutable state (the adaptive-gate cached delta is the canonical state case); **batched-vs-2-forward is a dispatch
|
||
detail *inside* one policy, not a separate mechanism** — which is why there is no standalone `BatchedCFG` axis to
|
||
collide with parallelism. **(2)** cfg-parallel is a *parallelism axis* — it shards `policy.predict`'s branches
|
||
across ranks and runs the **same rank-invariant combine** on every rank, and **composes under any policy**; a
|
||
request never owns both a batched-CFG policy variant *and* a `cfg` parallel group (declaring both is rejected,
|
||
§6.3.4). **(3)** companions are an *orchestrator pattern* — a request is split into companion sub-requests
|
||
*upstream* of diffusion, and the loop receives already-computed conditioning unchanged. Two caveats the slots must
|
||
respect: the combine runs in the **step body's numeric space** (Cosmos combines in **x0-space** post-EDM, not
|
||
noise-space — `denoising.py:845-933`); and **embedded-guidance/Flux is a degenerate single-branch identity-combine
|
||
policy** (guidance rides inside the forward kwarg), *not* "no CFG".
|
||
|
||
Which is why the scheduler's contract is deliberately smaller than the policy system: **a family may ship a
|
||
custom `step` body** — a hand-written function using samplers, CFG math, and conditioning utilities as a library
|
||
(the Cosmos3 port's denoise-engine pattern, legitimized). A custom step is still a `LoopStage`: scheduled,
|
||
budgeted, preempted, streamed, observed, and parity-gated exactly like a composed one, and the skeleton-owned
|
||
plumbing (offload dance, attention metadata, autocast, trajectory capture) is provided either way. Policy
|
||
factoring is the default because it deletes duplication; it is **not an admission requirement**. So the honest
|
||
form of the claim: a new model contributes *at minimum* a graph spec and a step body — policies where they fit —
|
||
and never edits shared loop code.
|
||
|
||
External validation, stated at its true scope: diffusers' **Modular Pipelines** independently converged on the
|
||
*policy* pillar — **Guiders** (CFG, PAG, APG, SkipLayer…, each with start/stop step scheduling) are exactly
|
||
pluggable CFG policies, and `CFGPolicy` stays adapter-compatible with their `BaseGuidance` interface so model
|
||
ports and guidance research transfer in both directions (§6.6). It does **not** validate loop inversion: their
|
||
loop blocks own their own iteration, invisible to any runtime (see §5). The honest position: **there is only
|
||
narrow, opt-in precedent — vllm-omni's `SupportsStepExecution` (`prepare_encode`/`denoise_step`/`step_scheduler`/
|
||
`post_decode`, `diffusion/models/interface.py:44-67`) is runtime-owned diffusion iteration at step granularity,
|
||
but Qwen-Image-only and off in every deploy; no surveyed system makes it the always-on default. This design
|
||
generalizes that contract to a universal, always-on one** (diffusers' loop blocks, by contrast, own their own
|
||
iteration), and the Phase-1 bit-identical parity gates are what carry that risk (risk 3).
|
||
|
||
#### 6.2.4 PipelineSpec: declarative graphs
|
||
|
||
Frozen, validated, registered per model family (the vllm-omni `PipelineConfig` shape, extended with loop nodes,
|
||
branches, and fan-out):
|
||
|
||
```python
|
||
COSMOS3_OMNI = PipelineSpec(
|
||
family="cosmos3",
|
||
components={"transformer": …, "vae": …, "sound_vae": Component(optional_for={…}, required_for={"t2vs"}),
|
||
"vision_encoder": …, "tokenizer": …}, # declared, eagerly loadable per task (kills P8)
|
||
nodes=[
|
||
Node("validate", InputValidation()),
|
||
Node("tokenize", Cosmos3Tokenize()), # text → TokenIds
|
||
Node("vis_cond", VAEEncodeConditioning(), when=task_in(I2V, V2W)), # image/video → clean latents
|
||
Node("reason", ARDecodeLoop(model="transformer", pathway="und"), # prompt upsampling / REASON
|
||
when=task_in(REASON) | opt("upsample_prompt")),
|
||
Node("pack", Cosmos3Pack()), # → PackedSeq artifact
|
||
Node("denoise", DenoiseLoop(model="transformer", # SAME module as "reason"
|
||
modalities=[VISION, ACTION?, SOUND?],
|
||
cfg=PerModalityCFG(interval_gated=True), sampler=FromConfig())),
|
||
Node("vdec", VAEDecode("vae")), # fan-out ↓
|
||
Node("adec", AudioVAEDecode("sound_vae"), when=task_in(T2VS)),
|
||
Node("assemble", AssembleOutput()), # → OmniOutput artifacts
|
||
],
|
||
edges=[…typed…],
|
||
)
|
||
```
|
||
|
||
Properties:
|
||
|
||
- **Branches** on `request.task` and request options replace geometry heuristics (P7).
|
||
- **Fan-out/fan-in** are real: video and audio decode in parallel after a joint denoise; thinker hidden-states can
|
||
feed both a text detokenizer and a talker (sglang-omni `wait_for/merge_fn` semantics).
|
||
- **Components are shared by reference.** Two loop nodes binding `"transformer"` get the same resident module —
|
||
this is the MoT requirement, impossible in engine-per-stage designs.
|
||
- **Linear pipelines are the degenerate case**; the existing stage lists translate mechanically.
|
||
- A separate **deploy config** (YAML) maps nodes → pools/devices/parallelism (vllm-omni's split of frozen pipeline
|
||
vs deployment), defaulting to "one pool, everything colocated".
|
||
|
||
### 6.3 Execution plane
|
||
|
||
#### 6.3.1 Engine and two-level scheduling
|
||
|
||
**Level 1 — engine scheduler (per request, per graph node).** Async intake → admission (validate, resolve
|
||
PipelineSpec, allocate request state) → dispatch nodes to worker pools as their inputs become ready; a request
|
||
state machine generalizing `multimodal_gen`'s `RequestTracker` (waiting → node-running → node-done → …), with
|
||
role-pool capacity dispatch (N:M:K encode/denoise/decode) available when configured.
|
||
|
||
**Level 2 — StepScheduler (per worker pool).** The new capability. Each pool runs an event loop over **ready loop
|
||
steps**:
|
||
|
||
```
|
||
while True:
|
||
work = select(ready_loops, policy) # rank0 decides, broadcast to pool (SPMD-consistent, as today)
|
||
├─ AR group: batch decode steps of all active ARDecodeLoops (continuous batching;
|
||
│ paged KV; chunked prefill for long prompts/vision contexts)
|
||
├─ Denoise group: pick one (or a shape-compatible batch of) DenoiseLoop step(s)
|
||
└─ interleave by priority/deadline; preemption only at step boundaries
|
||
```
|
||
|
||
Batching semantics are **paradigm-aware** (this is where "one scheduler" earns its keep):
|
||
|
||
- **AR steps**: classic continuous batching (sglang-style), batched across requests every iteration. CUDA-graph
|
||
decode per batch-size bucket.
|
||
- **Image diffusion**: cross-request batching with resolution buckets — the workload where batching pays most
|
||
(Flux2/SD3.5 serving).
|
||
- **Video diffusion**: batch-of-1 with TP/SP (plus the new CFG-parallel axis) stays the default (memory-bound); concurrency comes from
|
||
*interleaving* steps of multiple requests (a 4-step distilled job no longer queues behind a 50-step job) and,
|
||
later, Cosmos-framework-style token-budget packing of heterogeneous samples as an opt-in.
|
||
- **Hybrid (Cosmos3)**: reasoner decode steps join the AR group; the joint denoise joins the denoise group; both hit
|
||
the same resident weights, so the StepScheduler is also the **mode multiplexer** the MoT model needs. The
|
||
multiplexer needs a *parallelism* answer too, because the two loop types want disjoint layouts — denoise wants
|
||
SP(+CFG); AR decode is sequence-length-1, where SP has nothing to shard and CFG doesn't exist. The stated
|
||
answer, rather than a deferred contradiction: on a `[cfg × sp]` pool, **AR decode runs data-parallel across the
|
||
cfg×sp weight-replica axes** (every replica already holds full weights; the decode batch shards across them,
|
||
paged text-KV partitioned per replica — not duplicated), optionally over per-pathway TP subgroups (§6.3.3);
|
||
and **pure-REASON traffic is routable to differently-shaped pools** via cost classes and affinity routing, so
|
||
reasoner token latency is not gated by indivisible denoise steps except when a single request genuinely
|
||
interleaves both on the same weights. The Phase-4 gate measures pool efficiency (reasoner tokens/s/GPU at
|
||
target denoise throughput), not just the re-prefill strawman.
|
||
|
||
**Mechanics adopted from the vLLM v1 scheduler** (`vllm/v1/core/sched/scheduler.py` — production-proven, and a
|
||
cleaner formulation than our group split above):
|
||
|
||
- **Unified progress accounting — but the budget currency is cost, not counts.** v1 has no prefill/decode phases:
|
||
every request is `num_computed_tokens` catching up to a target under a per-iteration token budget, and that
|
||
works because a *token* maps near-linearly to work in the LLM regime. **A step does not.** A bidirectional
|
||
denoise step re-attends over the entire latent sequence at O(L²) with zero KV amortization — every step pays
|
||
full price — while AR prefill is causal-masked incremental work against a cache (O(n·C) for n new tokens on
|
||
context C), a decode step is ~O(C) per layer, and a chunked-causal video step sits between (one chunk vs cached
|
||
context). Counting "steps" would put items three orders of magnitude apart in one bucket. So we keep v1's
|
||
*accounting structure* — `steps_done`/`steps_target` progress counters, one admission/budget loop — but the
|
||
budget is **predicted GPU-time from a per-(model, phase, shape) cost model**: each schedulable item (a denoise
|
||
step at given L and CFG mode, an AR prefill chunk, a decode token-batch, an encoder forward) converts to the
|
||
common currency before admission. The cost model is policy-aware (batched-CFG ≈ 2× batch vs 2 forwards; VSA/
|
||
NATTEN sparsity changes the attention term; cache-dit changes the block count) and **it is the same cost model
|
||
we publish to Dynamo's Planner and router (ask A2, §6.3.6)** — one model, two consumers, calibrated online by
|
||
the Profiler observer.
|
||
- **Encoder work budgeted inside the same step — scoped to what diffusion actually needs.** v1 interleaves
|
||
encoder forwards with decode because VLM prefill demands it. Our shape is different: for diffusion, encoding is
|
||
a one-shot prologue per request, so the value here is narrower and specific — *waiting* requests' encode work
|
||
fills budget headroom alongside *running* requests' loop steps on the same pool (hiding encode latency under
|
||
denoise), with the full interleaved pattern reserved for omni workloads that genuinely stream multimodal
|
||
context mid-request. We take the budgeting mechanism, not the VLM workload assumption.
|
||
- **CPU discipline, applied where it bites.** v1's machinery (persistent `InputBatch` state allocated once with
|
||
sliced H2D copies, staged writes flushed once per step, **delta-only scheduler outputs**, async dispatch via
|
||
Futures) exists because LLM decode steps are ~10 ms and CPU time is the bottleneck. Our pressure is uneven: it
|
||
bites for the **AR decode group and batched image steps** (short, frequent) — adopt fully there — while video
|
||
denoise steps are 50–500 ms+ and tolerate a simpler dispatch path; we take the async-Future overlap everywhere
|
||
but do not pre-build LLM-grade batch plumbing for loops that don't need it.
|
||
- **Same-step speculative contract.** Draft tokens ride the scheduler output and are verified by rejection
|
||
sampling within the same step — the proven shape for `ARDecodeLoop` multi-token bundles (§12, open question 10).
|
||
|
||
Two properties make cost-currency scheduling *easier* here than vLLM's problem, and one makes it harder. Easier:
|
||
diffusion **baseline** costs are static and known at admission — (resolution, frames, steps, CFG mode, model)
|
||
determine per-step cost, flat across steps (no growing context) — with the honesty caveat the doc owes itself:
|
||
**admission always uses the conservative baseline**, because the design's own flagship features make realized cost
|
||
runtime-dependent (cache-dit skip decisions are residual comparisons unknowable in advance; VSA tile selection is
|
||
content-dependent; AR decode lengths are unbounded — budgeted at the `max_tokens` cap with refund on early EOS).
|
||
Runtime telemetry refines calibration and utilization accounting; it never licenses admission optimism. Harder:
|
||
**a denoise step is indivisible**, so the largest step bounds
|
||
iteration latency — one 30 s-1080p step can block an interactive Dreamverse segment regardless of budgets.
|
||
Mitigations, in order: don't co-schedule jumbo steps with latency-class work (cost classes map to pools — which
|
||
model-affinity pooling §9.4 wants anyway); shrink jumbo step wall-time with SP *within a node* (pools are
|
||
single-node in v1, §6.3.4 — SP degree caps at intra-node GPU count, stated rather than implied); and
|
||
admission-time SLO classes so the Planner (§6.3.5) scales pools per class instead of one pool absorbing both.
|
||
|
||
**Failure isolation, cancellation, and memory admission — the soundness conditions of multiplexing.**
|
||
Step-multiplexing changes the failure class categorically: today, one request per pool means one request's CUDA
|
||
error is its own problem; on a shared pool, a mid-step illegal access poisons the CUDA context and any in-flight
|
||
collectives for *every* co-scheduled tenant, including resident session caches. This is designed **with** Phase 2,
|
||
not after it:
|
||
|
||
- **Cancellation is first-class and common-path.** Vibe directing makes abandoning in-flight generation the
|
||
*normal* user action, not an edge case. Cancel takes effect at the next step boundary: queued steps drop,
|
||
`LoopState` and cache handles release, partial artifacts report with explicit `cancelled` status. The
|
||
flush/interrupt RPC (§9.1) is this mechanism, not a separate one.
|
||
- **Failure classification.** *Request-fatal* (admission/step allocation failure, NaN flagged by NaNWatch, errors
|
||
attributable to one request's step): abort that request via an **SPMD-consistent abort broadcast** — the exact
|
||
dual of the rank-0-decides scheduling broadcast — release its state; fan-out graphs deliver the named artifacts
|
||
completed so far plus a structured error (partial-artifact semantics). *Pool-fatal* (illegal memory access,
|
||
NCCL desync/timeout, context corruption): pool re-init — kill and restart workers (the **sentinel-fd
|
||
worker-death watch is inherited from Dreamverse's `gpu_pool.py:542-586` and explicitly preserved**, so Phase 2
|
||
is not a reliability regression for the flagship customer), re-establish collectives, invalidate all
|
||
pool-resident caches (sessions re-prefill from durable inputs; requests resume from serialized `LoopState`
|
||
where one exists, else requeue), and report through the Dynamo health surface (§6.3.5). vLLM v1 needed exactly
|
||
this machinery (`EngineDeadError`, `abort_requests`); we build it at the same layer.
|
||
- **Memory is the second budget axis.** vLLM's token budget is half its admission story; the other half is
|
||
`allocate_slots` in the same loop. We keep both halves: every schedulable item carries (predicted time,
|
||
**resident bytes, peak-activation bytes**) — latents, conditioning sets, CFG duplicates, and activation peaks
|
||
live in `RequestState` *outside* the CacheManager, so an **admission planner** bounds Σ resident + worst-case
|
||
peak (profiled per model × shape, vLLM-style) against the pool budget. Two requests that fit individually but
|
||
jointly OOM are rejected at admission, not discovered at step 37.
|
||
- **Preemption semantics, by state size.** "Preemption only at step boundaries" now says what happens to the
|
||
loser's multi-GB state: small resident states stay on-GPU; medium offload to pinned host memory; large or idle
|
||
drop and resume from serialized `LoopState` — with the re-encode/re-prefill economics priced by the cost model
|
||
(different economics from token-KV recompute, stated rather than assumed).
|
||
|
||
The non-serving path bypasses level 1 entirely: `VideoGenerator` submits one request and drives the pool to
|
||
completion synchronously — preserving today's latency profile and debuggability (G5, G6).
|
||
|
||
#### 6.3.2 CacheManager (unifying P5)
|
||
|
||
One manager, handle-based. First, the proportionality note (N5): **KV is the minority case here** — bidirectional
|
||
diffusion, the majority of today's traffic, allocates *no* KV at all, so the KV machinery below is lazy: pools
|
||
materialize only when a `PipelineSpec` declares KV-bearing loops (AR pathways, causal video, omni), and a pure
|
||
bidirectional deployment carries none of it.
|
||
|
||
For that KV-bearing minority, adversarial review caught this design importing a property vLLM does not have: the
|
||
single-`BlockPool`/no-fragmentation guarantee exists *because* physical bytes-per-block are uniform across all
|
||
groups — `kv_cache_utils.py` asserts a single page size (`get_uniform_page_size`), its docstring calls breaking
|
||
this "non-trivial due to memory fragmentation concerns", and groups differ only in tokens-per-block at equal byte
|
||
size. Our classes differ by **150–500× in natural granularity** (a text-KV page ≈ 64 KB/layer; a causal-Wan
|
||
latent-chunk slab is 9.6–32 MB/layer), and the demand classes are *workload-decoupled* (text-KV and chunk-KV vary
|
||
independently with request mix), so one unified pool either strands slab-sized memory under token traffic or
|
||
degenerates into static partitioning anyway — which is exactly what vLLM's one multi-page-size path (DeepseekV4)
|
||
does at startup. **Decision — simpler and honest: separate per-class pools with static budgets.** Each cache
|
||
class gets a budget carved at pool init from the deploy config (cost-model defaults, driven by what the
|
||
`PipelineSpec` declares): the text-KV class runs a vLLM-style paged block allocator; the chunk-KV class runs a
|
||
**slab allocator** (one latent-frame chunk per slab). Both sit behind the same `KVHandle` interface. The
|
||
fragmentation/deadlock argument: classes are statically partitioned, so cross-class fragmentation is impossible
|
||
by construction; within a class, granularity is uniform, so the standard arguments apply; rebalancing budgets is
|
||
a pool-drain operation — rare (class demand follows deployment workload mix, not per-request noise) but not free:
|
||
a drain is priced by the cost model like any quiescing barrier (§6.3.2's weight-transition pricing), and a slab
|
||
budget shrink must wait out or evict session-resident slab caches, so rebalancing is a scheduled operation, not a
|
||
hot-path one. (This is the original two-pool sketch returning with the justification it lacked — the single-pool detour
|
||
was built on a misread of vLLM.) Our cache classes:
|
||
|
||
- **Paged text KV** (paged class): used by `ARDecodeLoop` (Cosmos3 reasoner — replacing the
|
||
re-prefill-per-token loop; omni thinkers/talkers). No radix prefix tree in v1 of our build; block-hash prefix
|
||
reuse later (the allocator design accommodates it).
|
||
- **Chunked causal-video KV** (slab class): per-DiT-layer K/V for latent-frame chunks with
|
||
sink/local-window semantics — the formalization of `causal_wanvideo.py`'s ad-hoc dicts (Wan-Causal,
|
||
MatrixGame2/3, LongCat-VC, Cosmos3 world-model rollout). vLLM's `ChunkedLocalAttentionSpec` /
|
||
`SlidingWindowSpec` managers (out-of-window recycling, per-request admission caps) are the reference
|
||
*semantics*, re-expressed over slabs. A declared **training mode** disables mid-rollout recycling and keeps
|
||
grad-aware index snapshots (the `wan_causal.py:119-120,405-431` behavior, §8.2) — the inference recycling rules
|
||
are not silently applied under autograd. **MoT models fall out naturally**: the und pathway draws from the
|
||
paged class, the gen pathway from the slab class or nothing — independent budgets, no interference.
|
||
- **Feature caches**: text-embedding cache (content-hash keyed; reused across CFG branches, retries, and requests),
|
||
vision-encoder/deepstack cache (Cosmos3 image-conditioned reasoning), cache-dit (DBCache/FBCache) residual caches scoped
|
||
per `LoopState`. The embedding/encoder caches adopt vLLM's `EncoderCacheManager` design
|
||
(`mm_hash`-keyed, reference-counted with a freeable FIFO, budget-aware allocation). Two invalidation regimes,
|
||
not one: RL `update_weights` bumps a **weight epoch** that invalidates wholesale; but workflow-cloud traffic
|
||
patches *text encoders* per request (ComfyUI LoRAs carry `lora_te*` keys and a separate `strength_clip`), so
|
||
the cache key is **partitioned, not flushed**: `content-hash × encoder identity × adapter-set hash
|
||
(ids + strengths) × weight epoch`. Without the adapter component, two workflows sharing a prompt but differing
|
||
in te-LoRA stacks would silently serve each other stale embeddings — in the exact product whose trust claim is
|
||
reproducibility (§9.4).
|
||
- **Weight/adapter cache** (Phase 3+, pulled by multi-tenant workflow execution, §9.4): disk→CPU→GPU checkpoint and
|
||
LoRA tiers, LRU, content-hash-keyed `ModelSpec` resolution, fast patch/unpatch with stacked-LoRA semantics — and
|
||
**model-affinity scheduling** in the engine (route requests to pools where the checkpoint is already resident,
|
||
the diffusion analog of prefix-affinity routing). Crucially, **weight state is schedulable state**: components
|
||
are one resident copy per pool, so a patch/unpatch transition is a *pool-quiescing barrier* — drain in-flight
|
||
steps, apply/undo `W += scale·BA` shard-consistently across ranks (undo via CPU master-copy restore or 2×
|
||
backup, chosen per deploy memory budget), re-admit. Two interleaved loops requiring different patch states
|
||
cannot coexist on one pool; the StepScheduler therefore treats transitions as scheduled, costed operations —
|
||
the cost model carries a **weight-transition term**, and affinity scheduling exists precisely to minimize how
|
||
often it is paid (a quiesce-vs-queue policy decides whether a lone divergent request waits or forces the
|
||
barrier). Until this lands, the engine serves homogeneous model families per pool.
|
||
|
||
Handles (`KVHandle`, `FeatureRef`) live in `RequestState`; lifetimes are tied to the request (or to an explicit
|
||
session for interactive world models, where the chunk KV must persist across many requests of one game session).
|
||
|
||
#### 6.3.3 Model layer: MoT and packed sequences as first-class citizens
|
||
|
||
To make Cosmos3-class ports native rather than heroic (per §4.1):
|
||
|
||
- **`PackedSeq`** core type: modality segments, per-segment attention mode (causal / full / sparse-NATTEN /
|
||
temporal-causal), sample offsets, position-id planes (3D-MRoPE with FPS modulation), cond masks. Pack/unpack
|
||
utilities replace order-sensitive flat-latent slicing (one of the documented Cosmos3 port frictions, §3.2/P3).
|
||
- **Dual/multi-pathway layer conventions** in `fastvideo/layers/`: `MultiPathwayAttention` (per-pathway QKV/O +
|
||
shared attention over `PackedSeq`), `PathwayMLP` (dense or MoE per pathway), modality-routing by segment — TP
|
||
sharding defined per pathway. The Cosmos3 DiT port refactors onto these; future MoT/Transfusion-lineage models
|
||
reuse them.
|
||
- **Attention backend contract extended** to accept per-segment modes/masks, so backends (FA varlen, flex, NATTEN)
|
||
plug into MoT models the same way they plug into plain DiTs.
|
||
- **Sampler registry** (UniPC, EDM, Euler-FM, fixed-step distilled) with strict config validation that **rejects or
|
||
normalizes** incompatible checkpoint scheduler configs (the karras-sigmas NaN incident becomes a load-time error,
|
||
not a black video).
|
||
|
||
**Compile & graph position (informed by vLLM RFC #42770 and its discussion).** The largest engine just reversed
|
||
its compiler-first philosophy: full-graph `torch.compile` is being dropped in favor of manual fused ops in model
|
||
code, local compile for adjacent-op fusion, and **breakable CUDA graphs** (the SGLang technique, vLLM PR #42304 —
|
||
piecewise capture with no Dynamo/Inductor involvement). Our position, stated explicitly:
|
||
|
||
- **Never full-graph across the engine.** Per-component/per-block compile where it pays (today's per-component
|
||
flags, kept); manual fused ops are *permitted in model code* — the RFC's lesson — rather than forced through
|
||
shared abstractions.
|
||
- **Breakable CUDA graphs for loop step bodies — as an optimization tier, not a foundation.** The technique lets
|
||
cache-dit skip decisions and policy branches become break points between captured segments instead of
|
||
graph-invalidating recompiles, and video's batch-1 fixed shapes are the friendly case for capture. But it is
|
||
marked experimental in vLLM's own tree (`vllm/envs.py:724`), and interceptors × CFG branches × expert routing ×
|
||
observers-needing-eager multiply graph variants. The contract therefore is: **the eager path is the always-correct
|
||
baseline**; graphs are captured for a deliberately small matrix (per model, shape-bucket, branch-profile), and
|
||
interceptor-heavy configurations may simply run eager with the cost surfaced in metrics. The ≤2% Phase-2 latency
|
||
budget is gated on the Phase-1 loop-inversion overhead measurement, not on breakable graphs maturing upstream.
|
||
- **Two tiers, like vLLM's post-discussion resolution**: flagship (recipe, runtime) pairs (a FastWan/LTX-2 NVFP4
|
||
build on B200) may carry hand-fused, hardware-specific module variants, swapped in per `DeployConfig` without
|
||
touching the `PipelineSpec`; the 20+-family tail stays on shared layers + policies. The dissent in that RFC is
|
||
our guardrail for the tail: manual fusion doesn't scale across quant schemes and hardware, which is exactly what
|
||
the policy/registry layer is for.
|
||
- **Retire `fastvideo/forward_context.py` — staged, not deleted in place.** It is inherited from precisely the
|
||
global the RFC names as a compiler-compat hack, but today it is also the live attention-metadata transport for
|
||
`fastvideo/attention/layer.py:93` *and* the frozen legacy `training/` stack
|
||
(`training_pipeline.py:405`, `distillation_pipeline.py:672`) — deleting it in Phase 1 would violate N2's frozen
|
||
boundary. Staging: Phase 1 removes it from the **migrated inference path** (new loops carry `StageContext` +
|
||
`AttnMetadataProvider` + model-owned `RequestState` extensions — the same "model owns its state" conclusion
|
||
vLLM's `Model`/`ModelState` interface reaches), while the module itself survives as a compatibility shim fed by
|
||
the new context, so attention layers and every unmigrated/legacy caller are untouched; the file is deleted in
|
||
Phase 5 with the legacy paths.
|
||
|
||
*What a media engine would feed back to RFC #42770's `Model` interface* (we have standing to comment — our
|
||
experience is the evidence): (1) `prepare_inputs`/`forward` assume one phase per request; diffusion needs
|
||
multi-phase requests (encode once, then N denoise forwards) with **memoized cross-step state** (text embeddings
|
||
reused across all steps) — a small contract change that would let vllm-omni's diffusion stages implement the same
|
||
interface; (2) formalize encoder-output caching in the interface (their `EncoderCacheManager` exists but sits
|
||
outside the `Model` contract); (3) keep the interface tensor-type-agnostic (the vllm-metal request) so non-torch
|
||
runners can implement it.
|
||
|
||
#### 6.3.4 Placement, connectors, and external engines
|
||
|
||
- **Default**: one SPMD pool; node→pool mapping trivial; "connector" is an in-proc handoff (zero overhead).
|
||
**Pools are single-node in v1** — a load-bearing decision made explicitly: the engine never owns a cross-node
|
||
NCCL mesh inside one pool (no cross-node collective bring-up, no NCCL-timeout fault domains, no multi-node
|
||
drain protocol); multi-node scale is *multiple pools* fronted by Dynamo (§6.3.5). Consequences accepted: SP
|
||
degree ≤ intra-node GPUs, which bounds the jumbo-step mitigation (§6.3.1) and caps single-pool model scale —
|
||
the revisit trigger is a model or resolution that needs more. `RayDistributedExecutor`, today's multi-node
|
||
path, is **frozen with the legacy adapter** and retires at Phase 5; its use case is superseded by
|
||
Dynamo-fronted pools.
|
||
- **Role pools** (opt-in): encode / denoise / decode pools with capacity dispatch — port of `multimodal_gen`'s
|
||
disaggregation, now expressible per graph node rather than per hardcoded role.
|
||
- **Per-node placement** (opt-in): deploy YAML pins nodes to devices/pools (vision encoder on its own GPU; vocoder
|
||
on CPU); cross-pool edges select a `Connector` (transport: SHM/Mooncake/Mori/Yuanrong today, with NIXL as a
|
||
target — *not* "vllm-omni's NIXL/NCCL", which it does not use). The `chunk_ready` readiness signal is adopted
|
||
from vllm-omni's `OmniConnectorBase`; **credit-based flow control is sglang-omni's Relay design, not
|
||
vllm-omni's** (vllm-omni uses readiness-set *parking*, no credits) — so credit-flow here is a FastVideo
|
||
addition *over* vllm-omni's readiness model, composed with sglang-omni's proven design. For KV/cache-bearing
|
||
edges specifically, the
|
||
connector speaks a protocol shaped like vLLM's `KVConnectorBase_V1` (scheduler-side query/alloc/finish roles +
|
||
worker-side async `start_load`/`wait_for_layer`/`save` with layer-pipelined loading and `invalid_block_ids`
|
||
fault reporting) — the same interface NIXL/LMCache/Mooncake already implement, which is also what makes Dynamo
|
||
ask A4 (§6.3.6) an extension of an existing contract rather than a new one. **Adopt vllm-omni's TP-rank-aware +
|
||
CFG-branch-aware KV payload slicing** (`OmniKVTransferManager` `build_rank_aware_send_keys`/`recv_keys`): a
|
||
shared-weight MoT split across parallel groups must slice KV by rank *and* by CFG branch on the wire, not ship a
|
||
monolithic blob.
|
||
- **External-engine node** (opt-in, later): an `ARDecodeLoop`-shaped adapter that delegates a *separable* AR stage
|
||
(a 235B thinker) to a colocated sglang/vLLM instance over the connector interface. Useful for separable omni
|
||
models; never required for MoT models (which must stay native).
|
||
- **Declarative parallelism (adapted from vLLM-Omni RFC #4084).** Pool meshes and engine topology are described by
|
||
one stacked axis spec — e.g. `parallel: [dp(2), cfg(2), sp(4), tp(2)]` per pool — with collective-group
|
||
membership computed generically (the RFC's mesh-tree construction) instead of hand-wired per combination in
|
||
`fastvideo/distributed/parallel_state.py`. We adopt three properties outright: **(1) pre-flight validation** —
|
||
rank counts, group membership, and *ownership conflicts* are build errors; each axis records who realizes it
|
||
(e.g. CFG branches can be realized by a batching `CFGPolicy` (1-forward stack+chunk) *or* a `cfg` parallel group —
|
||
declaring both is rejected, the RFC's `l1_owner` / no-double-routing rule applied to our policy-vs-axis split); **(2) fail-loud** on
|
||
unsupported combinations rather than silent fallback; **(3) CPU-testability** — engine scheduler, graph dispatch,
|
||
and placement planning run against a fake worker pool so topology logic is unit-tested without GPUs (FastVideo CI
|
||
is GPU-starved; this is how the composition layer stays tested). The RFC's L1/L2/L3 intervention vocabulary —
|
||
batch routing / per-layer behavior / kernels — maps cleanly onto this design (L1 = engine + StepScheduler, L2 =
|
||
TP/SP/pathway-sharded layers, L3 = the attention-backend registry) and is reused for the extension system below.
|
||
We implement the spec natively rather than depending on `composable_parallel` (§11.11).
|
||
- **Axis vocabulary v1 + substrate** (informed by xDiT and torchtitan). Axes: `dp`, `cfg` (≤2), `sp = ulysses ×
|
||
ring`, `tp`, and **`pp_patch`** — xDiT's PipeFusion displaced-patch pipelining, which exploits diffusion's
|
||
step-to-step redundancy by reusing stale KV from the previous denoise step (mandatory warmup steps; wins at high
|
||
resolution and on weak interconnects; tolerable for few-step distilled models) and is **invalid for causal/AR
|
||
models** — an applicability condition the pre-flight validator enforces per `PipelineSpec`, since stale KV breaks
|
||
causality. xDiT's DistVAE needs no new axis: it is our decode role pool. Composition constraints travel with the
|
||
axes (e.g., `pp_patch × sp` requires KV replication inside attention). Implementation substrate: the spec
|
||
compiles to **PyTorch DeviceMesh** via a `ParallelDims`-style builder (torchtitan's pattern — product-of-degrees
|
||
validation, cached submeshes); the trainer consumes the mesh natively (FSDP2/DTensor), while engine hot paths
|
||
keep raw process groups where DTensor indirection isn't earned.
|
||
|
||
#### 6.3.5 Production scale: Dynamo as the first-class fleet partner
|
||
|
||
Everything above this line is a **single-deployment engine**: one pool (or a few colocated role pools) on one
|
||
node-group. Truly scaling — multi-node fleets, SLA-driven autoscaling, cross-node routing, cold starts, fleet
|
||
fault-recovery — is a different discipline, and the design's position is that **FastVideo never builds it**:
|
||
NVIDIA Dynamo (`./dynamo/`) is the named first-class partner for that layer. This is not aspirational — the
|
||
integration already exists in embryo: `dynamo/examples/diffusers/worker.py` registers FastVideo with
|
||
`register_llm(ModelInput.Text, ModelType.Videos, …)` and serves `VideoGenerator` through `@dynamo_endpoint`
|
||
(today behind an `asyncio.Lock`, one request at a time — precisely the limitation Phase 2 removes), and Dynamo
|
||
already fronts our sibling stacks (sglang's diffusion worker via `components/src/dynamo/sglang/init_diffusion.py`,
|
||
vllm-omni under `components/src/dynamo/vllm/omni/`). First-class means hardening that contract, not inventing one.
|
||
|
||
**Separation of concerns — the hard line:**
|
||
|
||
| Dynamo owns (FastVideo must not build) | FastVideo engine owns (Dynamo does not reach into) |
|
||
|---|---|
|
||
| Service discovery + model registration (etcd/NATS, K8s) | model execution, loops, policies, samplers |
|
||
| Cross-node request routing (KV/affinity/load-aware) | StepScheduler: step interleaving, AR batching, in-pool placement |
|
||
| SLA Planner: replica autoscaling (per-pool, prefill/decode ratios, K8s CRDs) | caches (paged KV, chunk KV, feature, weight fleet) and their in-node lifecycle |
|
||
| Fleet health, canaries, request migration, graceful-drain orchestration | consistency contract (C0–C2), parity gating, plugins |
|
||
| OpenAI-compatible fleet frontend + preprocessing | the per-pool API surface and `OmniEvent` streaming semantics |
|
||
| KVBM multi-tier token-KV offload; ModelExpress weight streaming (cold starts) | which weights/caches are worth offloading, and when |
|
||
|
||
**The contract (what "first-class backend" concretely requires of us — Phase 2/3 work items):**
|
||
|
||
1. **Worker surface** (Phase 2): the Dynamo worker wraps `AsyncEngine` instead of a locked `VideoGenerator` —
|
||
registration with `ModelType.Videos | Images` (+ `Chat` once omni lands), `worker_type` mapping our roles onto
|
||
theirs (Aggregated; Prefill/Decode for disagg), streaming handler (our `OmniEvent` chunks → their annotated
|
||
stream), health-check endpoint, graceful drain (finish in-flight loops, release pools) wired to
|
||
`DistributedRuntime`'s signal handling.
|
||
2. **Metrics for routing + Planner** (Phase 2): queue depth, pool utilization, and — important for video —
|
||
**per-request cost estimates** (a 5 s 480p distilled job and a 30 s 1080p job differ by orders of magnitude;
|
||
the Planner's SLA math needs the cost model we can derive from `PipelineSpec` × resolution × steps; LLM
|
||
backends get this implicitly from token counts, we must publish it). This is the same cost model the
|
||
StepScheduler uses as its internal budget currency (§6.3.1) — built once, consumed in-pool and at fleet level.
|
||
3. **Affinity events** (Phase 3, with the weight-fleet cache): Dynamo's KV-aware routing rests on engines
|
||
publishing cache events (`lib/kv-router/src/protocols.rs:627-646`:
|
||
`KvCacheEventData::Stored{parent_hash, blocks[{block_hash, tokens_hash}]}` / `Removed` / `Cleared`) into a
|
||
radix index. The index is hash-generic, but today's router derives keys from *token prefixes* — our two
|
||
affinity signals (**checkpoint residency** for the workflow cloud §9.4, **session residency** for Dreamverse
|
||
§9.1) need either a custom router policy (Dynamo exposes the router entrypoint for this) or an upstream
|
||
extension — formalized as **ask A1** (§6.3.6).
|
||
4. **Disagg alignment** (Phase 3): our encode/denoise/decode role pools map onto Dynamo's prefill/decode
|
||
coordination — the worker returns `disaggregated_params` in its terminal chunk and transfers tensors over
|
||
NIXL, which is already in our connector menu (§6.3.4); cross-node disagg then comes from contract
|
||
compatibility, not new infrastructure.
|
||
|
||
**What this buys, concretely:** multi-node serving without writing a router; SLA autoscaling that makes the
|
||
$5-subscription workflow cloud's unit economics work (engine-level multiplexing × fleet-level bin-packing);
|
||
ModelExpress weight streaming for fast cold-starts — which composes directly with the weight-fleet cache (§9.4)
|
||
when checkpoints migrate across the fleet; KVBM's GPU→CPU→SSD tiering pattern as the template for offloading
|
||
idle Dreamverse session caches (sessions are idle-heavy between user directions); and production ops (operator,
|
||
observability, fault tolerance) we never staff. Dreamverse's planned local-first `controller/` (Runpod/Modal
|
||
provisioning) remains the single-box story; at fleet scale it delegates to Dynamo rather than growing its own
|
||
orchestration.
|
||
|
||
#### 6.3.6 Asks of Dynamo (the partnership is two-way)
|
||
|
||
We have a direct working relationship with the Dynamo team, so Dynamo is **not treated as frozen**: where the
|
||
ecosystem design improves by changing Dynamo rather than working around it, we say so. Each ask below states the
|
||
gap, the proposed upstream change, who benefits beyond FastVideo (these are ecosystem gaps — vllm-omni,
|
||
sglang-diffusion, and every media/RL backend hit the same walls), and our fallback if it doesn't land. Roughly in
|
||
priority order:
|
||
|
||
- **A1 — Generic affinity routing (key spaces beyond token prefixes).** *Gap:* the KV router's radix index is
|
||
hash-generic, but keys are derived from request token prefixes — useless for our affinity signals. *Ask:*
|
||
generalize the event protocol (`lib/kv-router/src/protocols.rs`) to namespaced affinity events —
|
||
`{key_space: "checkpoint" | "session" | "lora_set" | "weight_version" | "kv_prefix", key_hash, resident,
|
||
capacity_hint}` — with router policies pluggable per key space. *Beneficiaries:* any diffusion/workflow backend
|
||
(checkpoint/LoRA residency — the §9.4 fleet cache), any session workload (world models, omni chat, Dreamverse),
|
||
and LLM serving itself (LoRA-set affinity). *Fallback:* custom router policy via Dynamo's router entrypoint;
|
||
worst case, least-loaded routing + engine-side redirects.
|
||
- **A2 — Heterogeneous request cost interface.** *Gap:* router load-accounting and Planner profiles infer cost
|
||
from token counts; a 5 s 480p distilled clip and a 30 s 1080p clip differ by orders of magnitude with identical
|
||
"prompt lengths." *Ask:* a standard per-request cost annotation (predicted GPU-seconds / memory class) that
|
||
engines publish and the router + Planner consume, with Planner profiling keyed on engine-declared cost features
|
||
(model, resolution, steps) instead of tokens. *Beneficiaries:* every media backend Dynamo fronts. *Fallback:*
|
||
encode cost classes as separate registered model names (crude bucketing).
|
||
- **A3 — Streaming media through the fleet frontend.** *Gap:* `ModelType.Videos` today returns one base64 MP4 blob
|
||
(the `examples/diffusers/worker.py` pattern); Dreamverse-class UX needs chunked fMP4/audio segments and progress
|
||
events end-to-end. *Ask:* first-class chunked-binary streaming (segments + progress + keep-alive) through the
|
||
frontend for media model types, mapping 1:1 onto our `OmniEvent` channel. *Fallback:* bypass the frontend for
|
||
streams (direct WebSocket to pools) and use Dynamo for routing/placement only — workable but forfeits unified
|
||
auth/observability.
|
||
- **A4 — Role-graph disaggregation (N roles, not two).** *Gap:* disagg coordination is hardwired to the
|
||
prefill→decode pair; our role pools are encode→denoise→decode (and omni graphs fan out further). The
|
||
`disaggregated_params` payload is already opaque, so the protocol nearly exists. *Ask:* generalize PrefillRouter
|
||
into a role router over engine-declared role graphs with NIXL hand-offs per edge. *Beneficiaries:* vllm-omni and
|
||
sglang-omni each hand-built exactly this orchestration (§5) — Dynamo could own the pattern once for the whole
|
||
ecosystem. *Fallback:* keep multi-role inside one Dynamo-visible worker (our engine-internal pools, §6.3.4) and
|
||
expose only aggregated workers to the fleet.
|
||
- **A5 — The RL weight plane: versioned update broadcast + staleness-aware routing.** *Gap:* ModelExpress streams
|
||
weights for cold starts; RL needs *hot* incremental updates — trainer publishes version N+1, the rollout fleet
|
||
converges, and the router knows who's on what. *Ask:* (a) ModelExpress incremental in-place weight-update
|
||
broadcast to a live worker set with per-worker `weight_version` tracking; (b) version/staleness-aware routing
|
||
(route rollouts to ≥version-K workers; drain stale ones) — A1's `weight_version` key space; (c) a Planner
|
||
objective plugin for **train↔rollout GPU elasticity** (scale the rollout pool against trajectory-queue depth and
|
||
the staleness bound rather than serving SLAs). *Beneficiaries:* every RL stack that fronts rollout with Dynamo —
|
||
this is miles' weight-transport + `weight_versions` machinery (§8.6) offered as fleet infrastructure.
|
||
*Fallback:* our own transports (§8.6) with Dynamo unaware of versions; works colocated, weakens fleet-scale
|
||
async RL.
|
||
- **A6 — KVBM generalization to engine-defined cache objects.** *Gap:* KVBM tiers token-KV blocks; our expensive
|
||
idle state is latent session caches and chunk-KV (Dreamverse sessions between directions, world-model games).
|
||
*Ask:* a generic cache-object API (register/pin/unpin, tier hints, eviction callbacks) so engine caches ride
|
||
KVBM's GPU→CPU→SSD machinery under fleet policy. *Fallback:* engine-side offload within the CacheManager
|
||
(single-node only).
|
||
- **A7 — Session lifecycle as a frontend/router primitive.** *Gap:* no session concept at the fleet layer; ours
|
||
(Dreamverse scenes, interactive world models, multi-turn omni) need create/attach/expire semantics pinned to
|
||
session-resident workers. *Ask:* session IDs as a first-class routing primitive (pairs with A1's `session` key
|
||
space + A6 for idle offload). *Fallback:* sticky routing by consistent hash at our API gateway (sglang-omni's
|
||
pattern), losing fleet-level visibility.
|
||
|
||
Sequencing with our phases: A2 + A3 matter by Phase 2 (Dynamo-grade worker + Dreamverse streaming); A1 + A4 by
|
||
Phase 3 (fleet cache + disagg); A5–A7 by Phase 4 (RL hardening + session serving). Each ask is independently
|
||
useful to Dynamo's existing backends — that's the test we apply before asking.
|
||
|
||
### 6.4 Extension system: hooks and plugins (cross-cutting)
|
||
|
||
Requirement: drop-in **optimization plugins** — [cache-dit](https://github.com/vipshop/cache-dit)
|
||
(DBCache, FBCache, TaylorSeer calibration) and successors — and drop-in **instrumentation** for debugging, profiling,
|
||
and numerical-precision alignment, without forking loop or model code per technique. Seeds already in-tree:
|
||
`fastvideo/hooks/` (`ModuleHookManager`/`ForwardHook`, plus `activation_trace.py` — zero-overhead-when-off,
|
||
regex-selected module taps with a `trace_step()` step-index context, built for port-parity debugging), and a
|
||
cautionary tale: `enable_teacache` is plumbed through the entire API surface (`pipeline_batch_info.py:182`,
|
||
OpenAI protocol, sampling params) yet has **no consumer in the shared denoise loop** — bolting optimizations into a
|
||
monolithic loop is exactly what didn't ship. The extension system makes such techniques additive.
|
||
|
||
Two tiers with different contracts:
|
||
|
||
**Tier 1 — Observers (read-only).** An event bus with stable, named hook points:
|
||
|
||
| Plane | Hook points |
|
||
|---|---|
|
||
| Request | `request_start/end`, `artifact_emitted`, `schedule_decision` |
|
||
| Pipeline | `node_start/end`, `loop_init/finalize`, `step_start/end` (step idx, timestep, state ref) |
|
||
| Execution | `model_forward pre/post`, module-level forward taps by name regex (existing `ModuleHookManager`), cache events (alloc/hit/evict), connector transfers |
|
||
|
||
Contract: observers cannot mutate state; the chain is assembled at pipeline build, so an unused hook point is
|
||
*literally absent* from the hot path (the activation-trace "zero overhead when off" rule, generalized); each
|
||
observer declares `needs_eager`, and the runtime drops compile/cudagraph capture only for the scopes it watches,
|
||
logging the cost. Built-ins shipped with the runtime:
|
||
|
||
- **ActivationTrace** — the existing module, promoted from env-var global to a per-request-scoped observer (env-var
|
||
UX preserved).
|
||
- **ParityAligner** — formalizes the Cosmos3 oracle methodology (N3) as infrastructure: *record mode* dumps named
|
||
taps per step/block from a reference run (official framework, or pre-change FastVideo); *compare mode* replays a
|
||
request with fixed seeds and reports the first divergence beyond per-tap tolerances. This is the engine behind
|
||
the Phase-1 "old loop vs new loop, bit-identical" gate, and the standing tool for every future port and precision
|
||
change (FP8/FP4 work included). The training side already runs the coarse version of this pattern in CI: PR #1396
|
||
pins per-method, **per-GPU-model** layer-0 grad-norm references at 10% tolerance
|
||
(`fastvideo/tests/train/methods/grad_norm_regression.py` — the causal-DFSFT refs differ ~9% between L40S and
|
||
GB200 while the finetune refs differ only ~0.1%: cross-device reduction order is a real *and method-dependent*
|
||
consistency variable, §8.5);
|
||
ParityAligner is that idea taken to per-tap, per-step resolution.
|
||
- **Profiler** — per node/step/block wall+CUDA timings, NVTX ranges, on-demand `torch.profiler` windows, memory
|
||
watermarks; exported through engine metrics.
|
||
- **NaNWatch** — first-NaN/Inf localization (module × step) for debugging.
|
||
|
||
**Tier 2 — Interceptors (compute-altering).** Explicit extension points — never monkeypatching (§11.10):
|
||
|
||
```python
|
||
class StepInterceptor(Protocol): # step-skip / cached-prediction schemes
|
||
def before_step(self, loop: LoopState) -> StepOverride | None: ... # may supply a predicted output, skipping the forward
|
||
def after_step(self, loop: LoopState, out) -> None: ... # update residual/calibration state
|
||
|
||
class BlockInterceptor(Protocol): # cache-dit class: FBCache, DBCache, TaylorSeer
|
||
def plan(self, loop: LoopState) -> BlockPlan: ... # which transformer blocks run vs skip this forward
|
||
def on_skipped_block(self, idx, h) -> torch.Tensor: ... # substitute cached / Taylor-extrapolated residual
|
||
```
|
||
|
||
- Block interception needs one model-side convention: in-tree DiTs execute their block stack through a
|
||
`run_blocks(self.blocks, h, ctx)` helper — a plain loop when nothing is bound (compile-friendly) — instead of
|
||
inlined `for` loops. Honest cost accounting: this is a **per-family refactor, not a rename** — today's DiTs
|
||
inline divergent block loops with different arguments and gradient-checkpointing paths (`wanvideo.py:712`,
|
||
`flux_2.py:1057`, `ltx2.py:2648`, `causal_wanvideo.py:560`). Adoption is per-family with ParityAligner gates
|
||
(Phase 1 scope = the Phase-1 families, Wan + Flux2); a family that hasn't converted simply **rejects block-level
|
||
plugins at build time** via capability negotiation — step-level interceptors still work everywhere.
|
||
- **State scoping is the correctness fix**: interceptor state lives in `LoopState.plugin_state[plugin_id]`, keyed
|
||
per request **and per CFG branch**. Reference implementations (cache-dit on diffusers, TeaCache forks) keep
|
||
residual state in module-level globals, which silently corrupts under concurrent requests — here
|
||
concurrency-safety is structural, which is what makes these optimizations usable in the serving engine at all.
|
||
- Build-time composition: ordered chain; conflicting interceptors (two block-skippers) rejected pre-flight — the
|
||
same validation stance as the parallelism spec; per-model compatibility declared in the `PipelineSpec` (a 4-step
|
||
distilled model rejects step-skip caches rather than producing garbage).
|
||
- Graph contract: interceptors declare `graph_safe`; graph-safe ones express skip decisions as captured tensor ops
|
||
or bucketed graph variants; others force eager for their scope only.
|
||
|
||
**cache-dit is the reference integration target.** It is the library sglang's multimodal serving already uses
|
||
(`multimodal_gen/runtime/cache/cache_dit_integration.py`: `enable_cache_on_transformer` /
|
||
`enable_cache_on_dual_transformer` built on cache-dit's `BlockAdapter` + `DBCacheConfig` /
|
||
`TaylorSeerCalibratorConfig`, configured by a `cache_dit_config` server arg). Two scars in that integration are
|
||
instructive: sglang must **monkeypatch `CachedContextManager.similarity`** to behave correctly on
|
||
sequence-parallel-sharded tensors, and must manually call `refresh_context_on_transformer` per request. Both needs
|
||
are structural in our interceptor contract — the `BlockInterceptor` adapter sees the parallel groups (so residual
|
||
comparison is group-aware by construction) and its state is `LoopState`-scoped (so per-request/per-branch lifecycle
|
||
is automatic, no manual refresh). The Wan2.2 dual-expert case (sglang's `dual_transformer` path) composes with our
|
||
`ExpertRouting` policy: one cache context per expert. We consume cache-dit as a dependency — its configs,
|
||
calibrators, and skip math — behind our extension point, keeping algorithm parity with the sglang fork.
|
||
|
||
**Packaging — and the trust boundary.** A `fastvideo.plugins` entry-point group + registry: third-party packages
|
||
(the cache-dit adapter, a vendor profiler) register without forking. **Enablement is DeployConfig-scoped only** —
|
||
entry points resolve at deploy load, never per request. The reasons are load-bearing in a multi-tenant engine:
|
||
plugins execute arbitrary code, and since the OpenAI protocol is *generated from the request schema*, a
|
||
per-request `plugins=[...]` field would derive third-party-code selection straight into the public API. Requests
|
||
therefore only *parameterize pre-enabled* plugins through per-plugin validated schemas
|
||
(`diffusion.plugin_params={"cache_dit": {"Fn": 8, "Bn": 8}}`), with two guards: exact-mode requests reject
|
||
`distribution_altering` parameterization outright (the §9.4 trust claim), and per-request plugin overhead —
|
||
including a `needs_eager` observer de-capturing scopes shared with co-scheduled tenants — is attributed in that
|
||
request's cost metrics, so noisy neighbors are visible and chargeable. Hook-point names are a versioned public
|
||
surface.
|
||
|
||
In RFC #4084's vocabulary: observers and interceptors are the L1/L2 intervention levels applied to *computation
|
||
shape* rather than parallelism (step/loop ≈ L1, block/layer ≈ L2), and L3 remains the attention/kernel backend
|
||
registry. One taxonomy, three uses — parallelism, optimization, instrumentation.
|
||
|
||
### 6.5 Entrypoints and compatibility
|
||
|
||
- `VideoGenerator` / `StreamingVideoGenerator` / CLI: unchanged signatures, now building `OmniRequest`s.
|
||
- New `AsyncEngine` (`generate(request) -> OmniOutput | AsyncIterator[OmniEvent]`).
|
||
- OpenAI-compatible server extends the existing `fastvideo/entrypoints/openai/`: `/v1/videos/generations`,
|
||
`/v1/images/generations`, `/v1/audio/speech`, `/v1/chat/completions` (omni: text+media in, text+media out —
|
||
lands with Phase 4),
|
||
SSE/WebSocket streaming. API-shape parity with `multimodal_gen` is an explicit goal so a future
|
||
upstream/downstream story (§11.1) stays open.
|
||
- **Workflow frontend** (Phase 3): the ComfyUI-workflow compiler (§9.4) is a peer entrypoint — workflow JSON parses
|
||
into `PipelineSpec` + `OmniRequest`s and enters the same admission path as every other client.
|
||
- **Legacy adapter**: `LegacyPipelineNode` wraps any existing `ComposedPipelineBase` as a single opaque graph node
|
||
(request → ForwardBatch shim → stages → artifacts). Every unmigrated model runs day one.
|
||
|
||
### 6.6 Model loading, configs, and the typed API
|
||
|
||
The user is mid-migration to a typed API (`fastvideo/api/`) and has flagged it — and config specification and model
|
||
loading generally — as open for redesign. Today there are **seven overlapping config surfaces**: arch configs
|
||
(`fastvideo/configs/models/`, with `param_names_mapping` — good, unchanged); pipeline configs
|
||
(`fastvideo/configs/pipelines/`); `FastVideoArgs` (81 fields) and its legacy subclass `TrainingArgs` (90 own + 81
|
||
inherited = 171); the
|
||
new-trainer YAML `TrainingConfig` (whose `distributed:` block re-declares the same parallelism fields); legacy
|
||
`SamplingParam` (75 fields, now at `fastvideo/api/sampling_param.py`); and the in-flight typed schema
|
||
(`fastvideo/api/schema.py`: `GenerationRequest`, `GeneratorConfig`, `ServerConfig`, `ParallelismConfig`,
|
||
`OffloadConfig`, `CompileConfig`, `InputConfig`, `OutputConfig`, …). The seams are load-bearing code:
|
||
`fastvideo/api/compat.py` is **651 lines of bidirectional translation** (`request_to_sampling_param`,
|
||
`generator_config_to_fastvideo_args`, `legacy_generate_call_to_request`, dotted-path explicit-field tracking via
|
||
`request_metadata.py`), and per-model knobs leak into the universal surface as side files
|
||
(`api/matrixgame2.py`, `api/matrixgame3.py`).
|
||
|
||
The in-flight work points the right way — typed request, generator config, and streaming event types
|
||
(`api/results.py`: `VideoProgressEvent`/`VideoPartialEvent`/`VideoFinalEvent` prefigure `OmniEvent`). Its problem
|
||
is that it sits *beside* the legacy surfaces, translating, rather than replacing them. Target: **four layers, one
|
||
owner each, everything else derived**:
|
||
|
||
| Layer | Owns | Absorbs / retires |
|
||
|---|---|---|
|
||
| `ArchConfig` | model structure + `param_names_mapping` | (unchanged) |
|
||
| `ModelSpec` + `PipelineSpec` (§6.2.4) | components and weights, graph, policies, samplers | pipeline configs' behavior knobs; registry name-detectors; `model_index.json` implicitness |
|
||
| `DeployConfig` | placement, stacked parallelism axes (§6.3.4), memory/offload, compile, plugins | the runtime thirds of `FastVideoArgs` and `GeneratorConfig`; `TrainingConfig.distributed` |
|
||
| `OmniRequest` (§6.1) | everything per-call, incl. per-model `ModelOptions` blocks | `GenerationRequest` (evolves into it), `SamplingParam`, the request half of `ForwardBatch` |
|
||
|
||
Derived views are *generated*, never hand-maintained: the OpenAI protocol models, CLI flags, and presets
|
||
(`api/presets.py` becomes per-model default `OmniRequest` fragments) all derive from the request schema; per-model
|
||
options are registered typed blocks rather than universal-schema fields (the `api/matrixgame2.py` pattern,
|
||
formalized). `compat.py` is migration scaffolding, but its retirement must be honest about who depends on it:
|
||
`VideoGenerator` itself currently routes through `generator_config_to_fastvideo_args` /
|
||
`legacy_generate_call_to_request` / `request_to_sampling_param` (`entrypoints/video_generator.py:31`), and the doc
|
||
promises those public signatures stay. So the policy is: **Phase 0 freezes the file** (no new entries, ever);
|
||
each phase deletes the translations its migrations obsolete (the `SamplingParam`/`ForwardBatch` bridges die as
|
||
families migrate); what legitimately remains until Phase 5 is the thin, generated legacy-signature shim — and at
|
||
Phase 5 the file reaches zero lines. Monotonic shrink per phase is the tracked metric, not a Phase-1 cliff.
|
||
|
||
**Model loading.** Replace implicit `model_index.json` + name-detector resolution with an explicit **`ModelSpec`
|
||
manifest** per checkpoint family: declared components with `required_for`/`optional_for` task sets (the Cosmos3
|
||
lazy-sound-VAE fix, P8), accepted weight layouts (diffusers-style, native, quantized variants), dtype/precision
|
||
policy, and the default `PipelineSpec` binding. Component loaders stay as they are (`TransformerLoader`,
|
||
`VAELoader`, … unchanged); resolution becomes manifest-first, with today's detectors as fallback for unmanifested
|
||
HF checkpoints. The same manifest is what `train/` `ModelBase.init_from` consumes — one loading path for inference,
|
||
training roles (student/teacher/critic), and RL rollout (§8).
|
||
|
||
**Hub interchange (diffusers Modular).** `ModelSpec` component entries serialize compatibly with diffusers'
|
||
`ComponentSpec` / `modular_model_index.json` (the modular-repo hub format), and guidance configs map onto their
|
||
Guider classes — so a modular-diffusers port and a FastVideo `PipelineSpec` share vocabulary, models published
|
||
either way load both ways, and HF remains the distribution rail for `(recipe, runtime)` artifacts (see the
|
||
artifact-registry known gap, §12). The same shape is appearing in vLLM core (RFC #42770): an HF-decoupled
|
||
`ModelConfig` plus a `Model` interface where the model owns its state and input preparation — the third
|
||
independent arrival at manifest-first loading and model-owned state, after diffusers Modular and this design.
|
||
|
||
**Schema stability tiers.** By Phase 3, at least four serialized surfaces have external consumers: `ModelSpec`
|
||
manifests (hub interchange, format co-owned with diffusers), compiled-workflow `PipelineSpec`s (content-hash-keyed
|
||
in the weight-fleet cache — a schema change silently changes hashes and invalidates fleet affinity), the
|
||
`OmniEvent` stream (Dreamverse's frontend; proposed as Dynamo ask A3's wire format), and per-node parameter
|
||
schemas. Every one carries a `schema_version`; each is **experimental** until the phase its first external
|
||
consumer ships (`OmniEvent`: Phase 2; `ModelSpec` + workflow specs: Phase 3), then
|
||
**stable-with-a-deprecation-window** (one minor release); workflow-spec hashes include the schema version so
|
||
affinity invalidation is explicit, never silent. The migration plan gates *behavior* at every phase — this is the
|
||
corresponding gate for *interfaces*.
|
||
|
||
---
|
||
|
||
## 7. Worked examples
|
||
|
||
**(a) Wan 2.2 T2V (today's bread and butter).** Graph: validate → text-encode → `DenoiseLoop(transformer,
|
||
cfg=AdaptiveGateCFG, routing=BoundaryTimestepRouting, attn=VSAProvider)` → vae-decode. Single pool, SP as today
|
||
(CFG-parallel available once the axis lands). Behavior and perf identical to today; the boundary-switch and VSA
|
||
hacks become declared policies.
|
||
|
||
**(b) Wan-Causal streaming T2V.** `ChunkRollout` outer loop holding a `ChunkKVPool` handle; inner
|
||
`DenoiseLoop(chunk)`; each finished chunk → incremental VAE decode → `emit` → SSE frames. The same graph with a
|
||
session-scoped KV handle serves MatrixGame-style interactive world models (actions arrive as streaming-input
|
||
updates, vllm-omni-style resumable requests).
|
||
|
||
**(c) LTX-2 two-stage A/V.** validate → text-encode → `DenoiseLoop(joint video+audio latents)` — likely a
|
||
**custom step body** under the §6.2.3 escape hatch, since LTX2's 1–4 guidance passes alter the network via forward
|
||
kwargs, a braid the slot skeleton doesn't own cleanly; per-modality CFG scales and STG remain policy-shaped if the
|
||
factoring proves out → upsample → `DenoiseLoop(refine, lora-swapped)` → fan-out {vae-decode video,
|
||
audio-vae+vocoder} → assemble (video+audio artifacts). Today's 11 hand-ordered stages become a graph with one fork
|
||
either way — the graph and loop contract don't care which way the step body lands.
|
||
|
||
**(d) Qwen-Omni-style thinker→talker→vocoder (separable omni).** Three nodes: `ARDecodeLoop(thinker,
|
||
emit=hidden_states)` →(stream)→ `ARDecodeLoop(talker, emit=codec_chunks)` →(stream)→ `Vocoder`. Single pool or
|
||
per-node placement; `async_chunk`-style streaming via `StepResult.emit` + connector readiness. Optionally the
|
||
thinker delegates to an external engine node.
|
||
|
||
**(e) Cosmos3 full omni (the forcing function).** The §6.2.4 spec: optional `ARDecodeLoop(reasoner)` with paged KV
|
||
(replacing re-prefill-per-token) → pack → joint `DenoiseLoop(vision[+action][+sound])` → fan-out decoders →
|
||
assemble {video, audio, action, text}. One resident MoT transformer bound to both loops; REASON task short-circuits
|
||
to text-only; world-model rollout = `ChunkRollout` with temporal-causal masks and clean-latent chunk KV.
|
||
|
||
**(f) Flux2 / Stable Audio serving.** Same `DenoiseLoop` with image/audio latents; StepScheduler batches
|
||
shape-bucketed requests; embedded-guidance and precision quirks are policies. FastVideo becomes a credible image/audio
|
||
server for free.
|
||
|
||
**(g) DiffusionNFT RL (landed: PR #1450).** The landed method is already *library-mode shaped* — rollouts run
|
||
in-process under the trainer's own kernels — but against a vendored loop: `DiffusionSampler` walks the full
|
||
25-step ODE (or re-noising `sde_reflow`) schedule over the frozen **old-policy** copy, conditional-only,
|
||
`attn_kind="dense"` pinned, and the objective is **likelihood-free**: only final clean latents survive the
|
||
rollout; training re-noises them at shuffled schedule timesteps and contrasts the student against an *implicit
|
||
negative policy* in prediction space, so per-step log-probs exist nowhere in the method. Under this design the
|
||
same method drives the shared `DenoiseLoop` with a sampler policy and `OutputSpec(return_final_latents=True)` —
|
||
deleting `rl/common/sampling.py` — and gains what the vendored loop cannot offer: CFG policies, distilled few-step
|
||
samplers, caches, and batched scheduling for the rollout phase (today's rollouts run with zero serving-grade
|
||
optimizations). Engine-client mode: seeded rollout `OmniRequest`s fan out to a rollout pool running the same loop,
|
||
and the trainer pushes `update_weights` between iterations — note the weights shipped are the *old policy*, a
|
||
decay-blended copy, a concrete `WeightSyncPlan` case (§8.6) — with sleep/wake reclaiming rollout memory during the
|
||
gradient phase. Consistency per §8.5's own rules: NFT is the *easy* rung — likelihood-free means no ratio exists
|
||
for drift to poison, and the landed discipline (pinned attention kind, pinned autocast dtype, seeded generators)
|
||
is C1 by construction; **the full ladder is for the GRPO-class methods that come next** — C1-pinned rollout (cache
|
||
acceleration off) bounds per-step log-prob drift to ε, trajectory *identity* is C2's conditioned replay guarantee,
|
||
not C1's; alternatively C0 with cache-dit on buys throughput and pays for it with TIS correction — never both at
|
||
once.
|
||
|
||
---
|
||
|
||
## 8. Training, post-training, and RL on the same substrate
|
||
|
||
FastVideo's differentiation — and its complexity tax — is that training and post-training live in the same repo as
|
||
inference, with RL now **landed** (PR [#1450](https://github.com/hao-ai-lab/FastVideo/pull/1450), DiffusionNFT for
|
||
Wan — no longer a theoretical reference: a 3,500-line PR on main spanning the 1,075-line method, a vendored
|
||
sampler, reward scorers, trainer hooks, tests, and three `.agents/skills` codifying its extension contract). The
|
||
question: does a fully unified design make sense? **Answer: unify the substrate, not the stacks** — and
|
||
for video models this is less a choice than a recognition: unlike LLMs, diffusion's inference optimizations are
|
||
themselves post-training artifacts (§8.3), so the loop inversion of §6.2.2 is the missing shared piece. One honest
|
||
qualification, learned from studying two production RL frameworks (`~/verl-omni` for diffusion RL, `~/miles` for LM RL): the industry's pain is not really
|
||
"many repos." It is **two runtimes with different kernels** — Megatron/FSDP-class training optimizations on one
|
||
side, vLLM/sglang-class inference optimizations on the other — and an entire mismatch-correction industry has grown
|
||
around that split (§8.4). That is simultaneously the strongest argument for what we are building and the sharpest
|
||
warning about it: one repo does not buy numerical consistency. One model definition + one kernel set + an
|
||
**enforced, measured consistency contract** does (§8.5).
|
||
|
||
### 8.1 What exists today (and what is duplicated)
|
||
|
||
Three consumers of the generative computation, with partial sharing:
|
||
|
||
| Layer | Shared today | Duplicated today |
|
||
|---|---|---|
|
||
| Models, component loaders + FSDP load, arch configs (`param_names_mapping`), schedulers, `fastvideo/distributed/`, attention backends, offload hooks | ✓ single-source across inference, `train/`, legacy `training/` | — |
|
||
| Validation generation | new stack's `ValidationCallback` instantiates the real inference pipeline and calls `forward()` | legacy stack rolls its own; the landed RL method adds a third path — `TrainingMethod.on_validation_begin()` + `rl/common/validation.py` re-implement prompt sharding, seeded sampling, and sample logging precisely because (per the hook's own docstring) these are "methods that intentionally avoid inference pipelines" |
|
||
| **Denoise/sampling loop** | — | **4 copies**: `pipelines/stages/denoising.py` (inference); `DMD2Method._student_rollout()` / `SelfForcingMethod._student_rollout_streaming()` (`train/methods/distribution_matching/`); inlined per-pipeline in legacy `training/*_distillation_pipeline.py` (~1200-line monoliths); `DiffusionSampler` in `train/methods/rl/common/sampling.py` (RL, landed with PR #1450) |
|
||
| CFG / uncond handling | — | 3 copies: `stages/denoising.py` (cond/uncond branching + CFG gating; `conditioning.py`'s forward is a no-op that only validates flags) vs `ModelBase.predict_noise(cfg_uncond=…)` vs legacy inline |
|
||
| Preprocessing | `pipelines/preprocess/*` already subclass `ComposedPipelineBase` and reuse `TextEncodingStage` → parquet latents/embeddings | trajectory extraction (`preprocess_pipeline_ode_trajectory.py`) is a separate path from `return_trajectory_latents` |
|
||
|
||
The fourth copy is no longer a prediction — PR #1450 landed it. `train/methods/rl/common/sampling.py` vendors
|
||
`DiffusionSampler`: an ODE walk delegating per-step math to the shared `FlowMatchEulerDiscreteScheduler.step`, plus
|
||
a re-noising `sde_reflow` mode that is a near line-for-line re-derivation of DMD2's rollout loop
|
||
(`dmd2.py:484–520`). Its own docstring states the reason: it *"intentionally does not call FastVideo's full
|
||
inference pipelines"* because that would bind the method to model-family pipeline classes like `WanDMDPipeline` —
|
||
and the accompanying skill (`.agents/skills/rlhf-training-abstractions/SKILL.md`) elevates this to standing policy
|
||
("do not bind RL methods to inference pipeline classes"), with a test asserting the RL YAML never references
|
||
`WanDMDPipeline`. The authors are right to refuse that dependency, and that is precisely the indictment: today's
|
||
only consumable units are family-bound pipeline classes, so every new post-training method must choose between a
|
||
wrong dependency and a private loop. The `add-rl-method` skill now mandates the vendored sampler for future RL
|
||
methods — which contains the duplication *within* `train/methods/rl/` but cements it outside the substrate, where
|
||
no inference optimization, CFG policy, or scheduler improvement will ever reach it.
|
||
|
||
Concretely the "fourth copy" is already **two** vendored RL/distill loops, not one: `DiffusionSampler`
|
||
(`rl/common/sampling.py:92-99` docstring, `:128-147` hand-rolled `for`-loop, `conditional=True`, `attn_kind=dense`)
|
||
*and* DMD2's `_student_rollout` (`dmd2.py:430-550`, `attn_kind=vsa`) — a second, independently vendored loop. What
|
||
each of these vendored loops **forgoes** by living outside the substrate is itemizable: (1) cross-group /
|
||
cross-request batching; (2) content-hash embedding/feature-cache reuse; (3) step scheduling and interleaving;
|
||
(4) cost-aware admission; (5) component-granular sleep/wake (it holds student + old + reference resident
|
||
simultaneously); (6) cache-dit / interceptor skip acceleration (forced dense, full 25-step ODE); (7) distilled
|
||
few-step sampler selection (a 4-step card collapses 25→4). A `DenoiseLoop` over substrate
|
||
step bodies is the third option this design exists to provide — and, per §8.4, it is the moat: rollout forward
|
||
*is* serve forward + capture, so every serving optimization is automatically a rollout optimization.
|
||
|
||
### 8.2 The decision: shared substrate, layered dependencies
|
||
|
||
Full unification — one framework that both trains and serves — is the wrong target: trainers own
|
||
optimizers/FSDP/grad-accumulation/checkpointing; the engine owns scheduling/caches/serving. What they genuinely
|
||
share is the **generative computation** and the **model substrate**:
|
||
|
||
```
|
||
substrate (shared): models/ + loaders/manifests + configs + schedulers + distributed/ + attention/
|
||
+ PipelineSpec graphs, Stage/LoopStage step bodies, sampler & CFG policies, PackedSeq
|
||
engine (inference): StepScheduler, queues, caches, serving, placement — depends on substrate
|
||
train (training): Trainer, methods, callbacks, data — depends on substrate (+ engine as client)
|
||
```
|
||
|
||
Rules and consequences:
|
||
|
||
- **`engine` never imports `train`** (the reverse is allowed). A substrate change is a change to both worlds — and
|
||
is gated as such (§12, risk 5).
|
||
- **Distillation rollouts call the shared loop — with one honest exception.** DMD2/KD methods drive
|
||
`DenoiseLoop.step` directly (grad-enabled where needed — step bodies are plain tensor programs, so autograd
|
||
composes) instead of re-implementing timestep iteration. **Self-forcing does not fit the inference step
|
||
contract** and is not forced into it: its rollout samples per-block exit indices broadcast across ranks, runs
|
||
no-grad steps to the exit, exactly *one* grad-enabled forward, then a separate no-grad `store_kv=True`
|
||
context-caching pass — grad-window flags and per-step cache-write control that would be training-only surface
|
||
in substrate code. It ships as a **custom training `LoopStage` in `train/`** (the §6.2.3 escape hatch),
|
||
consuming substrate pieces — model forwards, samplers, and `ChunkKVPool` in its declared **training mode**
|
||
(no mid-rollout recycling; grad-aware index snapshots so activation-checkpoint recompute doesn't double-advance
|
||
the cache, per `wan_causal.py:119-120, 405-431`). The dedup arithmetic stated honestly: shared step bodies where
|
||
they fit; custom loops stay custom but consume the substrate instead of vendoring their own sampling stack.
|
||
The trainer already concedes this shape in landed code: PR #1450 added `TrainingMethod.manages_optimization()` /
|
||
`managed_train_step(data_stream, …)` so DiffusionNFT's sample→score→inner-train outer epoch could own its own
|
||
optimizer cadence — method-owned loops are a supported trainer contract today, not a workaround this design
|
||
must invent.
|
||
- **Trajectory capture becomes one feature, not four.** `return_trajectory_latents`, the ODE-trajectory
|
||
preprocessing pipeline, RL rollout recording (the landed `SamplingResult` in `rl/common/sampling.py` —
|
||
latents + timesteps + sigmas — is the third ad-hoc capture path), and future log-prob capture for GRPO-class
|
||
methods all become `OutputSpec` options on the loop.
|
||
- **Preprocessing pipelines are loop-less graphs** of one-shot encode stages emitting parquet artifacts — they
|
||
already use the pipeline machinery, so this is a rename, not a rewrite.
|
||
- **Observers carry over for free**: ParityAligner / NaNWatch / Profiler attach to training rollouts identically —
|
||
first-class tooling for divergence hunts and precision (FP8/FP4, QAT) validation during training.
|
||
- The legacy `training/` stack stays frozen (per repo policy) and is unaffected until its models migrate to
|
||
`train/` on their own schedule; nothing here forces that migration.
|
||
|
||
### 8.3 Why diffusion is different: inference optimization *is* post-training
|
||
|
||
In the LLM world, inference optimization is mostly post-hoc and training-free: a lab ships frozen weights, and the
|
||
serving stack optimizes *around* them — PTQ quantization, KV-cache management, continuous batching, speculative
|
||
decoding, kernels. The artifact crossing the trainer→engine boundary is a checkpoint. That is why the two-runtime
|
||
split is tolerable there: the inference frontier can advance without touching training.
|
||
|
||
Video generation does not work that way. **The deployable model is itself a post-training artifact**, and nearly
|
||
every inference capability FastVideo ships is really a **(recipe, runtime) pair**:
|
||
|
||
| Inference capability | Training-side half | Runtime-side half |
|
||
|---|---|---|
|
||
| Few-step generation — the difference between unusable and usable | **Step distillation** (DMD2, consistency/rCM) — an active research frontier; recipes change quarterly | distilled samplers / timestep policies |
|
||
| Streaming + world models (AR video) | step distillation **plus a self-forcing variant on top** — the causal student *is created by training* | `ChunkKVPool`, causal rollout, streaming |
|
||
| Low-precision inference | **QAT** (our NVFP4 Attn-QAT line, upstreaming now) — PTQ alone doesn't hold quality; training must see deployment numerics | FP4/FP8 kernels + `PrecisionPolicy` |
|
||
| Sparse attention | **VSA** — trained sparsity (tile selection learned during training) | VSA kernel backend |
|
||
| Reward alignment | RL — DiffusionNFT landed (#1450, likelihood-free: needs only final-sample capture); GRPO-class next | samplers (ODE/SDE), final-latent + trajectory/log-prob capture |
|
||
|
||
Two structural consequences:
|
||
|
||
1. **The training loop embeds the inference loop — and that collocation is the central, durable moat.** Distillation
|
||
runs deployment-grade rollouts *inside* training: DMD needs student rollouts; self-forcing is literally "train
|
||
against your own KV-cached causal rollout" — the runtime feature must exist before the model class does; QAT's
|
||
fake-quant forward must match the deployed quant kernels. There is no valid sequencing of "build the inference
|
||
engine first, training later" — each is a dependency of the other, which is exactly why the substrate (§8.2) has
|
||
to come first, and why the denoise loop keeps getting copied (§8.1): every post-training method needs an
|
||
inference loop inside it, and absent a shared one, each grows its own. The payoff of collocating
|
||
inference + post-training/RL on one substrate: **rollout forward is serve forward + capture** — same loop, same
|
||
caches, same batcher, same numerics — so every serving optimization is automatically a rollout optimization and
|
||
there is exactly **one numerics surface** (no rollout-vs-train kernel gap to patch, §8.4).
|
||
|
||
**RL rollout is a strictly *better* cross-request batching case than open-world serving** — the killer point. A
|
||
GRPO/NFT group is K identical-config samples of one prompt: `num_video_per_prompt:24`,
|
||
`sample_train_batch_size:6`, `num_batches_per_epoch:48`, `num_steps:25`, single-frame 448×832, 4 GPUs
|
||
(`diffusion_nft_pick_clip.yaml:33-37,52-55,72-91`; `distributed_k_repeat_indices` with `repeats_per_prompt=24`,
|
||
`diffusion_nft.py:331-338`). Same shape, same schedule, same CFG branch, K=24-wide → **zero bucketing needed**
|
||
(serving must bucket heterogeneous resolutions/steps/CFG). The K samples also share *one* prompt embedding, so a
|
||
content-hash feature cache encodes the text **once** and reuses it across all 24 — a **24× text-encode
|
||
reduction** the vendored loop cannot express (it carries the embedding per-sample, `diffusion_nft.py:308-309`).
|
||
Per GPU per outer epoch = 6 prompts × 48 batches = 288 prompt-slots, each a 24-wide homogeneous denoise batch:
|
||
near-ideal co-batchable work the vendored Python loop runs **one sample at a time**.
|
||
|
||
2. **Train/serve skew generalizes the RL mismatch.** Any sampling-in-the-loop post-training optimizes the student
|
||
*under the rollout implementation used during training*. Serve it under different kernels/sampler/caching and
|
||
quality silently degrades — the diffusion analog of RL's off-policy bias, except it applies to essentially all
|
||
post-training and surfaces as quality regression rather than reward collapse (slower to notice, harder to
|
||
attribute). The consistency ladder (§8.5) is therefore **not an RL feature**: C1 (kernel-pinned) is the default
|
||
contract for *every* sampling-in-the-loop method — distill what you serve, serve what you distilled. (miles
|
||
reaches the same place from the LLM side: its INT4 QAT exists because rollout efficiency at 1TB scale required
|
||
training to see inference numerics.)
|
||
|
||
This reframes the unification question (§11.12): in LLMs, an inference engine can ignore training because
|
||
optimization is training-free; in diffusion, an inference engine that cannot run inside training loops is **cut
|
||
off from the actual optimization frontier**, and a training framework without a serving-grade rollout distills
|
||
models toward a toy sampler. FastVideo's "advantage that is also its disadvantage" is precisely that it occupies
|
||
the one position where the (recipe, runtime) pair can be co-designed and parity-gated. The RL argument of §8.4 is
|
||
a special case of this.
|
||
|
||
### 8.4 The lesson from RL systems: the two-runtime tax
|
||
|
||
**Why the split exists.** Training forwards want FSDP/Megatron sharding, activation checkpointing, high-precision
|
||
gradient accumulation, and sequence packing. Rollout forwards want KV caches, continuous batching, CUDA graphs,
|
||
quantization, and inference kernels. Run the *same weights* through both and the log-probs differ — different
|
||
kernels, precision paths, batching, reduction orders. The policy that generated the trajectories (π_rollout) is no
|
||
longer the policy being optimized (π_train); PPO-style ratios are computed against the wrong baseline; at scale
|
||
(especially MoE + FP8) this is a documented cause of training collapse. Everything below exists to patch that
|
||
wound:
|
||
|
||
| The tax | verl-omni (diffusion RL) | miles (LM RL) |
|
||
|---|---|---|
|
||
| **Dual model definitions** | Wan2.2 implemented *twice*: a rollout pipeline inside vLLM-Omni (`pipelines/wan22_dance_grpo/vllm_omni_rollout_adapter.py`, with trajectory/log-prob capture and dual-expert switching) and a diffusers/FSDP2 trainer engine (`workers/engine/fsdp/diffusers_impl.py`, VeOmni variant) | model exists in sglang *and* Megatron, bridged by weight-name iterators (Megatron fused params → HF names) |
|
||
| **Mismatch corrections** | `trainer/diffusion/rollout_correction.py`: *bypass mode* (trust rollout log-probs) vs *decoupled mode* (recompute under trainer + truncated importance sampling + rejection-sampling masks) | TIS/MIS in `backends/training_utils/loss_hub/corrections.py`: clamp `exp(logp_train − logp_rollout)` to configurable `[tis_clip_low, tis_clip]` bounds (CLI defaults `[0, 2.0]`; their MIS example config uses `[0.5, 2.0]`) or hard-mask out-of-range tokens (MIS); `tis_clipfrac` monitored as a health metric |
|
||
| **Determinism engineering** | deterministic rollout/reward/trainer modes; per-step seed derivation | a whole program for **bit-wise identical log-probs**: FlashAttention-3 + DeepGEMM batch-invariant kernels + deterministic sglang, both sides pinned to TransformerEngine |
|
||
| **MoE routing drift** | — | **R3 rollout routing replay**: capture per-token expert choices in sglang (`(seq−1, layers, top_k)` int32 — ~60 MB/sample at 32K×60×8) and replay them in the Megatron forward |
|
||
| **Precision drift** | — | **Unified FP8**: identical quantization recipe (blockwise / MXFP8) on both sides; weights dequantized to BF16 for transfer, re-quantized identically on arrival |
|
||
| **Weight-sync machinery** | FSDP2→vLLM-Omni LoRA collection + RPC; sleep levels (1 = keep VAE/text-encoder resident, 2 = full offload) | three transports behind one interface: Ray-IPC zero-copy (colocated), bucketed NCCL broadcast, P2P-RDMA with pinned-CPU staging; per-sample `weight_versions` staleness tags |
|
||
|
||
The punchline for FastVideo: verl-omni's Wan2.2 rollout adapter is a **re-implementation of code this repo already
|
||
owns** — external RL frameworks must rebuild our models inside other engines to RL-train them, then correct the
|
||
numerics. Doing RL natively means the model exists once and the corrections become a fallback, not a foundation.
|
||
|
||
**The moat made concrete.** Under this design, rollout = the *same* shared `DenoiseLoop` + a `CFGPolicy` (for NFT:
|
||
conditional-only / identity — `predict_noise` already carries `conditional:bool` + `cfg_uncond` as the policy knob,
|
||
`train/models/base.py:142-152`) + an `OutputSpec(capture=behavior)` + the step scheduler + the caches — so rollout
|
||
forward *is* serve forward + capture. Every serving optimization is automatically a rollout optimization, and there
|
||
is one numerics surface, no rollout-vs-train kernel gap to patch. The alternative is the **two-runtime tax** the
|
||
table above enumerates: verl-omni re-implements Wan inside vLLM-Omni and stitches it with `rollout_correction.py`'s
|
||
IS / rejection masks; miles' TIS/MIS, bitwise log-probs, R3 routing replay, and unified-FP8 are *all* mismatch
|
||
patches. FastVideo's own landed NFT confirms it from the other side — a vendored bare-model loop with zero
|
||
serving-grade optimizations (§8.1). Collocation at FastVideo's 1–30B FSDP2 scale **deletes the second runtime, the
|
||
correction layer, and the 3rd/4th denoise-loop copies** — and it makes the `(recipe, runtime)` flywheel (§8.3,
|
||
§9.5) honest, because preferences are then collected under the very serving profile that ships. This collocation of
|
||
inference + post-training (including RL) on one substrate is the central, durable moat. Three caveats the design
|
||
already names keep this from being hand-waving: old-policy weights are a `WeightSyncPlan` **role** (rollout samples
|
||
from `self.old`, decay-blended `diffusion_nft.py:858-862`); the dense-attn pin is a Precision/Attn **policy** on the
|
||
card; and likelihood-free NFT is the **C2 behavioral rung** (no log-probs to match — `old_deviate` / ref-MSE in
|
||
prediction space, `diffusion_nft.py:777-778`).
|
||
|
||
**What stays regardless** (intrinsic to RL, not artifacts of the split): trajectory buffers and rollout
|
||
management, group-relative advantage computation, multi-reward serving with async overlap, staleness control,
|
||
evaluation-during-training, checkpointing, fault tolerance (health-monitored engines). We adopt these; we delete
|
||
the rest. Two of them already exist landed in-repo as the baseline to build from: #1450's gathered per-prompt
|
||
group-advantage normalization (`diffusion_nft.py::_compute_advantages`) and its weighted `MultiRewardScorer`
|
||
(in-process, GPU-resident PickScore/CLIP — the async/HTTP serving half is the part still to come).
|
||
|
||
**The boundary condition, stated honestly.** Miles defaults to Megatron because its targets are ≥70B MoE models
|
||
(1TB-scale K2-Thinking with INT4 QAT). FastVideo's training targets are 1–30B DiTs/MoTs on FSDP2/HSDP — *that
|
||
scale assumption is what makes the single-runtime bet viable.* If a Megatron-class trainer ever becomes necessary
|
||
(100B+ MoT training), those models re-enter the two-runtime world — and the toolkit below (C0 corrections,
|
||
Behavior Record, weight transports) is designed to still apply as the fallback. This is a trigger to monitor, not
|
||
a reason to build for it now.
|
||
|
||
### 8.5 The bet: one model definition, two execution profiles, a consistency ladder
|
||
|
||
Even in one runtime there are honestly **two execution profiles**: the training forward (grad-enabled, activation
|
||
checkpointing, FSDP-gathered params, no caches) and the rollout forward (no-grad, CUDA graphs, KV/feature caches,
|
||
TP/SP-sharded, possibly quantized, possibly cache-dit-accelerated). Same repo ≠ same numerics — and the precise
|
||
mechanism matters. Thinking Machines' ["Defeating Nondeterminism in LLM
|
||
Inference"](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) is the definitive
|
||
analysis: forward passes contain no atomics and are *run-to-run deterministic for a fixed batch*; what breaks is
|
||
**batch invariance** — kernels switch reduction strategies with batch/shape (split reductions in RMSNorm, split-K
|
||
and tile/instruction selection in matmul, KV-split counts in attention), so a request's numerics depend on what it
|
||
was batched with. Compose that with variable load (or, in our case, with rollout-vs-training profile differences)
|
||
and you get nondeterminism — their headline: 1,000 temperature-0 completions of Qwen3-235B produced **80 unique
|
||
outputs**, first diverging at token 103; with batch-invariant kernels, 1,000/1,000 identical. Miles' bitwise
|
||
program (FA3 + DeepGEMM) and sglang's deterministic mode descend from this work. One codebase makes parity
|
||
tractable, not automatic — so consistency is a **declared, measured contract**, per run:
|
||
|
||
- **C0 — corrected.** Profiles free to differ: quantized rollout, cache-dit on, CUDA graphs. Trajectories store
|
||
rollout log-probs; objectives apply TIS/MIS (miles' clamp/mask semantics ported to per-step diffusion log-probs,
|
||
with verl-omni's rejection-sampling mask as an option); `logprob_diff` and clip-fraction are always-on training
|
||
telemetry. This is the throughput mode and the graceful-degradation mode.
|
||
- **C1 — kernel-pinned (default for RL).** Rollout-for-training pins the profile to the trainer's: same attention
|
||
backend, same `PrecisionPolicy`, distribution-altering interceptors disabled, eager or shape-matched graphs.
|
||
ParityAligner gates per-step log-prob diff ≤ ε in CI for likelihood-based methods (ε set per model family from
|
||
Phase-1 harness data, not guessed); the drift metric should sit at ~0 during training. "Trust the rollout
|
||
log-probs" (verl-omni's bypass mode) is then actually sound instead of hopeful. The landed DiffusionNFT (#1450)
|
||
is C1 by construction — in-process rollouts under the trainer's kernels, `attn_kind="dense"` hard-pinned on
|
||
rollout and all three training forwards, `FASTVIDEO_ATTENTION_BACKEND` defaulted to `FLASH_ATTN` in the launch
|
||
script — and it
|
||
already paid a C1-class bug tax that proves the discipline isn't free: `WanModel.predict_noise` had to stop
|
||
deriving autocast dtype from caller tensors because RL-created fp32 intermediates silently changed numerics
|
||
between rollout and gradient forwards (the `_get_training_dtype()` fix + `test_wan_training_dtype.py`). The CI
|
||
enforcement pattern also has a landed coarse precedent: #1396's per-method, device-keyed grad-norm regression
|
||
refs (10% rtol) — the same pin-a-golden-and-gate shape ParityAligner applies at per-step resolution.
|
||
- **C2 — bitwise.** Batch-invariant kernel set + deterministic mode + Behavior Record replay → bit-identical
|
||
log-probs between rollout and training forwards. The recipe is known (per the Thinking Machines post): fixed
|
||
reduction strategy for norms, one pinned matmul kernel config across shapes (~20% off cuBLAS peak, acceptable),
|
||
and attention with **fixed split-size** (not split-count) KV reductions plus a consistent KV layout (page-table
|
||
update before the kernel) so numerics are independent of how many tokens are processed at once — implementable
|
||
as a `torch.Library` op-swap (their `batch_invariant_ops`) behind our attention-backend registry as a
|
||
`deterministic` backend mode — and torchtitan already carries `set_batch_invariance()` utilities, so the kernel
|
||
set is landing in the PyTorch ecosystem proper, not just a vendor library. Measured cost in their vLLM demo:
|
||
26s → 42s (~1.6×) with the improved attention kernel. The payoff — stated with its conditions: in their RLVR
|
||
experiment, no correction collapses mid-run, importance weighting trains with KL ≈ 10⁻³, and bitwise
|
||
sampler-trainer identity holds KL flat at exactly 0. For diffusion, **kernels alone do not confer that**:
|
||
C2's "true on-policy" holds only as the conjunction of batch-invariant kernels *plus* the full Behavior Record
|
||
(sampler noise/RNG, guidance state, routing) *plus* C1's preconditions (matched precision, interceptors off,
|
||
same weight version, identical SDE log-prob arithmetic) — and it is a per-trajectory replay guarantee, verified
|
||
by ParityAligner, not a blanket property. C2 is our golden-run, divergence-debugging, and MoE-routing-parity
|
||
mode; C1+TIS covers daily training.
|
||
|
||
**Where batch variance actually bites FastVideo** (the post's lens applied to our workloads): the video denoise
|
||
path is effectively batch-1 with fixed shapes — accidentally near-batch-invariant already, our structural
|
||
advantage. The exposure is elsewhere: (a) **AR pathways** (Cosmos3 reasoner, omni thinkers) under continuous
|
||
batching, where batch composition changes every decode step — the blog's exact scenario, and why C1/C2 for AR
|
||
segments needs either batch-invariant decode kernels or frozen-batch rollout; (b) **image serving** with bucketed
|
||
cross-request batching; (c) **parallelism-degree differences** between rollout (SP/TP-sharded) and training
|
||
(FSDP-gathered) forwards — sharding is a batch-like axis that changes reduction layout, so C1 pins matched degrees
|
||
and C2 requires invariance across them; (d) **chunked causal video attention** (`ChunkKVPool`), where the
|
||
fixed-split-size principle applies directly: KV-chunk reductions must be invariant to how many chunks are resident,
|
||
or world-model rollouts won't replay bit-identically.
|
||
|
||
**Behavior Record (R3, generalized) — budgeted at real bytes.** Trajectories optionally capture *behavior
|
||
artifacts* beyond latents/log-probs: per-step RNG state (cheap — seeded generators in one shared loop reproduce
|
||
draws, so seeds suffice), sampler branch choices (cheap), and MoE routing decisions (**not cheap for diffusion**:
|
||
miles' ~60 MB/sample is per-token routing with one forward per token; diffusion re-routes the entire packed
|
||
sequence at *every denoise step, per CFG branch* — steps × branches × tokens × sparse-layers × top_k, which for a
|
||
Cosmos3-class request (~50K packed tokens, ~24 sparse layers, top_k 4, 35–50 steps × 2 branches) is **~1.3–1.9
|
||
GB/sample** int32, ~0.3–0.5 GB as uint8 — tens of GB per GRPO group). Consequently routing capture is an
|
||
**opt-in instrument for C2 goldens, MoE-parity debugging, and divergence hunts — not a training-scale default**;
|
||
routine RL runs at C1 (or C0+TIS) without it. If it is ever wanted at fleet scale, the GB-trajectory
|
||
streaming/storage design it forces is acknowledged as undesigned (§12 known gaps). The plumbing is an
|
||
`OutputSpec` feature of the loop (§6.2.2); the bytes are the cost, and they are stated.
|
||
|
||
**Plugins are policy-relevant.** cache-dit during rollout changes the sampled distribution — trajectories then come
|
||
from a behavior policy β ≠ π_θ. Contract: interceptors declare `distribution_altering: bool`; at C1+ such plugins
|
||
are disabled for rollout-for-training; at C0 they are permitted and stored log-probs must reflect the actual
|
||
(cached) computation, with IS correction in the objective. Enforced declaratively at pipeline build, not by
|
||
convention.
|
||
|
||
**Precision unification** is the same contract one level down (miles' FP8 lesson): rollout and trainer share one
|
||
`PrecisionPolicy` / quantization recipe — "train BF16, roll out FP8" is only legal at C0. This dovetails with the
|
||
in-repo Attn-QAT/FP4 work: QAT is exactly what makes low-precision rollout consistent with training by
|
||
construction.
|
||
|
||
### 8.6 Weights, parallelism duality, and placement
|
||
|
||
The trainer mesh (FSDP/HSDP) and the engine mesh (TP/SP/CFG) are different layouts over the same parameters — both
|
||
expressed in the stacked-axis spec (§6.3.4), e.g. `[hsdp_replicate(2), hsdp_shard(4)]` vs
|
||
`[dp(2), cfg(2), sp(2), tp(1)]`. The **`WeightSyncPlan`** is computed from **three inputs, not two** — axis specs
|
||
alone are necessary but not sufficient: (a) the pair of mesh specs; (b) **per-model weight-layout adapters**
|
||
covering what specs cannot express — fused QKV/MLP packing, MoE expert placement, quantized storage formats, LoRA
|
||
deltas, FSDP flattening, and checkpoint key translation (the `param_names_mapping`/state-dict-adapter lineage this
|
||
repo already maintains for conversion; torchtitan reaches the same conclusion — its fused-SwiGLU override needs
|
||
model-level state-dict adapters, and its own notes mark some redistributions as inexpressible,
|
||
`spmd_types.py:45`, `overrides/fused_swiglu.py:34`); (c) the transport. The plan is validated pre-flight and
|
||
CPU-testable with fake pools — replacing hand-written per-combination code (the bug class miles' weight-name
|
||
iterators and verl's resharding exist to manage), while being explicit that the per-model adapter is authored, not
|
||
derived. Phasing: Phase 2 ships the **colocated** cases (same-layout zero-copy; in-place reshard for the Phase-1
|
||
families' FSDP↔engine layouts, adapters hand-checked); the general cross-mesh plan over the declarative spec lands
|
||
with the spec in Phase 3.
|
||
|
||
- **Transports** (miles' menu, one interface): colocated same-layout → zero-copy views — the unified-runtime
|
||
dividend: often a literal no-op; colocated different-layout → in-place reshard; disaggregated → bucketed
|
||
collective broadcast or P2P-RDMA with pinned-CPU staging. `multimodal_gen`'s RL weights-updater is the in-family
|
||
precedent for the RPC surface. At fleet scale, weight distribution and version tracking should ride Dynamo's
|
||
ModelExpress rather than our own fan-out — that is ask A5 (§6.3.6): versioned incremental broadcast +
|
||
staleness-aware routing, with our transports as the colocated/fallback path. Two further precedents to reuse:
|
||
torchtitan's RL experiment syncs PolicyTrainer→vLLM via **TorchStore** push/pull with direct-RDMA state-dict fill
|
||
— the PyTorch-ecosystem transport to evaluate first for the disaggregated path (with its own caveat: DCP and
|
||
TorchStore do structure-matched fill, *not* automatic cross-mesh resharding — the `WeightSyncPlan` still owns the
|
||
layout math); and cosmos-rl's **buffer-model async sync** (a background thread on a side CUDA stream writes
|
||
incoming weights into a parameter-only clone; the live model swaps at safe points) is the pattern for
|
||
non-blocking colocated updates.
|
||
- **Sleep/wake is component-granular** (verl-omni's levels, generalized by our component registry): a rollout pool
|
||
can drop KV/feature caches and DiT weights while keeping VAE/text-encoder resident (level 1) or release
|
||
everything (level 2). Implementation reference: vLLM's `CuMemAllocator`
|
||
(`vllm/device_allocator/cumem.py`) — tag-based unmap/offload where our tags are component names (finer than
|
||
vLLM's global `weights`/`kv_cache` tags), with its documented caveats inherited: CUDA graphs invalidate across
|
||
sleep (recapture on wake), woken KV pools are zero-initialized (sessions re-prefill), and encoder/embedding
|
||
caches must `reset()` on weight update.
|
||
- **Staleness is first-class** (miles): every trajectory carries `weight_version`; `update_weights_interval` and a
|
||
max-staleness bound govern the async overlap loop (generate batch N+1 while training on N — the
|
||
`train_async.py` pattern); over-sampling with abort/truncation statuses handles stragglers, and loss masks zero
|
||
out incomplete samples. The landed NFT method (#1450) adds a wrinkle the plan must carry: its behavior policy is
|
||
not the student but a decay-blended **old** copy (and validation optionally swaps in the EMA shadow) — so a
|
||
`WeightSyncPlan` ships a declared *role* (student / EMA / blended old policy), never implicitly "the weights",
|
||
with the blend applied trainer-side before push or pool-side on arrival.
|
||
|
||
### 8.7 The RL layer on the engine
|
||
|
||
Components (all under `train/`, consuming the substrate and the engine — never the reverse). These are now the
|
||
**migration target for landed code**, not a blank-slate plan: `train/methods/rl/` exists on main (#1450) as a
|
||
1,729-line package — a 1,075-line `TrainingMethod` with advantages, objective math, and inner optimization
|
||
inlined, driving its own reusable-but-package-private sampler and reward modules — no `RolloutFn`, no
|
||
`TrajectoryBuffer`, consuming neither substrate loops nor engine — and its three `.agents`
|
||
skills (add-rl-method, add-reward-model, rlhf-training-abstractions) codify the current pre-§8.7 conventions.
|
||
The list below is what that module refactors into as the substrate and engine land:
|
||
|
||
- **`RolloutFn`** — miles' single best interface lesson: *users write one function*. Contract:
|
||
`(ctx: RolloutContext) -> list[Trajectory]`, where `ctx` provides an engine client (seeded generation, with
|
||
log-prob capture where the method's `OutputSpec` declares it — NFT-class methods don't), the data source, and
|
||
tool hooks. Covers single-shot diffusion sampling, N-sample GRPO groups,
|
||
multi-turn/agentic episodes, and future interleaved omni episodes — without touching the trainer.
|
||
- **`Trajectory`** (our `Sample`): request + seeds, per-step records as declared by the method's `OutputSpec`
|
||
(final latents only for NFT-class; latents + timesteps + log-probs for GRPO-class), Behavior
|
||
Record, rewards (scalar or dict), loss masks, `weight_version`, status (`COMPLETED/TRUNCATED/ABORTED`).
|
||
- **`TrajectoryBuffer`**: collection, filtering, group-by-prompt, conversion to training batches.
|
||
- **Reward layer** (verl-omni's shape): weighted multi-reward aggregation; sync in-process scorers and async HTTP
|
||
scorers; reward computation overlapped with rollout; reward models loadable as engine components — and later as
|
||
engine graph nodes scoring decoded artifacts. The landed `train/methods/rl/rewards/` is the right seed: thin
|
||
PickScore/CLIP scorers loading from HF (the #1438 draft's vendored HPSv3/VideoAlign repos were dropped before
|
||
landing — good hygiene to keep, §12 open question 13), a duck-typed `RewardScorer = Callable[[media, prompts],
|
||
Tensor]` contract, and weighted aggregation in `MultiRewardScorer` — what's missing is the async/HTTP/offload
|
||
half, since today's scorers sit GPU-resident on every training rank for the whole run.
|
||
- **Objectives library**: FlowGRPO/DanceGRPO (clipped PPO over per-step SDE log-probs; SDE-with-logprob is a
|
||
sampler policy, implemented once), DiffusionNFT (final latents + policy/ref forwards — extracted from the landed
|
||
#1450, whose objective math today lives inlined in the monolithic method class rather than as a reusable
|
||
module), Diffusion DPO (offline pairs), and AR GRPO/GSPO for token pathways — with cosmos-rl's stability kit
|
||
(AIPO one-sided clipping for async off-policy rollouts, dual-clip, KL-threshold off-policy masking) as the
|
||
C0-mode options. Shared group-relative advantage utils.
|
||
- **Two rollout modes, one set of trajectories**: *library mode* — the method drives substrate loops in-process
|
||
(required when backpropagating through sampling; the landed #1450 is library-mode *shaped* — in-process,
|
||
no-grad, trainer kernels — but drives its private `DiffusionSampler` instead of substrate loops, which is the
|
||
Phase-1 migration); *engine-client mode* — `RolloutClient` submits seeded
|
||
`OmniRequest`s to a colocated or disaggregated engine, inheriting queueing, batching, step interleaving, and
|
||
(at C0) cache acceleration. At fleet scale the disaggregated rollout pool is a **Dynamo-fronted fleet**: rollout
|
||
requests route through Dynamo (staleness/version-aware once ask A5 lands, §6.3.6), the Planner shifts GPUs
|
||
elastically between trainer and rollout pools, and weight updates propagate via ModelExpress. The consistency
|
||
ladder (§8.5) plus seeded RNG in `RequestState` is what makes the modes interchangeable without retuning.
|
||
- **Loops**: sync (rollout → reward → advantage → update → weight-sync) and async (overlapped, staleness-bounded)
|
||
— thin scripts over these components, deliberately the same shape as miles' `train.py`/`train_async.py`.
|
||
|
||
**Omni-RL — expressibility is established; the algorithm is not.** A hybrid episode over a MoT model (Cosmos3):
|
||
one trajectory containing AR reasoner segments (token log-probs) *and* denoise segments (per-step SDE log-probs),
|
||
rewarded jointly — one model, one trajectory, two loop types. Engine-per-stage RL frameworks cannot express this
|
||
without two engines and two weight copies; that structural advantage is real. But "optimized with mixed
|
||
objectives" names an **unsolved algorithm design**, stated as three open problems (open question 16): (1) *scale
|
||
mismatch* — token log-probs are O(1–10) nats over 10²–10³ tokens while per-step diffusion log-probs are Gaussian
|
||
densities over 10⁶–10⁷ latent dims; a joint clipped-ratio objective needs principled per-segment normalization
|
||
that none of the cited recipes (FlowGRPO/DanceGRPO/NFT/AIPO/GSPO) provides, and getting it wrong lets one
|
||
modality silently dominate the shared trunk; (2) *credit assignment* — the reasoner influences video reward only
|
||
through sampled discrete tokens re-entering as conditioning, a non-differentiable boundary, so token segments get
|
||
sparse trajectory-level signal while denoise segments get dense per-step ratios, both updating shared attention
|
||
weights with no interference analysis; (3) *reasoning regression* — RL-updating the und pathway on
|
||
video-correlated reward risks degrading reasoning; hybrid episodes need reference-model KL anchoring.
|
||
Accordingly, the Phase-4 deliverable is **hybrid trajectory capture demonstrated plus an objective design study**
|
||
— not a pilot that presumes the objective exists.
|
||
|
||
**What we deliberately do not build**: verl's general Ray hybrid-controller choreography (actor/critic/ref/reward
|
||
worker groups with cross-runtime resharding). At our scale the trainer drives a simple loop; if critic/ref models
|
||
arrive they are additional engine pools, not a new controller paradigm.
|
||
|
||
### 8.8 Config unification with training
|
||
|
||
`train/utils/training_config.py` keeps what is genuinely training's (optimizer, data, loop, checkpoint, tracker
|
||
sections) and **imports the shared layers** for everything else: the deploy/parallelism spec (§6.3.4) replaces its
|
||
`distributed:` block (today a field-for-field duplicate of `FastVideoArgs.tp_size/sp_size/hsdp_*`), and model/
|
||
pipeline identity comes from the same `ModelSpec`/`PipelineSpec` the engine loads (§6.6). One description of "what
|
||
model, how sharded" serves inference, training roles (student/teacher/critic), and rollout.
|
||
|
||
---
|
||
|
||
## 9. The application pull: Dreamverse, ComfyUI, and the workflow layer
|
||
|
||
Dreamverse (`apps/dreamverse/` — Next.js frontend + FastAPI runtime; open-sourced with the
|
||
[vibe-directing](https://haoailab.com/blogs/dreamverse/) and
|
||
[release](https://haoailab.com/blogs/fastvideo-dreamverse-release/) posts) looks out of place in an inference-runtime
|
||
design at first glance. It is the opposite: it is the **proof of the thesis and the generator of the requirements**.
|
||
"Vibe directing" — steering video through continuous natural-language revision, 30-second scenes composed of chained
|
||
5-second clips, ~4.55 s per 5 s 1080p clip on one B200 (faster than real-time playback) — is a product category that
|
||
exists *only because of (recipe, runtime) pairs* (§8.3): faster-than-realtime clips come from NVFP4 (QAT-backed) +
|
||
FA4 + `torch.compile` on the runtime side and from distillation recipes on the training side, streamed as fMP4 over
|
||
WebSocket with frame/audio conditioning carrying continuity between segments. The blog's own one-liner — *"better
|
||
creative work comes from a faster feedback loop, not just a better model"* — is §8.3 in product language.
|
||
|
||
### 9.1 What Dreamverse proves about the runtime gap
|
||
|
||
Read `apps/dreamverse/arch.md` and the division of labor is striking: the **app server owns "GPU assignment and
|
||
worker lifecycle," "actual generation queue semantics," warmup, and "stream chunk emission"**
|
||
(`dreamverse/gpu_pool.py`, `main.py`). The application layer is hand-building the engine plane of §6 because the
|
||
runtime doesn't provide it — P4 made manifest in a product, with a price tag: **one B200 per active user session**.
|
||
Dreamverse is simultaneously the best evidence for the engine (Phase 2 retires `gpu_pool.py` into an engine client
|
||
at single-session parity and runs the duty-cycle study; multi-session multiplexing lands at the Phase-3 gate) and
|
||
the best customer for nearly every §6 feature:
|
||
|
||
| Dreamverse need (shipping today, hand-rolled) | Engine feature it pulls |
|
||
|---|---|
|
||
| GPU pool + queue + warmup in the app server | engine scheduler, warmup requests, multi-tenant session multiplexing (§6.3.1) |
|
||
| fMP4 segment streaming over WebSocket, `ltx2_segment_start`/chunk events | `OmniEvent` streaming contract; loop `emit` (§6.2.2) |
|
||
| frame/audio conditioning chained across 5 s segments; init-image persistence | **session abstraction**: session-scoped latents/caches and continuation conditioning as first-class request inputs — the same machinery as interactive world-model sessions (§12, open question 9, which now has a production owner) |
|
||
| prompt-rewrite LLM (external provider today, with timeout/fallback) | AR graph node or external-engine node (§6.3.4); a future omni model collapses rewrite+generation into one resident model |
|
||
| LoRA controls per direction (#1420) | LoRA hot-swap on the engine path |
|
||
| SP for serving (#1424), NVFP4/FA4/compile | DeployConfig parallelism + PrecisionPolicy + plugins — per-app deploy YAML instead of bespoke wiring |
|
||
| safety filtering, prompt memory, session protocol | stays in the app — the workflow layer is a *client* of the engine, never merged into it |
|
||
|
||
That last row is the architectural rule this section adds: **the workflow layer composes engine requests; it does
|
||
not extend the engine.** Dreamverse keeps UX, prompt memory, safety, and session semantics; the engine absorbs
|
||
exactly the parts the app had to hand-roll (pool, queue, warmup, streaming transport, placement).
|
||
|
||
**Where multi-session capacity actually comes from** (a correction the review process forced — note that today's
|
||
`gpu_pool.py` deliberately admits one user per GPU, `is_available` false while `connected_users` is non-empty):
|
||
step interleaving improves *fairness*, not throughput — two always-on sessions are ~2× service demand, and no
|
||
scheduler conjures that away. The capacity claim rests on three real mechanisms: (1) **duty cycle** — directing
|
||
sessions are idle-heavy (the user types, watches, decides between segments), so time-averaged demand per session
|
||
is well below one GPU; (2) **admission control on the cost model** — sessions admit against measured per-segment
|
||
cost and a per-class latency SLO, with simultaneous segment requests queued within the SLO rather than
|
||
co-batched; (3) the distillation roadmap lowering per-segment cost over time. Accordingly the multi-session gate
|
||
is defined on a **recorded duty-cycle trace** of real session behavior — p95 segment latency within the SLO vs
|
||
the single-session baseline — never on synthetic always-on load.
|
||
|
||
**Transport triggers (from the LiveKit/WebRTC review).** fMP4-over-WebSocket remains the right transport for
|
||
segment streaming. WebRTC (LiveKit-class SFU) becomes the transport when any trigger fires: sub-second streaming
|
||
latency, one-to-many fanout, or **interactive world models** — the <100 ms motion-to-photon budget
|
||
(≈20–40 ms action round-trip + ≤40 ms/frame inference + ~10 ms encode/decode/render) is unreachable over TCP/WS
|
||
and needs WebRTC data channels as the action path. The engine contract that keeps both transports thin adapters:
|
||
`OmniEvent` optionally carries **raw frames + PTS** (not only encoded segments — the SFU SDK encodes in a native
|
||
thread from RGBA/I420 buffers), sessions accept an async action stream, and a flush/interrupt RPC discards stale
|
||
frames on a direction change. Until a trigger fires, the SFU/TURN/token operational weight is not paid.
|
||
|
||
### 9.2 The flywheel: where post-training, inference, and RL meet a product
|
||
|
||
Dreamverse's intended trajectory — post-training + inference + eventually RL composing one production-ready
|
||
workflow — closes the loop of §8:
|
||
|
||
```
|
||
distillation/QAT recipes (§8.3) ──► faster-than-realtime engine ──► vibe directing UX
|
||
▲ │
|
||
│ ▼
|
||
RL / DPO post-training (§8.7) ◄── preference data: kept vs discarded revisions,
|
||
edit chains, per-segment directions
|
||
```
|
||
|
||
Vibe-directing sessions emit *dense, naturally-labeled preference data* — every revision a user keeps over the one
|
||
they discarded is a DPO pair; every edit chain is a trajectory with human reward. No video lab has this loop in one
|
||
stack: labs with products lack open post-training; open frameworks lack products. The §8 RL layer is what turns
|
||
Dreamverse from a demo into a data engine, and the consistency ladder (§8.5) is what guarantees the preferences
|
||
collected under the serving profile actually transfer into training.
|
||
|
||
### 9.3 Positioning
|
||
|
||
The diffusion community is structurally more diverse than LLM serving — many base models, LoRA ecosystems,
|
||
workflow culture (ComfyUI) — and rewards stacks that compose models rather than serve one. "Vibe directing" names a
|
||
UX category the way "vibe coding" did, and the moat behind it is not the frontend: it is the only stack where the
|
||
app, the engine, and the post-training that makes the app possible are co-designed and parity-gated. Dreamverse
|
||
also functions as the runtime's production forcing function (sessions, safety, cost-per-user, consumer-GPU targets
|
||
— the Attn-QAT NVFP4-attention work aims at RTX 5090/4090/3090) — keeping the engine honest the way Cosmos3 keeps
|
||
the model layer honest (§4.1). Dreamverse's own planned controller/provider split (local-first compute over
|
||
Runpod/Modal, per its `AGENTS.md`) stays the single-box story; at fleet scale it delegates to Dynamo (§6.3.5)
|
||
rather than growing its own orchestration — provisioning is not our plane.
|
||
|
||
### 9.4 The ComfyUI question: the funnel, the compiler, and the accelerated cloud
|
||
|
||
ComfyUI (`~/ComfyUI`, 116K stars, GPL-3.0) is where the commodity diffusion community lives, and its tree tells us
|
||
exactly what it is and isn't: a **node-graph orchestrator** — 64 core nodes + 117 `comfy_extras` packs (Wan, Cosmos,
|
||
Hunyuan, Mochi video natively in-core), a runtime-patching model manager (`comfy/model_patcher.py`, LoRA hot-patch,
|
||
smart VRAM offload, FP8), **39 paid API-node providers in-tree** (`comfy_api_nodes/`: Kling, Runway, Veo, Sora… —
|
||
its monetization is orchestrating *other people's* models), 80 in-tree workflow blueprints (more via the registry) plus an App Mode that wraps
|
||
graphs in simplified UIs — and **no serving runtime**: a FIFO prompt queue, no cross-prompt batching, no
|
||
multi-tenancy, no distillation-awareness (`execution.py`, `server.py`). Structurally: ComfyUI is a frontend and
|
||
ecosystem; FastVideo is a (recipe, runtime) factory. Those are complements before they are competitors, which
|
||
shapes the play as **three sequenced motions**:
|
||
|
||
- **Motion A — embed (exists today, underinvested).** `comfyui/` in this repo already ships ComfyUI-FastVideo
|
||
custom nodes (multi-GPU `VideoGenerator` inside their graphs). This is the funnel: meet 116K-stars'-worth of
|
||
users inside their own tool and let the speed delta advertise itself. A "FastVideo Cloud" *API node* — slotting
|
||
into the same `comfy_api_nodes` pattern Kling/Runway use — is the cheapest distribution wedge their own platform
|
||
has standardized.
|
||
- **Motion B — compile (the core bet).** The decisive observation, stated at its defensible scope: **the
|
||
tier-1/tier-2 static sublanguage of ComfyUI workflows maps onto `PipelineSpec`** — within that vocabulary, both
|
||
are typed dataflow graphs over model components (the general claim "any workflow is a PipelineSpec" is false:
|
||
arbitrary custom nodes, Python-level patching, and dynamic graphs are exactly what the tiers exclude). The popular surface is small and maps almost 1:1 onto §6:
|
||
`CheckpointLoaderSimple→ModelSpec`, `KSampler→DenoiseLoop` (sampler/scheduler/CFG policies),
|
||
`LoraLoader→LoRA hot-swap`, `CLIPTextEncode→TextEncode node`, `VAEEncode/Decode→codec nodes`,
|
||
`ControlNetApply→ConditioningInjector`, `LatentUpscale→upsample stage`, `WanImageToVideo/EmptyCosmosLatentVideo→`
|
||
task presets. A clean-room **workflow compiler** (ComfyUI JSON → `PipelineSpec` + `OmniRequest`; no GPL exposure —
|
||
translate the format, never embed the code) over a **tiered core vocabulary** — tier 1 ≈ 20 nodes
|
||
(t2i/i2v + LoRA + upscale), tier 2 ≈ 40 (inpaint, ControlNet, video-conditioning variants) — covers the canonical
|
||
workflows, with **capability negotiation**: unsupported nodes produce a coverage report and a clear rejection,
|
||
not silent wrongness. The long tail of arbitrary custom nodes is explicitly out of
|
||
scope for v1 (option later: partition the graph and run unsupported subgraphs in a sandboxed headless ComfyUI
|
||
*sidecar process* — GPL-safe by process isolation — but that is a deliberate, deferred complexity).
|
||
- **Motion C — productize (Dreamverse Studio).** Per the thesis that workflows matter less as models improve —
|
||
which ComfyUI itself corroborates by shipping blueprints and App Mode to hide its own graphs — the destination is
|
||
not a node editor clone but *workflows as products*: the top community workflows as one-click, real-time
|
||
primitives inside Dreamverse (5 s 480p in ~2 s end-to-end today; consumer-GPU NVFP4 builds targeting
|
||
5090/4090-class cards), with cloud execution for whatever the local GPU can't hold.
|
||
|
||
**What this genuinely adds to the architecture** (and it is bounded — no plane is redesigned):
|
||
|
||
1. **The workflow compiler** — a new request-plane frontend, peer to the OpenAI server: parse, validate against the
|
||
supported vocabulary, emit graphs. The graph plane (§6.2.4) was built for exactly this shape.
|
||
2. **Heterogeneous checkpoint serving** — the one real new subsystem. Workflow users bring arbitrary checkpoints,
|
||
stacked LoRAs, ControlNets; the engine has so far assumed one resident model family per pool. Required: a
|
||
**weight/adapter cache** tier in the CacheManager (disk→CPU→GPU, LRU, content-hash keyed `ModelSpec` resolution),
|
||
fast patch/unpatch with ComfyUI-grade LoRA-stacking semantics, and **model-affinity scheduling** (route requests
|
||
to pools where the checkpoint is resident — the diffusion-serving analog of sglang's prefix-affinity routing).
|
||
The LoRA story, honestly sized: **affinity scheduling + patch/unpatch is primary** — group by (checkpoint,
|
||
adapter-set, strengths), with §6.3.2's weight-transition scheduling pricing the swaps. Punica-style batched
|
||
multi-LoRA is a *deferred optimization with a real semantic gap*: vLLM's `LoRARequest` is one integer id with
|
||
scaling baked into `lora_b` at registration, while ComfyUI traffic is N stacked LoRAs with continuous user-set
|
||
`strength_model`/`strength_clip` tweaked per generation — pushing that through Punica means registering every
|
||
(ordered-set × strengths) tuple as a synthetic adapter: near-zero cache hits across strength tweaks,
|
||
registration churn in the stacked weight slots, and concatenated ranks colliding with `max_lora_rank`. Batched
|
||
multi-LoRA earns a place only where adapter sets are small and strengths fixed — named fast-mode presets — not
|
||
as the general workflow-cloud mechanism.
|
||
3. **Equivalence modes as product trust** — the consistency machinery (§8.5) reused commercially, with the claims
|
||
scoped precisely. *Exact mode*: the user's base model and sampler math, deterministic seeds, reproducibility —
|
||
**within our runtime**; bit-parity with ComfyUI's own execution is never promised (its model patching,
|
||
sampler/noise details, and dtype/offload behavior differ). *Fast mode*: distilled weights, NVFP4/QAT, cache-dit
|
||
— dramatically faster, *labeled*, with equivalence reports that are **quality-metric comparisons against a
|
||
ComfyUI reference render** (SSIM-class + the §12 eval-system gap, which gates this product claim), not parity
|
||
assertions. This is the moat restated: ComfyUI Cloud and GPU rental can only run stock workflows on
|
||
rented silicon; substituting a faster model **credibly** requires owning the recipes (§8.3) — which is the one
|
||
thing an orchestrator structurally cannot do.
|
||
|
||
Risks, named: the compatibility treadmill (mitigation: tiered vocabulary + coverage reports, never chase the
|
||
custom-node long tail), output-mismatch expectations (mitigation: exact/fast labeling), GPL hygiene (clean-room
|
||
compiler; sidecar only ever as a separate process; the in-tree custom-node suite carries ComfyUI-side licensing),
|
||
and platform response (ComfyUI controls the registry and frontend; our defense is the recipe flywheel it cannot
|
||
replicate, §9.2). Decision recorded in §11.13.
|
||
|
||
---
|
||
|
||
## 10. Migration plan
|
||
|
||
Principles: every phase ships green (SSIM + parity suites); every phase deletes or freezes the code it replaces
|
||
(no third stack); models migrate family-by-family behind the legacy adapter. Three additions this plan makes
|
||
because the repo's own history demands them:
|
||
|
||
- **Enforcement, not intent.** The last freeze failed empirically: since `train/` landed (2026-03-09, #1159),
|
||
**19 commits modified the "frozen" `training/` stack** — including a brand-new `cosmos2_5_training_pipeline.py`
|
||
added *nine days after* `training/AGENTS.md` forbade new models there (#1227), plus world-model training
|
||
(#1179) and LongCat finetuning (#1244). This plan's freezes are therefore mechanical: a **CI path gate**
|
||
rejects new files under `fastvideo/training/` and new `DenoisingStage` subclasses once Phase 1 lands;
|
||
**CODEOWNERS** on legacy paths gives the freeze a human veto; every phase has a **named owner**.
|
||
- **The inflow rule.** New families land at ~1–2/month (Flux2-Klein and Lucy Edit in a single day; MatrixGame3,
|
||
MagiHuman, Stable Audio, Gen3C this spring). Untreated, the migration tail grows faster than phases retire it.
|
||
Rule: **from Phase-1 completion, new model ports land on the new abstractions** — loop inversion + policies +
|
||
the custom-step hatch suffice for standard diffusion families; until then, new ports land legacy *and* join the
|
||
migration checklist with the porting team owning the later migration. Exceptions are an explicit owner decision.
|
||
- **Sizing and dates.** Rough per-phase engineer-month estimates, validated at each kickoff: P−1: 2–3 · P0: 2–3 ·
|
||
P1: 6–9 · P2: 8–12 · P3: 8–12 · P4: 10–15 · P5: 4–6. Owners are assigned at the design review, and Phase 5 gets
|
||
a **calendar target** there — decision deadlines phrased as "by Phase 2" bind to that calendar, not to phase
|
||
labels.
|
||
|
||
- **Phase −1 — Land the baselines (prerequisite, not optional).**
|
||
Merge the Cosmos3 five-branch chain (`feat/cosmos3-tier-a-port` → `feat/cosmos3-reasoning`) to main, bringing
|
||
its 150-test parity suite into CI: the design's forcing function currently exists only as unmerged branches
|
||
(`pipelines/basic/cosmos3/` on main holds a stray `__pycache__`), and no substrate refactor should proceed
|
||
against an unmerged baseline — the alternative is months of side-branch rebasing across
|
||
ForwardBatch→RequestState and loop inversion, unowned and unsized. Seed SSIM references for the ~7 uncovered
|
||
families (Cosmos 2/2.5, Hunyuan, Hunyuan15(+SR), HYWorld, MagiHuman, Waypoint, MatrixGame-v1) so G5's
|
||
instrument has no vacuous passes. *Gate: cosmos3 suite green in CI on main; SSIM coverage = all shipped
|
||
families.*
|
||
- **Phase 0 — Typed I/O (low risk, immediate value).**
|
||
`OmniRequest`/`OmniOutput`/`TaskType` + artifact outputs; `VideoGenerator` shim; ForwardBatch adapter view. The
|
||
in-flight `fastvideo/api/` schema converges here instead of landing as another parallel surface —
|
||
`GenerationRequest` evolves into `OmniRequest`, `GeneratorConfig` into `DeployConfig`, and `compat.py` is
|
||
**frozen** — no new entries, monotonic shrink per phase, zero lines at Phase 5 (§6.6). Cosmos3 audio leaves
|
||
`batch.extra`. *Gate: all SSIM suites unchanged.*
|
||
- **Phase 1 — Loop inversion + policies + extension core.**
|
||
`DenoiseLoop`/`ARDecodeLoop` + policy registry; migrate Wan family (incl. causal) and Flux2 first — they jointly
|
||
exercise CFG variants, expert routing, chunk-KV, and image path; `DenoisingStage` subclasses shrink to specs +
|
||
policies. The extension system core lands here because the loop is being rebuilt anyway: observer bus,
|
||
Step/Block interceptors, the `run_blocks` convention, and the ParityAligner — with the **cache-dit adapter**
|
||
(DBCache + TaylorSeer behind `BlockInterceptor`, algorithm-parity with sglang's integration) as the first
|
||
interceptor; the dormant `enable_teacache` flag is retired in favor of plugin config, and
|
||
`fastvideo/forward_context.py` (the vLLM-inherited global, §6.3.3) retires in favor of `StageContext` +
|
||
`AttnMetadataProvider` + state extensions. `train/` distillation
|
||
methods begin adopting the shared step functions, and the landed DiffusionNFT (#1450) migrates off its vendored
|
||
`rl/common/sampling.py` `DiffusionSampler` onto them (§8.1) — a migration of shipped code with users, so it gets
|
||
its own parity evidence: identical seeded rollout latents before/after, plus reward-metric and grad-norm
|
||
regression neutrality. **CacheManager v0 ships here**: per-request chunk-KV slabs behind the
|
||
`KVHandle` seam — the Phase-4 pooled slab allocator is a swap behind the same handle (stated staging, not a
|
||
silent throwaway implementation).
|
||
*Gate: per-model SSIM + a step-level parity harness, built on ParityAligner (old loop vs new loop,
|
||
bit-identical); on the training side, the landed per-method grad-norm regression refs (#1396) extended to cover
|
||
the migrated methods, RL included.*
|
||
- **Phase 2 — Engine + StepScheduler.**
|
||
Async engine, queue, concurrency, image batching, streaming events; OpenAI server parity with `multimodal_gen`
|
||
(videos/images endpoints — chat arrives with omni in Phase 4, and **AR continuous batching is deliberately
|
||
deferred to Phase 4 per N5**: the pulling workload is the Cosmos3 reasoner; Phase 2 carries only the
|
||
`ARDecodeLoop` contract from Phase 1). **LTX-2's loops migrate here** as a linear graph (the degenerate case;
|
||
full fan-out waits for Phase 3) — required because Dreamverse runs LTX-2 and the dogfood path needs its loop
|
||
boundaries open. Rollout-service surface lands here too: `update_weights` RPC with the **colocated** transports
|
||
and hand-checked per-family adapters (§8.6 — the general cross-mesh `WeightSyncPlan` follows the spec in
|
||
Phase 3), component-granular sleep/wake, trajectory `weight_version` tagging, and the `RolloutClient` for RL
|
||
engine-client mode (§8.7). The **Dynamo-grade worker surface** ships here (§6.3.5), scoped to stock Dynamo —
|
||
registration wrapping `AsyncEngine` (retiring the locked `examples/diffusers/worker.py` pattern), health/drain,
|
||
cost metrics; streaming/cost integration beyond stock follows asks A2/A3 *or their fallbacks*, and the gate does
|
||
not depend on upstream landing. Offline path still bypasses the queue. **Dreamverse is the dogfooding customer
|
||
at Phase-2 scope**: single-session parity — `gpu_pool.py`/queue/warmup/stream-relay retire into engine-client
|
||
calls — plus a recorded duty-cycle concurrency study that sets the Phase-3 multi-session gate (§9.1). *Gate:
|
||
serving load tests; batch-of-1 video latency regression ≤ 2% (gated on the Phase-1 overhead measurement); an RL
|
||
smoke test runs the landed DiffusionNFT (#1450) in engine-client mode with C1-pinned rollouts — seeded
|
||
final-latent parity against its in-process path on Wan, per-step activation drift measured by ParityAligner and
|
||
within the Phase-1-calibrated ε (per-step *log-prob* drift becomes the gated metric when the first GRPO-class
|
||
method lands); Dreamverse runs single-session on the engine at parity; the engine deploys under
|
||
stock Dynamo as an aggregated `ModelType.Videos` worker and passes health/routing smoke tests.*
|
||
- **Phase 3 — Graphs + placement + workflow frontend.**
|
||
PipelineSpec graphs, fan-out models (LTX-2, Hunyuan15+SR), role pools + connectors (port `multimodal_gen`
|
||
disagg), deploy YAML, and the declarative stacked-parallelism spec with pre-flight validation and fake-pool CPU
|
||
tests (§6.3.4). The ComfyUI workflow compiler MVP lands here (it is a client of the graph plane): the tier-1
|
||
(~20-node) vocabulary with capability negotiation, plus the weight/adapter fleet cache and model-affinity
|
||
scheduling (§9.4). Dynamo integration deepens accordingly: affinity events and NIXL-aligned disagg bootstrap
|
||
(§6.3.5; asks A1/A4 in §6.3.6), and the general cross-mesh `WeightSyncPlan` lands with the spec (§8.6). *Gate:
|
||
LTX-2 A/V end-to-end (full fan-out graph); disagg throughput benchmark; topology test suite green on CPU; the
|
||
top community t2i+LoRA / i2v / upscale workflows compile and run with equivalence reports (quality-metric vs a
|
||
ComfyUI reference render, §9.4); **Dreamverse multi-session** — ≥2 concurrent sessions per GPU on the recorded
|
||
duty-cycle trace with p95 segment latency within the SLO set by the Phase-2 study (§9.1).* The study's published
|
||
load profile and targets double as the **§11.6 falsifier**: if step-level scheduling does not beat a
|
||
request-level baseline on those numbers, the simpler scheduler wins and the StepScheduler is descoped — the
|
||
complexity must earn itself on measurements, not on this document.
|
||
- **Phase 4 — Omni/MoT native + RL consistency.**
|
||
`PackedSeq` + multi-pathway layers; CacheManager (paged text KV + chunk KV + feature caches); Cosmos3 re-port onto
|
||
the new abstractions (reasoner gets KV cache; t2vs joint denoise; rollout); session-scoped world-model serving.
|
||
**AR continuous batching and paged-KV scheduling arrive here with their pulling workload** (the Cosmos3 reasoner
|
||
and omni chat — N5's condition met), as does `/v1/chat/completions`.
|
||
RL hardening lands alongside: the consistency ladder enforced end-to-end (C1 default, C2 bitwise mode for goldens
|
||
and MoE routing parity), Behavior Record capture/replay (an opt-in instrument, §8.5), TIS/MIS objectives as C0
|
||
fallback, and the omni-RL deliverable scoped per §8.7: **hybrid trajectory capture demonstrated + the objective
|
||
design study** (open question 16) — not a pilot that presumes the objective exists.
|
||
*Gate: Cosmos3 150-test parity suite passes on the new runtime; reasoner pool efficiency — tokens/s/GPU at
|
||
target concurrent denoise throughput (§6.3.1) — alongside the ≥10×-vs-re-prefill sanity floor; C1
|
||
rollout↔training drift ≈ 0 on a Wan RL run, with the drift dashboard live — per-step log-prob drift if a
|
||
GRPO-class method has landed by then, seeded final-latent parity for the likelihood-free NFT otherwise.*
|
||
- **Phase 5 — Deprecation, with its real precondition stated.**
|
||
Phase 5 is not just "remaining families migrated": the legacy `training/` stack is a *live consumer* of exactly
|
||
what Phase 5 deletes (`training_pipeline.py:39` imports `ComposedPipelineBase`/`ForwardBatch`/`LoRAPipeline`;
|
||
its validation instantiates real legacy pipelines running the legacy `DenoisingStage`;
|
||
`distillation_pipeline.py:31` likewise). So **legacy `training/` retirement is a Phase-5 precondition**: its
|
||
remaining users migrate to `train/` per the checklist, *then* `ComposedPipelineBase`, the legacy
|
||
`DenoisingStage`, `forward_context.py`, `compat.py`, and `RayDistributedExecutor` are deleted together. This is
|
||
where 4 loop copies → 1 actually completes — not before, and the document claims it nowhere earlier. Docs +
|
||
`.agents/` maps updated. *Gate: the deletion diff itself — a tree with one loop implementation.*
|
||
|
||
---
|
||
|
||
## 11. Trade-offs and alternatives considered
|
||
|
||
### 11.1 Adopt sglang `multimodal_gen` upstream instead of building here
|
||
|
||
FastVideo's runtime already has a serving-hardened fork living in sglang. Option: make sglang the runtime and
|
||
FastVideo a model/stage library.
|
||
|
||
- **Pros:** serving layer (server, scheduler process, disagg, warmup, RL weight updates) exists and is maintained;
|
||
one community runtime; router/gateway integration free.
|
||
- **Cons:** `multimodal_gen` inherited our P1–P3 problems and is diffusion-only — no AR loop, no KV cache, no
|
||
step-level scheduling; the hard 40% of this design (loop inversion, hybrid MoT, CacheManager) doesn't exist there
|
||
either, so we'd build it anyway — in a repo we don't govern, on sglang's release cadence, while FastVideo is the
|
||
research vehicle (Cosmos3, VSA, new ports land here first).
|
||
- **Decision:** build natively, but keep **API and concept parity** (request/queue/disagg shapes, server endpoints)
|
||
so the work upstreams cleanly later. The runtime layering (§6.0) is deliberately compatible with theirs.
|
||
*Open strategic question for the team: do we plan a re-convergence (upstream Phase 2's engine into
|
||
`multimodal_gen`), or accept a long-lived friendly fork? This needs an explicit owner-level decision by Phase 2.*
|
||
|
||
### 11.2 Multi-engine DAG composition only (vllm-omni / sglang-omni model)
|
||
|
||
- **Pros:** fastest route to separable omni (thinker/talker/vocoder); engine reuse; process isolation.
|
||
- **Cons:** expresses single-weights hybrids (Cosmos3 MoT, BAGEL lineage) only as **one opaque monolithic stage**
|
||
— vllm-omni's `bagel`/`lance` *are* real shared-weight MoT, but request-scheduled (`max_num_running_reqs=1`),
|
||
batch-incapable, with the AR+denoise interleaving invisible to the scheduler — so the runtime can't
|
||
schedule/batch/preempt the loops inside; cross-stage KV is a copy, not a shared live cache; weight duplication
|
||
for backbones split across *separate* stages; connector latency on the token path.
|
||
- **Decision:** adopt the *composition concepts* (declarative spec, connectors, per-stage schedulers, async_chunk
|
||
streaming) **inside** one runtime, and offer external-engine nodes as an opt-in — rather than making cross-engine
|
||
composition the foundation.
|
||
|
||
### 11.3 Status quo plus per-model monolithic stages
|
||
|
||
The Cosmos3 port proved a skilled engineer can ship an omni model inside today's abstractions in days. Why change?
|
||
|
||
- **Pros:** zero migration risk; maximum per-model freedom; no runtime tax.
|
||
- **Cons:** it buys exactly one request at a time with no KV cache — reasoning KV, batching, streaming, and prefix
|
||
reuse remain unbuilt (this document's assessment; the port's `PORT_STATUS.md` records 150/150 parity and makes no
|
||
production-gap claims of its own); every future omni/AR model re-pays the monolith cost; serving never
|
||
materializes.
|
||
- **Decision:** rejected as an end state; embraced as a migration mechanism (the legacy adapter *is* this option,
|
||
scoped per-family and temporary).
|
||
|
||
### 11.4 Big-bang rewrite vs incremental
|
||
|
||
A clean-room runtime would be simpler to design but violates G5 and repeats the two-training-stacks situation at
|
||
larger scale. **Decision:** incremental with phase gates and an explicit Phase-5 deletion milestone. The known risk
|
||
of "incremental" — the old path never dies — is mitigated by making each phase delete/freeze something.
|
||
|
||
### 11.5 Typed dataflow vs blackboard state
|
||
|
||
Full typed ports everywhere (every tensor an edge) is maximal safety but punishing verbosity for 20+ families;
|
||
the pure blackboard is what produced P3. **Decision:** typed artifacts on graph edges + typed per-model state
|
||
extensions inside nodes. Cost: two places state can live; mitigated by a lint rule — anything crossing a node
|
||
boundary must be an edge artifact.
|
||
|
||
### 11.6 Scheduler granularity
|
||
|
||
Request-level scheduling (multimodal_gen) is far simpler; step-level adds queue/dispatch complexity and an
|
||
SPMD-consistency obligation (decisions made on rank 0, broadcast — same discipline sglang and multimodal_gen
|
||
already use). **Decision:** step-level only where loops exist; one-shot nodes schedule at request granularity.
|
||
Without step-level scheduling there is no AR batching, no interleaving, no preemption, no MoT mode multiplexing —
|
||
it is the price of G1/G4. **The falsifier:** the Phase-2 duty-cycle study publishes a load profile and targets;
|
||
if at the Phase-3 gate step-level scheduling does not beat a request-level baseline on those numbers (≥2 concurrent
|
||
sessions per GPU, p95 segment latency within the study's SLO), the StepScheduler is descoped to request granularity
|
||
and the loop contract keeps only its streaming/preemption seams — the decision is measured, not assumed.
|
||
|
||
### 11.7 Cross-request batching for video diffusion
|
||
|
||
Batched video denoise (shape-bucketed or Cosmos-style token-budget packing) is real throughput but significant
|
||
complexity (ragged attention, per-sample steps, CFG alignment). **Decision:** defer; v1 video concurrency =
|
||
interleaving + parallelism. Revisit with data after Phase 2 (the StepScheduler interface already permits
|
||
multi-request denoise batches, so this is additive).
|
||
|
||
### 11.8 Build the KV manager vs embed sglang srt
|
||
|
||
srt's pools are excellent but token-centric and entangled with its scheduler/attention stack; our second cache class
|
||
(chunked causal-video KV with sink/local windows, session lifetimes) has no srt analog. **Decision:** native
|
||
implementation borrowing srt's allocator design; no radix prefix tree in v1 (content-hash embedding cache covers the
|
||
common reuse), reconsider radix when omni chat traffic justifies it.
|
||
|
||
### 11.9 Fleet layer: Dynamo as first-class partner vs alternatives
|
||
|
||
- **Build our own router/autoscaler:** rejected outright — it is a separate engineering discipline (Dynamo is a
|
||
Rust codebase with etcd/NATS discovery, radix-indexed KV routing, an SLA planner, KVBM tiering, ModelExpress
|
||
cold-start streaming), and every line of it we wrote would be a line not spent on the §8 substrate.
|
||
- **sglang-router only:** lighter, and remains a fine drop-in for simple single-model deployments — but it has no
|
||
planner, no disagg coordination, no multi-tier cache story.
|
||
- **Kubernetes-only (HPA + service mesh):** no request-content awareness — no affinity routing, no SLA planner
|
||
with per-request cost models, no KV/disagg coordination.
|
||
- **Decision: Dynamo first-class** (contract in §6.3.5). FastVideo already runs as a Dynamo `ModelType.Videos`
|
||
backend (`dynamo/examples/diffusers/worker.py`), and Dynamo already fronts sglang-diffusion and vllm-omni — the
|
||
fleet layer has standardized around exactly our ecosystem. Risks, honestly: Dynamo is NVIDIA-governed and its
|
||
router is token-prefix-centric today (our model/session-affinity routing is a collaboration item, open question
|
||
15). Because the relationship with the Dynamo team is direct, gaps are handled as upstream asks with stated
|
||
fallbacks (§6.3.6) rather than permanent workarounds. Mitigation regardless: the contract is deliberately thin
|
||
and OpenAI-shaped, so the engine stays deployable bare, behind sglang-router, or under any future orchestrator
|
||
without rework.
|
||
|
||
### 11.10 Explicit extension points vs monkeypatching (the cache-dit question)
|
||
|
||
cache-dit earns one-line adoption on diffusers (`cache_dit.enable_cache(pipe)`) by patching pipeline/transformer
|
||
internals at runtime. We could allow plugins the same freedom.
|
||
|
||
- **Pros:** zero model-side convention; instant compatibility with third-party techniques as published.
|
||
- **Cons:** patches break under `torch.compile`/cudagraph capture and TP-sharded modules; state lands in
|
||
module-level globals (unsafe under concurrent requests — disqualifying for a serving engine); no pre-flight
|
||
conflict detection between plugins; silent breakage whenever model internals shift.
|
||
- **Decision:** explicit, versioned hook points (§6.4) plus the `run_blocks` convention for in-tree models.
|
||
Module-level *forward hooks* remain available — but only to read-only observers, where the patching risks don't
|
||
apply. For cache-dit specifically: **depend on the library, not its patching path** — drive cache-dit's
|
||
configs/calibrators/skip math from behind our `BlockInterceptor`. sglang's integration is the precedent and the
|
||
cautionary tale at once: it adopts cache-dit via `BlockAdapter`, then has to monkeypatch
|
||
`CachedContextManager.similarity` for sequence-parallel tensors and hand-call `refresh_context_*` per request —
|
||
exactly the two things (group-aware comparison, request-scoped lifecycle) our interceptor contract provides
|
||
structurally (§6.4).
|
||
|
||
### 11.11 Adopt `composable_parallel` (RFC #4084) as a dependency vs native implementation
|
||
|
||
- **Pros of adopting:** shared maintenance; framework-agnostic by design (the RFC names SGLang as a second
|
||
backend); the spec/validation/fake-backend machinery arrives prebuilt.
|
||
- **Cons:** the RFC is deliberately Option-A *descriptive* — runtime routing stays with the host engine, and the
|
||
schemes we need most (per-node placement meshes, MoT pathway-sharded TP, chunk-KV-aware CP) are exactly the parts
|
||
deferred to follow-up RFCs; it is an open proposal, not shipped code; we already own a working
|
||
`parallel_state.py` and would be coupling our roadmap to a younger external contract.
|
||
- **Decision:** borrow the design — stacked axis specs, per-axis ownership markers, pre-flight validation,
|
||
CPU-testable composition — and implement natively, keeping our spec schema close enough to theirs that
|
||
converging on `composable_parallel` later stays cheap if it matures.
|
||
|
||
### 11.12 RL: single runtime vs hybrid-engine (verl-omni) vs dual-stack-with-corrections (miles)
|
||
|
||
- **Hybrid-engine (verl-omni):** reuse best-of-breed engines per role. For FastVideo this would mean
|
||
re-implementing *our own models* inside vLLM-Omni to roll them out — verl-omni literally carries a Wan2.2 rollout
|
||
re-implementation — then importing the correction machinery to patch the numerics. Absurd given we own an engine.
|
||
- **Dual-stack with corrections (miles):** the right call at ≥70B MoE scale where Megatron is non-negotiable; its
|
||
mismatch toolkit is world-class — but the toolkit exists to dress a wound we do not have to open at 1–30B scale.
|
||
- **Decision:** single runtime, two execution profiles, explicit consistency ladder (§8.5). Crucially, we adopt
|
||
miles' corrections (TIS/MIS) and Behavior Record replay as **built-in C0 degradation modes** rather than
|
||
pretending mismatch cannot occur — profiles will legitimately differ (quantized rollout, cache-dit on), and if a
|
||
Megatron-class trainer ever becomes necessary, the same toolkit is the escape hatch (§8.4 boundary condition).
|
||
And the diffusion-specific clincher (§8.3): inference optimization here *is* post-training — an engine that
|
||
cannot run inside training loops is cut off from its own optimization frontier, so single-runtime is less
|
||
optional for us than it would be for an LLM stack.
|
||
|
||
### 11.13 ComfyUI: embed, compile, or compete
|
||
|
||
- **Embed only** (custom nodes inside ComfyUI — exists today in `comfyui/`): cheapest distribution; but FastVideo
|
||
stays trapped under their queue (no batching, no sessions, no multi-tenancy) and the performance story is capped
|
||
by their runtime.
|
||
- **Compete head-on** (rebuild the node editor + ecosystem): rejected — 116K stars of network effects and a
|
||
thousand-pack custom-node ecosystem are not assailable frontally, and the graph-editor UX is precisely the layer
|
||
the market (including ComfyUI itself, via App Mode and blueprints) is abstracting away.
|
||
- **Compile** (clean-room workflow compiler onto the engine + accelerated cloud): the leverage play — it converts
|
||
their ecosystem into our demand without forking their UX battle, and its differentiation (credibly substituting
|
||
faster recipes, with equivalence reports) is the one move an orchestrator cannot copy (§9.4).
|
||
- **Decision: all three, sequenced and bounded.** A = funnel (maintain, add a FastVideo-Cloud API node); B = core
|
||
bet (Phase 3 MVP over the tier-1 vocabulary + weight-fleet cache; never chase the custom-node long tail —
|
||
tiered coverage with explicit rejection, sidecar only if demand proves it); C = Dreamverse Studio as the
|
||
destination (workflows-as-products, real-time as the wedge) — an evolution of §9.1–9.3, not a separate product.
|
||
Scope discipline: B's vocabulary is a *product* decision reviewed per tier, not an open-ended compatibility
|
||
promise.
|
||
|
||
---
|
||
|
||
## 12. Risks, concerns, and open questions
|
||
|
||
1. **Complexity creep.** Three planes, policies, graphs, the cache manager (per-class KV pools + feature + weight-fleet),
|
||
an extension system, a workflow compiler, and an RL layer — and the scope has *grown* with every
|
||
review pass of this document, which is itself the warning. *Mitigations:* phase gates with deletion milestones;
|
||
the additions are sequenced, not front-loaded (fleet cache + compiler in Phase 3, RL hardening in Phase 4, C2
|
||
kernels only where they pay); the offline path stays a ~hundred-line trace (request → linear graph → loops) that
|
||
a new contributor can read top-to-bottom; policies are plain objects, not a plugin framework.
|
||
2. **Hot-loop overhead / compile friendliness.** Per-step dispatch, policy indirection, observer/interceptor
|
||
chains, and emit handling must not cost milliseconds; cudagraph/compile capture needs stable step bodies.
|
||
*Mitigations:* policy *bindings* resolved at build time (policy *state* is per-request in `LoopState`, §6.2.3);
|
||
hook chains assembled at build and literally
|
||
absent when unused (§6.4), with `needs_eager`/`graph_safe` declarations scoping any capture loss; `step()`
|
||
bodies are the same tensor programs as today's loop bodies; capture per (node, shape-bucket, branch-profile),
|
||
with breakable CUDA graphs as the optimization tier (experimental upstream — the eager path is the correctness
|
||
guarantee, §6.3.3); Phase-2 gate includes a ≤2% batch-of-1 latency budget, itself gated on the Phase-1
|
||
loop-inversion overhead measurement.
|
||
3. **Always-on loop inversion has only narrow precedent — not the hybrid slice.** The one prior art is
|
||
vllm-omni's `SupportsStepExecution` (runtime-owned diffusion iteration at step granularity, but opt-in,
|
||
Qwen-Image-only, off in every deploy — §6.2.2); no surveyed system makes it the always-on default (vLLM/sglang
|
||
own AR token loops; diffusers' loop blocks own their own iteration — §6.2.3), and the hybrid AR+denoise
|
||
multiplexing on shared weights is the deepest end of a bet that is novel from its
|
||
first step. *Mitigations:* the Phase-1 parity gates are the load-bearing control (old loop vs new loop,
|
||
bit-identical, per family — this is where the novelty risk is actually retired); the Phase-1 overhead
|
||
measurement gates the latency story; the Cosmos3 port's 150-test bit-exact suite is the hybrid-slice prototype
|
||
harness, exercised early in Phase 4 before generalizing.
|
||
4. **SPMD-consistent scheduling.** All ranks of a pool must agree on every step decision; divergence deadlocks NCCL.
|
||
*Mitigations:* rank-0-decides + broadcast (existing pattern in multiproc executor and multimodal_gen); decision
|
||
inputs restricted to broadcast state; a debug mode asserts cross-rank decision hashes. The same channel carries
|
||
the **abort broadcast** (§6.3.1) — scheduling and failure isolation share one consistency mechanism by design.
|
||
5. **Two stacks during migration** (the very smell we're fixing). *Mitigations:* legacy adapter means zero
|
||
dual-maintained models — a family is either unmigrated (old code, untouched) or migrated (old code deleted);
|
||
per-family checklist tracked in `.agents/`; Phase 5 is a scheduled deletion, not an aspiration. The substrate
|
||
split (§8) raises the stakes: once `train/` methods consume the shared loops, a loop change breaks both worlds —
|
||
so training rollouts join the parity gates (trajectory-identical before/after) and substrate PRs run training
|
||
smoke tests, not just SSIM. The training-side gate layer already exists in CI to extend, not invent: #1396's
|
||
per-method, device-keyed layer-0 grad-norm regression refs (`fastvideo/tests/train/methods/`, Modal-wired) —
|
||
coarse 10%-rtol tripwires today, the natural carrier for the per-step parity assertions as methods migrate.
|
||
6. **CI/GPU cost.** Step-parity harnesses, serving load tests, SSIM suites, Cosmos3 parity on B200s. *Mitigations:*
|
||
step-level parity tests run on CPU/tiny configs (the Cosmos3 oracle methodology generalizes); GPU suites stay
|
||
per-family and gated to touched families — except substrate-wide PRs in Phases 0–2, where "touched" means all
|
||
of them and the full matrix runs as a budgeted cost (G5).
|
||
7. **Relationship with sglang `multimodal_gen`** (§11.1) — needs an explicit decision by Phase 2: upstream,
|
||
friendly fork, or shared core package. Drift here is a strategic cost regardless of architecture.
|
||
8. **Single runtime does not equal automatic numerical parity.** Two execution profiles remain
|
||
(grad/checkpointed/FSDP-gathered training forward vs no-grad/graphed/sharded/cached rollout forward), and the
|
||
failure mode is precisely characterized: kernels are deterministic per fixed batch but **not batch-invariant**
|
||
(Thinking Machines' analysis — reduction strategies change with batch/shape/sharding), so profile differences
|
||
alone break bit-equality; miles needed FA3 + DeepGEMM batch-invariant kernels to reach bitwise parity even
|
||
*within* matched stacks. *Mitigations:* the consistency ladder (§8.5) makes the level explicit per run instead
|
||
of implied; C1 is gated by ParityAligner drift CI (log-prob diff for likelihood-based methods, seeded
|
||
final-latent/activation parity for NFT-class); TIS/MIS and the Behavior Record ship as built-in C0
|
||
degradation; a batch-invariant `deterministic` kernel mode (op-swap, §8.5 C2) is scoped for goldens rather than
|
||
promised everywhere; drift metrics (`logprob_diff` and IS clip fraction where the objective has them,
|
||
prediction-space deviations like #1450's `old_deviate`/`old_kl_div` otherwise) are always-on training
|
||
telemetry, not post-mortem tools.
|
||
9. **Open: session semantics.** KV/cache lifetimes beyond a single request — interactive world models (game
|
||
sessions), multi-turn omni chat, and now Dreamverse scene continuation (chained-segment frame/audio conditioning,
|
||
§9.1), which makes this a production requirement with a concrete customer rather than a research nicety —
|
||
owner: Phase 4 design review, with Dreamverse's protocol as the reference workload.
|
||
10. **Open: speculative decoding for AR stages** (sglang-omni lacks it; vllm-omni inherits vLLM's). The
|
||
`ARDecodeLoop.step` contract should permit multi-token bundles from day one even if drafting lands later —
|
||
miles already ships speculative rollout with online-SFT'd draft models, evidence this matters for RL
|
||
throughput too. The contract shape to copy is vLLM v1's: draft tokens ride the scheduler output and are
|
||
rejection-sampled in the same step (§6.3.1).
|
||
11. **Open: quantization/FP8/FP4 interaction** with multi-pathway layers and the Attn-QAT work (linear/MLP FP4 paths
|
||
recently landed) — verify policy/layer factoring doesn't block per-pathway quantization configs. The
|
||
ParityAligner observer (§6.4) should become the standard harness for validating precision changes, and miles'
|
||
unified-FP8 lesson applies directly: rollout and trainer share one `PrecisionPolicy`/quant recipe —
|
||
"train BF16, roll out FP8" is only legal at consistency level C0 (§8.5).
|
||
12. **Open: hook-point surface stability.** Which hook points are public/versioned vs internal, and whether the
|
||
cache-dit adapter lives in-tree or upstream (vipshop/cache-dit already ships adapters for other frameworks) —
|
||
decide when the first external plugin lands (Phase 1–2).
|
||
13. **Open (narrowed): heavyweight reward-model packaging.** The original concern — the #1438 draft vendored the
|
||
full HPSv3/VideoAlign repos, trainers and deepspeed configs included — was resolved before landing: #1450 ships
|
||
only thin PickScore/CLIP scorers loading weights from HF, and the `add-reward-model` skill now governs
|
||
additions. What remains open is the heavyweight case: when HPSv3/VideoAlign/VLM-judge-class rewards land, do
|
||
they arrive as pip dependencies, pruned runtime-only ports, or engine components scored off-rank (§8.7)? The
|
||
landed pattern (GPU-resident scorer on every training rank) does not survive reward models with their own
|
||
multi-GB footprints.
|
||
14. **Open: workflow-compatibility tiering.** Which ComfyUI node vocabulary each tier covers, what the coverage
|
||
SLA is, whether/when the headless-ComfyUI sidecar for unsupported subgraphs is worth its operational cost, and
|
||
how exact-vs-fast equivalence is reported to users (§9.4) — decide with Phase 3 data from the top community
|
||
workflows.
|
||
15. **Open: which Dynamo asks land upstream, and when.** §6.3.6 enumerates seven (A1 affinity key spaces, A2 cost
|
||
interface, A3 media streaming, A4 role-graph disagg, A5 RL weight plane, A6 KVBM generalization, A7 sessions),
|
||
each with a fallback. Sequence them with the Dynamo team against our phases (A2/A3 → Phase 2, A1/A4 → Phase 3,
|
||
A5–A7 → Phase 4); commit fallbacks into the plan only for asks that miss their phase. Ask-sequencing needs an
|
||
owner and decision dates at the design review — and the A3 fallback (direct-WebSocket bypass) must be treated
|
||
as temporary by policy, or it hardens into exactly the permanent workaround §11.9 claims the relationship
|
||
avoids.
|
||
16. **Open: the omni-RL objective** (§8.7). Three named problems before any hybrid pilot: per-segment
|
||
normalization across O(1–10)-nat token log-probs vs 10⁶–10⁷-dim Gaussian step densities; credit assignment
|
||
across the non-differentiable sampled-token boundary into shared trunk weights; reasoning-regression control
|
||
(reference-KL anchoring for the und pathway). Owner: the RL lead, before Phase 4 scopes its deliverable.
|
||
|
||
### Known gaps — acknowledged, deliberately not yet designed
|
||
|
||
Named here so their absence reads as a decision, not an oversight. Each needs an owner and a design note before
|
||
the phase it gates:
|
||
|
||
- **Generation-quality evaluation system** (gates Phase 2/3 product claims). SSIM gates and ParityAligner cover
|
||
regression and numerical parity; they do not measure *quality*. "Fast mode is visually equivalent" (§9.4), RL
|
||
reward validation (§8.7), and distillation-recipe comparisons all need a real eval harness — VBench-class
|
||
benchmarks, reward-model scoring, human-preference sampling — with tracked baselines per model family.
|
||
- **The flywheel's data plane** (gates §9.2 → §8 RL). Preference events from Dreamverse need schema, storage,
|
||
lineage, filtering, and a consent/licensing policy before any RL method may train on them. The design covers
|
||
trajectory capture, not the data engineering or governance around it.
|
||
- **Tenancy, quotas, metering, and input hardening for the cloud API** (gates the §9.4 business). The engine emits
|
||
per-request cost; who meters, bills, and rate-limits is undesigned. Workflow JSON is untrusted input — resource
|
||
ceilings (resolution/steps/duration), schema validation, and abuse controls are required at the boundary.
|
||
- **Artifact/model registry for (recipe, runtime) outputs.** Distilled/QAT checkpoints are products; their
|
||
versioning, distribution (HF/S3/ModelExpress), provenance back to recipes, and license posture as derivatives of
|
||
differently-licensed bases are unspecified — interacts with the weight-fleet cache (§9.4) and A5 (§6.3.6).
|
||
- **Long-job resilience beyond the pool.** In-pool failure isolation, cancellation, partial artifacts, and
|
||
preemption are now designed (§6.3.1); what remains undesigned is durability *across* engine restarts (where
|
||
serialized `LoopState` persists, retention policy) and cross-pool job migration.
|
||
- **Behavior Record at training scale.** Full MoE-routing capture for Cosmos3-class requests is GB-per-sample
|
||
(§8.5); if it is ever used beyond goldens/debugging, the streaming/storage/transport design for GB-scale
|
||
trajectories from disaggregated rollout fleets does not exist.
|
||
- **One observability schema.** Drift telemetry (§8.5), profiler output (§6.4), cost metrics (§6.3.5), and engine
|
||
stats are each named where they arise; Phase 2 should consolidate them into a single telemetry spec.
|
||
|
||
---
|
||
|
||
## Appendix A — Source material
|
||
|
||
- FastVideo main: `fastvideo/pipelines/` (`composed_pipeline_base.py`, `stages/base.py:29`,
|
||
`stages/denoising.py:237`, `pipeline_batch_info.py:238`), `fastvideo/worker/`, `fastvideo/entrypoints/`,
|
||
`fastvideo/models/dits/causal_wanvideo.py:80-143`.
|
||
- Cosmos3 port: the full-omni branch is
|
||
https://github.com/hao-ai-lab/FastVideo/tree/feat/cosmos3-reasoning — top of the stacked chain
|
||
(`feat/cosmos3-tier-a-port` → `-i2v` → `-audio` → `-action` → `-reasoning`), covering T2V/I2V/T2I, joint
|
||
video+stereo-audio (t2vs), **action** (domain-aware pathway, joint vision+action+sound denoise), and greedy
|
||
text reasoning — 150/150 parity tests vs the official framework. Local worktree:
|
||
`/home/william5lin/FastVideo_cosmos3_port`; status doc `tests/local_tests/cosmos3/PORT_STATUS.md`.
|
||
- Official Cosmos3: `cosmos-framework/cosmos_framework/model/vfm/` (`mot/unified_mot.py`,
|
||
`mot/cosmos3_vfm_network.py`, `omni_mot_model.py`, `data/vfm/sequence_packing.py`, `inference/inference.py`).
|
||
- sglang: `~/sglang/python/sglang/multimodal_gen/` (runtime, disaggregation, pipelines_core) and
|
||
`~/sglang/python/sglang/srt/` (managers, mem_cache, model_executor, disaggregation).
|
||
- vllm-omni: `~/vllm-omni/vllm_omni/` (`config/stage_config.py`, `engine/orchestrator.py`,
|
||
`distributed/omni_connectors/`, `diffusion/diffusion_engine.py`).
|
||
- sglang-omni: `~/sglang-omni/sglang_omni/` (`config/schema.py`, `pipeline/`, `relay/`, `scheduling/`).
|
||
- NVIDIA Dynamo: `dynamo/` (`examples/diffusers/worker.py` — the existing FastVideo video worker;
|
||
`lib/bindings/python/` (`register_llm`, `@dynamo_endpoint`, `DistributedRuntime`);
|
||
`lib/kv-router/src/protocols.rs` (KV/affinity event format); `components/src/dynamo/sglang/init_diffusion.py`
|
||
and `components/src/dynamo/vllm/omni/` (sibling-stack backends); `components/src/dynamo/planner/` (SLA
|
||
autoscaler); `lib/kvbm-*` (multi-tier KV offload); `docs/design-docs/disagg-serving.md`). Contract analysis in
|
||
§6.3.5.
|
||
- vLLM-Omni RFC #4084 "Composable Parallel Strategies": https://github.com/vllm-project/vllm-omni/issues/4084
|
||
(open proposal by @kushanam, 2026-06-03; evaluated in §5, §6.3.4, §11.11).
|
||
- cache-dit (training-free DiT cache acceleration: DBCache/FBCache/TaylorSeer): https://github.com/vipshop/cache-dit
|
||
(integration model evaluated in §11.10).
|
||
- Existing FastVideo hook seeds: `fastvideo/hooks/` (`hooks.py` — `ModuleHookManager`/`ForwardHook`;
|
||
`activation_trace.py` — env-gated parity tracing with `trace_step`; `layerwise_offload.py`), and the dormant
|
||
`enable_teacache` flag (`fastvideo/pipelines/pipeline_batch_info.py:182`).
|
||
- sglang's cache-dit integration: `~/sglang/python/sglang/multimodal_gen/runtime/cache/cache_dit_integration.py`
|
||
(`enable_cache_on_transformer`/`_dual_transformer`, `_patch_cache_dit_similarity`, `refresh_context_*`;
|
||
`cache_dit_config` in `runtime/server_args.py:116`).
|
||
- Training stacks: `fastvideo/train/` (`trainer.py`, `methods/base.py`,
|
||
`methods/distribution_matching/{dmd2,self_forcing}.py`, `methods/rl/` (see below), `models/base.py`,
|
||
`callbacks/validation.py`, `utils/training_config.py`), legacy `fastvideo/training/*_pipeline.py`, preprocessing
|
||
`fastvideo/pipelines/preprocess/` (incl. `preprocess_pipeline_ode_trajectory.py`).
|
||
- RL (landed): PR #1450 "[feat] Add Wan RL DiffusionNFT training" (commit `5854aec2c`, 2026-06-11):
|
||
https://github.com/hao-ai-lab/FastVideo/pull/1450 — `fastvideo/train/methods/rl/{__init__.py, diffusion_nft.py,
|
||
common/{__init__,prompt_sampling,sampling,validation}.py, rewards/{__init__ (the`build_multi_reward_scorer`
|
||
factory), frame_rewards, media}.py}`, trainer hooks
|
||
(`manages_optimization`/`managed_train_step`/`on_validation_begin`), `ModelBase.decode_latents`,
|
||
`examples/train/configs/rl/wan/diffusion_nft_pick_clip.yaml`, and
|
||
`.agents/skills/{add-rl-method,add-reward-model,rlhf-training-abstractions}/SKILL.md`. (Supersedes the #1438
|
||
"Adam/dnft" draft cited in earlier revisions of this doc.)
|
||
- Training-side regression gates (landed): PR #1396 (commit `633d39356`) — per-method, device-keyed layer-0
|
||
grad-norm regression refs: `fastvideo/tests/train/methods/{grad_norm_regression.py, grad_norm_refs.json}`,
|
||
Modal wiring in `fastvideo/tests/modal/pr_test.py::run_train_framework_tests`.
|
||
- Typed API in flight: `fastvideo/api/` (`schema.py`, `sampling_param.py`, `compat.py` (651 lines), `presets.py`,
|
||
`results.py`, `request_metadata.py`, per-model `matrixgame2.py`/`matrixgame3.py`).
|
||
- verl-omni: `~/verl-omni/verl_omni/` (`trainer/diffusion/{ray_diffusion_trainer,rollout_correction,
|
||
diffusion_algos}.py`, `pipelines/wan22_dance_grpo/vllm_omni_rollout_adapter.py`,
|
||
`pipelines/schedulers/flow_match_sde.py`, `workers/rollout/vllm_rollout/vllm_omni_async_server.py`,
|
||
`workers/engine/{fsdp,veomni}/`, `reward_loop/reward_manager/multi.py`).
|
||
- miles: `~/miles/` (`train.py`, `train_async.py`, `miles/backends/training_utils/loss_hub/{losses,corrections}.py`
|
||
(TIS/MIS), `miles/backends/megatron_utils/update_weight/` (IPC/broadcast/P2P transports),
|
||
`miles/backends/megatron_utils/replay_utils.py` (R3), `miles/utils/types.py` (`Sample` with
|
||
`rollout_routed_experts`, `weight_versions`), `miles/rollout/base_types.py` (rollout-fn contract),
|
||
`docs/en/advanced/{miles-router,fp8-low-precision}.md`).
|
||
- ComfyUI: `~/ComfyUI` (GPL-3.0; `execution.py` + `comfy_execution/caching.py` (graph execution, output caching),
|
||
`nodes.py` (64 core nodes), `comfy_extras/nodes_{wan,cosmos,hunyuan,mochi,video_model}.py` (in-core video),
|
||
`comfy/{model_management,model_patcher,lora,multigpu}.py`, `comfy_api_nodes/` (39 paid API providers),
|
||
`blueprints/` (80 in-tree workflow templates; more via the registry), `QUANTIZATION.md`); FastVideo's existing custom-node suite: `comfyui/`
|
||
(this repo).
|
||
- Dreamverse: `apps/dreamverse/` (`arch.md`, `design.md`, `AGENTS.md`; runtime `dreamverse/{main,gpu_pool,
|
||
prompt_enhancer,session_init_image}.py`; `web/` Next.js frontend; `serve_configs/streaming_demo.yaml`); blogs:
|
||
https://haoailab.com/blogs/dreamverse/ (vibe directing) and https://haoailab.com/blogs/fastvideo-dreamverse-release/
|
||
(release; NVFP4/FA4/compile on B200, fMP4-over-WebSocket, frame/audio segment conditioning).
|
||
- xDiT: `~/xDiT/xfuser/` (`config/config.py` — ParallelConfig axes; `core/distributed/{parallel_state,
|
||
group_coordinator,runtime_state}.py`; `docs/methods/{pipefusion,usp,hybrid}.md`; DistVAE adapters in
|
||
`model_executor/models/runner_models/wan.py`).
|
||
- cosmos-rl: `~/cosmos-rl/cosmos_rl/` (`dispatcher/{controller,command}.py`, `policy/trainer/llm_trainer/
|
||
grpo_trainer.py` (AIPO/dual-clip/off-policy masking), `rollout/{rollout_base,worker/weight_sync}.py`
|
||
(RolloutBase registry; buffer-model async sync), `rollout/wfm_rollout/`, `utils/{distributed,parallelism_map}.py`).
|
||
- diffusers Modular: `~/diffusers/src/diffusers/modular_pipelines/` (`modular_pipeline.py` —
|
||
`LoopSequentialPipelineBlocks` line ~1294, `PipelineState`/`BlockState`; `modular_pipeline_utils.py` —
|
||
`ComponentSpec`; `components_manager.py`) and `src/diffusers/guiders/` (`guider_utils.py` — `BaseGuidance`).
|
||
- torchtitan: `~/torchtitan/torchtitan/` (`distributed/parallel_dims.py` — DeviceMesh builder + validation;
|
||
`distributed/{fsdp,activation_checkpoint}.py`; `components/checkpoint.py` — async DCP;
|
||
`experiments/rl/actors/{trainer,generator}.py` — TorchStore weight sync; batch-invariance utils in
|
||
`distributed/utils.py`).
|
||
- LiveKit agents: `~/agents/livekit-agents/livekit/agents/` (`voice/avatar/_runner.py` — `AvatarRunner`,
|
||
`AVSynchronizer`; `voice/avatar/_datastream_io.py` — data-stream/RPC control;
|
||
`examples/primitives/video-publisher.py` — `rtc.VideoSource.capture_frame`).
|
||
- Thinking Machines, "Defeating Nondeterminism in LLM Inference" (He et al., 2025):
|
||
https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ — batch-invariance analysis and the
|
||
true-on-policy (KL ≡ 0) RL result underpinning consistency level C2 (§8.5); companion library
|
||
`thinking-machines-lab/batch_invariant_ops` (torch.Library op-swap, FlexAttention-based deterministic vLLM mode).
|
||
- vLLM core: `~/vllm` (pulled 2026-06-10) + RFC #42770 "Changes in vLLM Model Development"
|
||
(https://github.com/vllm-project/vllm/issues/42770, Woosuk Kwon, 2026-05-15; discussion incl. the compile lead's
|
||
dissent and the two-tier head/tail resolution of 2026-05-28); linked: PR #42304 (breakable CUDA graph,
|
||
`VLLM_USE_BREAKABLE_CUDAGRAPH`), #43224 (porting compiler fusions to manual call sites), #24384 (HF-decoupled
|
||
config), `vllm/v1/worker/gpu/model_states/interface.py` (`ModelState`). FastVideo's inherited
|
||
`fastvideo/forward_context.py` is the in-repo instance of the hack this RFC deletes (staged retirement — 194
|
||
references across ~50 files; new-path StageContext in Phase 1, global frozen for legacy consumers, deletion at
|
||
Phase 5; §6.3.3).
|
||
v1 code studied in depth: `vllm/v1/engine/{async_llm,core,core_client,coordinator}.py` (process topology, DP
|
||
waves), `vllm/v1/core/sched/{scheduler,output}.py` (step-budget scheduling, in-step encoder budget, delta
|
||
outputs), `vllm/v1/core/{kv_cache_manager,kv_cache_coordinator,block_pool,encoder_cache_manager}.py`
|
||
(hybrid KV groups over one BlockPool; refcounted encoder cache), `vllm/v1/worker/gpu/{input_batch,states}.py`
|
||
(persistent batch, staged writes), `vllm/v1/cudagraph_dispatcher.py`,
|
||
`vllm/distributed/kv_transfer/kv_connector/v1/base.py` (`KVConnectorBase_V1`), `vllm/lora/` +
|
||
`punica_wrapper/` (batched multi-LoRA), `vllm/device_allocator/cumem.py` (tag-based sleep).
|