hao-ai-lab-FastVideo/plan.md

FastVideo Runtime — Aggressive Implementation Plan

Companion to design.md (v19) and design_summary.md · Stance: this plan trades interface stability for speed. Where it deviates from design.md's conservative migration (§10), the deviation is flagged with ⚡. design.md remains the architectural authority; this is the execution order.

---

1. Rules of engagement

We break, freely and early:

• The public Python API: generate_video(**kwargs) and SamplingParam are deleted, not deprecated.
• Config schemas: FastVideoArgs (1,272 lines, 81 fields) stops being a public or threaded surface.
• fastvideo/api/compat.py (651 lines): deleted in M1 ⚡ (design.md §6.6 shrinks it monotonically to Phase 5 — that policy existed only to honor signatures we are now licensed to break).
• CLI flags, YAML schemas, package layout, fastvideo.api exports, ComfyUI node params, every example.
• In-repo dependents (apps/dreamverse, comfyui/, examples/, scripts/) get fixed in the same PR train — we own them; no deprecation period, no shims.

We never break, at any speed:

• Numerics. Bit-identical loop parity and SSIM gates are not "conservative" — they are the definition of correct. Aggression applies to interfaces, never to outputs.
• Model coverage for the families that matter (tier list in §6 — the tail is a decision, not a casualty).
• The frozen legacy fastvideo/training/ stack (N2) and the bit-exact porting methodology (N3).
• External users get batched breakage: all user-visible breaks land in at most two releases (R1 = request/config cut, R2 = engine default), each with a migration guide and a fastvideo migrate codemod — never a drip.

2. The sequencing argument (answering "fix omni request first, then separate the planes?")

Yes to the first half. The second half should not be a project. The three planes are not separated by moving code into plane-named directories — today's monolithic stages would just get reshuffled and then rewritten when loops invert. The planes are born from two cuts, and a third that is really a config change:

1. The request-plane cut (M1) — OmniRequest becomes the only currency crossing the boundary. Everything behind it is implementation. This is your "fix the omni input and request first," and it goes first because it is low-risk, it defines the vocabulary every later stage consumes, and it gets the user-facing pain over with while the codebase is still familiar.
2. Loop inversion (M2) — this is the pipeline/execution plane separation. Once families expose init/step/finalize step bodies, something other than the family must own iteration; that owner is the executor, and the execution plane exists by construction. Before inversion there is nothing for an execution plane to schedule — "separating" it would be an empty directory.
3. The config cut (inside M1) — the real coupling between planes today is FastVideoArgs: one 81-field object threaded through entrypoints, pipelines, stages, and executors, mixing deploy-time, model-time, and request-time concerns. Splitting it into DeployConfig / ModelSpec / OmniRequest (design.md §6.6's four layers) is the single highest-leverage "separation" action, and it's schema work, not architecture work.

So the order is: M1 request+config cut → M2 loop inversion (planes now exist) → M3 engine on top. Plane separation is the outcome of M1+M2, not a milestone.

3. Milestones

Timeline assumes 3–4 engineers on the runtime critical path. Overlap is deliberate; gates are not ⚡-able.

M0 — Baselines, harness, enforcement (weeks 0–3, overlaps M1)

The license for everything aggressive afterward. Not skippable, not shrinkable. M0 does not block M1 (which changes no numerics) — the only hard rule is no family's M2 migration starts before its baseline exists.

• Merge the feat/cosmos3-reasoning chain (design.md sizes this alone at 2–3 engineer-months — it runs as its own track); seed SSIM references for the ~7 uncovered families.
• ParityAligner v0: record/compare named taps on current pipelines (it must exist before anything changes).
• The enforcement package, on day one (design.md §10 — the prior freeze was broken 19× for lack of exactly this): CI path gates (reject new fastvideo/training/ files now; reject new DenoisingStage subclasses once the first M2 family lands), CODEOWNERS on the frozen and migrating paths, a named owner per milestone, and the inflow rule — new model families land on the new abstractions from the first Wan/Flux2 landing onward.
• Announce the M1 freeze window for in-flight PRs touching fastvideo/api/, fastvideo_args.py, entrypoints.

Gate: every tier-A family has a recorded SSIM + activation baseline; CI gates live.

M1 — The request-plane cut (weeks 1–4) → breaking release R1

The typed API is partway there: VideoGenerator.generate(GenerationRequest) is already the documented primary entrypoint (generate_video carries a deprecation warning), and fastvideo/entrypoints/openai/ already serves POST /v1/videos and POST /v1/images. But the legacy path is still what's used: Dreamverse calls generate_video(**kwargs) (apps/dreamverse/dreamverse/video_generation.py:508), as do ComfyUI and most examples. M1 finishes the cut instead of bridging it:

• OmniRequest / OmniOutput / OmniEvent: evolve api/schema.py's GenerationRequest in place — typed modality parts, TaskType, per-model ModelOptions registered blocks (formalizing the api/matrixgame2.py pattern), seeds/priority/streaming flags. api/results.py's Video*Event types become OmniEvent.
• Config: four layers, one owner each (§6.6): extract DeployConfig (placement, parallelism axes, memory/ offload, compile, plugins) from the runtime third of FastVideoArgs + EngineConfig/ParallelismConfig; ModelSpec manifest v0 (manifest-first component resolution; today's name-detectors as fallback); OmniRequest absorbs every per-call field. CLI flags, OpenAI protocol models, and presets are generated from the schema.
• Delete ⚡: compat.py (651), sampling_param.py (411), generate_video(), the fastvideo.api legacy exports, FastVideoArgs as a public type. Internally it survives as a boundary-constructed shim for as long as anything still receives it: migrated families drop it per-family in M2, but unmigrated tier-B stages (LegacyPipelineNode) and the frozen training/ stack (whose TrainingArgs subclasses it) carry it until M6 — it dies as a type with the tail, not before.
• Fix in-train: ComfyUI nodes (legacy-API callers), all examples/ (~75 files, mostly mechanical), scripts/, docs. Ship fastvideo migrate (codemod: old kwargs/YAML → OmniRequest/DeployConfig).
• Internals unchanged: ForwardBatch is built from OmniRequest at the boundary; the executor and stages are untouched in M1.

Gate: all SSIM suites unchanged; Dreamverse + ComfyUI + examples green on the new surface; R1 notes + codemod published.

M2 — Loop inversion (weeks 4–10): the pipeline plane is born

• DenoiseLoop / ARDecodeLoop with init/step/finalize; runtime owns iteration; custom-step escape hatch from day one (the Cosmos3-port and self-forcing pattern is legitimate, §6.2.3).
• Family order (each lands step body + policies, and deletes its legacy stage code in the same PR ⚡ — continuous deletion, no end-of-plan cliff): Wan 2.1/2.2 + Flux2 first, jointly — design.md's rationale stands: together they exercise CFG variants, expert routing, chunk-KV, and the image path, so the step-body contract freezes only after all four are exercised → Wan-causal (self-forcing student) → LTX-2 → HunyuanVideo → Stable Audio → remaining image families. Unmigrated families keep running via LegacyPipelineNode.
• Policies: CFGPolicy (absorbs the 3 CFG copies), AttnMetadataProvider, FlowShiftPolicy, PrecisionPolicy.
• Extension core lands with the loop (it's why the loop is being rebuilt): observer bus, ParityAligner promoted to per-request observer, Profiler/NaNWatch, and cache-dit as the first interceptor (retiring enable_teacache).
• forward_context.py off the migrated inference path (194 references across ~68 files today: ~8 importer files in frozen training/, the rest spread across train/ models, tier-B inference stages, quantization, and tests); the module survives as a shim for frozen training/ and unmigrated tier-B stages until M6 — what M2 guarantees is that no migrated family and no new code touches it.
• train/ migrates per-family, immediately behind inference: DMD2 and the landed DiffusionNFT (#1450) adopt the shared step functions as each family's body lands — rl/common/sampling.py's loop is deleted, #1396 grad-norm refs extended to the migrated methods (RL included).

*Gate, per family: old-vs-new loop bit-identical (ParityAligner) + SSIM + a recorded loop-overhead / batch-of-1 latency measurement (the baseline M3 gates against); for train/: seeded rollout latents identical, reward metrics

• grad-norms neutral. No family is ever dual-maintained.*

M3 — Execution plane: engine + scheduler (weeks 8–14, overlaps M2) → breaking release R2

• AsyncEngine (queue, admission, cancellation-as-common-path, failure isolation); offline VideoGenerator keeps its name, becomes a thin sync wrapper that can bypass the queue.
• StepScheduler v0: multiplexes denoise steps across requests in a pool; budget currency = predicted GPU-time from a calibrated per-(model, phase, shape) cost table (the cost model matures later; the currency is right from day one). Carries the ARDecodeLoop contract; AR batching itself waits for its workload (N5).
• CacheManager v0: per-request chunk-KV slabs behind KVHandle; CFG-parallel axis (2-branch in practice).
• Dynamo stock worker (registration, health/drain, cost metrics), retiring the locked dynamo/examples/diffusers/worker.py pattern.
• Dreamverse hard-cut (per design.md Phase 2; the aggressive delta is doing it in one PR): gpu_pool.py, queue, warmup, and stream relay deleted and replaced by engine-client calls; the duty-cycle concurrency study runs on the result.
• Colocated weight-sync RPC + component-granular sleep/wake + RolloutClient (engine-client RL mode for #1450).

Gate: serving load tests; batch-of-1 latency regression ≤ 2% vs the M2-recorded measurement; Dreamverse single-session parity; RL engine-client seeded final-latent parity vs in-process; deploys under stock Dynamo.

M4 — Graphs, parallelism, multi-session (weeks 14–20)

• PipelineSpec graph IR: per-family pipeline classes shrink to spec + step body + policies (create_pipeline_stages() retires); LTX-2 and Hunyuan15+SR land as real fan-out graphs.
• Role pools + connectors (port multimodal_gen's disagg state machine); declarative stacked-parallelism axes compiled to DeviceMesh; general cross-mesh WeightSyncPlan.
• ComfyUI workflow→spec compiler MVP (tier-1 ~20-node vocabulary) + weight/adapter fleet cache.

Gate (design.md Phase 3's, in full): ≥2 Dreamverse sessions/GPU on the recorded duty-cycle trace, p95 within SLO — this is also where the loop-inversion falsifier is evaluated (see §7); LTX-2 A/V full-fan-out end-to-end; disaggregated-vs-colocated throughput benchmark; CPU-only topology validation suite; ComfyUI tier-1 workflows compile and run with equivalence reports; spec-built pipelines SSIM-identical to M2 loop versions.

M5 — Omni/MoT native + RL hardening (weeks 20–30)

• Cosmos3 re-port onto specs: packed factored sequences, dual-pathway attention, reasoner paged KV, joint denoise, world-model ChunkRollout; /v1/chat/completions; AR continuous batching arrives with this workload (N5).
• Consistency ladder enforced end-to-end: C1 default in CI, C2 bitwise mode for goldens, Behavior Record opt-in; first GRPO-class method lands on the engine-client rollout path (log-prob drift becomes the gated metric).

Gate: Cosmos3 150-test parity suite on the new runtime; reasoner pool efficiency — tokens/s/GPU at target concurrent denoise throughput, with the ≥10×-vs-re-prefill sanity floor; C1 drift ≈ 0 on a Wan RL run with the drift dashboard live.

M6 — The tail and the precondition (week 30+)

Continuous deletion (M1/M2) shrinks the final phase but does not eliminate it: what remains by M5 is the tier-B tail on LegacyPipelineNode and the frozen training/ stack — which is a live consumer of ComposedPipelineBase and forward_context, so its retirement is the precondition, exactly as design.md Phase 5 states. M6 = execute the §6 tail decision (migrate or deprecate each tier-B family), retire training/ per the checklist, then delete ComposedPipelineBase, the legacy DenoisingStage, forward_context.py, FastVideoArgs/TrainingArgs, and RayDistributedExecutor together. 4 loop copies → 1.

4. Breakage manifest (user-visible)

Release	What breaks	Replacement	Aid
R1 (M1)	generate_video(prompt, **kwargs), SamplingParam, FastVideoArgs as public type, fastvideo.api legacy exports, CLI flag names, YAML config schema, streaming event types (Video*Event → OmniEvent, schema_version'd from day one)	VideoGenerator.generate(OmniRequest), DeployConfig, generated CLI/protocol, OmniEvent	fastvideo migrate codemod, migration guide, R0 pinned
R2 (M3)	Default execution path becomes the engine (offline bypass preserved); server lifecycle (queue/admission semantics, job states)	AsyncEngine	guide; OmniEvent schema unchanged from R1
after R2	nothing user-visible — M4/M5 are additive	—	—

5. Deviations from design.md §10, stated honestly

design.md	this plan	why it's safe now
Phase 0 keeps VideoGenerator/CLI signatures; compat.py shrinks to Phase 5	M1 breaks signatures, deletes compat.py ⚡	the only argument for the shim was signature stability — explicitly revoked
Legacy code deleted at Phase 5	per-family deletion at parity, M2 onward ⚡	parity gate is per-family anyway; carrying dead code to a final phase only invites the 19×-broken-freeze failure mode
Phases strictly sequential	M2/M3 overlap ⚡	the engine consumes step bodies, not finished families; the step-body contract freezes at the Wan+Flux2 landing
Phases −1 through 4 sized at 36–54 engineer-months	~21–28 engineer-months (3–4 eng × 30 wks) ⚡	the delta is real deleted work — no compat maintenance, no adapter upkeep, no dual-stack carry — plus M2/M3 overlap; treat 30 weeks as the aggressive case and 36–40 as the planning case
Unchanged	parity/SSIM gates (restored in full at every milestone), enforcement package (CI path gates, CODEOWNERS, inflow rule — now at M0), train/RL migration timing (design.md Phase 1 already migrates NFT), Dreamverse hard-cut (Phase 2 already prescribes it), N2/N3/N5, cost-model currency, Dynamo asks + fallbacks, schema versioning	aggression budget is spent on interfaces only

6. Decisions needed before M0

1. Tier the model zoo. Tier A (migrated, coverage guaranteed): Wan 2.1/2.2, Wan-causal/self-forcing, LTX-2, Flux2, HunyuanVideo, Stable Audio, Cosmos3 (contingent on the M0 merge — it is not on main today), image families. Tier B (runs on LegacyPipelineNode until someone claims it, candidate for deprecation at M6): gen3c, matrixgame2/3, longcat, the rest. Approve or edit the split — it bounds M2.
2. Release framing. R1 as v0.3.0 (pre-1.0 semantics, loud notes) vs holding breaks for a v1.0 story. Recommendation: v0.3.0 now — waiting taxes every milestone.
3. Freeze windows. M1 freezes api//args/entrypoints PRs ~2 weeks; M2 freezes per-family stage PRs while that family migrates (days each). Needs maintainer sign-off.
4. Staffing. Critical path is M2's per-family step bodies — parallelizable per family after the Wan+Flux2 reference lands. 3–4 engineers ≈ 30 weeks to M5 in the aggressive case (design.md's own sizing implies 36–40 weeks at the same staffing — see §5); 2 engineers ≈ stretch ~1.5×. The Cosmos3-chain merge (M0) is its own 2–3 engineer-month track and should be staffed separately from the runtime critical path.

7. Risks specific to the aggressive posture

• In-flight PR collisions with layout/schema moves → freeze windows (above) + landing schema cuts at milestone starts, not ends.
• Community churn at R1 (ComfyUI users, script users) → codemod covers the mechanical 90%; the 10% that isn't mechanical (kwargs with changed semantics) is enumerated in the guide; previous version stays pinned and installable.
• Parity harness becomes the bottleneck — every aggressive deletion is licensed by it. Mitigation: it is the first deliverable (M0), and per-family migration PRs are template-driven (record → port → compare → delete).
• Overlap risk (M2/M3): the engine team building against a moving step-body contract → the contract (init/step/finalize + StepResult) freezes at the first family (Wan), enforced by the same schema-version discipline as external surfaces.
• The known unknown: loop inversion at scheduler granularity has no production precedent (design.md §1). The falsifier stands, on design.md §11.6's schedule: the M3 duty-cycle study publishes the targets; the falsifier is evaluated at the M4 gate — if step-level multiplexing can't beat request-level serialization on real Dreamverse traces, StepScheduler retreats to request-level dispatch and the loop contract keeps only its streaming/preemption seams, with no family code changing — step bodies and the M1/M2 cuts retain full value.