SolitaryThinker
53d63afa76
Unify the four design docs (design.md, designv2.md, design_v3.md, designv4.md) into a single authoritative v2/README.md: the (recipe, runtime) thesis, driven loops, planes, one-WorkUnit scheduler, the parity ladder + interleave gate, training-on-shared-loops, the weight-sharing topology catalog, the current GPU status (20+ models + the BAGEL/Qwen-Omni/Cosmos3 trio verified), and a prioritized roadmap. Recast design_summary.md as a one-page design philosophy pointing to it. Add .agents/exploration/mstar-v2-roadmap.md (the adversarially-verified M* Walk-Graph gap analysis driving the roadmap). Delete the four superseded design docs.
57 lines
4.4 KiB
Markdown
57 lines
4.4 KiB
Markdown
# FastVideo — Design Philosophy
|
||
|
||
One page on *why* FastVideo is built the way it is. The full architecture, the as-built status, and the
|
||
forward roadmap live in **[`v2/README.md`](v2/README.md)** — this is the philosophy beneath it.
|
||
|
||
---
|
||
|
||
**A deployable model is a post-training artifact.** Unlike an LLM — where inference optimizes frozen weights
|
||
after the fact — a *usable* video/omni model is *created* by training: step distillation for latency, QAT for
|
||
precision, distillation + self-forcing for causal/world models. So every inference capability is a
|
||
**(recipe, runtime) pair**: the weights and the loop that produced-and-assumes them are one versioned object.
|
||
This is the source of the moat — whoever owns *both* sides of the pair owns the optimization frontier — and it
|
||
is why training and serving cannot be two systems.
|
||
|
||
**The work is loops, not `forward()`.** Denoise timesteps, AR decode, chunked rollout, VAE tiles, encoder
|
||
chunks, audio tokens, reward batches, optimizer steps, media chunks — video and omni inference is iteration. A
|
||
runtime that collapses everything to a single `forward` can't schedule, batch, cancel, stream, reserve memory
|
||
for, or capture the behavior of what actually runs. So loops are first-class, and they are **driven**: the
|
||
model describes the next step it needs, the runtime decides when and with whom it runs, the model folds the
|
||
result back. The model keeps content-adaptive control flow; the runtime keeps admission, batching, streaming,
|
||
and behavior capture. Per-request state lives in typed `LoopState`, never in module globals — so interleaving
|
||
requests through one model instance cannot smear state, by construction.
|
||
|
||
**The model is the center; everything else is a view over it.** A typed `ModelCard` owns components, loops,
|
||
the recipe, and the parity contract. Programs compose a card's loops into a task; Workflows compose cards into
|
||
pipelines; the scheduler runs the *steps* of all loops as `WorkUnit`s under one currency (predicted GPU-time,
|
||
because a bidirectional denoise step and an AR token are ~1000× apart and incommensurable in counts);
|
||
deployment places and routes; products stream artifacts. None of them define model semantics — they reference
|
||
the Model Plane. One resident instance can run many loop types on shared weights, which is what makes omni/MoT
|
||
native rather than a DAG that doubles weights.
|
||
|
||
**Correctness is a typed contract, not a hope.** Caches are correct by *key* — if a field can change output
|
||
semantics it is in the key, so reuse is partitioned, never blindly flushed. Parity between the train-forward
|
||
and the serve-forward is *measured* on a declared ladder (component → loop → behavioral → distribution →
|
||
artifact-quality), never assumed. And the non-negotiable gate is **interleave bit-parity**: N requests
|
||
interleaved at step granularity must be bit-identical to running them serially — the test the whole
|
||
loop-inversion bet lives or dies on.
|
||
|
||
**One substrate for inference, training, and RL.** The rollout forward *is* the serve forward plus capture —
|
||
same loop, same caches, same batcher, same numerics — so every serving optimization is automatically a rollout
|
||
optimization, and there is one numerics surface the ladder measures rather than a correction layer papering
|
||
over it. The engine doubles as the RL rollout engine under a strict rule: `training` consumes the engine; the
|
||
**engine never imports `training`**.
|
||
|
||
**Borrow aggressively; copy nothing as the core.** vLLM/SGLang scheduling, vLLM-Omni/SGLang-Omni omni serving,
|
||
Dynamo fleet orchestration, diffusers components, xDiT parallelism, TorchTitan mesh discipline,
|
||
verl-omni/miles RL lessons, ComfyUI workflows, Dreamverse/LiveKit sessions — each contributes a take, none is
|
||
the center. Deployment orchestration (Dynamo) sits *above* the engine, never inside it. Extensions are
|
||
versioned hook points, never monkeypatching. New frontier capabilities arrive as a card, a method, a loop, a
|
||
workflow, or a controller — **not a rewrite**.
|
||
|
||
> A model card is a (recipe, runtime) pair with a parity obligation. The model owns loop semantics; the runtime
|
||
> owns loop lifecycle. One resident instance runs many loops; one scheduler runs their steps in one currency.
|
||
> Caches are correct by key; parity is correct by test; the interleave gate is non-negotiable. Training records
|
||
> behavior on the same loops it serves. Deployment places and routes; products stream artifacts; neither defines
|
||
> the model.
|