Add v2/platform/: the (recipe, runtime) backend membrane that lets CPU, GPU,
and other devices coexist behind one dispatch substrate.
- Two tuple-keyed registries: COMPONENTS(kind, device, variant) for
weight-bearing components and KERNELS(op, device, arch, variant) for
stateless primitives, each with an availability predicate and an enumerable
manifest (declared-but-unavailable cells listed without importing torch).
- Platform: detected (device, arch) owning the device + arch fallback chains
and a per-platform cached KernelTable; detect() is honest (CPU/numpy unless
torch+CUDA are actually present). Arch fallback is monotonic — only degrades
to older/portable archs, never a newer binary-incompatible one.
- Seams wired: ModelInstance.component() -> platform.build_component(spec,self)
with spec.factory as the numpy terminal rung (existing cards untouched); the
wan21 denoise thunk dispatches solver ops through model.platform.kernels.
- Three backends: cpu (numpy terminal + parity oracle), accel (pure-python
stand-in proving cross-device resolution, arch fallback, and the oracle),
torch_cuda (declared-but-unavailable; no faked GPU).
- test_platform.py: 16 tests — detection, terminal rung, arch-fallback walk +
monotonicity, device precedence, variant fallback, component override +
per-kind device fallback, the parity oracle (accel == cpu, bit_identical via
the C1 ladder), and matrix enumeration without importing torch.
Scope is honestly bounded in the README/docstrings: only wan21-denoise routes
through the kernel table and only the dit kind is overridden today (each a
one-line adoption); the torch/CUDA path and a cudagraph workspace-safety
contract are declared/deferred, not implemented. 183 tests pass.
Three more capabilities on distinct untested surfaces (167 tests pass, no new runtime primitive).
A — Adapter plane (§9.19): one base + swappable LoRA/ControlNet adapters, selected per request
(DiffusionParams.adapters); AdapterDenoiseLoop applies each active adapter's velocity delta. Per-request
selection changes output, multi-LoRA composes, ControlNet conditions on a control image, mixed-adapter
requests interleave without smearing, hot-swap changes generation, cache key partitions by adapter stack
(the adapter_versions field, previously declared-only). ToyLoRA/ToyControlNet; models/adapters/. 6 tests.
B — Non-linear workflows (§9.17): ParallelWorkflow (fan-out: one input → N models → merged) and
BestOfNWorkflow (generate N → score with the served reward card → return best; inference-time scaling).
The shapes a linear chain can't express. 4 tests.
D — RL→distill flywheel (§9.18): run_flywheel RL-improves the base (NFT), then distills FROM the RL'd model
(DMD2 teacher = RL'd policy) into a faster card, recording the base→rl→distilled provenance chain in
RecipeSpec.parents. The distilled student is measurably closer to the RL'd teacher than the base; the
distilled card serves few-step. training/flywheel.py. 4 tests.
designv4 §9.17–§9.19 + layout/closing/counts (167 tests, 29 files).
The last audit stress test. A cheap draft model proposes K tokens, the target verifies
them in one batched step, and SpeculativeARLoop accepts the matching prefix + one target
correction — a variable accepted-length per round (a ragged AR loop the model owns).
- Exactness: the emitted sequence equals the target's OWN greedy decode for any draft
quality (every accepted token is one the target would produce; the correction is the
target's token) — the speedup is free.
- Speedup scales with accept rate: draft-agree 0.3→1x, 0.7→3x, 1.0→4x=K tokens/round
(fewer verify_rounds, the expensive model's latency steps, for the same output).
- Two components (draft + target) co-scheduled on one resident instance; each round an
AR_TOKEN WorkUnit.
models/speculative/ (loop+card+program); backend ToyTargetModel/ToyDraftModel with a
shared length-dependent target formula (no degenerate fixed point). 5 tests; full suite
153 passed. designv4 §9.16 + counts (153 tests, 26 files).
The remaining design_v3 probes (all except 8b speculative decoding). All fit with no new runtime
primitive (148 tests pass).
#6 LTX-2 joint audio+video (§9.11) — LTX-2 declared an audio_vae required_for t2vs but never used
it; now a single 2-stage denoise carries a synchronized audio latent (conditioned on video),
applies per-modality CFG (guidance_per_modality), and decodes via video VAE + audio VAE → video +
audio. Gated on requesting audio, so the T2V path is byte-identical (existing tests untouched).
ToyAudioVAE; build_ltx2_av_program. 5 tests.
#4 Live weight-sync under in-flight serving (§9.14) — WeightSyncController makes the freeze → drain →
transfer → bump version + invalidate → resume lifecycle explicit. Tests: a mid-flight swap corrupts
(the hazard); draining first leaves the in-flight request bit-identical to baseline while a
post-sync request reflects new weights; transformer-only sync, so the frozen text-encoder cache
survives. The RL flywheel's hardest correctness. 3 tests.
#5 Reward-model-as-a-served-card (§9.15) — a reward model is a card (scorer + a score loop emitting
REWARD_BATCH units); ServedRewardScorer drop-in-replaces the numpy scorer so any RL method becomes
RLHF/RLAIF with no method change. ToyRewardModel; models/reward/. 4 tests.
#7 Content-adaptive control flow (§9.12) — CacheDiTDenoiseLoop (isolated WanDenoiseLoop subclass)
reuses the cached velocity when predictions barely change (cache-dit skip) and early-exits on
convergence — variable step count; interleave parity holds across ragged loops. models/adaptive/. 4 tests.
#8a Nested workflows (§9.13) — a workflow stage can invoke another workflow (engine.run routes ids);
requires/validate recurse; cycles caught at registration + a run-time guard (engine._wf_running).
build_t2i_i2v_extend_workflow. 5 tests.
designv4 §9.11–§9.15 + falsifier/layout/closing updates (148 tests, 25 files). Also removed a
pre-existing unused import in ltx2/loop.py.
Three more example folders alongside inference/, all CPU/numpy, self-contained
(sys.path bootstrap), public API only, every script verified to run green.
training/ (7) — one per method, all via the uniform method.train_step seam:
01 finetune · 02 dmd2 distillation · 03 diffusion_nft (likelihood-free RL,
samples from the old policy, feature-cache reuse) · 04 self-forcing (causal
chunk_rollout) · 05 joint LM+generator RL (UniRL; joint + prompt-only) ·
06 N-way joint RL (per_expert vs shared credit) · 07 end-to-end workflow RL
(T2I+I2V from one final-video reward).
omni/ (4) — 01 Cosmos3 (reason→joint denoise, shared MoT) · 02 BAGEL
(text→image, shared MoT; scheduler prices both WorkUnit kinds) · 03 Qwen-Omni
(thinker→talker→vocoder, three separate experts, text+audio) · 04 interleave
parity across AR + diffusion loop types.
workflows/ (2) — 01 cross-model T2I→I2V workflow (image provably conditions the
video) · 02 workflow as a first-class servable (requires/validate, address by
id, register_workflows catalog, WorkflowRegistry).
Each folder has a README indexing its scripts.
Five self-contained, runnable scripts (CPU/numpy) for the Wan2.1-1.3B card on the
v2 runtime, each bootstrapping the repo onto sys.path so they run from anywhere:
- 01_basic_t2v.py minimal path: build engine → T2V request → run → video
- 02_params_and_reproducibility.py DiffusionParams knobs + seeded bit-identical reproducibility
- 03_streaming.py per-denoise-step preview chunks (OutputSpec stream)
- 04_concurrent_interleaved.py step-interleaved batching + interleave parity gate + cache reuse
- 05_async_serving.py AsyncEngine: concurrent generate, event stream, step-boundary cancel
+ README.md indexing them. All five run green; use only the public API.
- v2/README.md: point to designv4.md as the unified design (design_v3 as
north star); refresh the scope table (joint/N-way/workflow RL, Qwen-Omni
cascade, cross-model Workflow, tiled VAE co-scheduling, WorldModelSession),
package layout (program/Workflow, runtime/session, the 7 methods, new model
dirs), the demonstrated-stress-tests list (§9.3–§9.10), and counts (49→127,
20 files). Sessions moved out of "out of scope"; WebRTC wire stays out.
- designv4.md: drop two intermediate absolute suite totals (milestone "91/97
passed") in favor of "zero regressions" so the only absolute count is the
current 127 (intro + layout) — no stale numbers.
Targets the three design_v3 claims that were most load-bearing AND least
exercised (sessions/realtime, training-plane boundary, the WorkUnit-generality
falsifier). All fit with no new runtime primitive (127 tests pass).
1. Interactive world-model session (runtime/session.py, §9.8) — the Session
plane had ZERO coverage. WorldModelSession drives the causal chunk_rollout
loop as a long-lived session: persistent cross-request world state on the
Session.kv_handle, frame streaming, transactional step-boundary cancellation
(a cancelled act leaves the world resumable), no cross-session smearing.
Added only a continuation seam to the chunk loop (init seeds context from a
world_context slot; default empty = unchanged one-shot path). 5 tests.
2. End-to-end RL over a cross-model workflow (training/methods/workflow_rl.py,
§9.9) — trains BOTH flux-t2i and wan-i2v from ONE final-video reward. Rolls
out the whole workflow with SDE capture in both instances; the same final
advantage drives FlowGRPO PPO on each stage's transformer; two WeightSyncPlans
on two instances. The earlier model (T2I) is trained by a reward on the final
video — end-to-end credit across a model boundary — proven causal by a control
(constant reward => zero advantage => nothing moves). 4 tests.
3. Heterogeneous WorkUnit co-scheduling (models/tiled/, §9.10) — the §17
falsifier. VAETileLoop makes VAE decode a loop of VAE_TILE units; tiling is
exact (== one-shot, C0), and VAE_TILE + DIFFUSION_STEP pipelines interleave
bit-identically and co-run in one batch. Validates the mechanism; the
economic half (does it pay) stays a port-time measurement. 4 tests.
designv4 §9.8–§9.10 + falsifier/layout/closing updates (127 tests, 20 files).
Answers "what's the right way to name/register custom pipelines like T2I→I2V":
treat a Workflow like a card — a stable namespaced id in the same servable
namespace, declared dependencies, and a two-level registry. No new concepts;
mirrors how cards are registered (and vllm-omni's pipeline_registry).
- program/workflow.py: Workflow gains `requires` (the cards it composes, derived
from stages) and `validate(engine)` (fail-fast if a required card is absent,
P7). New WorkflowRegistry: declarative workflow_id -> builder catalog for
out-of-tree/ad hoc use.
- runtime/engine.py: `_workflows` registry + register_workflow (validates deps,
rejects id collision with a model_id) + serves(); engine.run routes a request
whose model_id is a workflow to workflow.run — addressable exactly like a model.
Single-model hot path untouched.
- runtime/async_engine.py + serving/server.py: serves() and /models include
workflows (discoverable as servables).
- models/__init__.py: declarative `_WORKFLOWS` catalog (the cross-model analog of
_BUILDERS) + register_workflows() helper; build_image_video_engine now registers
the workflow too. Adding a custom pipeline = one catalog line.
- Naming convention: dotted/namespaced workflow_id (`image_video.t2i_i2v`),
distinct from kebab model ids, collision-checked. Renamed from `t2i_then_i2v`.
- tests (+5, 12 total in the file): addressable by id, requires/validate, id
collision, registry catalog, register_workflows helper. Full suite 114 passed.
- designv4 §9.6: the naming & registration convention documented.
Two more pipelines stress-testing the design, plus a BAGEL-placement note in
designv4. Both fit with no new runtime primitive (109 tests pass).
Pipeline 1 — cross-model T2I→I2V (program/workflow.py, models/image_video/):
- Realizes ProgramKind.WORKFLOW as a thin multi-instance orchestrator ABOVE the
engine (the hot path stays single-instance). A Program composes one model's
loops; a Workflow chains full engine.run calls across distinct cards, threading
artifacts. (LTX-2 already covers same-card multi-stage; cross-model — FLUX→Wan
— is the new capability the single-instance runner can't express.)
- flux-t2i (text→image) and wan-i2v (text+image→video) cards; the I2V program
folds the conditioning image into text_embeds so WanDenoiseLoop is unchanged.
- Each model keeps its own interleave-parity guarantee (crossing instances is a
Workflow boundary, not a loop step). 7 tests incl. video-depends-on-image.
Pipeline 2 — N-way joint RL (training/methods/joint_multi_rl.py, models/multi_expert/):
- JointMultiExpertRL generalizes UnifiedRLMethod (N=2) to N refiner LMs + a
generator: one reward → one group advantage → N token-PG updates + 1 FlowGRPO
PPO update, N+1 independent WeightSyncPlans. Proves the substrate was already
N-ready (card holds N components/loops; per-component weight-sync; dict grad
targets) — only the method body looped over two; now it loops over a list.
- credit="per_expert" learns all N cleanly; credit="shared" (faithful to UniRL)
works but is noisier — the honest multi-agent credit-assignment result, a
reward-shaping choice, not a substrate limit. 6 tests (N=1,3,4; prompt-only).
Fix — flow_sde_ml_velocity (loop/sampler.py): the toy FlowGRPO generator update
targeted the velocity the model already produced (a no-op once guidance_scale=1
was set for the C2 identity; the unified generator moved only on ~1e-7 noise).
The correct PG surrogate targets the max-likelihood velocity of the realized
sample — nonzero at ratio==1. Both UniRL and N-way generators now learn for real;
the C2 ratio==1 identity still holds (measured before the update).
BAGEL: MoT/shared-weight (one transformer on both loops), same row as Cosmos3;
real BAGEL's co-resident experts are expressible via the expert-routing policy
(partial sharing) — captured in designv4 §2.3.
Ports vllm-omni's canonical qwen2_5_omni omni-speech cascade as a v2 card:
a third weight-sharing topology — three disjoint experts (thinker, talker,
vocoder) on three loop types (ar_decode → ar_decode → audio_decode) in one
request, with chained cross-stage conditioning and streaming codec→waveform.
vllm-omni runs these as three opaque request-scheduled stages; v2 makes every
thinker token, talker token, and vocoder chunk a runtime-visible WorkUnit.
- models/backend.py: ToyTalker (a genuinely distinct AR expert, weight-salted)
+ ToyVocoder (streaming code2wav: codec tokens → waveform chunks).
- models/omni/vocoder_loop.py: VocoderLoop filling the pre-declared
LoopKind.AUDIO_DECODE / WorkUnitKind.AUDIO_CHUNK slot.
- models/omni/ar_loop.py: ARDecodeLoop gains a configurable prompt_slot so two
chained AR loops don't collide on the prefill slot (thinker vs talker).
- models/qwen_omni/: card (3 experts/3 loops) + program (tokenize → thinker →
emit_text → thinker→talker full-payload hand-off → talker → talker→vocoder →
vocoder → emit_audio). Cross-stage hand-offs are explicit Program nodes, the
model-native form of vllm-omni's custom_process_input_func.
- _enums.py: Capability.TEXT_TO_SPEECH.
- tests/test_thinker_talker.py: 6 tests incl. three-loop interleave parity,
cascade conditioning, AUDIO_CHUNK streaming. Full suite 97 passed.
designv4.md: §2.3 topology table extended to four topologies; new §9.5 on the
cascade; reference-synthesis + package layout updated.
Stress-tests the v2 Card/Loop/Program design with a UniRL/PromptRL-style
joint RL recipe: a prompt-refiner LM expert and a flow generator expert,
two separate experts driven by two loop types in one request, both updated
simultaneously from a single RL reward.
- loop/sampler.py: flow_sde_step_with_logprob — FlowGRPO SDE rollout sampler
(per-step Gaussian log-prob), distinct from the deterministic ODE serve step.
- request/params.py: gated sde_rollout/sde_noise_scale on DiffusionParams so
the serve path stays byte-identical (default ODE).
- models/wan21/loop.py: gated SDE-rollout capture in WanDenoiseLoop.advance.
- models/backend.py: ToyPromptRefiner — a real REINFORCE categorical policy
(the Qwen role), separate weights from the generator.
- models/unified/: the unified card+program — two disjoint experts (llm +
transformer) on ar_decode + diffusion_denoise; the topological opposite of
the Cosmos3 MoT card, same vocabulary.
- training/methods/unified_rl.py: joint GRPO — one reward -> group advantage
-> LM token policy gradient + DiT FlowGRPO PPO; two LRs; prompt-only/joint
flag; reuses the shared diffusion loop for rollout.
- training/weight_sync.py: WeightSyncPlan gains a component scope so the two
experts version + cache-invalidate independently (LM sync never flushes the
frozen text-encoder feature cache).
- tests/test_unified_rl.py: 9 tests incl. likelihood-based C2 identity, the
two-loop interleave parity gate, joint vs prompt-only. Full suite 91 passed.
designv4.md: unified design doc reflecting v2 as built+tested, with the joint
RL stress test as the validating case study (the design held — new card +
new method, no new runtime primitive).
Directory rename (git mv, history preserved) plus rewrite of all references: absolute imports in
tests, the zero-dep runner, docstrings, comments, and the README. No behavior change.
Run: python3 -m pytest v2/tests/ -q ; python3 v2/run_tests.py ; python3 -m v2.examples
Review confirmed the core bets (concurrent disaggregation is bit-identical, design conformance holds,
Dynamo genuinely optional, engine stays step-scheduled). Fixes for the untested failure paths:
- HIGH: pool capacity (RolePool.in_flight) no longer leaks when a disaggregated request is cancelled
or errors mid-occupancy — DisaggregatedRunner.close() releases the occupied pool and AsyncEngine._run
calls it in a finally (a cancel on a capacity-1 denoiser no longer bricks the pool).
- credit flow-control: cross-pool transfer wraps acquire/release in try/finally (no credit leak on a
failing transfer); slot is re-homed only on a successful fetch.
- AsyncEngine: duplicate in-flight request_id is rejected (was a deadlock); submit() cancels the driver
task when the consumer abandons the stream (client disconnect → no orphaned compute); bounded
per-request history (no unbounded _events/_states/_results/_runners growth).
- cancellation is common-path on the OFFLINE path too (cancel check at the top of every runner.tick()).
- HTTP server: read timeout (slowloris guard → 408), body-size cap (→ 413), invalid Content-Length
(→ 400), explicit StreamReader limit, and aclose() of the SSE generator on client disconnect.
- build_deployment_card no longer aliases one mutable CostModel across replica cards (dataclasses.replace),
so online calibration of one worker's cost doesn't mutate another's.
- video-job tasks tracked (not fire-and-forget); server.close() cancels/drains them; jobs dict bounded;
fleet affinity map bounded.
6 regression tests added for these paths. 82 tests pass (pytest + zero-dep runner).
Builds the full serving layer the design files specify, instead of deferring it to Dynamo:
- transport/ (§7.3): pluggable Connectors (in-proc zero-copy / SHM-fake copy) with chunk_ready
readiness (vllm-omni) AND credit-based flow control (sglang-omni Relay); KVConnector protocol shape;
TransferManifest.
- runtime/ (§6, §13; plan M3/M4): AsyncEngine — request queue, lifecycle state machine
(waiting→running→completed/cancelled/failed), live AsyncIterator[OmniEvent] streaming, common-path
cancellation, step-level concurrency. RolePool + DisaggregatedRunner (encoder→denoiser→decoder,
capacity-aware dispatch, cross-pool transfers via connectors); disaggregated output is bit-identical
to inline. No-progress detection (no busy-spin).
- deploy/ (§14, §6.3.5-6): DeploymentCard; OUR OWN LocalFleet (discovery, health/drain, least-loaded
/ cost-model / sticky-affinity routing) so we never rely on Dynamo; DynamoWorkerAdapter +
FakeDynamoRuntime export the SAME card + cost model so Dynamo CAN front us — one object, two consumers.
- serving/ (§6.3.5, §12): framework-free stdlib-asyncio OpenAI server (our own version of the
vllm-omni pattern): /v1/chat/completions (SSE), /v1/images/generations, /v1/videos (async job+poll)
+ /v1/videos/sync, /v1/models, /health, /metrics. A thin shim over the STEP-scheduled engine — the
runtime-visible loop scheduler vllm-omni's request-scheduled opaque DIFFUSION stage lacks.
15 serving tests (real-socket HTTP+SSE via stdlib asyncio, disagg==inline, fleet routing, Dynamo
contract, cancellation); 76 tests pass total (pytest + zero-dep runner). ~7900 LOC.
One resident MoT instance runs BOTH an ar_decode loop and a diffusion_denoise loop on shared weights
(the §16 claim no DAG-of-engines can express), with both loops runtime-visible: the scheduler prices
ar_token AND diffusion_step WorkUnits — unlike vllm-omni's opaque DIFFUSION stage the scheduler never
sees inside.
- ARDecodeLoop: token decode until EOS/max_tokens, paged text-KV — the omni AR pathway (loop/§5).
- ToyMoTDiT: one module exposing an und pathway (ar_forward) AND a gen pathway (denoise __call__);
ToyTokenizer. Binding both loops to one instance = shared weights, no duplication.
- models/cosmos3/: tokenize → reason(ar_decode) → pack(tokens→conditioning) → diffusion_denoise →
vae_decode; sound_vae declared optional_for non-t2vs (the lazy-component P8 fix, not an env-var hack).
- models/bagel/: the canonical vllm-omni model — generate_text(ar_decode) → generate_image(diffusion),
text+image outputs, both loops step-scheduled.
- The diffusion loop is WanDenoiseLoop reused (one loop definition bound to the MoT module).
- build_omni_engine() + an omni worked example; 7 omni tests (shared-instance, both-kinds-scheduled,
interleave parity across loop types, lazy sound_vae). 61 tests pass (pytest + zero-dep runner).
- Admission fails fast with AdmissionInfeasible on infeasible/deadlocked reservations instead of a
10M-iteration busy-spin: no-progress detection in run_to_completion/run_interleaved via a real
progress token, plus feasibility pre-checks (need > pool capacity).
- Compute budget is now a refundable concurrency gate (release() refunds spent), not a
never-refunded lifetime cap that silently deadlocks.
- §7.1: the text-encoder feature CacheKey carries adapter_versions + precision (no stale serve across
te-LoRA stacks); per-component weight versions mean a transformer-only RL weight sync no longer
flushes the frozen text-encoder cache (component-scoped invalidation, not wholesale).
- Interleave gate flags symmetric-empty output instead of passing it vacuously.
- skipped_steps counted only when the override is actually consumed; BatchScheduler wired for round
batch-accounting (metric renamed stepped_units); ResidualCache.get cleanup; stream chunks carry a
latent preview payload; dead progress-vars removed.
- 5 regression tests added for the previously-untested paths. 54 tests pass (pytest + zero-dep runner).
A scoped, CPU-testable realization of design_v3.md — the architecture where the atomic unit
is a typed (recipe, runtime) ModelCard, the model owns loop semantics while the runtime owns
loop lifecycle, one resident instance runs many loops, and training records behavior on the
same loops it serves.
Implements:
- card/ loop/ runtime/ cache/ memory/ parallel/ parity/ extend/ program/ request/ training/
spanning design_v3 §4-§13: ModelCard + validate(); driven loops (init/next/advance/finalize);
step-interleaving Engine with reservation-before-admission + per-class caches keyed by CacheKey;
the C0-C4 consistency ladder + the non-negotiable batch-of-N interleave parity gate.
- Inference: Wan2.1-1.3B (T2V), LTX2.3 (two-stage distilled, shared transformer), Wan-causal
(chunk rollout + slab-KV streaming).
- Training (Wan2.1-1.3B), each driving the SAME loops the engine serves: finetune (flow-match),
DMD2 (teacher/critic distribution matching), DiffusionNFT (likelihood-free C2, samples from the
decay-blended old policy, group-relative advantages, shared-prompt cache reuse), self-forcing
(causal chunk loop). The engine never imports training (the §10 dependency rule, grep-verified).
numpy-only core (no torch/GPU here); heavy Wan/LTX forwards are deterministic toy stand-ins with
lazy torch-adapter seams (ComponentSpec.load_id/factory) for a GPU box. 49 tests pass via pytest
and a zero-dependency runner; interleave parity verified (and a buggy module-global interceptor
provably breaks it). Omni-ready spine (ar_decode/chunk_step loop kinds, multi-loop instances,
LoopState.extension) for the phase-2 Cosmos3 + vllm-omni omni ports.
Run: python3 -m pytest mini_fastvideo/tests/ -q ; python3 -m mini_fastvideo.examples
SolitaryThinker