hao-ai-lab-FastVideo/mini_fastvideo

mini-fastvideo

A scoped, CPU-testable realization of the design_v3.md model-native runtime — the architecture where omni is native, train and serve are the same loops by construction, and correctness is a typed contract you can sign.

It is "mini" in two senses: (1) it supports a deliberate subset of models/methods, and (2) the heavy neural forwards are small numpy stand-ins so the whole thing runs and is tested on a laptop with no GPU and no torch. The architecture is real; the kernels are toys. On a GPU box you swap ComponentSpec.factory for the torch adapters (which wrap the real fastvideo.models.* modules) and nothing else in the runtime, scheduler, caches, parity, or training code changes — that is the whole point of the (recipe, runtime) separation.

Scope (phase 1)

	Supported
Inference	Wan2.1-1.3B (T2V, bidirectional diffusion) · LTX2.3 (two-stage distilled: 8-step base → upsample → 3-step refine) · Wan-causal (chunked/streaming rollout, the self-forcing student)
Training (all on Wan2.1-1.3B)	finetuning (flow-match) · DMD2 distillation · DiffusionNFT (likelihood-free RL) · self-forcing (causal, on Wan-causal)

Phase 2 (designed-for, not yet built): a Cosmos3 omni card and a canonical vllm-omni-style MoT model. The spine is already omni-ready — see Omni readiness below.

The one invariant (design_v3 §1)

Model cards own components, loops, recipes, and parity.
Programs compose loops into tasks.
The scheduler executes the steps of loops as WorkUnits under one budget.
Caches are correct by key, not by hope.
Training records behavior on the same loops it serves.

Package layout → design_v3 §18

Package	design_v3	What it is	Validated by
card/	§4	ModelCard = (components, loops, recipe, parity) as one versioned, validatable object; ModelInstance (one resident card; many loops bind shared components by reference)	tests/test_contracts.py
loop/	§5	the driven-loop contract init/next/advance/finalize; typed WorkPlan/StepResult/LoopState; the single LoopRunner driver; policies (CFG/flow-shift/precision/expert/conditioning); the flow-match sampler	test_policies.py, test_inference.py
runtime/	§6	Engine (serial + step-interleaved execution), layered scheduler (AdmissionController, BatchScheduler), the RuntimeLoopContext (the inversion point), cost currency	test_admission.py, test_interleave_gate.py
cache/	§7	typed CacheKey (partitioned by adapter+weights), per-class pools (feature/residual/slab_kv/paged_kv) behind one manager	test_cache.py
memory/	§7.3	tagged pools, reservation-before-admission, component-granular sleep/wake	test_admission.py
parallel/	§8	ParallelPlan (named axes), fake DeviceMesh builder, pre-flight validation (cfgp≤2, pp_patch invalid for causal, ownership conflicts)	test_contracts.py
parity/	§9	ParityAligner (record/compare taps), the C0–C4 consistency ladder, and the non-negotiable batch-of-N interleave gate	test_parity.py, test_interleave_gate.py
extend/	§11	observers (Profiler, NaNWatch) + interceptors (cache-dit-style ResidualSkipInterceptor) with state per request & per CFG branch	test_extensions.py
program/	§13	Program of typed nodes (ModelLoopNode/ComponentNode) + edges; when= branches replace geometry heuristics	test_inference.py
request/	§12	typed Request/Session/Artifact/Stream/CancelScope; TaskType declared, never inferred	used everywhere
training/	§10	rollout = serve loop + capture; BehaviorRecord; WeightSyncPlan (declared role); rewards; the 4 methods	test_training.py
models/	§4	the concrete (recipe, runtime) cards: wan21/, ltx2/, wan_causal/ + the numpy toy backend	all

Enforced boundary (design_v3 §10, §18): training/ imports card/loop/runtime/models; the engine never imports training. card/ imports no runtime.

What is actually demonstrated (each backed by a test)

• Loop inversion is safe under interleaving. Two+ concurrent requests, interleaved at step granularity, are bit-identical to running them serially — for all three models (test_interleave_gate.py). A deliberately buggy module-global interceptor breaks this gate, while the correct per-request one passes (test_extensions.py) — the structural proof of §5.1/§11.
• The (recipe, runtime) pair is enforced. A card whose recipe assumes_loop a loop it doesn't declare fails validate() (test_contracts.py).
• One resident instance runs many loops on shared weights. LTX2's base + refine loops bind the same transformer by reference (models/ltx2/card.py).
• Caches are correct by key. Same prompt + different te-LoRA stack ⇒ different key (partitioned, not stale); an update_weights bump invalidates wholesale (test_cache.py).
• Admission is reservation-first. Two requests that fit alone but jointly OOM ⇒ one is deferred, and deferral does not change outputs (test_admission.py).
• Train ≡ serve by construction. Every training method drives the same loop the engine serves, in the ROLLOUT profile; DiffusionNFT is likelihood-free (C2), samples from the decay-blended old policy (not the student), uses group-relative advantages, and encodes the shared prompt once per K-sample group via the feature cache (test_training.py).
• pp_patch parallelism is rejected for causal models (stale KV breaks causality, test_contracts.py).

Run it

cd /Users/willlin/src/FastVideo-mini
python3 -m pytest mini_fastvideo/tests/ -q      # 49 tests
python3 mini_fastvideo/run_tests.py             # same suite, ZERO deps (no pytest needed)
python3 -m mini_fastvideo.examples              # the design_v3 §15 worked examples

Requires only Python 3.10+ and numpy.

Running the real models (GPU)

Each ComponentSpec records the real load_id (e.g. fastvideo.models.dits.wanvideo:WanTransformer3DModel). To run real Wan/LTX:

1. install torch + the parent fastvideo package + weights;
2. replace each ComponentSpec.factory with a lazy adapter that loads the real module and exposes the same call surface the loops use (a velocity/predict_noise forward; vae.decode; text_encoder.encode);
3. swap the numpy flow-match sampler for the real FlowUniPCMultistepScheduler (Wan) / distilled scheduler (LTX) inside the loop's step body. The loops, policies, scheduler, caches, parity gates, and training methods are unchanged.

Omni readiness (phase 2)

The spine already carries the minimal concessions so Cosmos3 / vllm-omni MoT is additive, not a refactor: LoopKind/WorkUnitKind include ar_decode/ar_token/chunk_step; ModelInstance runs multiple loops against shared components; LoopSpec.shared_weight_components + extension_schema; LoopState.extension for packed factored sequences; paged_kv cache class; the interleave gate generalizes to any loop. Adding an omni card means: a Cosmos3PackedSequence state extension, an ar_decode loop bound to the same transformer as the diffusion_denoise loop, and a program ar_decode(reasoner) → pack → diffusion_denoise(joint) → decoders.

Deliberately out of scope (per design_v3 non-goals)

Fleet/Dynamo orchestration, the ComfyUI workflow compiler, WebRTC realtime transport, full LLM-grade AR serving (radix trees, speculative decoding), and real distributed parallelism (the mesh builder is a CPU fake). These are named in design_v3 as the fleet's job or as later phases.