7fd58f26ea1bf200f2e5d51b4e711f8d5cc268b7
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 SolitaryThinker
|
7fd58f26ea |
[fix] correct GPU adapters against real fastvideo API (cross-check findings)
Adversarial cross-check of the written-not-run torch adapters against the real fastvideo source confirmed the interface contracts (DiT returns bare velocity tensor; timestep=sigma*1000; encode().mode() + bare decode; .last_hidden_state; no fused solver kernel) but caught a wrong construction layer. Fixed in code: - Construction: WanTransformer3DModel / AutoencoderKLWan have NO from_pretrained. Replace it with the real FastVideo loaders (TransformerLoader / VAELoader / TextEncoderLoader + TokenizerLoader, each load(model_path, fastvideo_args)). The loader resolves the class from the checkpoint config — so UMT5-vs-T5 is chosen correctly instead of hardcoded (was BLOCKER #1/#4/#5). - Text encoder: wrap the forward in set_forward_context(...) — the (U)MT5 attention reads global state via get_forward_context(); a bare call mis-encodes (was BLOCKER #2). Drop the wrong padding="max_length". - VAE: apply latent normalization the DiT expects — (z-mean)*inv_std on encode, inverse on decode, with latents_std stored as its reciprocal; shift_factor before decode (was BLOCKER #3). Skipping it yields washed-out video, not error. The remaining unknowns are genuinely box-dependent (FastVideoArgs fields, shift_factor placement, exact tokenizer kwargs, FSDP) — GPU_BRINGUP.md reconciled to mark what's now fixed-in-code vs what still needs the box. 204 CPU tests pass. |
||
|
SolitaryThinker
|
15b6278e9f |
[feat] real torch/CUDA backend (written-not-run) behind the cuda cells
Implement the GPU backend the substrate was built for: Platform.detect() -> cuda resolves real torch adapters + torch solver ops instead of the numpy rungs, with the existing loops/policies/scheduler/training unchanged. WRITTEN-NOT-RUN: this environment has no GPU/torch, so the torch code is grounded in the verbatim real fastvideo APIs (DiT forward signature confirmed from source) but cannot be executed/verified here. It is gated available=False (CPU mini stays green; importing the backends never imports torch), with every on-box confirm point marked `# BRINGUP` and an ordered checklist in platform/backends/GPU_BRINGUP.md. - torch_adapters.py: TorchWanDiT / TorchWanVAE / TorchT5Encoder wrap the real module named by each card's load_id and bridge it to the mini's duck-typed surface (numpy<->torch at the boundary; loop math stays numpy fp32). DiT weight-surface (copy_from/blend_from/clone) for serving sync; mse_grad_step raises (GPU training is a separate workstream). - torch_kernels.py: flow_match_step / flow_sde_step as plain torch elementwise. Grounded conclusion from the kernel audit: fastvideo-kernel ships NO fused solver kernel (only attention/norm/quant primitives), so the cuda solver is torch, registered at arch generic with an honest source string. - torch_cuda.py: rewritten as lazy trampolines (torch imported only inside builder/kernel bodies). Adds the missing vae + text_encoder cuda components (they'd otherwise silently fall back to the toy) and corrects the dishonest "fastvideo-kernel:flow_*" labels. - ComponentSpec.checkpoint: the weights source for the torch adapter (risk A; the one field the cards didn't carry). Empty on the CPU toys. - 7 CPU-verifiable wiring tests: honest registration/sources, torch-free import, cuda-resolves-real-cells-not-toy, build-fails-loudly-without-torch. 204 tests pass (CPU). The torch path needs a GPU box to verify (GPU_BRINGUP.md). |
||
|
SolitaryThinker
|
cd0fbbd3a0 |
[feat] static-buffer capture form for the cudagraph step body (Path A)
Close the loudest deferred gap from the cudagraph audit: the capturable step now binds its I/O to address-stable static buffers (modeling real CUDA static I/O buffers), instead of allocating fresh arrays per call. - StaticWorkspace: address-stable buffers allocated once per capture key; bind() copies the current step's inputs in place via np.copyto, which RAISES on a shape/dtype mismatch — turning the weak peak_activation_bytes proxy into a real key-soundness backstop (a step whose shape doesn't fit can't replay an incompatible graph). Output written into a static buffer too. - WorkPlan.graph_fn / graph_inputs: a capturable step exposes its deterministic op-structure as graph_fn(model, workspace) reading EVERY per-step input (latent, sigmas, conditioning, scale) from the workspace — never from closure over per-step data — plus the dict of current values. Loops without both stay on the eager path. wan21 factors a shared _velocity() so graph_fn and the eager run stay bit-identical. - Capturer dispatch captures/replays via graph_fn against the keyed workspace; the workspace is shared per key on the instance. Correct under the engine's synchronous step execution (bind+graph_fn atomic per dispatch, output returned as a copy) — proven by the batch-of-N interleave gate running two same-key requests through the shared workspace bit-identically. A concurrent/multi-stream executor would need a per-stream pool (documented). - 2 new tests (no-static-form eager-break, static-buffer shape-mismatch raises); workspace collision test split into bytes-proxy vs shape-backstop. 197 tests pass. |
||
|
SolitaryThinker
|
02151584cd |
[feat] piecewise CUDA-graph capture/replay at the step boundary (Path A)
Wire the capture/replay lifecycle into the driven-loop step boundary — the other half of Path A (hand-fused kernels behind the registry + piecewise cudagraphs, no compiler). Models and tests the correctness-critical control logic; replay re-runs the current step thunk (CPU models the lifecycle, not the GPU speedup). - GraphCapturer on the instance (cross-request cache), wired into RuntimeLoopContext.execute and gated by LoopSpec.graph_capture == "breakable_cudagraph". Capture key = (device, arch, loop, shape_sig, resident-weight-versions, graph_key). Eager-break for non-capturable (SDE) and interceptor-overridden steps. Never executes a stored thunk, so interleaved requests can't smear state. - WorkPlan.capturable / graph_key. wan21 sets capturable=not sde and folds compute dtype (shape_sig.dtype) + CFG branch set + expert + scheduler-precision into the key — closing a cross-precision key-collision corruption path a non-fp32 build would otherwise hit on a real GPU (audit finding). - Version-in-key auto-invalidation + real eviction: set_weights_version evicts the synced component's graphs (duck-typed, so card/ imports no runtime), preventing a GPU graph leak across FlowGRPO's per-iteration syncs. - register_kernel gains the workspace_bytes capture-safety contract (declared in the matrix; cuda cells declare real scratch, numpy reference is 0). - 11 tests: capture-once/replay-many, eager-break (SDE + override), capture ≡ pure-eager bit-identical, recapture on shape + weight-version change with eviction, eager-loop gating, accel-backend capture, and capturer unit tests (key discrimination, eager-break, workspace-collision safety net, eviction). Honestly deferred (bite a real GPU, not the CPU tests): the static-buffer refactor of the step body, admission budgeting of capture cost (GRAPH_CAPTURE), and per-card opt-in beyond wan21 — all documented in cudagraph.py + README. 195 tests pass. |
||
|
SolitaryThinker
|
51bb5e349a |
[feat] route all diffusion loops + RL recompute through the kernel table
Finish the kernel seam across the board so the platform's KernelTable is the universal solver-dispatch path, not just wan21. - Loops: ltx2, wan_causal, adapters, adaptive now resolve the flow-match solver via model.platform.kernels.get(FLOW_MATCH_STEP) instead of importing the numpy sampler directly (wan21 already did). On CPU this is bit-identical (the cpu kernel IS the old function); a GPU/accel backend now overrides every loop. - RL: the FlowGRPO log-prob recompute in unified_rl / joint_multi_rl / workflow_rl dispatches FLOW_SDE_STEP through the platform, pinned to the SAME kernel the rollout used (C2 kernel-pinning — otherwise the PPO ratio biases on a real GPU where rollout and recompute kernels could differ). - accel backend: add a kind-generic AccelComponent wrapper and override the vae component too (text_encoder left unregistered to keep the device->cpu fallback demonstrated), closing the "only dit is overridden" gap. - tests: vae override assertion; a second-loop (wan-causal chunk rollout) parity oracle proving accel == cpu bit-identical beyond wan21. README scope updated. 184 tests pass. |
||
|
SolitaryThinker
|
1bdbf1d5b3 |
[feat] multi-backend dispatch substrate (device/arch/kernel registries)
Add v2/platform/: the (recipe, runtime) backend membrane that lets CPU, GPU, and other devices coexist behind one dispatch substrate. - Two tuple-keyed registries: COMPONENTS(kind, device, variant) for weight-bearing components and KERNELS(op, device, arch, variant) for stateless primitives, each with an availability predicate and an enumerable manifest (declared-but-unavailable cells listed without importing torch). - Platform: detected (device, arch) owning the device + arch fallback chains and a per-platform cached KernelTable; detect() is honest (CPU/numpy unless torch+CUDA are actually present). Arch fallback is monotonic — only degrades to older/portable archs, never a newer binary-incompatible one. - Seams wired: ModelInstance.component() -> platform.build_component(spec,self) with spec.factory as the numpy terminal rung (existing cards untouched); the wan21 denoise thunk dispatches solver ops through model.platform.kernels. - Three backends: cpu (numpy terminal + parity oracle), accel (pure-python stand-in proving cross-device resolution, arch fallback, and the oracle), torch_cuda (declared-but-unavailable; no faked GPU). - test_platform.py: 16 tests — detection, terminal rung, arch-fallback walk + monotonicity, device precedence, variant fallback, component override + per-kind device fallback, the parity oracle (accel == cpu, bit_identical via the C1 ladder), and matrix enumeration without importing torch. Scope is honestly bounded in the README/docstrings: only wan21-denoise routes through the kernel table and only the dit kind is overridden today (each a one-line adoption); the torch/CUDA path and a cudagraph workspace-safety contract are declared/deferred, not implemented. 183 tests pass. |
||
|
SolitaryThinker
|
de295403c7 |
[feat] Adapter plane, non-linear workflows, RL→distill flywheel
Three more capabilities on distinct untested surfaces (167 tests pass, no new runtime primitive). A — Adapter plane (§9.19): one base + swappable LoRA/ControlNet adapters, selected per request (DiffusionParams.adapters); AdapterDenoiseLoop applies each active adapter's velocity delta. Per-request selection changes output, multi-LoRA composes, ControlNet conditions on a control image, mixed-adapter requests interleave without smearing, hot-swap changes generation, cache key partitions by adapter stack (the adapter_versions field, previously declared-only). ToyLoRA/ToyControlNet; models/adapters/. 6 tests. B — Non-linear workflows (§9.17): ParallelWorkflow (fan-out: one input → N models → merged) and BestOfNWorkflow (generate N → score with the served reward card → return best; inference-time scaling). The shapes a linear chain can't express. 4 tests. D — RL→distill flywheel (§9.18): run_flywheel RL-improves the base (NFT), then distills FROM the RL'd model (DMD2 teacher = RL'd policy) into a faster card, recording the base→rl→distilled provenance chain in RecipeSpec.parents. The distilled student is measurably closer to the RL'd teacher than the base; the distilled card serves few-step. training/flywheel.py. 4 tests. designv4 §9.17–§9.19 + layout/closing/counts (167 tests, 29 files). |
||
|
SolitaryThinker
|
34834eb4f4 |
[feat] #8b speculative (draft-verify) decoding — exact + lower-latency AR
The last audit stress test. A cheap draft model proposes K tokens, the target verifies them in one batched step, and SpeculativeARLoop accepts the matching prefix + one target correction — a variable accepted-length per round (a ragged AR loop the model owns). - Exactness: the emitted sequence equals the target's OWN greedy decode for any draft quality (every accepted token is one the target would produce; the correction is the target's token) — the speedup is free. - Speedup scales with accept rate: draft-agree 0.3→1x, 0.7→3x, 1.0→4x=K tokens/round (fewer verify_rounds, the expensive model's latency steps, for the same output). - Two components (draft + target) co-scheduled on one resident instance; each round an AR_TOKEN WorkUnit. models/speculative/ (loop+card+program); backend ToyTargetModel/ToyDraftModel with a shared length-dependent target formula (no degenerate fixed point). 5 tests; full suite 153 passed. designv4 §9.16 + counts (153 tests, 26 files). |
||
|
SolitaryThinker
|
bd36d69680 |
[feat] Five more stress tests: LTX-2 A/V, weight-sync, served reward, cache-dit, nested workflows
The remaining design_v3 probes (all except 8b speculative decoding). All fit with no new runtime primitive (148 tests pass). #6 LTX-2 joint audio+video (§9.11) — LTX-2 declared an audio_vae required_for t2vs but never used it; now a single 2-stage denoise carries a synchronized audio latent (conditioned on video), applies per-modality CFG (guidance_per_modality), and decodes via video VAE + audio VAE → video + audio. Gated on requesting audio, so the T2V path is byte-identical (existing tests untouched). ToyAudioVAE; build_ltx2_av_program. 5 tests. #4 Live weight-sync under in-flight serving (§9.14) — WeightSyncController makes the freeze → drain → transfer → bump version + invalidate → resume lifecycle explicit. Tests: a mid-flight swap corrupts (the hazard); draining first leaves the in-flight request bit-identical to baseline while a post-sync request reflects new weights; transformer-only sync, so the frozen text-encoder cache survives. The RL flywheel's hardest correctness. 3 tests. #5 Reward-model-as-a-served-card (§9.15) — a reward model is a card (scorer + a score loop emitting REWARD_BATCH units); ServedRewardScorer drop-in-replaces the numpy scorer so any RL method becomes RLHF/RLAIF with no method change. ToyRewardModel; models/reward/. 4 tests. #7 Content-adaptive control flow (§9.12) — CacheDiTDenoiseLoop (isolated WanDenoiseLoop subclass) reuses the cached velocity when predictions barely change (cache-dit skip) and early-exits on convergence — variable step count; interleave parity holds across ragged loops. models/adaptive/. 4 tests. #8a Nested workflows (§9.13) — a workflow stage can invoke another workflow (engine.run routes ids); requires/validate recurse; cycles caught at registration + a run-time guard (engine._wf_running). build_t2i_i2v_extend_workflow. 5 tests. designv4 §9.11–§9.15 + falsifier/layout/closing updates (148 tests, 25 files). Also removed a pre-existing unused import in ltx2/loop.py. |
||
|
SolitaryThinker
|
37544c68a5 |
[docs] Update v2/README to designv4 + current scope (127 tests)
- v2/README.md: point to designv4.md as the unified design (design_v3 as north star); refresh the scope table (joint/N-way/workflow RL, Qwen-Omni cascade, cross-model Workflow, tiled VAE co-scheduling, WorldModelSession), package layout (program/Workflow, runtime/session, the 7 methods, new model dirs), the demonstrated-stress-tests list (§9.3–§9.10), and counts (49→127, 20 files). Sessions moved out of "out of scope"; WebRTC wire stays out. - designv4.md: drop two intermediate absolute suite totals (milestone "91/97 passed") in favor of "zero regressions" so the only absolute count is the current 127 (intro + layout) — no stale numbers. |
||
|
SolitaryThinker
|
88e4a4355c |
[feat] Three stress tests: interactive sessions, workflow RL, heterogeneous co-scheduling
Targets the three design_v3 claims that were most load-bearing AND least exercised (sessions/realtime, training-plane boundary, the WorkUnit-generality falsifier). All fit with no new runtime primitive (127 tests pass). 1. Interactive world-model session (runtime/session.py, §9.8) — the Session plane had ZERO coverage. WorldModelSession drives the causal chunk_rollout loop as a long-lived session: persistent cross-request world state on the Session.kv_handle, frame streaming, transactional step-boundary cancellation (a cancelled act leaves the world resumable), no cross-session smearing. Added only a continuation seam to the chunk loop (init seeds context from a world_context slot; default empty = unchanged one-shot path). 5 tests. 2. End-to-end RL over a cross-model workflow (training/methods/workflow_rl.py, §9.9) — trains BOTH flux-t2i and wan-i2v from ONE final-video reward. Rolls out the whole workflow with SDE capture in both instances; the same final advantage drives FlowGRPO PPO on each stage's transformer; two WeightSyncPlans on two instances. The earlier model (T2I) is trained by a reward on the final video — end-to-end credit across a model boundary — proven causal by a control (constant reward => zero advantage => nothing moves). 4 tests. 3. Heterogeneous WorkUnit co-scheduling (models/tiled/, §9.10) — the §17 falsifier. VAETileLoop makes VAE decode a loop of VAE_TILE units; tiling is exact (== one-shot, C0), and VAE_TILE + DIFFUSION_STEP pipelines interleave bit-identically and co-run in one batch. Validates the mechanism; the economic half (does it pay) stays a port-time measurement. 4 tests. designv4 §9.8–§9.10 + falsifier/layout/closing updates (127 tests, 20 files). |
||
|
SolitaryThinker
|
f1dead1361 |
[feat] Register cross-model workflows as first-class named servables
Answers "what's the right way to name/register custom pipelines like T2I→I2V": treat a Workflow like a card — a stable namespaced id in the same servable namespace, declared dependencies, and a two-level registry. No new concepts; mirrors how cards are registered (and vllm-omni's pipeline_registry). - program/workflow.py: Workflow gains `requires` (the cards it composes, derived from stages) and `validate(engine)` (fail-fast if a required card is absent, P7). New WorkflowRegistry: declarative workflow_id -> builder catalog for out-of-tree/ad hoc use. - runtime/engine.py: `_workflows` registry + register_workflow (validates deps, rejects id collision with a model_id) + serves(); engine.run routes a request whose model_id is a workflow to workflow.run — addressable exactly like a model. Single-model hot path untouched. - runtime/async_engine.py + serving/server.py: serves() and /models include workflows (discoverable as servables). - models/__init__.py: declarative `_WORKFLOWS` catalog (the cross-model analog of _BUILDERS) + register_workflows() helper; build_image_video_engine now registers the workflow too. Adding a custom pipeline = one catalog line. - Naming convention: dotted/namespaced workflow_id (`image_video.t2i_i2v`), distinct from kebab model ids, collision-checked. Renamed from `t2i_then_i2v`. - tests (+5, 12 total in the file): addressable by id, requires/validate, id collision, registry catalog, register_workflows helper. Full suite 114 passed. - designv4 §9.6: the naming & registration convention documented. |
||
|
SolitaryThinker
|
cfc2ad7487 |
[feat] Cross-model T2I→I2V workflow + N-way joint RL over arbitrary experts
Two more pipelines stress-testing the design, plus a BAGEL-placement note in designv4. Both fit with no new runtime primitive (109 tests pass). Pipeline 1 — cross-model T2I→I2V (program/workflow.py, models/image_video/): - Realizes ProgramKind.WORKFLOW as a thin multi-instance orchestrator ABOVE the engine (the hot path stays single-instance). A Program composes one model's loops; a Workflow chains full engine.run calls across distinct cards, threading artifacts. (LTX-2 already covers same-card multi-stage; cross-model — FLUX→Wan — is the new capability the single-instance runner can't express.) - flux-t2i (text→image) and wan-i2v (text+image→video) cards; the I2V program folds the conditioning image into text_embeds so WanDenoiseLoop is unchanged. - Each model keeps its own interleave-parity guarantee (crossing instances is a Workflow boundary, not a loop step). 7 tests incl. video-depends-on-image. Pipeline 2 — N-way joint RL (training/methods/joint_multi_rl.py, models/multi_expert/): - JointMultiExpertRL generalizes UnifiedRLMethod (N=2) to N refiner LMs + a generator: one reward → one group advantage → N token-PG updates + 1 FlowGRPO PPO update, N+1 independent WeightSyncPlans. Proves the substrate was already N-ready (card holds N components/loops; per-component weight-sync; dict grad targets) — only the method body looped over two; now it loops over a list. - credit="per_expert" learns all N cleanly; credit="shared" (faithful to UniRL) works but is noisier — the honest multi-agent credit-assignment result, a reward-shaping choice, not a substrate limit. 6 tests (N=1,3,4; prompt-only). Fix — flow_sde_ml_velocity (loop/sampler.py): the toy FlowGRPO generator update targeted the velocity the model already produced (a no-op once guidance_scale=1 was set for the C2 identity; the unified generator moved only on ~1e-7 noise). The correct PG surrogate targets the max-likelihood velocity of the realized sample — nonzero at ratio==1. Both UniRL and N-way generators now learn for real; the C2 ratio==1 identity still holds (measured before the update). BAGEL: MoT/shared-weight (one transformer on both loops), same row as Cosmos3; real BAGEL's co-resident experts are expressible via the expert-routing policy (partial sharing) — captured in designv4 §2.3. |
||
|
SolitaryThinker
|
97fedaf432 |
[feat] Add Qwen-Omni thinker→talker→vocoder model (3 experts, 3 loops)
Ports vllm-omni's canonical qwen2_5_omni omni-speech cascade as a v2 card: a third weight-sharing topology — three disjoint experts (thinker, talker, vocoder) on three loop types (ar_decode → ar_decode → audio_decode) in one request, with chained cross-stage conditioning and streaming codec→waveform. vllm-omni runs these as three opaque request-scheduled stages; v2 makes every thinker token, talker token, and vocoder chunk a runtime-visible WorkUnit. - models/backend.py: ToyTalker (a genuinely distinct AR expert, weight-salted) + ToyVocoder (streaming code2wav: codec tokens → waveform chunks). - models/omni/vocoder_loop.py: VocoderLoop filling the pre-declared LoopKind.AUDIO_DECODE / WorkUnitKind.AUDIO_CHUNK slot. - models/omni/ar_loop.py: ARDecodeLoop gains a configurable prompt_slot so two chained AR loops don't collide on the prefill slot (thinker vs talker). - models/qwen_omni/: card (3 experts/3 loops) + program (tokenize → thinker → emit_text → thinker→talker full-payload hand-off → talker → talker→vocoder → vocoder → emit_audio). Cross-stage hand-offs are explicit Program nodes, the model-native form of vllm-omni's custom_process_input_func. - _enums.py: Capability.TEXT_TO_SPEECH. - tests/test_thinker_talker.py: 6 tests incl. three-loop interleave parity, cascade conditioning, AUDIO_CHUNK streaming. Full suite 97 passed. designv4.md: §2.3 topology table extended to four topologies; new §9.5 on the cascade; reference-synthesis + package layout updated. |
||
|
SolitaryThinker
|
3ec511419f |
[feat] UniRL/PromptRL joint LM+generator RL stress test + designv4
Stress-tests the v2 Card/Loop/Program design with a UniRL/PromptRL-style joint RL recipe: a prompt-refiner LM expert and a flow generator expert, two separate experts driven by two loop types in one request, both updated simultaneously from a single RL reward. - loop/sampler.py: flow_sde_step_with_logprob — FlowGRPO SDE rollout sampler (per-step Gaussian log-prob), distinct from the deterministic ODE serve step. - request/params.py: gated sde_rollout/sde_noise_scale on DiffusionParams so the serve path stays byte-identical (default ODE). - models/wan21/loop.py: gated SDE-rollout capture in WanDenoiseLoop.advance. - models/backend.py: ToyPromptRefiner — a real REINFORCE categorical policy (the Qwen role), separate weights from the generator. - models/unified/: the unified card+program — two disjoint experts (llm + transformer) on ar_decode + diffusion_denoise; the topological opposite of the Cosmos3 MoT card, same vocabulary. - training/methods/unified_rl.py: joint GRPO — one reward -> group advantage -> LM token policy gradient + DiT FlowGRPO PPO; two LRs; prompt-only/joint flag; reuses the shared diffusion loop for rollout. - training/weight_sync.py: WeightSyncPlan gains a component scope so the two experts version + cache-invalidate independently (LM sync never flushes the frozen text-encoder feature cache). - tests/test_unified_rl.py: 9 tests incl. likelihood-based C2 identity, the two-loop interleave parity gate, joint vs prompt-only. Full suite 91 passed. designv4.md: unified design doc reflecting v2 as built+tested, with the joint RL stress test as the validating case study (the design held — new card + new method, no new runtime primitive). |
||
|
SolitaryThinker
|
145082cf74 |
[refactor] rename package mini_fastvideo → v2
Directory rename (git mv, history preserved) plus rewrite of all references: absolute imports in tests, the zero-dep runner, docstrings, comments, and the README. No behavior change. Run: python3 -m pytest v2/tests/ -q ; python3 v2/run_tests.py ; python3 -m v2.examples |