SolitaryThinker
f1dc587c74
One resident MoT instance runs BOTH an ar_decode loop and a diffusion_denoise loop on shared weights (the §16 claim no DAG-of-engines can express), with both loops runtime-visible: the scheduler prices ar_token AND diffusion_step WorkUnits — unlike vllm-omni's opaque DIFFUSION stage the scheduler never sees inside. - ARDecodeLoop: token decode until EOS/max_tokens, paged text-KV — the omni AR pathway (loop/§5). - ToyMoTDiT: one module exposing an und pathway (ar_forward) AND a gen pathway (denoise __call__); ToyTokenizer. Binding both loops to one instance = shared weights, no duplication. - models/cosmos3/: tokenize → reason(ar_decode) → pack(tokens→conditioning) → diffusion_denoise → vae_decode; sound_vae declared optional_for non-t2vs (the lazy-component P8 fix, not an env-var hack). - models/bagel/: the canonical vllm-omni model — generate_text(ar_decode) → generate_image(diffusion), text+image outputs, both loops step-scheduled. - The diffusion loop is WanDenoiseLoop reused (one loop definition bound to the MoT module). - build_omni_engine() + an omni worked example; 7 omni tests (shared-instance, both-kinds-scheduled, interleave parity across loop types, lazy sound_vae). 61 tests pass (pytest + zero-dep runner).
33 lines
1.7 KiB
Python
33 lines
1.7 KiB
Python
"""Cosmos3 omni program (design_v3 §15b):
|
|
|
|
tokenize → reason (ar_decode) → pack(tokens→cond) → diffusion_denoise → vae_decode
|
|
|
|
The reasoner and the denoiser bind the SAME resident ``transformer``; the scheduler is the mode
|
|
multiplexer. This is the workload no DAG-of-engines can express (splitting them doubles the weights
|
|
and severs the shared state).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from ...program import ComponentNode, ModelLoopNode, Program, ProgramKind
|
|
from ..omni import emit_text_node, pack_cond_from_tokens, tokenize_node, vae_decode_node
|
|
|
|
|
|
def build_cosmos3_program() -> Program:
|
|
return Program(
|
|
program_id="cosmos3.omni", kind=ProgramKind.INLINE,
|
|
nodes=[
|
|
ComponentNode("tokenize", fn=tokenize_node, writes=("prompt_tokens",)),
|
|
ModelLoopNode("reason", loop_id="ar_decode", output_slot="reason_out",
|
|
reads=("prompt_tokens",), writes=("reason_out",)),
|
|
ComponentNode("emit_text", fn=emit_text_node("reason_out", "reason_text"),
|
|
reads=("reason_out",), writes=("reason_text",)),
|
|
ComponentNode("pack", fn=pack_cond_from_tokens("reason_out"),
|
|
reads=("reason_out",), writes=("text_embeds", "neg_text_embeds")),
|
|
ModelLoopNode("denoise", loop_id="diffusion_denoise", output_slot="denoise_out",
|
|
reads=("text_embeds",), writes=("denoise_out",)),
|
|
ComponentNode("vae_decode", fn=vae_decode_node("denoise_out", "video"),
|
|
reads=("denoise_out",), writes=("video",)),
|
|
],
|
|
output_artifacts={"video": "video", "text": "reason_text", "latents": "denoise_out"},
|
|
).validate()
|