SolitaryThinker
8240970ea6
Builds the full serving layer the design files specify, instead of deferring it to Dynamo: - transport/ (§7.3): pluggable Connectors (in-proc zero-copy / SHM-fake copy) with chunk_ready readiness (vllm-omni) AND credit-based flow control (sglang-omni Relay); KVConnector protocol shape; TransferManifest. - runtime/ (§6, §13; plan M3/M4): AsyncEngine — request queue, lifecycle state machine (waiting→running→completed/cancelled/failed), live AsyncIterator[OmniEvent] streaming, common-path cancellation, step-level concurrency. RolePool + DisaggregatedRunner (encoder→denoiser→decoder, capacity-aware dispatch, cross-pool transfers via connectors); disaggregated output is bit-identical to inline. No-progress detection (no busy-spin). - deploy/ (§14, §6.3.5-6): DeploymentCard; OUR OWN LocalFleet (discovery, health/drain, least-loaded / cost-model / sticky-affinity routing) so we never rely on Dynamo; DynamoWorkerAdapter + FakeDynamoRuntime export the SAME card + cost model so Dynamo CAN front us — one object, two consumers. - serving/ (§6.3.5, §12): framework-free stdlib-asyncio OpenAI server (our own version of the vllm-omni pattern): /v1/chat/completions (SSE), /v1/images/generations, /v1/videos (async job+poll) + /v1/videos/sync, /v1/models, /health, /metrics. A thin shim over the STEP-scheduled engine — the runtime-visible loop scheduler vllm-omni's request-scheduled opaque DIFFUSION stage lacks. 15 serving tests (real-socket HTTP+SSE via stdlib asyncio, disagg==inline, fleet routing, Dynamo contract, cancellation); 76 tests pass total (pytest + zero-dep runner). ~7900 LOC.
22 lines
854 B
Python
22 lines
854 B
Python
"""Execution plane — engine + scheduler (design_v3 §6)."""
|
|
from __future__ import annotations
|
|
|
|
from .async_engine import AsyncEngine, RequestState
|
|
from .context import RuntimeLoopContext
|
|
from .disaggregated import DisaggregatedRunner
|
|
from .engine import Engine, ProgramRunner
|
|
from .pools import DeployConfig, PoolSet, RolePool, RolePoolSpec, wan_t2v_disaggregated
|
|
from .scheduler import (
|
|
AdmissionController,
|
|
AdmissionInfeasible,
|
|
BatchScheduler,
|
|
SchedulerMetrics,
|
|
StepTicket,
|
|
WorkUnit,
|
|
)
|
|
|
|
__all__ = ["Engine", "ProgramRunner", "RuntimeLoopContext", "AdmissionController",
|
|
"AdmissionInfeasible", "BatchScheduler", "SchedulerMetrics", "StepTicket", "WorkUnit",
|
|
"AsyncEngine", "RequestState", "DisaggregatedRunner",
|
|
"DeployConfig", "PoolSet", "RolePool", "RolePoolSpec", "wan_t2v_disaggregated"]
|