fix: audit follow-ups for the next-free-GPU dispatcher

Lifecycle:
- Queue clears and deletes leave a run whose master prompt is currently
  executing alone, as ComfyUI leaves the running prompt alone; previously an
  executing orchestrator-only head run could be cancelled out from under its
  waiting Collector.
- The sweep now also drops worker-assigned runs whose queued delegate prompt
  left the queue without the Collector finishing, releasing their result
  queues instead of leaking them and polling forever.
- Completed runs release their result queues, and orchestration no longer
  pre-creates result queues when load balance picks the master alone.

Transport:
- A websocket timeout during connect, before anything was sent, is a plain
  failure the dispatcher may retry elsewhere; only timeouts after the prompt
  was sent remain AmbiguousDispatchError.

Efficiency and cleanup:
- One WorkerSnapshot per worker per cycle answers every run's "is my job
  there" question (queue first, history only when needed).
- Sweep and cancellation snapshot queued/running prompt ids once per pass.
- probe_worker_queues reuses dispatch._probe_worker_queue; the dead
  aggregate idle/busy probe helper is gone.

Rebased on "never replay load-balanced runs on missing history": an
unresolved submission stays on hold until delivery or positive evidence,
so there is no rejection path to fix and no bounded hold.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
speedyrulz
2026-10-20 20:54:55 -04:00
co-authored by Claude Fable 5.1
parent 62743bf8e1
commit 09c4554dd0
6 changed files with 321 additions and 110 deletions
+6
View File
@@ -98,6 +98,12 @@ async def _dispatch_via_websocket(worker_url, payload, client_id, timeout=60.0):
try:
async with asyncio.timeout_at(deadline):
ws = await session.ws_connect(ws_url, heartbeat=20)
except asyncio.TimeoutError as exc:
# Nothing was sent: a definite failure the caller may retry elsewhere.
raise RuntimeError("Worker websocket connect timed out before dispatch; nothing was sent") from exc
try:
async with asyncio.timeout_at(deadline):
await ws.send_json(ws_payload)
while True:
msg = await ws.receive()