fix: audit follow-ups for the next-free-GPU dispatcher

Lifecycle:
- Queue clears and deletes leave a run whose master prompt is currently
  executing alone, as ComfyUI leaves the running prompt alone; previously an
  executing orchestrator-only head run could be cancelled out from under its
  waiting Collector.
- The sweep now also drops worker-assigned runs whose queued delegate prompt
  left the queue without the Collector finishing, releasing their result
  queues instead of leaking them and polling forever.
- Completed runs release their result queues, and orchestration no longer
  pre-creates result queues when load balance picks the master alone.

Transport:
- A websocket timeout during connect, before anything was sent, is a plain
  failure the dispatcher may retry elsewhere; only timeouts after the prompt
  was sent remain AmbiguousDispatchError.

Efficiency and cleanup:
- One WorkerSnapshot per worker per cycle answers every run's "is my job
  there" question (queue first, history only when needed).
- Sweep and cancellation snapshot queued/running prompt ids once per pass.
- probe_worker_queues reuses dispatch._probe_worker_queue; the dead
  aggregate idle/busy probe helper is gone.

Rebased on "never replay load-balanced runs on missing history": an
unresolved submission stays on hold until delivery or positive evidence,
so there is no rejection path to fix and no bounded hold.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
speedyrulz
2026-10-20 20:54:55 -04:00
co-authored by Claude Fable 5.1
parent 62743bf8e1
commit 09c4554dd0
6 changed files with 321 additions and 110 deletions
+5 -1
View File
@@ -566,6 +566,9 @@ async def orchestrate_distributed_execution(
# collect-only prompt once the worker delivers, not now, so the master
# executor stays free for the next full workflow.
hold_delegate_for_worker = None
# Load balance picked the master alone: no worker will ever deliver, so
# no result queues are needed for this run.
master_runs_alone = False
if load_balance_requested and not image_list_mode:
candidate_workers = list(active_workers)
if not delegate_master:
@@ -617,6 +620,7 @@ async def orchestrate_distributed_execution(
# Master selected as least busy; run master workload only.
active_workers = []
delegate_master = False
master_runs_alone = True
trace_debug(
execution_trace_id,
"Load-balance selected master for execution (workers skipped).",
@@ -711,7 +715,7 @@ async def orchestrate_distributed_execution(
manager.ensure_capacity(len(serverless_jobs))
# Do not leave collector queues behind when offline compilation/media validation fails.
if not image_list_mode:
if not image_list_mode and not master_runs_alone:
for job_id in job_id_map.values():
await _ensure_distributed_queue(job_id)