fix: audit follow-ups for the next-free-GPU dispatcher
Lifecycle: - Queue clears and deletes leave a run whose master prompt is currently executing alone, as ComfyUI leaves the running prompt alone; previously an executing orchestrator-only head run could be cancelled out from under its waiting Collector. - The sweep now also drops worker-assigned runs whose queued delegate prompt left the queue without the Collector finishing, releasing their result queues instead of leaking them and polling forever. - Completed runs release their result queues, and orchestration no longer pre-creates result queues when load balance picks the master alone. Transport: - A websocket timeout during connect, before anything was sent, is a plain failure the dispatcher may retry elsewhere; only timeouts after the prompt was sent remain AmbiguousDispatchError. Efficiency and cleanup: - One WorkerSnapshot per worker per cycle answers every run's "is my job there" question (queue first, history only when needed). - Sweep and cancellation snapshot queued/running prompt ids once per pass. - probe_worker_queues reuses dispatch._probe_worker_queue; the dead aggregate idle/busy probe helper is gone. Rebased on "never replay load-balanced runs on missing history": an unresolved submission stays on hold until delivery or positive evidence, so there is no rejection path to fix and no bounded hold. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
62743bf8e1
commit
09c4554dd0
@@ -566,6 +566,9 @@ async def orchestrate_distributed_execution(
|
||||
# collect-only prompt once the worker delivers, not now, so the master
|
||||
# executor stays free for the next full workflow.
|
||||
hold_delegate_for_worker = None
|
||||
# Load balance picked the master alone: no worker will ever deliver, so
|
||||
# no result queues are needed for this run.
|
||||
master_runs_alone = False
|
||||
if load_balance_requested and not image_list_mode:
|
||||
candidate_workers = list(active_workers)
|
||||
if not delegate_master:
|
||||
@@ -617,6 +620,7 @@ async def orchestrate_distributed_execution(
|
||||
# Master selected as least busy; run master workload only.
|
||||
active_workers = []
|
||||
delegate_master = False
|
||||
master_runs_alone = True
|
||||
trace_debug(
|
||||
execution_trace_id,
|
||||
"Load-balance selected master for execution (workers skipped).",
|
||||
@@ -711,7 +715,7 @@ async def orchestrate_distributed_execution(
|
||||
manager.ensure_capacity(len(serverless_jobs))
|
||||
|
||||
# Do not leave collector queues behind when offline compilation/media validation fails.
|
||||
if not image_list_mode:
|
||||
if not image_list_mode and not master_runs_alone:
|
||||
for job_id in job_id_map.values():
|
||||
await _ensure_distributed_queue(job_id)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user