WorkerSnapshot tracked a single failure flag, so a failed /history request
made every later query in the same cycle return None, even for runs the
already-fetched /queue showed as running. A held run that was still
running then accumulated missed probes and had its collect-only delegate
queued before delivery, parking a participating master on it.
Queue and history failures are now tracked separately: queue evidence is
always consulted first, a history failure only affects runs that need
history, and neither endpoint is retried within the snapshot's cycle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lifecycle:
- Queue clears and deletes leave a run whose master prompt is currently
executing alone, as ComfyUI leaves the running prompt alone; previously an
executing orchestrator-only head run could be cancelled out from under its
waiting Collector.
- The sweep now also drops worker-assigned runs whose queued delegate prompt
left the queue without the Collector finishing, releasing their result
queues instead of leaking them and polling forever.
- Completed runs release their result queues, and orchestration no longer
pre-creates result queues when load balance picks the master alone.
Transport:
- A websocket timeout during connect, before anything was sent, is a plain
failure the dispatcher may retry elsewhere; only timeouts after the prompt
was sent remain AmbiguousDispatchError.
Efficiency and cleanup:
- One WorkerSnapshot per worker per cycle answers every run's "is my job
there" question (queue first, history only when needed).
- Sweep and cancellation snapshot queued/running prompt ids once per pass.
- probe_worker_queues reuses dispatch._probe_worker_queue; the dead
aggregate idle/busy probe helper is gone.
Rebased on "never replay load-balanced runs on missing history": an
unresolved submission stays on hold until delivery or positive evidence,
so there is no rejection path to fix and no bounded hold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keep ambiguous submissions tracked until positive job-specific evidence or result delivery confirms acceptance. Bounded or cleared history cannot prove rejection, even on repeated checks.
Add regressions using the real worker history lookup, retain late-result and cancellation coverage, and document the intentional unresolved hold.
Second round of review follow-ups for the next-free-GPU dispatcher.
Ambiguous acceptance is no longer inferred from aggregate queue depth. The
dispatcher asks the worker about the specific run: a prompt carrying one
of its job ids in the worker's running or pending queue, or in its recent
history, means the run was accepted and its result is collected (or
reported) there; only after consecutive checks find no trace of it is the
original prompt restored. A worker that cannot be asked keeps the run on
hold. The same question decides when a held delegate prompt must be queued
because the worker finished (or lost) the run without delivering.
An eagerly dispatched run whose acknowledgement was lost is now collected
after the worker delivers: the record remembers that no master prompt was
ever queued for it, so acceptance queues its collector instead of treating
it like a collector that is already executing.
Cancellation now reaches the queue-insertion boundary: queue_prompt_payload
takes an insert_if guard that is checked under the queue mutex right before
the item is put, after validation has yielded. The delegate requeue passes
the run's cancellation flag, so a clear or delete that lands during
validation refuses the insertion instead of being discovered afterwards.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review follow-ups for the next-free-GPU dispatcher.
Ambiguous acceptance: the worker transport now raises a typed
AmbiguousDispatchError when the request may have reached the worker but no
acknowledgement came back (websocket closed or timed out before the ack,
unreadable /prompt acknowledgement, request failure after it could have
been received). A connection refusal or an error status stays a definite
failure. The dispatcher no longer restores such a run to the master queue:
it is held as unresolved, claimable by nobody, until the worker delivers,
is seen busy (accepted: collected as usual) or is seen idle without
delivering (never accepted: the original prompt goes back in its place).
While the worker is unreachable the run stays on hold. The immediate
dispatch path does the same and still reports the error to the caller.
Cancellation during a hand-off: the dispatcher hooks the master queue's
wipe_queue and delete_queue_item so a clear or delete reaches runs that
are out of the queue for a hand-off. Preparation and submission are now
separate steps; a run cancelled before submission is dropped together with
its result queues and never sent, while work a worker already accepted
stays owned until its result arrives or the worker is seen done, and is
then discarded instead of reappearing. A Collector that fails or is
interrupted while waiting for its assignment cancels its run the same way,
so the preallocated result queue is removed and an in-flight dispatch
stops before submission. Discarded job ids are remembered so a late
worker cannot recreate their result queue through /distributed/prepare_job.
Also updates the frozen Collector load_balance tooltip in the v1 node
contract fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When a run went to a worker, the master's collect-only delegate prompt was
queued at once. ComfyUI's single executor then sat on that prompt waiting
for the worker, so every full workflow queued behind it waited too: with
one worker and a participating master, the master GPU only started once
the worker's first job finished.
On a participating master the delegate prompt is now held back and queued
only once the worker posts its final result (job_complete is_last), so the
master executor keeps pulling full workflows meanwhile. If the worker is
seen idle or unreachable for several probes without delivering, the
delegate is queued anyway so the Collector's timeout path reports it. Both
the immediate dispatch to an idle worker and dispatcher claims use this;
an orchestrator-only master, which has nothing else to run, still queues
the delegate immediately so the run stays visible in its queue.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two gaps in the next-free-GPU dispatcher:
A run was registered with the dispatcher before the master prompt reached
the ComfyUI queue. Prompt validation yields to the event loop, so a sweep
in that window saw the run as neither queued nor running and dropped it.
Runs are now registered unqueued and ignored by sweep and claim until the
orchestrator marks them queued; a failed queue still discards them.
When a worker had been dispatched but the master's delegate prompt could
not be re-queued, the run's record was removed, leaving nothing to collect
the worker's result. The record now stays assigned to the worker with the
requeue pending, and the dispatcher retries it with backoff until the
delegate prompt is back on the master queue; tracking ends only when the
Collector finishes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
With load_balance enabled, a run was pinned to a participant at queue time
by comparing worker queue depths, so a faster GPU went idle while a slower
one still held a backlog. Runs queued while every participant is busy now
stay unassigned in the master's prompt queue; a dispatcher watches the
candidate workers and hands the oldest waiting run to whichever one frees
up first, re-queueing the master side as a delegate prompt under the same
prompt id. A participating master takes whatever waiting run its executor
reaches first. Idle participants are still used immediately.
Upscale workflows and serverless pools keep the previous queue-depth
selection.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
