Second round of review follow-ups for the next-free-GPU dispatcher.
Ambiguous acceptance is no longer inferred from aggregate queue depth. The
dispatcher asks the worker about the specific run: a prompt carrying one
of its job ids in the worker's running or pending queue, or in its recent
history, means the run was accepted and its result is collected (or
reported) there; only after consecutive checks find no trace of it is the
original prompt restored. A worker that cannot be asked keeps the run on
hold. The same question decides when a held delegate prompt must be queued
because the worker finished (or lost) the run without delivering.
An eagerly dispatched run whose acknowledgement was lost is now collected
after the worker delivers: the record remembers that no master prompt was
ever queued for it, so acceptance queues its collector instead of treating
it like a collector that is already executing.
Cancellation now reaches the queue-insertion boundary: queue_prompt_payload
takes an insert_if guard that is checked under the queue mutex right before
the item is put, after validation has yielded. The delegate requeue passes
the run's cancellation flag, so a clear or delete that lands during
validation refuses the insertion instead of being discovered afterwards.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
With load_balance enabled, a run was pinned to a participant at queue time
by comparing worker queue depths, so a faster GPU went idle while a slower
one still held a backlog. Runs queued while every participant is busy now
stay unassigned in the master's prompt queue; a dispatcher watches the
candidate workers and hands the oldest waiting run to whichever one frees
up first, re-queueing the master side as a delegate prompt under the same
prompt id. A participating master takes whatever waiting run its executor
reaches first. Idle participants are still used immediately.
Upscale workflows and serverless pools keep the previous queue-depth
selection.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
