fix: settle unacknowledged submissions by asking the worker about the run itself

Second round of review follow-ups for the next-free-GPU dispatcher.

Ambiguous acceptance is no longer inferred from aggregate queue depth. The
dispatcher asks the worker about the specific run: a prompt carrying one
of its job ids in the worker's running or pending queue, or in its recent
history, means the run was accepted and its result is collected (or
reported) there; only after consecutive checks find no trace of it is the
original prompt restored. A worker that cannot be asked keeps the run on
hold. The same question decides when a held delegate prompt must be queued
because the worker finished (or lost) the run without delivering.

An eagerly dispatched run whose acknowledgement was lost is now collected
after the worker delivers: the record remembers that no master prompt was
ever queued for it, so acceptance queues its collector instead of treating
it like a collector that is already executing.

Cancellation now reaches the queue-insertion boundary: queue_prompt_payload
takes an insert_if guard that is checked under the queue mutex right before
the item is put, after validation has yielded. The delegate requeue passes
the run's cancellation flag, so a clear or delete that lands during
validation refuses the insertion instead of being discovered afterwards.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
speedyrulz
2026-10-20 19:50:59 -04:00
co-authored by Claude Fable 5.1
parent 31d66b0dc8
commit 44cfb46a72
6 changed files with 362 additions and 77 deletions
+6 -1
View File
@@ -380,7 +380,12 @@ def _register_waiting_run(prompt_index, job_id_map, candidates, *, master_may_ru
delegate_prompt = _build_master_prompt(
prompt_index, [worker["id"]], job_id_map, master_url, True, workflow_meta
)
await queue_prompt_payload(delegate_prompt, workflow_meta, client_id, prompt_id=prompt_id)
# Validation yields to the loop; a clear or delete that lands meanwhile
# must stop the insertion itself, not just the attempt.
await queue_prompt_payload(
delegate_prompt, workflow_meta, client_id, prompt_id=prompt_id,
insert_if=lambda: not record.cancelled,
)
record = PendingRun(
prompt_id,