fix: settle unacknowledged submissions by asking the worker about the run itself
Second round of review follow-ups for the next-free-GPU dispatcher. Ambiguous acceptance is no longer inferred from aggregate queue depth. The dispatcher asks the worker about the specific run: a prompt carrying one of its job ids in the worker's running or pending queue, or in its recent history, means the run was accepted and its result is collected (or reported) there; only after consecutive checks find no trace of it is the original prompt restored. A worker that cannot be asked keeps the run on hold. The same question decides when a held delegate prompt must be queued because the worker finished (or lost) the run without delivering. An eagerly dispatched run whose acknowledgement was lost is now collected after the worker delivers: the record remembers that no master prompt was ever queued for it, so acceptance queues its collector instead of treating it like a collector that is already executing. Cancellation now reaches the queue-insertion boundary: queue_prompt_payload takes an insert_if guard that is checked under the queue mutex right before the item is put, after validation has yielded. The delegate requeue passes the run's cancellation flag, so a clear or delete that lands during validation refuses the insertion instead of being discovered afterwards. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
31d66b0dc8
commit
44cfb46a72
@@ -380,7 +380,12 @@ def _register_waiting_run(prompt_index, job_id_map, candidates, *, master_may_ru
|
||||
delegate_prompt = _build_master_prompt(
|
||||
prompt_index, [worker["id"]], job_id_map, master_url, True, workflow_meta
|
||||
)
|
||||
await queue_prompt_payload(delegate_prompt, workflow_meta, client_id, prompt_id=prompt_id)
|
||||
# Validation yields to the loop; a clear or delete that lands meanwhile
|
||||
# must stop the insertion itself, not just the attempt.
|
||||
await queue_prompt_payload(
|
||||
delegate_prompt, workflow_meta, client_id, prompt_id=prompt_id,
|
||||
insert_if=lambda: not record.cancelled,
|
||||
)
|
||||
|
||||
record = PendingRun(
|
||||
prompt_id,
|
||||
|
||||
Reference in New Issue
Block a user