Lifecycle:
- Queue clears and deletes leave a run whose master prompt is currently
executing alone, as ComfyUI leaves the running prompt alone; previously an
executing orchestrator-only head run could be cancelled out from under its
waiting Collector.
- The sweep now also drops worker-assigned runs whose queued delegate prompt
left the queue without the Collector finishing, releasing their result
queues instead of leaking them and polling forever.
- Completed runs release their result queues, and orchestration no longer
pre-creates result queues when load balance picks the master alone.
Transport:
- A websocket timeout during connect, before anything was sent, is a plain
failure the dispatcher may retry elsewhere; only timeouts after the prompt
was sent remain AmbiguousDispatchError.
Efficiency and cleanup:
- One WorkerSnapshot per worker per cycle answers every run's "is my job
there" question (queue first, history only when needed).
- Sweep and cancellation snapshot queued/running prompt ids once per pass.
- probe_worker_queues reuses dispatch._probe_worker_queue; the dead
aggregate idle/busy probe helper is gone.
Rebased on "never replay load-balanced runs on missing history": an
unresolved submission stays on hold until delivery or positive evidence,
so there is no rejection path to fix and no bounded hold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review follow-ups for the next-free-GPU dispatcher.
Ambiguous acceptance: the worker transport now raises a typed
AmbiguousDispatchError when the request may have reached the worker but no
acknowledgement came back (websocket closed or timed out before the ack,
unreadable /prompt acknowledgement, request failure after it could have
been received). A connection refusal or an error status stays a definite
failure. The dispatcher no longer restores such a run to the master queue:
it is held as unresolved, claimable by nobody, until the worker delivers,
is seen busy (accepted: collected as usual) or is seen idle without
delivering (never accepted: the original prompt goes back in its place).
While the worker is unreachable the run stays on hold. The immediate
dispatch path does the same and still reports the error to the caller.
Cancellation during a hand-off: the dispatcher hooks the master queue's
wipe_queue and delete_queue_item so a clear or delete reaches runs that
are out of the queue for a hand-off. Preparation and submission are now
separate steps; a run cancelled before submission is dropped together with
its result queues and never sent, while work a worker already accepted
stays owned until its result arrives or the worker is seen done, and is
then discarded instead of reappearing. A Collector that fails or is
interrupted while waiting for its assignment cancels its run the same way,
so the preallocated result queue is removed and an in-flight dispatch
stops before submission. Discarded job ids are remembered so a late
worker cannot recreate their result queue through /distributed/prepare_job.
Also updates the frozen Collector load_balance tooltip in the v1 node
contract fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Preserve source values without extra wires, collect explicit per-item overrides, and expose ordered bundles through Get Value. Keep legacy sockets and no-override transport compatible.
* feat: add Runpod serverless image collector pools
* fix: prevent worker settings clipping and remove preview copy
* fix: initialize worker settings before mounting the form
* fix: validate complete serverless requests against 10 MB limit
* fix: simplify serverless cards and keep status dots circular
* fix: match serverless endpoint box to master card
* fix: preserve serverless capacity and recovery across job failures
Retain durable reservations until provider termination or explicit recovery, protect unresolved pools and credentials, and add scoped reconciliation. Consolidate lifecycle and secret policies and split prompt/media stages with regression coverage.
* feat: add dynamic per-image list queue with Start and End nodes
* fix(image-list): reject upstream data bypasses around Start
* fix(image-list): make optional End text a connection-only input
* fix(image-list): isolate unavailable workers during preflight
* fix(image-list): bound acknowledgements and configure run deadline
* fix(media-sync): preserve output and temp source roots
* fix(image-list): own job lifecycle and preserve worker preparation
* feat(image-list): let master participate unless orchestrator-only
* feat(image-list)!: replace fixed Start extras with growable value_N slots
Distributed Image List Start no longer has reference_image, text,
strength, seed or per-slot mode dropdowns. It now takes images, mask and
up to 10 generic value_N inputs (IMAGE, MASK, STRING, INT, FLOAT,
BOOLEAN) that grow via native V3 Autogrow, each with a matching output.
- One value is shared by every image; one per image pairs in order; any
other count fails before dispatch. Only images is split into frames.
- Outputs are declared up front (image, mask, item_index, context,
value_1..value_10). web/imageListStart.js hides unused value outputs,
mirrors input types, and follows Autogrow compaction so downstream
links keep their value.
- Image-list protocol bumped to v2; older workers fail preflight.
Breaking: saved workflows using the old Start inputs must recreate Start.
* fix(serverless): disable Clear saved API key while the worker is enabled
The checkbox read "Clear saved API key (disable pool first)", but the
panel never defines "pool" and saving with it ticked on an enabled worker
was rejected by the backend.
- Disable the checkbox while the worker is enabled, with a tooltip saying
to turn the worker off first. Also disable it when no key is saved.
- Use "worker" instead of "pool" in user-facing serverless errors, the
serverless guide and the README entry.
* feat(serverless): replace Concurrent jobs with a single Images per run setting
- Remove the separate Concurrent jobs field; rename Jobs per run to
Images per run (1-16) with a tooltip explaining that Runpod's max
workers decides how many GPUs run them.
- Backend uses a fixed per-worker limit of 16 outstanding jobs, so all
images of a run are submitted at once. Unresolved jobs still count
against it. The limit is constant, so it never has to change while
jobs are unresolved.
- Legacy 'concurrency' in saved configs and older browser tabs is
accepted and ignored/dropped; it no longer triggers the busy guard.
- Revert the enable toggle and show the reason when the backend rejects it.
- Trim a pasted api.runpod.ai/v2/<id> URL to the endpoint ID; plainer
endpoint-ID error otherwise.
- Confirm before a type change deletes a saved Runpod API key.
- Update docs and tests.
* feat: add Runpod serverless image collector pools
* fix: prevent worker settings clipping and remove preview copy
* fix: initialize worker settings before mounting the form
* fix: validate complete serverless requests against 10 MB limit
* fix: simplify serverless cards and keep status dots circular
* fix: match serverless endpoint box to master card
* fix: preserve serverless capacity and recovery across job failures
Retain durable reservations until provider termination or explicit recovery, protect unresolved pools and credentials, and add scoped reconciliation. Consolidate lifecycle and secret policies and split prompt/media stages with regression coverage.
* fix(serverless): disable Clear saved API key while the worker is enabled
The checkbox read "Clear saved API key (disable pool first)", but the
panel never defines "pool" and saving with it ticked on an enabled worker
was rejected by the backend.
- Disable the checkbox while the worker is enabled, with a tooltip saying
to turn the worker off first. Also disable it when no key is saved.
- Use "worker" instead of "pool" in user-facing serverless errors, the
serverless guide and the README entry.
* feat(serverless): replace Concurrent jobs with a single Images per run setting
- Remove the separate Concurrent jobs field; rename Jobs per run to
Images per run (1-16) with a tooltip explaining that Runpod's max
workers decides how many GPUs run them.
- Backend uses a fixed per-worker limit of 16 outstanding jobs, so all
images of a run are submitted at once. Unresolved jobs still count
against it. The limit is constant, so it never has to change while
jobs are unresolved.
- Legacy 'concurrency' in saved configs and older browser tabs is
accepted and ignored/dropped; it no longer triggers the busy guard.
- Revert the enable toggle and show the reason when the backend rejects it.
- Trim a pasted api.runpod.ai/v2/<id> URL to the endpoint ID; plainer
endpoint-ID error otherwise.
- Confirm before a type change deletes a saved Runpod API key.
- Update docs and tests.
