Lifecycle:
- Queue clears and deletes leave a run whose master prompt is currently
executing alone, as ComfyUI leaves the running prompt alone; previously an
executing orchestrator-only head run could be cancelled out from under its
waiting Collector.
- The sweep now also drops worker-assigned runs whose queued delegate prompt
left the queue without the Collector finishing, releasing their result
queues instead of leaking them and polling forever.
- Completed runs release their result queues, and orchestration no longer
pre-creates result queues when load balance picks the master alone.
Transport:
- A websocket timeout during connect, before anything was sent, is a plain
failure the dispatcher may retry elsewhere; only timeouts after the prompt
was sent remain AmbiguousDispatchError.
Efficiency and cleanup:
- One WorkerSnapshot per worker per cycle answers every run's "is my job
there" question (queue first, history only when needed).
- Sweep and cancellation snapshot queued/running prompt ids once per pass.
- probe_worker_queues reuses dispatch._probe_worker_queue; the dead
aggregate idle/busy probe helper is gone.
Rebased on "never replay load-balanced runs on missing history": an
unresolved submission stays on hold until delivery or positive evidence,
so there is no rejection path to fix and no bounded hold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Second round of review follow-ups for the next-free-GPU dispatcher.
Ambiguous acceptance is no longer inferred from aggregate queue depth. The
dispatcher asks the worker about the specific run: a prompt carrying one
of its job ids in the worker's running or pending queue, or in its recent
history, means the run was accepted and its result is collected (or
reported) there; only after consecutive checks find no trace of it is the
original prompt restored. A worker that cannot be asked keeps the run on
hold. The same question decides when a held delegate prompt must be queued
because the worker finished (or lost) the run without delivering.
An eagerly dispatched run whose acknowledgement was lost is now collected
after the worker delivers: the record remembers that no master prompt was
ever queued for it, so acceptance queues its collector instead of treating
it like a collector that is already executing.
Cancellation now reaches the queue-insertion boundary: queue_prompt_payload
takes an insert_if guard that is checked under the queue mutex right before
the item is put, after validation has yielded. The delegate requeue passes
the run's cancellation flag, so a clear or delete that lands during
validation refuses the insertion instead of being discovered afterwards.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review follow-ups for the next-free-GPU dispatcher.
Ambiguous acceptance: the worker transport now raises a typed
AmbiguousDispatchError when the request may have reached the worker but no
acknowledgement came back (websocket closed or timed out before the ack,
unreadable /prompt acknowledgement, request failure after it could have
been received). A connection refusal or an error status stays a definite
failure. The dispatcher no longer restores such a run to the master queue:
it is held as unresolved, claimable by nobody, until the worker delivers,
is seen busy (accepted: collected as usual) or is seen idle without
delivering (never accepted: the original prompt goes back in its place).
While the worker is unreachable the run stays on hold. The immediate
dispatch path does the same and still reports the error to the caller.
Cancellation during a hand-off: the dispatcher hooks the master queue's
wipe_queue and delete_queue_item so a clear or delete reaches runs that
are out of the queue for a hand-off. Preparation and submission are now
separate steps; a run cancelled before submission is dropped together with
its result queues and never sent, while work a worker already accepted
stays owned until its result arrives or the worker is seen done, and is
then discarded instead of reappearing. A Collector that fails or is
interrupted while waiting for its assignment cancels its run the same way,
so the preallocated result queue is removed and an in-flight dispatch
stops before submission. Discarded job ids are remembered so a late
worker cannot recreate their result queue through /distributed/prepare_job.
Also updates the frozen Collector load_balance tooltip in the v1 node
contract fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When a run went to a worker, the master's collect-only delegate prompt was
queued at once. ComfyUI's single executor then sat on that prompt waiting
for the worker, so every full workflow queued behind it waited too: with
one worker and a participating master, the master GPU only started once
the worker's first job finished.
On a participating master the delegate prompt is now held back and queued
only once the worker posts its final result (job_complete is_last), so the
master executor keeps pulling full workflows meanwhile. If the worker is
seen idle or unreachable for several probes without delivering, the
delegate is queued anyway so the Collector's timeout path reports it. Both
the immediate dispatch to an idle worker and dispatcher claims use this;
an orchestrator-only master, which has nothing else to run, still queues
the delegate immediately so the run stays visible in its queue.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two gaps in the next-free-GPU dispatcher:
A run was registered with the dispatcher before the master prompt reached
the ComfyUI queue. Prompt validation yields to the event loop, so a sweep
in that window saw the run as neither queued nor running and dropped it.
Runs are now registered unqueued and ignored by sweep and claim until the
orchestrator marks them queued; a failed queue still discards them.
When a worker had been dispatched but the master's delegate prompt could
not be re-queued, the run's record was removed, leaving nothing to collect
the worker's result. The record now stays assigned to the worker with the
requeue pending, and the dispatcher retries it with backoff until the
delegate prompt is back on the master queue; tracking ends only when the
Collector finishes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
With load_balance enabled, a run was pinned to a participant at queue time
by comparing worker queue depths, so a faster GPU went idle while a slower
one still held a backlog. Runs queued while every participant is busy now
stay unassigned in the master's prompt queue; a dispatcher watches the
candidate workers and hands the oldest waiting run to whichever one frees
up first, re-queueing the master side as a delegate prompt under the same
prompt id. A participating master takes whatever waiting run its executor
reaches first. Idle participants are still used immediately.
Upscale workflows and serverless pools keep the previous queue-depth
selection.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Use the existing images/audio Collector for typed serverless pools. Preserve saved legacy_png defaults, map them to serverless png, and retain every transfer format with exact opt-in. Remove the separate Media Collector.
Send direct exact frames through bounded raw tensor multipart transport without 8-bit fallback. Reject incompatible legacy pools, invalid payloads and incomplete exact results. Cover real CPU HTTP round trips, native ComfyUI execution, mixed participant ordering and failure paths.
Preserve source values without extra wires, collect explicit per-item overrides, and expose ordered bundles through Get Value. Keep legacy sockets and no-override transport compatible.
* feat: add Runpod serverless image collector pools
* fix: prevent worker settings clipping and remove preview copy
* fix: initialize worker settings before mounting the form
* fix: validate complete serverless requests against 10 MB limit
* fix: simplify serverless cards and keep status dots circular
* fix: match serverless endpoint box to master card
* fix: preserve serverless capacity and recovery across job failures
Retain durable reservations until provider termination or explicit recovery, protect unresolved pools and credentials, and add scoped reconciliation. Consolidate lifecycle and secret policies and split prompt/media stages with regression coverage.
* feat: add dynamic per-image list queue with Start and End nodes
* fix(image-list): reject upstream data bypasses around Start
* fix(image-list): make optional End text a connection-only input
* fix(image-list): isolate unavailable workers during preflight
* fix(image-list): bound acknowledgements and configure run deadline
* fix(media-sync): preserve output and temp source roots
* fix(image-list): own job lifecycle and preserve worker preparation
* feat(image-list): let master participate unless orchestrator-only
* feat(image-list)!: replace fixed Start extras with growable value_N slots
Distributed Image List Start no longer has reference_image, text,
strength, seed or per-slot mode dropdowns. It now takes images, mask and
up to 10 generic value_N inputs (IMAGE, MASK, STRING, INT, FLOAT,
BOOLEAN) that grow via native V3 Autogrow, each with a matching output.
- One value is shared by every image; one per image pairs in order; any
other count fails before dispatch. Only images is split into frames.
- Outputs are declared up front (image, mask, item_index, context,
value_1..value_10). web/imageListStart.js hides unused value outputs,
mirrors input types, and follows Autogrow compaction so downstream
links keep their value.
- Image-list protocol bumped to v2; older workers fail preflight.
Breaking: saved workflows using the old Start inputs must recreate Start.
* fix(serverless): disable Clear saved API key while the worker is enabled
The checkbox read "Clear saved API key (disable pool first)", but the
panel never defines "pool" and saving with it ticked on an enabled worker
was rejected by the backend.
- Disable the checkbox while the worker is enabled, with a tooltip saying
to turn the worker off first. Also disable it when no key is saved.
- Use "worker" instead of "pool" in user-facing serverless errors, the
serverless guide and the README entry.
* feat(serverless): replace Concurrent jobs with a single Images per run setting
- Remove the separate Concurrent jobs field; rename Jobs per run to
Images per run (1-16) with a tooltip explaining that Runpod's max
workers decides how many GPUs run them.
- Backend uses a fixed per-worker limit of 16 outstanding jobs, so all
images of a run are submitted at once. Unresolved jobs still count
against it. The limit is constant, so it never has to change while
jobs are unresolved.
- Legacy 'concurrency' in saved configs and older browser tabs is
accepted and ignored/dropped; it no longer triggers the busy guard.
- Revert the enable toggle and show the reason when the backend rejects it.
- Trim a pasted api.runpod.ai/v2/<id> URL to the endpoint ID; plainer
endpoint-ID error otherwise.
- Confirm before a type change deletes a saved Runpod API key.
- Update docs and tests.
* feat: add Runpod serverless image collector pools
* fix: prevent worker settings clipping and remove preview copy
* fix: initialize worker settings before mounting the form
* fix: validate complete serverless requests against 10 MB limit
* fix: simplify serverless cards and keep status dots circular
* fix: match serverless endpoint box to master card
* fix: preserve serverless capacity and recovery across job failures
Retain durable reservations until provider termination or explicit recovery, protect unresolved pools and credentials, and add scoped reconciliation. Consolidate lifecycle and secret policies and split prompt/media stages with regression coverage.
* fix(serverless): disable Clear saved API key while the worker is enabled
The checkbox read "Clear saved API key (disable pool first)", but the
panel never defines "pool" and saving with it ticked on an enabled worker
was rejected by the backend.
- Disable the checkbox while the worker is enabled, with a tooltip saying
to turn the worker off first. Also disable it when no key is saved.
- Use "worker" instead of "pool" in user-facing serverless errors, the
serverless guide and the README entry.
* feat(serverless): replace Concurrent jobs with a single Images per run setting
- Remove the separate Concurrent jobs field; rename Jobs per run to
Images per run (1-16) with a tooltip explaining that Runpod's max
workers decides how many GPUs run them.
- Backend uses a fixed per-worker limit of 16 outstanding jobs, so all
images of a run are submitted at once. Unresolved jobs still count
against it. The limit is constant, so it never has to change while
jobs are unresolved.
- Legacy 'concurrency' in saved configs and older browser tabs is
accepted and ignored/dropped; it no longer triggers the busy guard.
- Revert the enable toggle and show the reason when the backend rejects it.
- Trim a pasted api.runpod.ai/v2/<id> URL to the endpoint ID; plainer
endpoint-ID error otherwise.
- Confirm before a type change deletes a saved Runpod API key.
- Update docs and tests.
Worker prompts drop everything downstream of DistributedCollector /
UltimateSDUpscaleDistributed, so the master re-added a PreviewImage
(or PreviewAudio) to give ComfyUI an output node. That preview wrote
every result frame as a PNG into the worker's temp directory: for a
362-frame 1344x768 video, ~2 GB per run (RAM when temp is on tmpfs)
plus CPU time spent PNG-encoding frames nobody looks at.
- Add DistributedWorkerSink: an output node that discards its input.
It is is_dev_only, so it stays out of node search and menus.
- The master probes each worker with GET /object_info/DistributedWorkerSink
while preparing its prompt and uses the sink when it is registered.
Workers on older versions (or a failed probe) keep the PreviewImage /
PreviewAudio terminator, so mixed versions keep working.
Verified with a real master plus one updated and one old worker:
the updated worker ran EmptyImage -> Collector -> Sink and left its
temp directory empty; the old worker still received PreviewImage.
Claude-Session: https://claude.ai/code/session_0166t8T3CVaxZkJbP2RtSpo6
Co-authored-by: ymiyad <cab-13-might@icloud.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
