WorkerSnapshot tracked a single failure flag, so a failed /history request
made every later query in the same cycle return None, even for runs the
already-fetched /queue showed as running. A held run that was still
running then accumulated missed probes and had its collect-only delegate
queued before delivery, parking a participating master on it.
Queue and history failures are now tracked separately: queue evidence is
always consulted first, a history failure only affects runs that need
history, and neither endpoint is retried within the snapshot's cycle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lifecycle:
- Queue clears and deletes leave a run whose master prompt is currently
executing alone, as ComfyUI leaves the running prompt alone; previously an
executing orchestrator-only head run could be cancelled out from under its
waiting Collector.
- The sweep now also drops worker-assigned runs whose queued delegate prompt
left the queue without the Collector finishing, releasing their result
queues instead of leaking them and polling forever.
- Completed runs release their result queues, and orchestration no longer
pre-creates result queues when load balance picks the master alone.
Transport:
- A websocket timeout during connect, before anything was sent, is a plain
failure the dispatcher may retry elsewhere; only timeouts after the prompt
was sent remain AmbiguousDispatchError.
Efficiency and cleanup:
- One WorkerSnapshot per worker per cycle answers every run's "is my job
there" question (queue first, history only when needed).
- Sweep and cancellation snapshot queued/running prompt ids once per pass.
- probe_worker_queues reuses dispatch._probe_worker_queue; the dead
aggregate idle/busy probe helper is gone.
Rebased on "never replay load-balanced runs on missing history": an
unresolved submission stays on hold until delivery or positive evidence,
so there is no rejection path to fix and no bounded hold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keep ambiguous submissions tracked until positive job-specific evidence or result delivery confirms acceptance. Bounded or cleared history cannot prove rejection, even on repeated checks.
Add regressions using the real worker history lookup, retain late-result and cancellation coverage, and document the intentional unresolved hold.
Second round of review follow-ups for the next-free-GPU dispatcher.
Ambiguous acceptance is no longer inferred from aggregate queue depth. The
dispatcher asks the worker about the specific run: a prompt carrying one
of its job ids in the worker's running or pending queue, or in its recent
history, means the run was accepted and its result is collected (or
reported) there; only after consecutive checks find no trace of it is the
original prompt restored. A worker that cannot be asked keeps the run on
hold. The same question decides when a held delegate prompt must be queued
because the worker finished (or lost) the run without delivering.
An eagerly dispatched run whose acknowledgement was lost is now collected
after the worker delivers: the record remembers that no master prompt was
ever queued for it, so acceptance queues its collector instead of treating
it like a collector that is already executing.
Cancellation now reaches the queue-insertion boundary: queue_prompt_payload
takes an insert_if guard that is checked under the queue mutex right before
the item is put, after validation has yielded. The delegate requeue passes
the run's cancellation flag, so a clear or delete that lands during
validation refuses the insertion instead of being discovered afterwards.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review follow-ups for the next-free-GPU dispatcher.
Ambiguous acceptance: the worker transport now raises a typed
AmbiguousDispatchError when the request may have reached the worker but no
acknowledgement came back (websocket closed or timed out before the ack,
unreadable /prompt acknowledgement, request failure after it could have
been received). A connection refusal or an error status stays a definite
failure. The dispatcher no longer restores such a run to the master queue:
it is held as unresolved, claimable by nobody, until the worker delivers,
is seen busy (accepted: collected as usual) or is seen idle without
delivering (never accepted: the original prompt goes back in its place).
While the worker is unreachable the run stays on hold. The immediate
dispatch path does the same and still reports the error to the caller.
Cancellation during a hand-off: the dispatcher hooks the master queue's
wipe_queue and delete_queue_item so a clear or delete reaches runs that
are out of the queue for a hand-off. Preparation and submission are now
separate steps; a run cancelled before submission is dropped together with
its result queues and never sent, while work a worker already accepted
stays owned until its result arrives or the worker is seen done, and is
then discarded instead of reappearing. A Collector that fails or is
interrupted while waiting for its assignment cancels its run the same way,
so the preallocated result queue is removed and an in-flight dispatch
stops before submission. Discarded job ids are remembered so a late
worker cannot recreate their result queue through /distributed/prepare_job.
Also updates the frozen Collector load_balance tooltip in the v1 node
contract fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When a run went to a worker, the master's collect-only delegate prompt was
queued at once. ComfyUI's single executor then sat on that prompt waiting
for the worker, so every full workflow queued behind it waited too: with
one worker and a participating master, the master GPU only started once
the worker's first job finished.
On a participating master the delegate prompt is now held back and queued
only once the worker posts its final result (job_complete is_last), so the
master executor keeps pulling full workflows meanwhile. If the worker is
seen idle or unreachable for several probes without delivering, the
delegate is queued anyway so the Collector's timeout path reports it. Both
the immediate dispatch to an idle worker and dispatcher claims use this;
an orchestrator-only master, which has nothing else to run, still queues
the delegate immediately so the run stays visible in its queue.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two gaps in the next-free-GPU dispatcher:
A run was registered with the dispatcher before the master prompt reached
the ComfyUI queue. Prompt validation yields to the event loop, so a sweep
in that window saw the run as neither queued nor running and dropped it.
Runs are now registered unqueued and ignored by sweep and claim until the
orchestrator marks them queued; a failed queue still discards them.
When a worker had been dispatched but the master's delegate prompt could
not be re-queued, the run's record was removed, leaving nothing to collect
the worker's result. The record now stays assigned to the worker with the
requeue pending, and the dispatcher retries it with backoff until the
delegate prompt is back on the master queue; tracking ends only when the
Collector finishes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
With load_balance enabled, a run was pinned to a participant at queue time
by comparing worker queue depths, so a faster GPU went idle while a slower
one still held a backlog. Runs queued while every participant is busy now
stay unassigned in the master's prompt queue; a dispatcher watches the
candidate workers and hands the oldest waiting run to whichever one frees
up first, re-queueing the master side as a delegate prompt under the same
prompt id. A participating master takes whatever waiting run its executor
reaches first. Idle participants are still used immediately.
Upscale workflows and serverless pools keep the previous queue-depth
selection.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Merge main while retaining exact transfer, serverless cleanup and image assembly error propagation. Update RGB fixtures and verify mixed RGB/RGBA exact HTTP transport without value loss.
Use the existing images/audio Collector for typed serverless pools. Preserve saved legacy_png defaults, map them to serverless png, and retain every transfer format with exact opt-in. Remove the separate Media Collector.
Send direct exact frames through bounded raw tensor multipart transport without 8-bit fallback. Reject incompatible legacy pools, invalid payloads and incomplete exact results. Cover real CPU HTTP round trips, native ComfyUI execution, mixed participant ordering and failure paths.
Preserve source values without extra wires, collect explicit per-item overrides, and expose ordered bundles through Get Value. Keep legacy sockets and no-override transport compatible.
* feat: add Runpod serverless image collector pools
* fix: prevent worker settings clipping and remove preview copy
* fix: initialize worker settings before mounting the form
* fix: validate complete serverless requests against 10 MB limit
* fix: simplify serverless cards and keep status dots circular
* fix: match serverless endpoint box to master card
* fix: preserve serverless capacity and recovery across job failures
Retain durable reservations until provider termination or explicit recovery, protect unresolved pools and credentials, and add scoped reconciliation. Consolidate lifecycle and secret policies and split prompt/media stages with regression coverage.
* feat: add dynamic per-image list queue with Start and End nodes
* fix(image-list): reject upstream data bypasses around Start
* fix(image-list): make optional End text a connection-only input
* fix(image-list): isolate unavailable workers during preflight
* fix(image-list): bound acknowledgements and configure run deadline
* fix(media-sync): preserve output and temp source roots
* fix(image-list): own job lifecycle and preserve worker preparation
* feat(image-list): let master participate unless orchestrator-only
* feat(image-list)!: replace fixed Start extras with growable value_N slots
Distributed Image List Start no longer has reference_image, text,
strength, seed or per-slot mode dropdowns. It now takes images, mask and
up to 10 generic value_N inputs (IMAGE, MASK, STRING, INT, FLOAT,
BOOLEAN) that grow via native V3 Autogrow, each with a matching output.
- One value is shared by every image; one per image pairs in order; any
other count fails before dispatch. Only images is split into frames.
- Outputs are declared up front (image, mask, item_index, context,
value_1..value_10). web/imageListStart.js hides unused value outputs,
mirrors input types, and follows Autogrow compaction so downstream
links keep their value.
- Image-list protocol bumped to v2; older workers fail preflight.
Breaking: saved workflows using the old Start inputs must recreate Start.
* fix(serverless): disable Clear saved API key while the worker is enabled
The checkbox read "Clear saved API key (disable pool first)", but the
panel never defines "pool" and saving with it ticked on an enabled worker
was rejected by the backend.
- Disable the checkbox while the worker is enabled, with a tooltip saying
to turn the worker off first. Also disable it when no key is saved.
- Use "worker" instead of "pool" in user-facing serverless errors, the
serverless guide and the README entry.
* feat(serverless): replace Concurrent jobs with a single Images per run setting
- Remove the separate Concurrent jobs field; rename Jobs per run to
Images per run (1-16) with a tooltip explaining that Runpod's max
workers decides how many GPUs run them.
- Backend uses a fixed per-worker limit of 16 outstanding jobs, so all
images of a run are submitted at once. Unresolved jobs still count
against it. The limit is constant, so it never has to change while
jobs are unresolved.
- Legacy 'concurrency' in saved configs and older browser tabs is
accepted and ignored/dropped; it no longer triggers the busy guard.
- Revert the enable toggle and show the reason when the backend rejects it.
- Trim a pasted api.runpod.ai/v2/<id> URL to the endpoint ID; plainer
endpoint-ID error otherwise.
- Confirm before a type change deletes a saved Runpod API key.
- Update docs and tests.
* feat: add Runpod serverless image collector pools
* fix: prevent worker settings clipping and remove preview copy
* fix: initialize worker settings before mounting the form
* fix: validate complete serverless requests against 10 MB limit
* fix: simplify serverless cards and keep status dots circular
* fix: match serverless endpoint box to master card
* fix: preserve serverless capacity and recovery across job failures
Retain durable reservations until provider termination or explicit recovery, protect unresolved pools and credentials, and add scoped reconciliation. Consolidate lifecycle and secret policies and split prompt/media stages with regression coverage.
* fix(serverless): disable Clear saved API key while the worker is enabled
The checkbox read "Clear saved API key (disable pool first)", but the
panel never defines "pool" and saving with it ticked on an enabled worker
was rejected by the backend.
- Disable the checkbox while the worker is enabled, with a tooltip saying
to turn the worker off first. Also disable it when no key is saved.
- Use "worker" instead of "pool" in user-facing serverless errors, the
serverless guide and the README entry.
* feat(serverless): replace Concurrent jobs with a single Images per run setting
- Remove the separate Concurrent jobs field; rename Jobs per run to
Images per run (1-16) with a tooltip explaining that Runpod's max
workers decides how many GPUs run them.
- Backend uses a fixed per-worker limit of 16 outstanding jobs, so all
images of a run are submitted at once. Unresolved jobs still count
against it. The limit is constant, so it never has to change while
jobs are unresolved.
- Legacy 'concurrency' in saved configs and older browser tabs is
accepted and ignored/dropped; it no longer triggers the busy guard.
- Revert the enable toggle and show the reason when the backend rejects it.
- Trim a pasted api.runpod.ai/v2/<id> URL to the endpoint ID; plainer
endpoint-ID error otherwise.
- Confirm before a type change deletes a saved Runpod API key.
- Update docs and tests.
Worker prompts drop everything downstream of DistributedCollector /
UltimateSDUpscaleDistributed, so the master re-added a PreviewImage
(or PreviewAudio) to give ComfyUI an output node. That preview wrote
every result frame as a PNG into the worker's temp directory: for a
362-frame 1344x768 video, ~2 GB per run (RAM when temp is on tmpfs)
plus CPU time spent PNG-encoding frames nobody looks at.
- Add DistributedWorkerSink: an output node that discards its input.
It is is_dev_only, so it stays out of node search and menus.
- The master probes each worker with GET /object_info/DistributedWorkerSink
while preparing its prompt and uses the sink when it is registered.
Workers on older versions (or a failed probe) keep the PreviewImage /
PreviewAudio terminator, so mixed versions keep working.
Verified with a real master plus one updated and one old worker:
the updated worker ran EmptyImage -> Collector -> Sink and left its
temp directory empty; the old worker still received PreviewImage.
Claude-Session: https://claude.ai/code/session_0166t8T3CVaxZkJbP2RtSpo6
Co-authored-by: ymiyad <cab-13-might@icloud.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(collector): compressed worker -> master result transfer
Workers sent every frame as an uncompressed PNG (compress_level=0),
base64-encoded inside a JSON request, one request per frame. For cloud
workers this makes result download (egress) the dominant transfer cost:
a 121-frame 720p batch is ~446 MB per worker.
Add `transfer_format` / `transfer_quality` optional widgets to
Distributed Collector:
- legacy_png (default): unchanged behaviour
- png, ffv1: lossless, bit-identical master tensors to legacy_png
- jpeg, webp: lossy stills
- h264, h265, av1: lossy video of the whole batch
Compressed results are posted as multipart/form-data chunks (binary, no
base64, ~32 MB soft cap per request, configurable with
COMFYUI_DISTRIBUTED_TRANSFER_CHUNK_BYTES). The master decodes each chunk
off the event loop and enqueues one item per frame, so collection,
ordering and audio handling are unchanged. Encoding also runs in an
executor on the worker.
Compatibility:
- /distributed/job_complete still accepts the legacy JSON envelope.
- A worker talking to an older master gets a 400 on the first chunk and
resends everything as legacy_png.
- Missing video encoders fall back to h264 (lossy) or png (ffv1).
Measured on 121 frames at 1280x720: png 163 MB (2.7x), ffv1 128 MB
(3.5x), h264 q90 2.4 MB (186x, 40.2 dB), av1 q90 0.4 MB.
Also clips IMAGE values before 8-bit quantization on the new path so
out-of-range values no longer wrap around.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0166t8T3CVaxZkJbP2RtSpo6
* fix(collector): wire transfer options through V3 node; bound finalized payload size
Addresses review on #97.
V3 integration
- Add optional `transfer_format` / `transfer_quality` inputs to the
registered nodes/v3.py DistributedCollector schema (legacy defaults) and
accept/forward them in execute() to the runtime helper.
- tests/test_v3_collector_transfer.py checks the registered node's public
schema and argument forwarding without a ComfyUI checkout; the opt-in
real-runtime acceptance (tests/helpers/v3_runtime_check.py) now allows
exactly these appended optional inputs, verifies their contract, and
checks forwarding through the real executor.
Payload size bound
- encode_chunks() enforces limits on finalized bytes (after encoder flush
and muxer finalization). Encoder lookahead kept the pre-flush buffer
near zero, so a chunk could hold the whole batch and finalize far above
the limit. Oversized video chunks are re-encoded with proportionally
fewer frames; the next chunk is sized from observed bytes/frame.
- Multi-frame chunks stay within TRANSFER_MAX_CHUNK_BYTES (32 MiB); a
single frame may use up to TRANSFER_MAX_REQUEST_BYTES (96 MiB, new,
COMFYUI_DISTRIBUTED_TRANSFER_MAX_REQUEST_BYTES) minus envelope reserve.
A frame that cannot fit raises FrameTooLargeError.
- The worker encodes all chunks before sending, so an unfittable frame
fails before anything is delivered (no partial batch on the master).
- Audio rides on the last chunk only when the request stays in bounds;
otherwise it is sent as a final audio-only completion.
- Legacy fallback triggers only on an older master's rejection
("Invalid JSON payload"); other errors, e.g. size limits, raise with a
pointer to the limit settings instead of resending as legacy_png.
- The master returns 413 when a request exceeds --max-upload-size.
Tests: codec flush-overshoot and single-frame cases, worker fallback /
nothing-sent / audio placement, and real aiohttp server tests with
client_max_size covering bounded high-entropy video, 413 reporting, and
unfittable frames.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0166t8T3CVaxZkJbP2RtSpo6
---------
Co-authored-by: ymiyad <ymiyad@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: migrate distributed nodes to native ComfyUI V3 API
* fix: hide Distributed Value backing fields in Nodes 2.0
Use native widget visibility without converting backing fields into sockets. Preserve widget types, serializers and options identity; cover edits, reload, worker refresh and linked widgets.
* fix: isolate local worker databases and avoid port conflicts
* fix: harden local worker startup preflight and database paths
Handle auto-launch before the master binds, allow TIME_WAIT restarts without accepting active listeners, resolve relative paths in the worker cwd, and round-trip SQLite URLs through SQLAlchemy. Add regressions and remove the README additions.
