robertvoy-ComfyUI-Distributed/docs/serverless-media.md

Serverless images and audio

Distributed Collector (DistributedCollector) collects IMAGE and/or AUDIO from direct workers and serverless pools. There is no separate DistributedMediaCollector; replace that removed node with DistributedCollector for image/audio workflows. The Collector has no MASK, STRING, or native VIDEO output sockets.

comfy-media-v1 is an explicit endpoint profile, not an automatically detected capability. It returns typed image/audio items to the Collector. The standalone bridge's generic contract also supports MASK, STRING, and native ComfyUI VIDEO, but that does not add those outputs to the Collector. Omitting media_profile still selects the image-only legacy-image-v1 contract. See legacy serverless setup for credentials and recovery.

Use the Collector with a typed endpoint

1. Prepare an endpoint with the standalone media bridge and the workflow's models/custom nodes. The endpoint needs only the bridge, not the full ComfyUI-Distributed extension.
2. In the serverless worker settings, set Endpoint media profile to Typed media v1 (requires endpoint bridge) (media_profile: "comfy-media-v1"). Start with Images per run = 1. This existing setting counts remote jobs, not the number of typed items each job returns.
3. Use Distributed Collector (DistributedCollector), connect images, audio, or both, and connect the corresponding outputs downstream. Image-only and audio-only workflows use the same node.
4. Queue through Distributed. Delegate Only excludes local upstream generation; normal collection includes the master's local values.

Test Saved Endpoint checks provider authentication/health, not bridge installation, model availability, or typed execution. Selecting a profile does not install or upgrade anything on the endpoint.

The Collector accepts optional images and audio inputs, including ComfyUI list inputs, and returns two ordinary outputs—not list outputs:

Socket / output index	Native type	Aggregation
images / 0	IMAGE	Concatenate BHWC tensors along the batch dimension
audio / 1	AUDIO	Concatenate waveforms along the sample dimension

For each connected socket, order is master first, then prepared participant order, then that participant's item order—never completion order. Delegate-only mode omits the master. Image dimensions/channels and audio batch/channel dimensions must be compatible for concatenation. Use a common audio sample rate; the Collector does not resample or align streams. Audio-only collection returns None for images; when no audio is supplied it returns a silent placeholder, not an empty list.

The typed wire contract carries 1–64 items per connected socket. The entire remote response is validated and decoded before any of its results are delivered; missing, duplicate, mismatched, or corrupt typed results fail collection. Validated wire lists are then combined into the Collector's ordinary outputs.

Direct workers, legacy-image-v1 pools and comfy-media-v1 pools can share an image-only Collector run. Audio requires direct workers or comfy-media-v1 pools. If any Collector requests exact, a selected legacy-image-v1 pool is rejected before dispatch instead of silently returning 8-bit images.

Image return format and quality

The Collector's saved transfer_format default remains legacy_png, with transfer_quality 90. For comfy-media-v1, the compiler maps legacy_png to png, so normal serverless returns use compressed 8-bit PNG. exact is opt-in. These controls affect remote IMAGE returns only, not master-local images, uploaded inputs or AUDIO. The bridge's generic MASK, STRING and native VIDEO codecs are also unaffected.

• exact: preserve supported floating-point dtype and values, with no image quantization. Available for direct workers and comfy-media-v1 pools; it is never downgraded to an 8-bit format. An older master that rejects exact transfer causes an error, not a legacy retry. Update both sides before selecting it.
• legacy_png, png, ffv1: lossless 8-bit image pixels, not exact float tensors. For typed serverless returns, image values are clipped to [0,1] and quantized; legacy_png is sent as png. Direct workers retain the original JSON/base64 framing for legacy_png.
• jpeg, webp: lossy still-image compression.
• h264, h265 (experimental), av1: lossy video encoding of each IMAGE tensor's frame batch. This is an image-return encoding, not a native VIDEO output.
• Quality 1–100 affects lossy formats only, using the existing Collector's quality-to-CRF mapping for video. Dimensions, frame order and channel count are preserved for typed serverless returns; alpha is carried separately where needed, not silently dropped.

Both master and endpoint must have this bridge revision for compressed image returns. Pillow handles still formats and CPU/software PyAV encoders handle video; unavailable codecs fail clearly rather than silently changing the selection. Compression can reduce result egress and inline response size, but does not relax encoded/decoded byte budgets or the input request limit. Compressed IMAGE batches are capped at 240 frames and 8192 pixels per dimension, with a conservative 256 MiB aggregate decode budget (including codec scratch and alpha). Direct workers have different limits and non-exact compatibility fallbacks; see result transfer. The bridge's internal MediaExportV1 default is still exact for generic callers that omit controls; the Collector compiler explicitly supplies png for the normal saved default.

Inputs and fidelity

Automatic local-file transfer supports stock LoadImage, plus explicit stock LoadAudio (audio filename) and LoadVideo (file filename) adapters. Audio and video loaders require a literal filename, no extra controls, and output slot 0. Files must be under ComfyUI's input/output/temp roots. Custom file loaders are not supported; detected unsupported file references fail rather than silently becoming endpoint-local paths. Models and custom nodes are never transferred.

• LoadImage uses the existing bounded file/base64 image staging path.
• LoadAudio is decoded on the master using native PCM semantics with bounded accumulation, then transferred as a waveform and sample rate. It does not resample; integer PCM conversion follows ComfyUI's f32_pcm behavior.
• LoadVideo transfers opaque file bytes as native VideoFromFile, without decoding frames or transcoding the container.
• Typed tensor transport is tensor-le-v1: contiguous little-endian float16, bfloat16, float32, or float64 bytes with exact dtype/shape/layout metadata. Only dense, finite tensors with positive dimensions are supported. IMAGE has 1, 3, or 4 channels. Bridge export explicitly detaches/stages tensors to CPU; the pure codec itself rejects non-CPU storage. Remote decoding returns independently owned CPU values. STRING transport is exact UTF-8 (utf8-v1).

Bridge-level native VIDEO support

The generic bridge can import/export native VIDEO, and the typed input adapter can stage LoadVideo for upstream processing. This is not a Collector VIDEO output or a generic video converter.

The adapter supports the pinned native classes exactly, not arbitrary subclasses or objects exposing save_to() / get_components():

• VideoFromComponents: images, optional matching BHW or BHW1 alpha, optional audio, positive rational frame rate, bounded JSON metadata, positive integer bit depth, and sRGB, HDR, or HDR PQ color space are preserved.
• VideoFromFile: regular local files within allowed media roots or an exact BytesIO; opaque bytes, finite signed start/duration, and optional four-integer crop are preserved without normalization.
• VideoFromList: 1–64 flat exact VideoFromFile segments, plus optional complete_audio, in order.

Unknown classes, incompatible native state, nested/component segment lists, unsupported tensors, invalid metadata, and oversized payloads are rejected. There is no fallback transcode, frame extraction, color conversion, or container repair/validation. A downstream native consumer still needs to read the original container. Decoded file-backed VIDEO values own independent BytesIO storage; component tensors also own their storage, so staging-file cleanup cannot invalidate a returned VIDEO.

Size and operational limits

• The typed contract caps each artifact at 256 MiB, aggregate artifact bytes at 512 MiB, and each artifact list at 128 entries. These are validation ceilings, not a promise that inline requests can carry that much data.
• The built-in master packs inputs inline and enforces 10,000,000 bytes for the complete serialized Runpod /run request, including workflow, policy and base64. Legacy image reads also retain their 10,000,000-byte aggregate guard. There is no built-in master-side large-input uploader in PR2.
• The endpoint defaults to inline results and caps its result JSON at 10,000,000 bytes. Larger results require an operator-supplied publisher returning expiring public HTTPS artifact URLs. See the bridge guide for bounded download/upload responsibilities and retention.
• The generic five-kind typed envelope is a serverless bridge contract. Direct workers support Collector images/audio, including opt-in exact IMAGE transport, not generic MASK/STRING/VIDEO Collector outputs. The typed compiler rejects unsupported distributed/batch nodes and collector-to-collector dependencies. Downstream processing stays on the master. Image-list scheduling is a separate execution mode.

Timeout is not proof that Comfy stopped

Provider cancellation, a handler timeout, or cancellation of its Python task does not establish that the local Comfy prompt stopped. The bridge never posts /interrupt: the pinned core route checks a queue snapshot and then interrupts globally, so even a prompt ID does not make it race-free.

If submission/termination or callback teardown cannot be confirmed, the bridge raises MediaExecutionUncertain and retains owned input/exporter files—even after terminal Comfy history if a downloader/publisher ignores cancellation. Callback teardown has a separate 10-second grace; late callback exit does not clean retained files. Callbacks must be cooperative async code; synchronous event-loop blocking cannot be hard-timed-out.

The endpoint operator must quarantine the owning worker/environment, reconcile the local prompt and readers or terminate that environment, and establish that they have stopped before cleanup or worker reuse. Do not automatically retry the request or turn every handler exception into a safely reusable terminal provider job. Master-side provider status/reconciliation alone is not proof that a surviving local Comfy process or callback has stopped.

These docs describe the implementation, not a test result or live Runpod/provider or GPU end-to-end validation.