robertvoy-ComfyUI-Distributed/docs/serverless-media.md

# Typed serverless media (PR2)

`comfy-media-v1` adds typed list results, exact by default, for `IMAGE`, `AUDIO`, `MASK`,
`STRING`, and native ComfyUI `VIDEO`. It is an **explicit endpoint profile**, not
an automatically detected capability. Omitting `media_profile` still selects
`legacy-image-v1`; existing image collectors and handlers keep their old contract.
See [legacy serverless setup](serverless-workers.md) for credentials and recovery.

## Use a typed collector

1. Prepare an endpoint with the [standalone media bridge](../serverless_bridge/README.md)
   and the workflow's models/custom nodes. The endpoint needs only the bridge,
   **not the full ComfyUI-Distributed extension**.
2. In the serverless worker settings, set **Endpoint media profile** to
   **Typed media v1 (requires endpoint bridge)** (`media_profile: "comfy-media-v1"`).
   Start with **Images per run = 1**. This existing setting counts remote jobs,
   not the number of typed items each job returns.
3. Use **Distributed Media Collector** (`DistributedMediaCollector`), connect at
   least one typed input, and connect its outputs to the appropriate downstream
   nodes. For literal text, connect a `PrimitiveString` node to `texts`.
4. Queue through Distributed. **Delegate Only** excludes local upstream
   generation; normal collection includes the master's local values.

**Test Saved Endpoint** checks provider authentication/health, not bridge
installation, model availability, or typed execution. Selecting a profile does
not install or upgrade anything on the endpoint.

The collector has five optional inputs and five **list outputs**, in this order:

| Socket / output index | Native type | Each list item |
| --- | --- | --- |
| `images` / 0 | `IMAGE` | One BHWC tensor, including its batch dimension |
| `audio` / 1 | `AUDIO` | `{ "waveform": BCT_tensor, "sample_rate": positive_int }` |
| `masks` / 2 | `MASK` | One BHW tensor |
| `texts` / 3 | `STRING` | Exact text, including whitespace or an empty string |
| `video` / 4 | `VIDEO` | One supported native video object |

For each connected socket, order is **master first, then the prepared participant
order, then that participant's item order**—never completion order. Delegate-only
mode omits the master. An unconnected socket returns `[]`. A connected remote
socket must return 1–64 items; an empty string is an item, an empty list is not.
Lists are not concatenated into one tensor, resampled, resized, or aligned across
sockets. Different items may have different valid shapes or audio sample rates.
The entire remote response is validated and decoded before any of its results
are delivered; missing, duplicate, mismatched, or corrupt results fail collection.

## Image return format and quality

`transfer_format` defaults to **`exact`** and `transfer_quality` to **90**.
These controls affect **remote IMAGE returns only**, not master-local images,
uploaded inputs, MASK, AUDIO, STRING or native VIDEO values.

- `exact`: preserve floating-point dtype and values, with no image quantization.
- `legacy_png`, `png`, `ffv1`: lossless **8-bit image pixels**, not exact float
  tensors. Image values are clipped to [0,1] and quantized as in the existing
  Collector. `legacy_png` uses PNG without compression; `png` compresses it.
- `jpeg`, `webp`: lossy still-image compression.
- `h264`, `h265`, `av1`: lossy video encoding of each IMAGE tensor's frame batch.
  This does not transcode the native `video` socket or combine separate list items.
- Quality 1–100 affects lossy formats only, using the existing Collector's
  quality-to-CRF mapping for video. Dimensions, frame order and channel count
  are preserved; alpha is carried separately where needed, not silently dropped.

Both master and endpoint must have this bridge revision for compressed image
returns. Pillow handles still formats and CPU/software PyAV encoders handle video;
unavailable codecs fail clearly rather than silently changing the selection.
Compression can reduce result egress and inline response size, but does not relax
encoded/decoded byte budgets or the input request limit. Compressed IMAGE batches
are capped at 240 frames and 8192 pixels per dimension, with a conservative
256 MiB aggregate decode budget (including codec scratch and alpha). The original Collector's
settings, default and fallback behavior are unchanged.

## Inputs and fidelity

Automatic local-file transfer supports stock `LoadImage`, plus explicit stock
`LoadAudio` (`audio` filename) and `LoadVideo` (`file` filename) adapters. Audio and
video loaders require a literal filename, no extra controls, and output slot 0.
Files must be under ComfyUI's input/output/temp roots. Custom file loaders are not
supported; detected unsupported file references fail rather than silently
becoming endpoint-local paths. Models and custom nodes are never transferred.

- `LoadImage` uses the existing bounded file/base64 image staging path.
- `LoadAudio` is decoded on the master using native PCM semantics with bounded
  accumulation, then transferred as a waveform and sample rate. It does not
  resample; integer PCM conversion follows ComfyUI's `f32_pcm` behavior.
- `LoadVideo` transfers opaque file bytes as native `VideoFromFile`, without
  decoding frames or transcoding the container.
- Typed tensor transport is `tensor-le-v1`: contiguous little-endian float16,
  bfloat16, float32, or float64 bytes with exact dtype/shape/layout metadata.
  Only dense, finite tensors with positive dimensions are supported. IMAGE has
  1, 3, or 4 channels. Bridge export explicitly detaches/stages tensors to CPU;
  the pure codec itself rejects non-CPU storage. Remote decoding returns
  independently owned CPU values. STRING transport is exact UTF-8 (`utf8-v1`).

### Native VIDEO is not a generic video converter

The adapter supports the pinned native classes **exactly**, not arbitrary
subclasses or objects exposing `save_to()` / `get_components()`:

- `VideoFromComponents`: images, optional matching BHW or BHW1 alpha, optional
  audio, positive rational frame rate, bounded JSON metadata, positive integer
  bit depth, and `sRGB`, `HDR`, or `HDR PQ` color space are preserved.
- `VideoFromFile`: regular local files within allowed media roots or an exact
  `BytesIO`; opaque bytes, finite signed start/duration, and optional four-integer
  crop are preserved without normalization.
- `VideoFromList`: 1–64 flat exact `VideoFromFile` segments, plus optional
  `complete_audio`, in order.

Unknown classes, incompatible native state, nested/component segment lists,
unsupported tensors, invalid metadata, and oversized payloads are rejected.
There is no fallback transcode, frame extraction, color conversion, or container
repair/validation. A downstream native consumer still needs to read the original
container. Decoded file-backed VIDEO values own independent `BytesIO` storage;
component tensors also own their storage, so staging-file cleanup cannot
invalidate a returned VIDEO.

## Size and operational limits

- The typed contract caps each artifact at **256 MiB**, aggregate artifact bytes
  at **512 MiB**, and each artifact list at **128** entries. These are validation
  ceilings, not a promise that inline requests can carry that much data.
- The built-in master packs inputs inline and enforces **10,000,000 bytes for the
  complete serialized Runpod `/run` request**, including workflow, policy and
  base64. Legacy image reads also retain their 10,000,000-byte aggregate guard.
  There is **no built-in master-side large-input uploader** in PR2.
- The endpoint defaults to inline results and caps its result JSON at
  **10,000,000 bytes**. Larger results require an operator-supplied publisher
  returning expiring public HTTPS artifact URLs. See the bridge guide for
  bounded download/upload responsibilities and retention.
- Remote typed transport is **serverless only**. Direct-worker typed transport
  is not implemented. Do not mix a typed collector with legacy image collectors,
  unsupported distributed/batch nodes, or collector-to-collector dependencies.
  Downstream processing stays on the master. Image-list scheduling is separate
  PR3 work, not part of this contract.

## Timeout is not proof that Comfy stopped

Provider cancellation, a handler timeout, or cancellation of its Python task
**does not establish that the local Comfy prompt stopped**. The bridge never
posts `/interrupt`: the pinned core route checks a queue snapshot and then
interrupts globally, so even a prompt ID does not make it race-free.

If submission/termination or callback teardown cannot be confirmed, the bridge
raises `MediaExecutionUncertain` and retains owned input/exporter files—even
after terminal Comfy history if a downloader/publisher ignores cancellation.
Callback teardown has a separate 10-second grace; late callback exit does not
clean retained files. Callbacks must be cooperative async code; synchronous
event-loop blocking cannot be hard-timed-out.

The endpoint operator must **quarantine the owning worker/environment, reconcile
the local prompt and readers or terminate that environment, and establish that
they have stopped before cleanup or worker reuse**. Do not automatically retry
the request or turn every handler exception into a safely reusable terminal
provider job. Master-side provider status/reconciliation alone is not proof
that a surviving local Comfy process or callback has stopped.

These docs describe the implementation and its offline tests; they do not claim
live Runpod/provider or GPU end-to-end validation.