robertvoy-ComfyUI-Distributed/docs/serverless-workers.md

# Runpod serverless pools (preview)

A **Serverless (Runpod)** worker is a queue-based endpoint/pool, not a permanent
ComfyUI machine. Runpod chooses the actual GPU workers. Distributed submits one
normal ComfyUI API workflow per logical slot and collects the returned images.
The endpoint workers **do not need ComfyUI-Distributed installed**.

## Setup

1. Add a worker in the Distributed panel and select **Serverless (Runpod)**.
2. Enter its name, Runpod **endpoint ID** (not a URL), and API key.
3. Start with **Jobs per run = 1**, **Concurrent jobs = 1**, and **Timeout = 1800**.
4. Save, then choose **Test Saved Endpoint**. This checks authentication and
   endpoint health only; it does not start GPU inference or prove model/node compatibility.
5. Enable the pool when you want queued Distributed workflows to use it.
6. Use an image **Distributed Collector** and queue through Distributed as usual.
   Enable **Delegate Only** if the master should collect without generating locally.

Keys are write-only. Leave the key field blank to retain a saved key, or select
**Clear saved API key** to remove it (disable the pool first). A saved key is
required before enabling a pool. Use HTTPS or a trusted local connection when
entering keys. Like ComfyUI itself, the panel must not be exposed to untrusted
users: anyone able to queue arbitrary workflows can run code and spend endpoint credits.

## Endpoint contract

The endpoint must accept this generic input shape; named application modes are
not used:

```json
{
  "input": {
    "workflow": {
      "1": {"class_type": "EmptyImage", "inputs": {"width": 32, "height": 32, "batch_size": 1, "color": 0}},
      "2": {"class_type": "SaveImage", "inputs": {"images": ["1", 0], "filename_prefix": "example"}}
    },
    "images": []
  }
}
```

For image inputs, `images` contains `{ "name": "input.png", "image": "<base64>" }`
entries and the handler must rewrite `LoadImage` references to the staged files.
The adapter transfers local input images; it does not copy models or install nodes.
All referenced models and custom nodes must already exist in the endpoint image.

On completion, Runpod's `output.images` must contain artifact records such as:

```json
{
  "filename": "example_00001_.png",
  "subfolder": "",
  "artifact_type": "output",
  "artifact_path": "example_00001_.png",
  "type": "base64",
  "data": "<base64>"
}
```

Image data can alternatively use `type: "s3_url"`, `"gcs_url"`, or `"url"`
with an HTTPS URL in `data`/`url`. Signed artifact URLs are downloaded without
Runpod credentials; private-network destinations and unsafe redirects are refused.
The handler must preserve injected `SaveImage.filename_prefix` values so results
can be matched to the right collector and slot. Duplicate basenames in different
subfolders remain different artifact identities.

A Runpod job marked `COMPLETED` is still rejected if `output.error` or
`output.errors` is nonempty, required collector outputs are missing, or image
artifacts are invalid. Failures are surfaced rather than silently returning an
incomplete serverless batch.

## Scheduling and cancellation

- **Jobs per run** adds logical collector participants. Each receives its own
  Distributed Seed/Value overrides. It is not a physical GPU count.
- **Concurrent jobs** bounds this master's outstanding submissions per pool,
  including retained jobs after a restart. Ambiguous submissions and unconfirmed
  cancellations keep their capacity reservation until reconciled. Other
  applications using the same endpoint are unaffected.
- **Load balance** selects one participant, so it does not fan out every pool slot.
- **Timeout** includes waiting for a local slot, Runpod queue/cold-start time,
  execution, and output retrieval. Provider policy limits are submitted as well.
- A zero-worker endpoint can be healthy: Runpod scales from zero on submission.
- Interrupting a collector or using the panel cancellation action requests
  cancellation only for jobs submitted by this master. It never purges the endpoint.
- A network failure during submission is **not retried automatically**: the job
  may already be billable. Check the Runpod job list before queuing again.

## Persistence and recovery

Credentials live outside `gpu_config.json` and Git in the master ComfyUI user
folder under `distributed/serverless/credentials.json`, with restrictive POSIX
permissions. They are omitted from public configuration, workflow exports, and
browser configuration state. The first version requires a POSIX (Linux/macOS)
master for serverless secret storage; existing non-serverless workers remain
available on Windows.

`distributed/serverless/jobs.json` records provider job IDs and sanitized states,
not image data, workflow payloads, or API keys. A restarted master cannot resume
an interrupted collector. It does **not** automatically resubmit jobs. Retained
IDs are available through `/distributed/serverless/status/<worker_id>`; the pool's
cancel action can request cancellation of known pre-restart jobs. If no job ID
was received, check Runpod manually. Cancellation is best-effort and may race
with completion; `CANCEL_REQUESTED` is not a provider-confirmed cancellation.

Unresolved records protect the pool's endpoint, concurrency, API key, worker type,
and deletion even when no local task remains. Disabling a pool is still allowed;
it does not cancel existing jobs. On restart, never-submitted `WAITING` slots are
closed locally, while interrupted `SUBMITTING` records without IDs become
`UNKNOWN` and remain protected.

### Reconcile or acknowledge recovery

Recovery uses the same trusted-master API as the panel; it never resubmits a job.

1. Read `GET /distributed/serverless/status/<worker_id>`. The `jobs` list includes
   every unresolved record with its local `record_id`, provider `job_id` (if
   received), and state. These records remain visible if endpoint health fails.
2. Request cancellation from the panel, or send
   `POST /distributed/serverless/reconcile` with the body below to **read** the
   retained provider IDs. Only a confirmed terminal state releases protection.
   Network errors, expired IDs, or ambiguous responses do not.

   ```json
   {"worker_id": "YOUR_POOL_ID"}
   ```

3. If no ID was returned, or Runpod no longer retains its status, inspect Runpod
   manually first. **Only after verifying that the job is no longer running or
   queued**, send `POST /distributed/serverless/resolve` for that one record:

   ```json
   {"worker_id": "YOUR_POOL_ID", "record_id": "LOCAL_RECORD_ID", "provider_checked": true}
   ```

   This records `RESOLVED`: an explicit operator acknowledgment, **not** a
   provider-confirmed cancellation. It cannot resolve a live local task and does
   not stop remote compute. It releases the record's reservation and may let
   already-waiting jobs proceed. Never automate this acknowledgment as a retry
   mechanism; an unverified job can still be billable.

Pool changes become available once all its local tasks and unresolved records
are finished. No credentials or ledger files need to be edited by hand.

## First-version limits

- Image collectors only. Audio/video transfer and audio collectors are rejected.
- Coordinated `UltimateSDUpscaleDistributed` workloads and unsupported Distributed
  node types are rejected before submission.
- Downstream post-collector processing remains local; the remote graph contains
  only the upstream compute required for image saves.
- Inputs must be supported images under ComfyUI's input/output/temp roots.
- Raw image input reads are guarded at 10,000,000 bytes total, and the complete
  serialized `/run` JSON request body (workflow, base64 images, and policy) must
  also fit within 10,000,000 bytes. Base64 adds about 33%, so the usable raw-image
  budget is lower and depends on workflow size. Large-media/object-storage upload
  support is not included.
- No remote GPU discovery, endpoint provisioning, model synchronization, automatic
  scaling changes, or inbound master callback/tunnel is needed.

This feature is verified with offline provider/transport tests. A successful
health test is not a paid end-to-end generation test; test your deployed handler
and model inventory with one slot before increasing concurrency.