Commit Graph
19 Commits
Author SHA1 Message Date
Robert Wojciechowski 59ccbabe79 Merge main into feat/runpod-serverless-workers
Resolve nodes/collector.py: keep the serverless manager accessor and
main's compressed result transfer. Update test stubs for the merged
code: fetch_worker_has_node (#99) in the serverless integration test,
and the real aiohttp.web in the transfer integration test.
2026-10-13 22:16:16 +00:00
Robert Wojciechowski 06be0e5dee feat(serverless): replace Concurrent jobs with a single Images per run setting
- Remove the separate Concurrent jobs field; rename Jobs per run to
  Images per run (1-16) with a tooltip explaining that Runpod's max
  workers decides how many GPUs run them.
- Backend uses a fixed per-worker limit of 16 outstanding jobs, so all
  images of a run are submitted at once. Unresolved jobs still count
  against it. The limit is constant, so it never has to change while
  jobs are unresolved.
- Legacy 'concurrency' in saved configs and older browser tabs is
  accepted and ignored/dropped; it no longer triggers the busy guard.
- Revert the enable toggle and show the reason when the backend rejects it.
- Trim a pasted api.runpod.ai/v2/<id> URL to the endpoint ID; plainer
  endpoint-ID error otherwise.
- Confirm before a type change deletes a saved Runpod API key.
- Update docs and tests.
2026-10-13 21:33:02 +00:00
Robert Wojciechowski 7060e31edb fix(serverless): disable Clear saved API key while the worker is enabled
The checkbox read "Clear saved API key (disable pool first)", but the
panel never defines "pool" and saving with it ticked on an enabled worker
was rejected by the backend.

- Disable the checkbox while the worker is enabled, with a tooltip saying
  to turn the worker off first. Also disable it when no key is saved.
- Use "worker" instead of "pool" in user-facing serverless errors, the
  serverless guide and the README entry.
2026-10-13 21:01:57 +00:00
40678a51c4 feat(worker): terminate pruned worker prompts with a hidden no-op sink (#99)
Worker prompts drop everything downstream of DistributedCollector /
UltimateSDUpscaleDistributed, so the master re-added a PreviewImage
(or PreviewAudio) to give ComfyUI an output node. That preview wrote
every result frame as a PNG into the worker's temp directory: for a
362-frame 1344x768 video, ~2 GB per run (RAM when temp is on tmpfs)
plus CPU time spent PNG-encoding frames nobody looks at.

- Add DistributedWorkerSink: an output node that discards its input.
  It is is_dev_only, so it stays out of node search and menus.
- The master probes each worker with GET /object_info/DistributedWorkerSink
  while preparing its prompt and uses the sink when it is registered.
  Workers on older versions (or a failed probe) keep the PreviewImage /
  PreviewAudio terminator, so mixed versions keep working.

Verified with a real master plus one updated and one old worker:
the updated worker ran EmptyImage -> Collector -> Sink and left its
temp directory empty; the old worker still received PreviewImage.


Claude-Session: https://claude.ai/code/session_0166t8T3CVaxZkJbP2RtSpo6

Co-authored-by: ymiyad <cab-13-might@icloud.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:27:19 +11:00
Robert Wojciechowski 87c0cda9f6 fix: preserve serverless capacity and recovery across job failures
Retain durable reservations until provider termination or explicit recovery, protect unresolved pools and credentials, and add scoped reconciliation. Consolidate lifecycle and secret policies and split prompt/media stages with regression coverage.
2026-10-11 06:56:01 +00:00
Robert Wojciechowski a281d03c6d fix: validate complete serverless requests against 10 MB limit 2026-10-11 05:52:23 +00:00
Robert Wojciechowski 1093818327 feat: add Runpod serverless image collector pools 2026-10-11 04:55:01 +00:00
Robert Wojciechowski cedc8d4591 feat: support audio-only distributed workflows (#87)
Support image, audio, or combined distributed collection with guarded audio-only worker, master, endpoint, preview, and delegate-only paths.
2026-07-12 20:36:43 +10:00
Robert Wojciechowski 6e7818bee6 fix: preserve delegate-only master config inputs (#86)
Fixes #85
2026-06-05 17:12:25 +10:00
Robert Wojciechowski c297b34b94 Add DistributedValue node with per-worker overrides 2026-02-28 04:53:20 +00:00
Robert Wojciechowski b95675d1ff Move load balancing to collector/orchestrator and remove queue node 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 0dae73ec7d fix: rewrite synced media inputs to worker-local paths 2026-02-28 04:53:20 +00:00
Robert Wojciechowski df606af80e fix: sync LoadVideo 'file' media refs for remote workers 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 9058cf061d fix: sync audio media refs for worker prompt validation 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 04169ac651 fix: correct divider chunking and improve distributed validation/audio transport 2026-02-28 04:53:20 +00:00
Robert Wojciechowski e4c945e536 chore: commit all pending distributed changes 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 6de2050b3f fix(tests): restore config error contract and stub compatibility 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 5eacd17aeb refactor: implement REFACTOR_PLAN_2 across distributed stack 2026-02-24 01:38:41 +00:00
Robert Wojciechowski f0fe702f80 Commit all current refactor changes 2026-02-23 23:47:23 +00:00