Commit Graph
28 Commits
Author SHA1 Message Date
Robert Wojciechowski 5bb14fba97 Merge feat/runpod-serverless-workers (now including main) into feat/dynamic-image-list-queue
Resolve api/queue_orchestration.py, nodes/v3.py and
tests/helpers/v3_runtime_check.py to keep both the worker sink (#99)
and the image-list nodes. Stub nodes.image_list in the V3 collector
transfer test.
2026-10-13 22:17:30 +00:00
Robert Wojciechowski 59ccbabe79 Merge main into feat/runpod-serverless-workers
Resolve nodes/collector.py: keep the serverless manager accessor and
main's compressed result transfer. Update test stubs for the merged
code: fetch_worker_has_node (#99) in the serverless integration test,
and the real aiohttp.web in the transfer integration test.
2026-10-13 22:16:16 +00:00
Robert Wojciechowski 00a9ba05a3 Merge feat/runpod-serverless-workers into feat/dynamic-image-list-queue 2026-10-13 21:33:28 +00:00
Robert Wojciechowski 06be0e5dee feat(serverless): replace Concurrent jobs with a single Images per run setting
- Remove the separate Concurrent jobs field; rename Jobs per run to
  Images per run (1-16) with a tooltip explaining that Runpod's max
  workers decides how many GPUs run them.
- Backend uses a fixed per-worker limit of 16 outstanding jobs, so all
  images of a run are submitted at once. Unresolved jobs still count
  against it. The limit is constant, so it never has to change while
  jobs are unresolved.
- Legacy 'concurrency' in saved configs and older browser tabs is
  accepted and ignored/dropped; it no longer triggers the busy guard.
- Revert the enable toggle and show the reason when the backend rejects it.
- Trim a pasted api.runpod.ai/v2/<id> URL to the endpoint ID; plainer
  endpoint-ID error otherwise.
- Confirm before a type change deletes a saved Runpod API key.
- Update docs and tests.
2026-10-13 21:33:02 +00:00
Robert Wojciechowski 7060e31edb fix(serverless): disable Clear saved API key while the worker is enabled
The checkbox read "Clear saved API key (disable pool first)", but the
panel never defines "pool" and saving with it ticked on an enabled worker
was rejected by the backend.

- Disable the checkbox while the worker is enabled, with a tooltip saying
  to turn the worker off first. Also disable it when no key is saved.
- Use "worker" instead of "pool" in user-facing serverless errors, the
  serverless guide and the README entry.
2026-10-13 21:01:57 +00:00
40678a51c4 feat(worker): terminate pruned worker prompts with a hidden no-op sink (#99)
Worker prompts drop everything downstream of DistributedCollector /
UltimateSDUpscaleDistributed, so the master re-added a PreviewImage
(or PreviewAudio) to give ComfyUI an output node. That preview wrote
every result frame as a PNG into the worker's temp directory: for a
362-frame 1344x768 video, ~2 GB per run (RAM when temp is on tmpfs)
plus CPU time spent PNG-encoding frames nobody looks at.

- Add DistributedWorkerSink: an output node that discards its input.
  It is is_dev_only, so it stays out of node search and menus.
- The master probes each worker with GET /object_info/DistributedWorkerSink
  while preparing its prompt and uses the sink when it is registered.
  Workers on older versions (or a failed probe) keep the PreviewImage /
  PreviewAudio terminator, so mixed versions keep working.

Verified with a real master plus one updated and one old worker:
the updated worker ran EmptyImage -> Collector -> Sink and left its
temp directory empty; the old worker still received PreviewImage.


Claude-Session: https://claude.ai/code/session_0166t8T3CVaxZkJbP2RtSpo6

Co-authored-by: ymiyad <cab-13-might@icloud.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:27:19 +11:00
Robert Wojciechowski dcddf23db2 feat(image-list)!: replace fixed Start extras with growable value_N slots
Distributed Image List Start no longer has reference_image, text,
strength, seed or per-slot mode dropdowns. It now takes images, mask and
up to 10 generic value_N inputs (IMAGE, MASK, STRING, INT, FLOAT,
BOOLEAN) that grow via native V3 Autogrow, each with a matching output.

- One value is shared by every image; one per image pairs in order; any
  other count fails before dispatch. Only images is split into frames.
- Outputs are declared up front (image, mask, item_index, context,
  value_1..value_10). web/imageListStart.js hides unused value outputs,
  mirrors input types, and follows Autogrow compaction so downstream
  links keep their value.
- Image-list protocol bumped to v2; older workers fail preflight.

Breaking: saved workflows using the old Start inputs must recreate Start.
2026-10-13 09:18:50 +11:00
Robert Wojciechowski 8653e34254 feat(image-list): let master participate unless orchestrator-only 2026-10-12 07:33:00 +00:00
Robert Wojciechowski a34969e6f3 fix(image-list): own job lifecycle and preserve worker preparation 2026-10-11 21:20:22 +00:00
Robert Wojciechowski 8e0ac3befe fix(media-sync): preserve output and temp source roots 2026-10-11 21:20:22 +00:00
Robert Wojciechowski eac29590bb fix(image-list): bound acknowledgements and configure run deadline 2026-10-11 21:07:19 +00:00
Robert Wojciechowski bac16afaad fix(image-list): isolate unavailable workers during preflight 2026-10-11 20:57:20 +00:00
Robert Wojciechowski 3e280a0e29 feat: add dynamic per-image list queue with Start and End nodes 2026-10-11 08:49:23 +00:00
Robert Wojciechowski 87c0cda9f6 fix: preserve serverless capacity and recovery across job failures
Retain durable reservations until provider termination or explicit recovery, protect unresolved pools and credentials, and add scoped reconciliation. Consolidate lifecycle and secret policies and split prompt/media stages with regression coverage.
2026-10-11 06:56:01 +00:00
Robert Wojciechowski a281d03c6d fix: validate complete serverless requests against 10 MB limit 2026-10-11 05:52:23 +00:00
Robert Wojciechowski 1093818327 feat: add Runpod serverless image collector pools 2026-10-11 04:55:01 +00:00
Robert Wojciechowski cedc8d4591 feat: support audio-only distributed workflows (#87)
Support image, audio, or combined distributed collection with guarded audio-only worker, master, endpoint, preview, and delegate-only paths.
2026-07-12 20:36:43 +10:00
Robert Wojciechowski 6e7818bee6 fix: preserve delegate-only master config inputs (#86)
Fixes #85
2026-06-05 17:12:25 +10:00
Robert Wojciechowski c297b34b94 Add DistributedValue node with per-worker overrides 2026-02-28 04:53:20 +00:00
Robert Wojciechowski b95675d1ff Move load balancing to collector/orchestrator and remove queue node 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 0dae73ec7d fix: rewrite synced media inputs to worker-local paths 2026-02-28 04:53:20 +00:00
Robert Wojciechowski df606af80e fix: sync LoadVideo 'file' media refs for remote workers 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 9058cf061d fix: sync audio media refs for worker prompt validation 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 04169ac651 fix: correct divider chunking and improve distributed validation/audio transport 2026-02-28 04:53:20 +00:00
Robert Wojciechowski e4c945e536 chore: commit all pending distributed changes 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 6de2050b3f fix(tests): restore config error contract and stub compatibility 2026-02-28 04:53:20 +00:00
Robert Wojciechowski 5eacd17aeb refactor: implement REFACTOR_PLAN_2 across distributed stack 2026-02-24 01:38:41 +00:00
Robert Wojciechowski f0fe702f80 Commit all current refactor changes 2026-02-23 23:47:23 +00:00