9 Commits
Author SHA1 Message Date
Robert Wojciechowski 6eebb3a0ef fix: never replay load-balanced runs on missing history
Keep ambiguous submissions tracked until positive job-specific evidence or result delivery confirms acceptance. Bounded or cleared history cannot prove rejection, even on repeated checks.

Add regressions using the real worker history lookup, retain late-result and cancellation coverage, and document the intentional unresolved hold.
2026-10-21 00:06:28 +00:00
speedyrulzandClaude Fable 5.1 049565081d fix: keep a participating master free while a worker runs a load-balanced job
When a run went to a worker, the master's collect-only delegate prompt was
queued at once. ComfyUI's single executor then sat on that prompt waiting
for the worker, so every full workflow queued behind it waited too: with
one worker and a participating master, the master GPU only started once
the worker's first job finished.

On a participating master the delegate prompt is now held back and queued
only once the worker posts its final result (job_complete is_last), so the
master executor keeps pulling full workflows meanwhile. If the worker is
seen idle or unreachable for several probes without delivering, the
delegate is queued anyway so the Collector's timeout path reports it. Both
the immediate dispatch to an idle worker and dispatcher claims use this;
an orchestrator-only master, which has nothing else to run, still queues
the delegate immediately so the run stays visible in its queue.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-20 18:50:25 -04:00
speedyrulzandClaude Fable 5.1 366a8bc9a7 feat: hand load-balanced runs to the next free GPU
With load_balance enabled, a run was pinned to a participant at queue time
by comparing worker queue depths, so a faster GPU went idle while a slower
one still held a backlog. Runs queued while every participant is busy now
stay unassigned in the master's prompt queue; a dispatcher watches the
candidate workers and hands the oldest waiting run to whichever one frees
up first, re-queueing the master side as a delegate prompt under the same
prompt id. A participating master takes whatever waiting run its executor
reaches first. Idle participants are still used immediately.

Upscale workflows and serverless pools keep the previous queue-depth
selection.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-20 13:25:26 -04:00
Robert Wojciechowski a9ef36f951 feat: unify Collector media transport and add direct exact returns
Use the existing images/audio Collector for typed serverless pools. Preserve saved legacy_png defaults, map them to serverless png, and retain every transfer format with exact opt-in. Remove the separate Media Collector.

Send direct exact frames through bounded raw tensor multipart transport without 8-bit fallback. Reject incompatible legacy pools, invalid payloads and incomplete exact results. Cover real CPU HTTP round trips, native ComfyUI execution, mixed participant ordering and failure paths.
2026-10-14 03:41:05 +00:00
Robert Wojciechowski 32ac027e0e docs: align selected copy with current behavior 2026-07-12 11:13:22 +00:00
Robert Wojciechowski ff7bfb5ef5 Update docs for load balance, logs, and Distributed Value 2026-02-28 04:53:20 +00:00
Robert Wojciechowski f0fe702f80 Commit all current refactor changes 2026-02-23 23:47:23 +00:00
Robert Wojciechowski 7f24cb0c7a Refactor distributed orchestration and frontend modules 2026-02-23 11:11:40 +00:00
robertvoy 5dd10cef39 organise docs 2025-12-28 10:48:54 +11:00