The master's wait-for-workers loop could block forever if a worker
crashed without cleanup, leaving a job in "Claimed" state. The
_release_timed_out_jobs() was only called during claim_job, never
during the wait loop.
Fixes:
- Call release_timed_out_jobs() every 5s during the wait loop to
reclaim jobs from dead workers (no heartbeat)
- Add configurable wait_timeout (default 600s/10min) from distribution
config — after timeout, force-reclaim all remaining claimed jobs and
proceed to finalization instead of looping forever
- Add public release_timed_out_jobs() method to DistributionManager
- Remaining jobs after timeout are marked completed and abandoned with
a warning, unblocking the ComfyUI queue
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The LoRA field in job configs uses a combo string format:
"name:model_str:clip_str + name2:model_str:clip_str"
Parts can also be folder references like "FolderName/:0.90:0.90"
or random selection syntax "Folder/[3,0.5]".
_extract_required_models() was passing the entire combo string as a
single filename, causing WinError 123 (invalid path) because colons
are illegal in Windows filenames.
Now splits on " + ", extracts the filename before ":", and handles:
- Individual LoRA files: synced from master
- Folder references (trailing /): expanded locally via folder_paths
- Random syntax ([count,strength]): expanded locally via folder_paths
Same fix applied to distribution_manager._get_upcoming_models() which
builds the pre-fetch list for workers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Seed: expanded per-gen behavior + new full run seed behavior (random/increment/decrement before/after entire run)
- Canvas: fix flicker during simultaneous pan+zoom via requestAnimationFrame batching
- Upscaling: 3 session-level modes (HiRes fix, model upscale, model+HiRes) with tiled VAE support
- Cooldown: configurable GPU breaks between generations with optional VRAM clearing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Worker downloads missing models from master via HTTP before processing.
Background pre-fetch thread downloads upcoming models during job execution.
Distribution manager scans pending queue for unique model references.
Job claim responses include upcoming_models list when sync enabled.
Master passes sync_models_to_workers flag to workers via start_worker.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each config array can now have its own resolution list displayed as chips with
categorized preset dropdown (SD 1.5/SDXL/Flux × Square/Portrait/Landscape) plus
custom W×H input. Resolutions flow through configs_json, expand in the Cartesian
product, and override the sampler's global resolutions_json on a per-config basis.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Worker registration now retries up to 5 times with 3s delay on 503,
handling the case where master is still starting up (e.g. pre-encoding).
2. Worker claim_job on 503 no longer immediately stops - gives master up to
5 poll cycles to become active before giving up (only when no jobs processed yet).
3. When master pre-encoding is enabled, worker skips build_prompt_with_triggers()
entirely (avoids CivitAI lookups and LoRA file hashing). Master now includes
the final prompt text (with triggers) in the job dict for metadata use.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
