Commit Graph
17 Commits
Author SHA1 Message Date
Jason HokuandClaude Opus 4.6 833158fab6 fix: worker model sync parses LoRA combo strings correctly
The LoRA field in job configs uses a combo string format:
"name:model_str:clip_str + name2:model_str:clip_str"
Parts can also be folder references like "FolderName/:0.90:0.90"
or random selection syntax "Folder/[3,0.5]".

_extract_required_models() was passing the entire combo string as a
single filename, causing WinError 123 (invalid path) because colons
are illegal in Windows filenames.

Now splits on " + ", extracts the filename before ":", and handles:
- Individual LoRA files: synced from master
- Folder references (trailing /): expanded locally via folder_paths
- Random syntax ([count,strength]): expanded locally via folder_paths

Same fix applied to distribution_manager._get_upcoming_models() which
builds the pre-fetch list for workers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 14:03:55 -07:00
Jason HokuandClaude Opus 4.6 d5fd56541b fix: lazy-import torch/safetensors in network gateway, update stale comments
Move torch and safetensors imports inside huggingface_vae_decode() so
CivitAI and Distribution functions still work if those packages are
missing. Update stale "Uses urllib" comments to reference the gateway.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 02:27:20 -07:00
Jason HokuandClaude Opus 4.6 3da8a07f80 refactor: migrate distribution_worker.py to network gateway
All 6 urllib call patterns (download, register, claim, submit,
heartbeat, fail) now use gateway functions from network_utils.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 02:20:34 -07:00
JasonHokuandClaude Opus 4.6 91b8598ac3 feat: add seed settings, canvas flicker fix, upscaling & GPU cooldown
- Seed: expanded per-gen behavior + new full run seed behavior (random/increment/decrement before/after entire run)
- Canvas: fix flicker during simultaneous pan+zoom via requestAnimationFrame batching
- Upscaling: 3 session-level modes (HiRes fix, model upscale, model+HiRes) with tiled VAE support
- Cooldown: configurable GPU breaks between generations with optional VRAM clearing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 14:00:06 -07:00
JasonHokuandClaude Opus 4.6 836a7b3b70 fix: distributed worker missing 8 sampling parameters causing different outputs
The worker's generate_image() call was only passing basic parameters
(seed, steps, cfg, sampler, scheduler, denoise, attention_mode) but
missing all advanced/model-specific parameters that the local path
passes. This caused workers to produce different images than local runs.

Added missing parameters:
- model_sampling_override (noise schedule for Flux/SD3/AuraFlow)
- model_sampling_shift (shift values for model sampling)
- model_sampling_flux_max_shift / model_sampling_flux_base_shift
- use_advanced_sampling (SamplerCustomAdvanced vs standard KSampler)
- advanced_guider / advanced_scheduler
- flux_guidance_value
- width / height (for Flux2Scheduler computation)

All parameters were already present in the job config dict from
expand_configs() but were never forwarded to generate_image().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 16:22:30 -08:00
JasonHokuandClaude Opus 4.6 1e43776994 fix: free GPU memory before VAE decode + NaN diagnostics
Worker now clones latent and frees UNet tensors + CUDA cache before
VAE decode, giving Dynamic VRAM clean memory for VAE loading.
decode_latent_with_vae now detects NaN in input/output, logs diagnostic
info, retries with float32, and handles bfloat16->numpy conversion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 14:25:09 -08:00
JasonHokuandClaude Opus 4.6 3003fb8193 fix: add torch.inference_mode to worker thread for Dynamic VRAM compat
Worker threads run outside ComfyUI's execution engine which wraps
everything in torch.inference_mode(). Without it, Dynamic VRAM's lazy
weight loading produces NaN during VAE decode, causing black images.
Matches ComfyUI's execution.py:703 pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 14:06:56 -08:00
JasonHokuandClaude Opus 4.6 fddffd7a8c Add worker model sync and upcoming models pre-fetching
Worker downloads missing models from master via HTTP before processing.
Background pre-fetch thread downloads upcoming models during job execution.
Distribution manager scans pending queue for unique model references.
Job claim responses include upcoming_models list when sync enabled.
Master passes sync_models_to_workers flag to workers via start_worker.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 15:15:54 -08:00
JasonHoku 82a875fbbd distribution worker fix remote vae v2 2026-02-28 22:13:50 -08:00
JasonHoku 2c15fe035a fix remote vae on distribution worker 2026-02-28 21:56:12 -08:00
JasonHokuandClaude Opus 4.6 d87947bb11 Fix worker 503 race condition and skip lora trigger lookups with master encoding
1. Worker registration now retries up to 5 times with 3s delay on 503,
   handling the case where master is still starting up (e.g. pre-encoding).
2. Worker claim_job on 503 no longer immediately stops - gives master up to
   5 poll cycles to become active before giving up (only when no jobs processed yet).
3. When master pre-encoding is enabled, worker skips build_prompt_with_triggers()
   entirely (avoids CivitAI lookups and LoRA file hashing). Master now includes
   the final prompt text (with triggers) in the job dict for metadata use.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 19:53:47 -08:00
JasonHoku 25e4c657f0 fix lora triggerwords 2026-02-27 16:53:20 -08:00
JasonHoku 822373227c Optional Text Encoder On Master Only & Serve To Workers & More Distrib Logging 2026-02-26 20:11:59 -08:00
JasonHoku 5d3a76b246 distribution numpy img save fix, lora triggerword pathing fix 2026-02-26 17:16:02 -08:00
JasonHoku 2d5acaa7ca fix promptserver requests 2026-02-26 16:50:11 -08:00
JasonHoku 473c8509a3 add distribution logging 2026-02-26 16:22:21 -08:00
JasonHoku cfb75e7922 Create Run Distribution System 2026-02-26 15:37:55 -08:00