diff --git a/ProjectStructure.md b/ProjectStructure.md index f8d2669..0522cf8 100644 --- a/ProjectStructure.md +++ b/ProjectStructure.md @@ -36,9 +36,9 @@ ComfyUI-Ultimate-Sampler-Grid/ ├── config_builder_node.py # "Config Builder" Node ├── json_text_node.py # "Smart JSON" Node │ -├── generation_orchestrator.py # The "Conductor" (Loop Logic) +├── generation_orchestrator.py # The "Conductor" (Loop Logic, ~1700 lines) ├── image_generation.py # KSampler & VAE Wrappers -├── model_loader.py # Checkpoint/LoRA Loading Logic +├── model_loader.py # Checkpoint/LoRA/VAE/GGUF Loading Logic ├── model_cache.py # 3-Tier Model Caching System ├── remote_vae.py # Remote VAE Offloading │ @@ -52,15 +52,19 @@ ComfyUI-Ultimate-Sampler-Grid/ ├── manifest_utils.py # Manifest Read/Write/Merge Helpers ├── html_generator.py # Reads /resources/ to build the Dashboard HTML │ +├── distribution_manager.py # Distributed Job Queue Coordinator (Master) +├── distribution_worker.py # Worker Thread for Remote Processing +├── distribution_routes.py # Distribution REST API Endpoints +│ ├── web/ # [ComfyUI Integration Layer] │ ├── dashboard.js # Registers Dashboard Node & Message Handling -│ ├── config_builder.js # Registers Builder Node & Custom UI Widget │ ├── smart_json_text.js # Registers JSON Node & Syntax Highlighting │ └── conf_builder/ # [Config Builder UI Modules] │ ├── conf-builder-main.js # Main entry, widget registration │ ├── conf-builder-ui-components.js # UI rendering (dropdowns, sliders, cards) │ ├── conf-builder-config-management.js # Save/Load/Duplicate config logic -│ └── conf-builder-utilities.js # State-to-JSON conversion, iteration counting +│ ├── conf-builder-utilities.js # State-to-JSON conversion, iteration counting +│ └── conf-builder-distribution.js # Distribution settings UI (workers, toggles) │ └── resources/ # [The Dashboard SPA Core] ├── template.html # HTML Skeleton for the generated report @@ -143,6 +147,17 @@ All routes registered in `__init__.py` via `PromptServer.instance.routes`. | POST | `/config_tester/export_favorites` | Export favorited images `{session_name, pack_metadata, organize_by_prompt, organize_by_lora}` | | POST | `/config_tester/scan_directory` | Scan external dir, create symlinks, use built-in `/view` `{directory_path, session_name}` | +**Distribution APIs** (`/distribution/`, registered in `distribution_routes.py`): + +| Method | Route | Purpose | +|---|---|---| +| GET | `/distribution/claim_job` | Worker claims next pending job (200 + job JSON, 204 = no jobs, 503 = inactive) | +| POST | `/distribution/submit_result` | Worker uploads completed image (multipart: {metadata, image}) | +| GET | `/distribution/status` | Get distribution status, worker stats, job counts | +| POST | `/distribution/register_worker` | Worker registration with master | +| GET | `/distribution/download_model` | Download model file for model sync `{category, filename}` | +| POST | `/distribution/heartbeat` | Worker keepalive signal | + --- ### **1. The Backend: Core Logic & Orchestration** @@ -157,9 +172,9 @@ All routes registered in `__init__.py` via `PromptServer.instance.routes`. * **Optional Inputs:** `optional_model` (MODEL), `optional_clip` (CLIP), `optional_vae` (VAE), `optional_positive` (CONDITIONING), `optional_negative` (CONDITIONING), `optional_latent` (LATENT) -* **`generation_orchestrator.py`** (The Conductor — largest file, ~987 lines) -* **Role:** Manages the generation loop. -* **Key Functions:** `run_generation_loop()`, `check_if_job_completed()`, `setup_session_directories()`, `load_model_by_type()`, `get_model_cache_key()`, `calculate_clip_hash()` +* **`generation_orchestrator.py`** (The Conductor — largest file, ~1700 lines) +* **Role:** Manages the generation loop, distributed processing, and master text encoding. +* **Key Functions:** `run_generation_loop()`, `check_if_job_completed()`, `setup_session_directories()`, `load_model_by_type()`, `get_model_cache_key()`, `calculate_clip_hash()`, `_conditioning_to_serializable()`, `_serializable_to_conditioning()`, `_preencode_all_conditionings()`, `_run_distributed_generation()`, `_send_distribution_status()` * **Key Logic:** * **Smart Skip:** `check_if_job_completed()` matches against manifest items by seed/resolution/sampler/scheduler/steps/cfg/denoise. Skips model/lora/prompt matching when optional inputs connected. * **Lookahead:** Triggers `model_cache.preload_lora_model()` / `preload_base_model()` for the *next* job while the current one runs. @@ -168,6 +183,9 @@ All routes registered in `__init__.py` via `PromptServer.instance.routes`. * **Node Change Detection:** Detects when connected upstream nodes have changed for smarter job resuming. * **Pre-encoding:** Single model → batch-encode all unique prompts upfront. Multi-model → re-encode per model switch. * **VAE Switching:** Flushes pending batch before switching VAE. Tracks `default_model_vae` for reverting to "Default". +* **Distribution:** `_run_distributed_generation()` manages the distributed pipeline — populates job queue, optionally pre-encodes conditionings, notifies workers, and processes master's share of jobs. +* **Master Text Encoding:** `_preencode_all_conditionings()` groups configs by (model, LoRA), loads each combination, encodes all unique prompts, and serializes via `_conditioning_to_serializable()`. Handles multi-entry conditionings (AND combinator). +* **Per-Worker Logging:** Final summary prints each worker's completed/failed job counts from `manager.get_status()["workers"]`. @@ -181,6 +199,7 @@ All routes registered in `__init__.py` via `PromptServer.instance.routes`. * **Role:** `UltimateConfigBuilder` node. All state stored in single `lora_config` widget as JSON string. * **Key Functions:** `generate_config()`, `process_lora_array()`, `get_available_sessions()`, `expand_lora_folders()`, `lookup_lora_triggers()` * **Key Logic:** Python reads ONLY `lora_config` widget — all other INPUT_TYPES widgets (samplers, schedulers, steps, cfg) are vestigial and ignored. Also defines API routes for trigger/metadata lookup and model lists. +* **Distribution Output:** When distribution is enabled, wraps configs in `{"configs": [...], "_distribution": {...}}` with keys: `enabled`, `worker_urls`, `claim_timeout` (default 600), `use_master_encoding`, `sync_models_to_workers`. When disabled, still wraps in `{"configs": [...]}` — `sampler_node.py` must unwrap in both cases. * **`html_generator.py`** (The Builder) @@ -247,6 +266,39 @@ All routes registered in `__init__.py` via `PromptServer.instance.routes`. * **Key Logic:** Packs ComfyUI-compatible metadata into PNG for CivitAI uploads. Maintains SHA256 hash cache at `benchmarks/model_hashes.json`. +--- + +### **2b. The Backend: Distribution System** + +*Located in the root directory. Enables multi-machine parallel processing.* + +* **`distribution_manager.py`** (The Job Coordinator) +* **Role:** Thread-safe distributed job queue running on the master. Manages job lifecycle: PENDING → CLAIMED → COMPLETED/FAILED. +* **Classes:** `JobState` (constants), `DistributedJob` (individual job), `DistributionManager` (coordinator) +* **Key Methods:** `populate_jobs()`, `claim_job(worker_id)`, `complete_job(job_id)`, `get_stats()`, `set_encoded_conditionings()`, `get_encoded_conditionings_for_job()` +* **Key Logic:** + * Thread-safe with `RLock` for concurrent worker access + * Job timeout handling (default 600s) — re-queues timed-out jobs as PENDING + * `MAX_FAIL_COUNT = 3` — jobs that fail 3 times are abandoned + * `_job_to_dict()` attaches pre-encoded conditionings to job claim response if available + * Tracks per-worker stats (completed, failed, last_seen) + +* **`distribution_worker.py`** (The Remote Worker) +* **Role:** Background thread running on remote ComfyUI instances. Polls master for jobs, processes locally, uploads results. +* **Class:** `WorkerThread(threading.Thread)` — daemon thread +* **Key Methods:** `run()` (main loop), `_process_job()`, `_download_model_from_master()`, `_prefetch_models()`, `stop()` +* **Key Logic:** + * Polls `/distribution/claim_job` every 2 seconds + * Caches model/CLIP/VAE between jobs (reuses when same model) + * Checks for `encoded_positive`/`encoded_negative` in job data — uses master conditionings if available, falls back to local CLIP encoding otherwise + * Model sync: downloads models from master via `/distribution/download_model` if enabled + * Exits after 30 consecutive empty polls or 503 response (master finished) + +* **`distribution_routes.py`** (The API Layer) +* **Role:** Registers `/distribution/*` API endpoints on the PromptServer. +* **Global State:** `_distribution_manager` (active manager, master mode), `_worker_thread` (active worker, worker mode) — both thread-safe via getters/setters +* **Key Endpoints:** `claim_job` (GET), `submit_result` (POST multipart), `status` (GET), `register_worker` (POST), `download_model` (GET), `heartbeat` (POST) + --- ### **Key Data Shapes** @@ -274,10 +326,46 @@ All routes registered in `__init__.py` via `PromptServer.instance.routes`. "model_prompt_prefix": "", "model_prompt_suffix": "", "attention_modes": ["default"], + "resolutions": [], "seed_behavior": "fixed", "text_encoders": [], "clip_type": "stable_diffusion", - "gguf_options": {} + "gguf_options": {}, + "model_sampling_override": "None", + "model_sampling_shift": 3.0, + "use_advanced_sampling": false, + "advanced_guider": "Basic", + "use_flux_guidance": false, + "flux_guidance_value": 3.5 +} +``` + +**Distribution Config** (embedded in `config_builder_node.py` output as `_distribution` key): +```json +{ + "enabled": true, + "worker_urls": ["http://192.168.1.100:8188", "http://192.168.1.101:8188"], + "claim_timeout": 600, + "use_master_encoding": false, + "sync_models_to_workers": false +} +``` + +**Top-level state fields** (in `conf-builder-main.js` default state, outside `config_arrays`): +```json +{ + "session_name": "my_session", + "config_name": "my_config", + "auto_save": false, + "include_none": true, + "label_mode": "full", + "global_positive_groups": [], + "global_negative": "", + "distribution_enabled": false, + "worker_urls": [], + "claim_timeout": 600, + "use_master_encoding": false, + "sync_models_to_workers": false } ``` @@ -327,6 +415,25 @@ iframe postMessage({type: 'toggle_fullscreen', node_id}) → dashboard.js listener → toggle .dashboard-fullscreen class ``` +**Distribution Status Updates (Server → Dashboard iframe):** +``` +generation_orchestrator → PromptServer.send_sync("ultimate_grid.distribution_status") + → dashboard.js api.addEventListener → postMessage({type: 'distribution_status'}) + → iframe displays worker activity +``` + +**Distribution Job Flow (Master ↔ Worker):** +``` +Master: + 1. generation_orchestrator → DistributionManager.populate_jobs() + 2. [Optional] _preencode_all_conditionings() → manager.set_encoded_conditionings() + 3. POST /distribution/register_worker on each worker + 4. Workers poll GET /distribution/claim_job + 5. manager._job_to_dict() attaches encoded_positive/encoded_negative if available + 6. Worker processes job → POST /distribution/submit_result (multipart: metadata + image) + 7. Master saves image + updates manifest +``` + --- ### **3. The Frontend: ComfyUI Integration** @@ -334,9 +441,14 @@ iframe postMessage({type: 'toggle_fullscreen', node_id}) *Located in `/web/`.* * **`dashboard.js`** -* **Role:** Registers the Dashboard node (`UltimateGridDashboard`). Listens for `ultimate_grid.update` and `ultimate_grid.progress` server events. -* **Key Logic:** Forwards events to matching iframe via `postMessage()`. Auto-loads session when update received for unloaded session. Handles fullscreen toggle via `.dashboard-fullscreen` CSS class. +* **Role:** Registers the Dashboard node (`UltimateGridDashboard`). Listens for `ultimate_grid.update`, `ultimate_grid.progress`, and `ultimate_grid.distribution_status` server events. +* **Key Logic:** Forwards events to matching iframe via `postMessage()`. Auto-loads session when update received for unloaded session. Handles fullscreen toggle via `.dashboard-fullscreen` CSS class. **Auto-loads on node creation** with 500ms delay via `forceLoadSession()` so the dashboard shows content immediately (landing page or session grid) instead of a blank black box. * **Widgets:** "RELOAD / SHOW SESSION" button → `forceLoadSession()`, "DELETE SESSION" button → POST `/config_tester/delete_session` +* **Event Listeners (Singletons):** Each uses `window.__ultimateGrid*ListenerInstalled` guard to prevent duplicate registration: + * `ultimate_grid.update` — Live data updates to iframe + * `ultimate_grid.progress` — ETA/progress bar updates + * `ultimate_grid.distribution_status` — Distribution worker status + * `window "message"` — Fullscreen toggle from iframe * **`config_builder.js`** (entry point, loads `conf_builder/` modules) @@ -359,9 +471,14 @@ iframe postMessage({type: 'toggle_fullscreen', node_id}) * **`conf_builder/conf-builder-ui-components.js`** (The Widget Library) -* **Key Exports:** `createSearchableSelect()`, `createSlider()`, `createInputGroup()`, `getStyles()` +* **Key Exports:** `createSearchableSelect()`, `createSlider()`, `createInputGroup()`, `createTopBar()`, `createSidebar()`, `createSectionHeader()`, `getStyles()` +* **`conf_builder/conf-builder-distribution.js`** (Distribution Settings UI) +* **Role:** Renders the distribution settings section in the Config Builder. +* **Key Exports:** `renderDistributionSection()`, `renderDistributionSettingsSection()` +* **Features:** Worker URL management (add/remove), connection testing via `/distribution/test_worker`, status indicators (green/yellow/red dots), claim timeout slider (30-900s, default 600), toggles for "Use Master Text Encoding" and "Sync Models to Workers". + --- @@ -430,7 +547,7 @@ iframe postMessage({type: 'toggle_fullscreen', node_id}) |---|---| | `sampler_node.py` | `run_tests()`, `find_existing_match()`, `get_latent_channels()`, `IS_CHANGED()` | | `config_builder_node.py` | `generate_config()`, `process_lora_array()`, `get_available_sessions()`, `expand_lora_folders()` | -| `generation_orchestrator.py` | `run_generation_loop()`, `check_if_job_completed()`, `setup_session_directories()`, `load_model_by_type()` | +| `generation_orchestrator.py` | `run_generation_loop()`, `check_if_job_completed()`, `setup_session_directories()`, `load_model_by_type()`, `_preencode_all_conditionings()`, `_conditioning_to_serializable()`, `_serializable_to_conditioning()`, `_run_distributed_generation()` | | `batch_encoding.py` | `encode_prompt_with_combinators()`, `batch_encode_prompts()` | | `config_utils.py` | `expand_configs()`, `parse_prompt_input_nested()`, `prepare_input_jobs()`, `parse_lora_definition()` | | `model_loader.py` | `load_checkpoint()`, `load_loras()`, `load_vae_by_name()`, `load_diffusion_model_and_clip()` | @@ -439,6 +556,9 @@ iframe postMessage({type: 'toggle_fullscreen', node_id}) | `manifest_utils.py` | `load_existing_manifest()`, `save_manifest()`, `merge_manifest_user_changes()` | | `directory_scanner.py` | `scan_directory_for_images()`, `parse_a1111_parameters()` | | `metadata_packer.py` | `pack_metadata_into_image()`, `extract_metadata_from_image()`, `calculate_file_hash()` | +| `distribution_manager.py` | `DistributionManager.populate_jobs()`, `.claim_job()`, `.complete_job()`, `.get_stats()`, `.set_encoded_conditionings()`, `.get_encoded_conditionings_for_job()` | +| `distribution_worker.py` | `WorkerThread.run()`, `._process_job()`, `._download_model_from_master()`, `._prefetch_models()`, `.stop()` | +| `distribution_routes.py` | `get_distribution_manager()`, `set_distribution_manager()`, endpoint handlers for `/distribution/*` | **JavaScript (by file):** @@ -447,7 +567,8 @@ iframe postMessage({type: 'toggle_fullscreen', node_id}) | `conf-builder-utilities.js` | `convertStateToConfigs()`, `getIterationCount()`, `parseLoraString()`, `buildLoraString()`, `refreshAllConfigBuilders()` | | `conf-builder-config-management.js` | `renderUI()`, `updatePreview()`, `createConfigArrayElement()`, `createModelElement()` | | `conf-builder-ui-components.js` | `createSearchableSelect()`, `createSlider()`, `getStyles()` | -| `conf-builder-main.js` | `ensureModulesLoaded()`, `node.saveState()`, `node.renderUI()`, `node.loadSession()` | +| `conf-builder-main.js` | `ensureModulesLoaded()`, `node.saveState()`, `node.renderUI()`, `node.loadSession()`, `node.triggerAutoSave()` | +| `conf-builder-distribution.js` | `renderDistributionSection()`, `renderDistributionSettingsSection()` | | `logic_ui.js` | `toggleCogMenu()`, `toggleFiltersPopup()`, `initFilters()` | | `logic_utils.js` | `loadSession()`, `exportFavorites()`, `scanDirectory()`, `triggerGen()`, `toggleFullscreen()` | | `logic_virtual.js` | `renderDOM()`, `updateVisibleItems()`, `calculateVisibleRange()`, `autoFitZoom()`, `goToImage()` | @@ -458,19 +579,23 @@ iframe postMessage({type: 'toggle_fullscreen', node_id}) ### **Summary of Data Flow** -1. **User Config:** You set parameters in the **Sampler Node** (Python). -2. **Expansion:** `generation_orchestrator.py` expands this into a list of jobs. -3. **Generation:** The backend loops through jobs, using **ModelCache** and **ConditioningCache** for speed. -4. **Storage:** Images are saved to disk; metadata is appended to `manifest.json`. -5. **Compilation:** `html_generator.py` reads the `manifest.json` and all `.js` files in `/resources/`, baking them into a single `dashboard_html` string. -6. **Display:** The **Dashboard Node** (JS) renders this HTML in an iframe. -7. **Boot:** `logic_init.js` wakes up, loads the data into `logic_state.js`, and `logic_virtual.js` starts rendering cards. -8. **Interaction:** You click "Favorite" in the UI -> `logic_events.js` sends an API request to `__init__.py` -> Python updates `manifest.json` on disk. +1. **User Config:** You set parameters in the **Config Builder** (visual UI) or **Sampler Node** (JSON text). +2. **Config Output:** `config_builder_node.py` wraps configs in `{"configs": [...], "_distribution": {...}}`. The `sampler_node.py` unwraps the `"configs"` key and extracts distribution settings. +3. **Expansion:** `generation_orchestrator.py` expands configs into a full list of jobs (Cartesian product). +4. **Distribution (optional):** If enabled, `_run_distributed_generation()` populates the job queue, optionally pre-encodes conditionings, and notifies workers. Workers poll for jobs, process locally, and upload results. +5. **Generation:** The backend loops through its share of jobs, using **ModelCache** and **ConditioningCache** for speed. +6. **Storage:** Images are saved to disk; metadata is appended to `manifest.json`. +7. **Compilation:** `html_generator.py` reads the `manifest.json` and all `.js` files in `/resources/`, baking them into a single `dashboard_html` string. +8. **Display:** The **Dashboard Node** (JS) renders this HTML in an iframe. Auto-loads on node creation. +9. **Boot:** `logic_init.js` calls `init()` which calls `showSessionLandingIfEmpty()`. If no data, the session landing page is shown. If data exists, `logic_virtual.js` renders the grid. +10. **Interaction:** You click "Favorite" in the UI -> `logic_events.js` sends an API request to `__init__.py` -> Python updates `manifest.json` on disk. --- ### **Critical Development Gotchas** +> ⚠️ **GOLDEN RULE: DO NOT REMOVE ANY CODE. DO NOT REMOVE ANY COMMENTS. ONLY CHANGE WHAT IS NECESSARY.** + 1. **Dual Data Path** — Frontend `convertStateToConfigs()` (in `conf-builder-utilities.js`) and Python `generate_config()` (in `config_builder_node.py`) are **independent implementations** that both transform `node.state` into config JSON. Both must stay in sync when adding new config fields or the UI preview will mismatch the actual output. 2. **Widget Bridge** — The `lora_config` widget (STRING type) holds the **entire** node state as serialized JSON. Python `generate_config()` ignores all other widget params and reads only `lora_config`. The `samplers`, `schedulers`, `steps`, `cfg` widgets in `INPUT_TYPES` are vestigial (kept for ComfyUI schema compliance only). @@ -493,4 +618,105 @@ iframe postMessage({type: 'toggle_fullscreen', node_id}) 11. **Field Name Inconsistency: `favorite` vs `favorited`** — `manifest_utils.py` `merge_manifest_user_changes()` uses the key `"favorite"`. But `__init__.py` routes like `save_manifest` and `export_favorites` check `"favorited"`. When touching favorite/unfavorite logic, check which key name the specific code path uses. -12. **Prompt Expansion is Recursive** — `parse_prompt_input_nested()` in `config_utils.py` supports arbitrarily deep nested arrays. Flat list = OR (options). List containing sub-lists = AND (Cartesian product). This is used by both the node text inputs and per-config prompt groups from the builder UI. \ No newline at end of file +12. **Prompt Expansion is Recursive** — `parse_prompt_input_nested()` in `config_utils.py` supports arbitrarily deep nested arrays. Flat list = OR (options). List containing sub-lists = AND (Cartesian product). This is used by both the node text inputs and per-config prompt groups from the builder UI. + +13. **Config Builder Output is ALWAYS Wrapped** — `config_builder_node.py` always outputs `{"configs": [...]}` (with optional `"_distribution"` key). The `sampler_node.py` MUST unwrap the `"configs"` key before passing to `expand_configs()`. Both the `_distribution` present AND absent cases must be handled (see lines 269-286 of `sampler_node.py`). + +14. **Dashboard Auto-Load on Creation** — `web/dashboard.js` calls `forceLoadSession()` with 500ms delay when a dashboard node is created. The `get_session_html` endpoint in `__init__.py` returns a full HTML template with empty manifest for non-existent sessions (NOT a 404). This ensures the landing page shows immediately. + +15. **`logic_pipeline.js` Legacy Init Guard** — Lines 489-505 have a `DOMContentLoaded` handler that calls `loadSession()` ONLY when `fullManifest.items.length > 0`. Without this guard, `loadSession()` triggers an alert popup for non-existent sessions. The proper initialization path is `logic_init.js` → `init()` → `showSessionLandingIfEmpty()`. + +16. **Distribution Default Values Must Stay in Sync** — `claim_timeout` default (600) appears in: `distribution_manager.py` (constructor), `config_builder_node.py` (fallback), `conf-builder-distribution.js` (slider default), `conf-builder-main.js` (default state). All four must match. Same for `use_master_encoding` default (false). + +17. **Conditioning Serialization Uses np.copy()** — `_serializable_to_conditioning()` in `generation_orchestrator.py` must call `np.frombuffer(...).copy()` before `torch.from_numpy()` because `np.frombuffer()` returns read-only arrays that are incompatible with GPU tensor operations. + +18. **Multi-Entry Conditioning (AND combinator)** — The AND combinator creates multiple `[tensor, {pooled_output}]` entries in a conditioning list. Serialization in `_conditioning_to_serializable()` must iterate ALL entries, not just `[0]`. Deserialization in `_serializable_to_conditioning()` must reconstruct the full list. + +19. **State Migration for New Fields** — When adding new fields to the Config Builder default state in `conf-builder-main.js` (around line 89), also add a migration check (around line 363) to backfill the default for existing saved workflows. Without migration, old workflows will have `undefined` for the new field. + +20. **ComfyUI Registry Security** — No `import requests`, no `subprocess`/`os.system`/`eval()`/`exec()`, no runtime `pip install`, no custom file-serving endpoints, no code obfuscation. Use `urllib.request` for HTTP. Use ComfyUI's `/view` endpoint for serving files. Sanitize all user-supplied paths. See top of this file for full rules. + +--- + +### **Notes for AI Development** + +This section provides critical context for AI assistants working on this codebase. + +**Architecture Mental Model:** +``` +┌─────────────────────────────────────────────────────────────┐ +│ ComfyUI Browser Tab │ +│ ┌──────────────────┐ ┌──────────────────────────────────┐│ +│ │ Config Builder │ │ Dashboard Node ││ +│ │ (conf-builder-*) │ │ (dashboard.js) ││ +│ │ Direct DOM widget │ │ ┌──────────────────────────────┐││ +│ │ │ │ │ iframe (srcdoc) │││ +│ │ │ │ │ Dashboard SPA │││ +│ │ │ │ │ (resources/logic_*.js) │││ +│ │ │ │ │ postMessage() ↕ │││ +│ │ │ │ └──────────────────────────────┘││ +│ └──────────────────┘ └──────────────────────────────────┘│ +│ ↕ widget value ↕ server events │ +├─────────────────────────────────────────────────────────────┤ +│ ComfyUI Server (Python) │ +│ ┌──────────────┐ ┌────────────────┐ ┌────────────────┐ │ +│ │config_builder│ │sampler_node.py │ │__init__.py │ │ +│ │_node.py │→ │ │ │(API endpoints) │ │ +│ └──────────────┘ │ ↓ │ └────────────────┘ │ +│ │generation_ │ │ +│ │orchestrator.py │ │ +│ │ ↓ │ │ +│ ┌─────────┐ │model_loader + │ ┌─────────────────┐ │ +│ │ModelCache│←────│image_generation│ │DistributionMgr │ │ +│ └─────────┘ │ ↓ │ │(master jobs) │ │ +│ │html_generator │ └─────────────────┘ │ +│ └────────────────┘ ↕ HTTP │ +│ ┌─────────────────┐ │ +│ │WorkerThread │ │ +│ │(remote workers) │ │ +│ └─────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +**File Dependency Graph (Python):** +``` +sampler_node.py + → generation_orchestrator.py (main entry) + → config_utils.py (expand_configs, prepare_input_jobs) + → model_loader.py (load_checkpoint, load_loras) + → image_generation.py (generate_image, flush_batch) + → batch_encoding.py (encode_prompt_with_combinators) + → trigger_words.py (build_prompt_with_triggers) + → manifest_utils.py (save_manifest, load_existing_manifest) + → model_cache.py (ModelCache) + → conditioning_cache.py (ConditioningCache) + → remote_vae.py (RemoteVAEDecodeWorker) + → html_generator.py (get_html_template) + → distribution_manager.py (DistributionManager) + → distribution_worker.py (WorkerThread) + +config_builder_node.py + → lora_utils.py (trigger word lookup, CivitAI API) + +__init__.py + → all node classes (sampler, dashboard, config_builder, json_text) + → distribution_routes.py (distribution API endpoints) + → html_generator.py (on-demand HTML generation) + → metadata_packer.py (export favorites) + → directory_scanner.py (scan external directories) + → manifest_utils.py (save/load manifests) +``` + +**Server Event Types:** + +| Event Name | Sender | Data Fields | Purpose | +|---|---|---|---| +| `ultimate_grid.update` | `generation_orchestrator.py` | `session_name, node, manifest, meta, new_items` | New images generated | +| `ultimate_grid.progress` | `generation_orchestrator.py` | `session_name, progress_pct, eta_str, current_job, total_jobs, avg_duration, total_generated, total_elapsed, finish_time` | ETA/progress bar | +| `ultimate_grid.distribution_status` | `generation_orchestrator.py` | `session_name, workers, stats` | Distribution worker status | + +**Testing Changes:** +- **Python backend files:** Restart ComfyUI +- **`web/` JS files:** Refresh browser (Ctrl+F5) +- **`resources/` JS/CSS/HTML files:** Dashboard HTML must be regenerated — re-run the sampler node or call `/config_tester/get_session_html` +- **Config Builder state fields:** Also update migration checks in `conf-builder-main.js` \ No newline at end of file diff --git a/README.md b/README.md index d7ba2ca..67a89a0 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,12 @@ Stop guessing which Sampler, Scheduler, Prompt, Denoise, Model, Lora or CFG valu - [API Integration (Config Builder)](#api-integration-config-builder) - [File Locations](#file-locations) - [Keyboard Shortcuts](#keyboard-shortcuts) +- [Distributed Processing](#-distributed-processing) + - [Setting Up Distribution](#setting-up-distribution) + - [Master Text Encoding](#master-text-encoding) + - [Model Sync](#model-sync) + - [Distribution Troubleshooting](#distribution-troubleshooting) +- [Notes for AI Development](#-notes-for-ai-development) - [Changelog](#-changelog) - [License](#-license) - [Credits](#-credits) @@ -128,6 +134,17 @@ Stop guessing which Sampler, Scheduler, Prompt, Denoise, Model, Lora or CFG valu * **Recursive Cartesian Prompt Builder:** Visual chip-based prompt editor with `{option1|option2|option3}` syntax that expands into all combinations. Nest multiple groups for exponential prompt variation. * **Model & LoRA On/Off Switches:** Toggle individual models and LoRAs on/off without removing them from the config. Bypassed entries are omitted from both the preview JSON and the generated config output. +### 🌐 Distributed Processing (Multi-Machine) +* **Multi-Worker Distribution:** Split large grid runs across multiple ComfyUI instances. A master node coordinates job distribution while workers process images in parallel and upload results back. +* **One-Click Setup:** Enable distribution in the Config Builder's Distribution Settings section. Add worker URLs (e.g., `http://192.168.1.100:8188`) and toggle "Enable Distribution." +* **Worker Connection Testing:** Test connectivity to each worker with a single click — status dots show green (reachable), yellow (testing), or red (error). +* **Master Text Encoding:** Optional "Use Master Text Encoding" toggle — the master pre-encodes ALL unique prompts (including LoRA-specific combinations) and sends the encoded conditionings to workers. Workers skip CLIP encoding entirely, saving significant time on prompt-heavy grids. +* **Model Sync:** Optional "Sync Models to Workers" toggle — automatically transfers checkpoint and LoRA files from the master to workers that don't have them. +* **Smart Job Queue:** Thread-safe job distribution with automatic re-queuing on worker timeout (configurable, default 600s) and max 3 retries before abandoning a job. +* **Per-Worker Logging:** Final generation summary shows each worker's completed/failed job counts for full visibility into the distributed run. +* **Live Distribution Status:** Dashboard receives real-time distribution status updates showing worker activity and progress. +* **Backward Compatible:** Workers gracefully fall back to local CLIP encoding if master-encoded conditionings aren't available. + ### 🎨 Interactive Dashboard (The "IDE") * **Infinite Canvas with Pan/Zoom:** Google Maps-style navigation with mouse drag, mousewheel zoom, and keyboard shortcuts. * **Virtual Scrolling:** Ultra-optimized rendering handles thousands of images smoothly by only loading visible items — scroll through 5000+ images without lag. @@ -184,18 +201,52 @@ The Config Builder node is included automatically — no separate installation i **File Structure:** ``` ComfyUI/custom_nodes/ComfyUI-Ultimate-Auto-Sampler/ -├── config_builder_node.py # Config Builder backend -├── js/conf_builder/ -│ ├── conf-builder-main.js # Entry point & node registration -│ ├── conf-builder-utilities.js # Data fetching & parsing -│ ├── conf-builder-ui-components.js # Reusable UI elements -│ └── conf-builder-config-management.js # Rendering & state logic -├── trigger_words.py # LoRA trigger word handling -├── batch_encoding.py # CLIP batch encoding with caching -├── manifest_utils.py # Manifest file management -├── model_loader.py # Model/LoRA loading and patching -├── image_generation.py # Image generation and sampling -└── generation_orchestrator.py # Main orchestration layer +├── __init__.py # Server routes, API endpoints, node mappings +├── sampler_node.py # Main "Sampler Grid" node +├── dashboard_node.py # "Dashboard Viewer" node +├── config_builder_node.py # "Config Builder" node backend +├── json_text_node.py # "Smart JSON" node +│ +├── generation_orchestrator.py # Main orchestration layer & generation loop +├── image_generation.py # KSampler & VAE wrappers +├── model_loader.py # Checkpoint/LoRA/VAE loading and patching +├── model_cache.py # 3-tier model caching system +├── remote_vae.py # Remote VAE offloading +│ +├── batch_encoding.py # CLIP batch encoding with combinator support +├── conditioning_cache.py # Disk-based conditioning cache +├── lora_utils.py # LoRA searching & validation +├── trigger_words.py # CivitAI trigger word fetching & prompt assembly +├── config_utils.py # Config parsing, expansion & Cartesian products +├── directory_scanner.py # External directory session scanner +├── metadata_packer.py # PNG/WebP metadata embedding for exports +├── manifest_utils.py # Manifest read/write/merge helpers +├── html_generator.py # Reads /resources/ to build dashboard HTML +│ +├── distribution_manager.py # Distributed job queue coordinator (master) +├── distribution_worker.py # Worker thread for remote processing +├── distribution_routes.py # Distribution API endpoints +│ +├── web/ # [ComfyUI Integration Layer] +│ ├── dashboard.js # Dashboard node registration & messaging +│ ├── smart_json_text.js # JSON node & syntax highlighting +│ └── conf_builder/ # [Config Builder UI Modules] +│ ├── conf-builder-main.js # Entry point, widget registration +│ ├── conf-builder-ui-components.js # Reusable UI (dropdowns, sliders, cards) +│ ├── conf-builder-config-management.js # Save/Load/state rendering logic +│ ├── conf-builder-utilities.js # Data fetching, caching, calculations +│ └── conf-builder-distribution.js # Distribution settings UI +│ +└── resources/ # [Dashboard SPA Core] + ├── template.html # HTML skeleton for dashboard + ├── report.css # Styling (infinite canvas, cards, modals) + ├── logic_state.js # Global state variables + ├── logic_utils.js # API & helper functions + ├── logic_ui.js # UI renderer (DOM creation, modals) + ├── logic_virtual.js # Virtual scroll engine & viewport + ├── logic_pipeline.js # Sorting & filtering pipeline + ├── logic_events.js # Event listeners (messages, keyboard) + └── logic_init.js # Initialization & startup ``` --- @@ -1169,6 +1220,19 @@ Scan an external directory for image sessions and load them into the dashboard. } ``` +### Distribution API (registered via `distribution_routes.py`) + +These endpoints are used internally by the distribution system: + +| Method | Route | Purpose | +|--------|-------|---------| +| GET | `/distribution/claim_job` | Worker claims next pending job (200 + job JSON, 204 = no jobs, 503 = inactive) | +| POST | `/distribution/submit_result` | Worker uploads completed image (multipart: metadata + image) | +| GET | `/distribution/status` | Get distribution status and worker stats | +| POST | `/distribution/register_worker` | Worker registration with master | +| GET | `/distribution/download_model` | Download model file for model sync | +| POST | `/distribution/heartbeat` | Worker keepalive signal | + --- ## File Locations @@ -1229,8 +1293,161 @@ ComfyUI/output/benchmarks/model-data/ --- +## 🌐 Distributed Processing + +Distribute large grid runs across multiple ComfyUI instances for parallel image generation. + +### Setting Up Distribution + +1. **Ensure Workers Are Running:** Each worker must be a separate ComfyUI instance with this node suite installed and running. +2. **Open Config Builder:** In the Config Builder node, scroll to the **Distribution Settings** section. +3. **Add Worker URLs:** Enter the full URL of each worker (e.g., `http://192.168.1.100:8188`). Click **+ Add Worker** for additional workers. +4. **Test Connections:** Click the test button next to each worker URL. A green dot means the worker is reachable. +5. **Enable Distribution:** Toggle the distribution switch ON. +6. **Run Generation:** Queue the workflow. The master node will coordinate job distribution automatically. + +**How It Works:** +``` +Master (your machine): + 1. Expands configs into individual jobs + 2. [Optional] Pre-encodes all prompts (Master Text Encoding) + 3. Notifies workers to start polling + 4. Processes its own jobs while workers claim theirs + 5. Workers upload completed images back to master + 6. Master saves everything to the session manifest +``` + +### Master Text Encoding + +When enabled, the master pre-encodes ALL unique positive/negative prompts for every (model, LoRA) combination before distributing jobs. Workers receive the encoded conditionings in their job claim response and skip CLIP encoding entirely. + +**Benefits:** +- Workers don't need to load text encoders at all +- Eliminates duplicate CLIP encoding across workers +- Significant time savings for prompt-heavy grids + +**Enable:** Toggle "Use Master Text Encoding" in Distribution Settings. + +**Technical Notes:** +- Encoded conditionings are serialized as base64-encoded NumPy arrays (~640KB per job) +- Multi-entry conditionings (AND combinator) are fully supported +- Workers fall back to local encoding if pre-encoded data is unavailable (backward compatible) + +### Model Sync + +When enabled, the master automatically transfers checkpoint and LoRA files to workers that don't have them. + +**Enable:** Toggle "Sync Models to Workers" in Distribution Settings. + +### Distribution Troubleshooting + +* **Workers not connecting:** Verify the worker URL is correct (include port). Check firewalls. Both master and worker must have this node suite installed. +* **Jobs timing out:** Increase the "Claim Timeout" slider (default 600s). First-time model loads on workers can take several minutes. +* **Duplicate images:** If a worker times out but actually completes the job, duplicates may appear. Increase claim timeout to prevent this. +* **Workers not using pre-encoded prompts:** Ensure "Use Master Text Encoding" is enabled. Check worker console for `"Using master pre-encoded conditionings"` log message. +* **Model sync failing:** Ensure both master and worker have sufficient disk space. Check worker console for download errors. + +--- + +## 🤖 Notes for AI Development + +This section provides critical context for AI assistants (Claude, GPT, etc.) working on this codebase. + +### Critical Constraints + +**DO NOT REMOVE ANY CODE. DO NOT REMOVE ANY COMMENTS. ONLY CHANGE WHAT IS NECESSARY.** This is the #1 rule for all code changes. The codebase has many interconnected parts and removing code often breaks things in unexpected ways. + +### Architecture Mental Model + +This is a **hybrid application** with three layers: +1. **Python Backend** — Runs inside ComfyUI's server process. Handles generation, caching, file I/O, and API endpoints. +2. **JS Frontend Bridge** (`web/` directory) — Runs in the ComfyUI browser tab. Registers custom nodes, manages widget state, and forwards server events. +3. **Dashboard SPA** (`resources/` directory) — Runs **inside an iframe** in the dashboard node. A standalone app with its own state management, virtual scrolling, and event handling. + +### Key Gotchas for AI Development + +1. **Config Builder output is ALWAYS wrapped** — `config_builder_node.py` outputs `{"configs": [...], "_distribution": {...}}`. The `sampler_node.py` MUST unwrap the `"configs"` key before passing to `expand_configs()`. If you add new top-level keys to the config output, update the extraction logic in `sampler_node.py` lines 269-286. + +2. **Two independent config-to-JSON implementations** — `convertStateToConfigs()` in `conf-builder-utilities.js` (JS preview) and `generate_config()` in `config_builder_node.py` (Python output) both transform `node.state` into config JSON. **Both must stay in sync** when adding new config fields. + +3. **Dashboard runs in an iframe** — The dashboard HTML is set via `iframe.srcdoc`. All JS/CSS is inlined (no external requests). Communication between the parent ComfyUI page and the iframe uses `postMessage()`. The `dashboard.js` file bridges server events to iframe messages. + +4. **Dashboard auto-loads on node creation** — `web/dashboard.js` calls `forceLoadSession()` with a 500ms delay when a dashboard node is created. The `get_session_html` API endpoint returns a full HTML template with empty manifest for non-existent sessions (NOT a 404). + +5. **`logic_pipeline.js` has a legacy init block** — Lines 489-505 have a `DOMContentLoaded` handler that calls `loadSession()` only when `fullManifest.items.length > 0`. The proper initialization happens in `logic_init.js` which calls `init()`. Don't add a second unconditional `loadSession()` call — it causes alert popups for empty sessions. + +6. **Manifest merge on save** — `save_manifest()` always reloads from disk first to preserve user favorites/rejected/notes. This is intentional for concurrent access (generation + user interaction). Don't bypass this. + +7. **`onNodeCreated` must be synchronous** — In `conf-builder-main.js`, async work runs inside fire-and-forget `(async () => { ... })()`. Making `onNodeCreated` async breaks ComfyUI widget registration. + +8. **Optional inputs disable caches** — When `optional_model`/`optional_clip`/`optional_positive`/`optional_negative` are connected to the sampler node, conditioning cache is disabled and `IS_CHANGED()` returns `NaN` (forces re-execution every time). + +9. **Distribution conditionings are serialized with np.copy()** — `np.frombuffer()` returns read-only arrays. Must call `.copy()` before `torch.from_numpy()` or GPU tensor operations will fail. + +10. **Default state migration** — When adding new fields to the Config Builder state (in `conf-builder-main.js`), also add a migration check in the `onNodeCreated` function (around line 363) to backfill the default value for existing saved workflows. + +11. **claim_timeout default is 600** — This value appears in `distribution_manager.py`, `config_builder_node.py`, `conf-builder-distribution.js`, and `conf-builder-main.js`. Keep all four in sync. + +12. **ComfyUI Registry security** — No `import requests`, no `subprocess`/`eval`/`exec`, no custom file-serving endpoints. Use `urllib.request` for HTTP and ComfyUI's `/view` endpoint for serving files. See `ProjectStructure.md` for full rules. + +### File Dependencies (Import Graph) + +``` +sampler_node.py + → generation_orchestrator.py (main entry point) + → config_utils.py (expand_configs, prepare_input_jobs) + → model_loader.py (load_checkpoint, load_loras) + → image_generation.py (generate_image, flush_batch) + → batch_encoding.py (encode_prompt_with_combinators) + → trigger_words.py (build_prompt_with_triggers) + → manifest_utils.py (save_manifest, load_existing_manifest) + → model_cache.py (ModelCache) + → conditioning_cache.py (ConditioningCache) + → remote_vae.py (RemoteVAEDecodeWorker) + → html_generator.py (get_html_template) + → distribution_manager.py (DistributionManager) + → distribution_worker.py (WorkerThread) + +config_builder_node.py + → lora_utils.py (trigger word lookup) + +__init__.py + → all node classes + → distribution_routes.py (distribution API endpoints) + → html_generator.py (on-demand HTML generation) + → metadata_packer.py (export favorites) + → directory_scanner.py (scan external directories) + → manifest_utils.py (save/load manifests) +``` + +### Server Event Types + +| Event Name | Sender | Data | Purpose | +|---|---|---|---| +| `ultimate_grid.update` | `generation_orchestrator.py` | `{session_name, node, manifest, meta, new_items}` | New images generated | +| `ultimate_grid.progress` | `generation_orchestrator.py` | `{session_name, progress_pct, eta_str, ...}` | ETA/progress updates | +| `ultimate_grid.distribution_status` | `generation_orchestrator.py` | `{session_name, workers, stats}` | Distribution status | + +### Testing Changes + +After modifying backend Python files, restart ComfyUI. After modifying `web/` JS files, refresh the browser (Ctrl+F5). After modifying `resources/` JS/CSS/HTML files, the dashboard HTML must be regenerated — either re-run the sampler node or call the `/config_tester/get_session_html` endpoint. + +--- + ## 📝 Changelog +### Update 3/6/26 — Distributed Processing & Dashboard Improvements +* 🌐 **Multi-Worker Distribution:** Split grid runs across multiple ComfyUI instances with automatic job coordination, claiming, timeout handling, and result upload. +* 🧠 **Master Text Encoding:** Master pre-encodes all prompts and sends conditionings to workers, eliminating duplicate CLIP encoding. +* 📦 **Model Sync:** Optionally transfer checkpoint and LoRA files from master to workers automatically. +* 📊 **Per-Worker Logging:** Final summary shows each worker's completed/failed job counts. +* ⚙️ **Distribution Settings UI:** Config Builder section with worker URL management, connection testing, and toggles for text encoding and model sync. +* 🏠 **Session Landing Page:** Dashboard now auto-loads and shows a session picker on startup instead of a blank black box. +* 🔄 **Dashboard Auto-Load:** Dashboard iframe automatically loads session HTML on node creation with 500ms delay. +* 🛠️ **Non-Existent Session Handling:** `get_session_html` endpoint returns a full template with empty manifest instead of 404. +* ⏱️ **Claim Timeout Update:** Default claim timeout increased from 300s to 600s across all files. +* 🤖 **AI Development Notes:** Added comprehensive development notes to README and ProjectStructure for AI-assisted development. + ### Update 2/16/26 — Config Builder & Dashboard Overhaul * 🎛️ **VAE Selection & Iteration:** Test multiple VAEs per config with searchable dropdown and folder expansion. VAEs are a full Cartesian dimension. * 🧠 **Attention Mode Testing:** Grid-test different attention implementations (`default`, `sageattn`, `xformers`) as a combinatorial dimension. @@ -1309,5 +1526,5 @@ Created for the ComfyUI community. Special thanks to all contributors and tester **Support development:** [Buy me a coffee on Ko-fi](https://ko-fi.com/jasonhoku) ☕ -**Version:** 2.0 -**Last Updated:** February 2026 +**Version:** 3.0 +**Last Updated:** March 2026