From d4384781dc518c439db343466e0e37865104be4e Mon Sep 17 00:00:00 2001 From: Robert Date: Wed, 25 Jun 2025 11:59:35 +1000 Subject: [PATCH 01/41] Initial commit --- README.md | 165 +++++++++++++++ __init__.py | 23 +++ gpu_config.json | 35 ++++ launch_distributed.bat | 119 +++++++++++ multigpu.py | 286 ++++++++++++++++++++++++++ web/gpupanel.js | 452 +++++++++++++++++++++++++++++++++++++++++ 6 files changed, 1080 insertions(+) create mode 100644 README.md create mode 100644 __init__.py create mode 100644 gpu_config.json create mode 100644 launch_distributed.bat create mode 100644 multigpu.py create mode 100644 web/gpupanel.js diff --git a/README.md b/README.md new file mode 100644 index 0000000..7b78a9e --- /dev/null +++ b/README.md @@ -0,0 +1,165 @@ +# ComfyUI-Distributed + +> ⚠️ **DEVELOPMENT STATUS**: This project is currently in active development and is not yet production-ready. Features may change, and bugs may exist. Use at your own risk. + +A custom node extension for ComfyUI that enables distributed image generation across multiple GPUs through a master-worker architecture. + +## Overview + +ComfyUI-Distributed extends ComfyUI with the ability to distribute batch processing across multiple GPUs, allowing for linear scaling of image generation performance. It achieves this by coordinating multiple ComfyUI instances, each running on a different GPU, with one instance acting as the master coordinator. + +## Features + +- **Multi-GPU Distributed Processing**: Leverage all available GPUs simultaneously +- **Master-Worker Architecture**: Automatic coordination between GPU instances +- **Dynamic Worker Management**: Enable/disable GPUs on-the-fly through the UI +- **Execution Modes**: + - **Parallel Mode**: All GPUs process simultaneously for maximum speed + - **Staggered Mode**: Sequential GPU startup with configurable delays to manage VRAM usage +- **VRAM Management**: Built-in tools to clear GPU memory across all instances +- **Web UI Integration**: Seamless control panel integrated into ComfyUI's interface + +## How It Works + +1. Launch multiple ComfyUI instances, each assigned to a different GPU +2. One instance acts as the master (typically on CUDA device 0) +3. Add the "Multi-GPU Collector" node to your workflow +4. The extension automatically distributes batches across available GPUs +5. Results are collected and combined by the master instance + +For example, with 4 GPUs and a batch size of 2, you'll generate 8 images in parallel. + +## Requirements + +- ComfyUI installation +- Multiple NVIDIA GPUs +- Python 3.8+ +- Windows (for the provided launcher script) or Linux/Mac with manual setup + +## Installation + +1. Clone this repository into your ComfyUI custom nodes directory: + ```bash + cd ComfyUI/custom_nodes + git clone https://github.com/yourusername/ComfyUI-Distributed.git + ``` + +2. No additional dependencies required - uses ComfyUI's existing libraries + +## Usage + +### Quick Start (Windows) + +1. Copy the `launch_distributed.bat` file to your ComfyUI root directory (alongside other .bat files like `run_nvidia_gpu.bat`) + +2. Run the launcher script from the ComfyUI root directory: + ```bash + launch_distributed.bat + ``` + This will automatically start ComfyUI instances for each GPU + +3. Open the master instance (usually http://localhost:8188) + +4. Add the "Multi-GPU Collector" node to your workflow + +5. Use the GPU control panel in the sidebar to: + - Select which GPUs to use + - Choose between parallel or staggered execution + - Clear VRAM when needed + +### Manual Setup (All Platforms) + +Start each ComfyUI instance with the appropriate CUDA device: + +```bash +# Master instance (GPU 0) +CUDA_VISIBLE_DEVICES=0 python main.py --port 8188 + +# Worker instance (GPU 1) +CUDA_VISIBLE_DEVICES=1 python main.py --port 8189 --enable-cors-header + +# Worker instance (GPU 2) +CUDA_VISIBLE_DEVICES=2 python main.py --port 8190 --enable-cors-header + +# Continue for additional GPUs... +``` + +### Configuration + +The system uses a `gpu_config.json` file for configuration. Default configuration: + +```json +{ + "master": { + "port": 8188, + "device": 0, + "enabled": true + }, + "workers": [ + { + "port": 8189, + "device": 1, + "enabled": true, + "extra_args": "--enable-cors-header" + }, + { + "port": 8190, + "device": 2, + "enabled": true, + "extra_args": "--enable-cors-header" + } + ], + "settings": { + "stagger_delay": 2.0, + "job_timeout": 120 + } +} +``` + +## Workflow Integration + +The Multi-GPU Collector node integrates seamlessly with existing ComfyUI workflows: + +1. Connect the node after your image generation pipeline +2. The node will automatically detect batch sizes and distribute work +3. All GPUs will process their assigned batches +4. Results are collected and passed to the next node as a single batch + +## Performance Considerations + +- **Parallel Mode**: Best for workflows with consistent VRAM usage +- **Staggered Mode**: Ideal for memory-intensive workflows or when GPUs have different VRAM capacities +- **Batch Size**: Total generation = batch_size × number_of_active_GPUs + +## Troubleshooting + +- **Workers not connecting**: Ensure CORS headers are enabled (`--enable-cors-header`) +- **VRAM errors**: Use staggered mode with appropriate delays +- **Timeout errors**: Increase `job_timeout` in configuration +- **Port conflicts**: Ensure each instance uses a unique port + +## Development + +This project is under active development. Contributions are welcome! + +### Known Issues + +- Windows Terminal required for the launcher script +- Manual configuration needed for more than 4 GPUs +- Some custom nodes may not be compatible with distributed processing + +### Planned Features + +- Linux/Mac launcher scripts +- Automatic GPU detection and configuration +- Load balancing based on GPU capabilities +- Support for remote workers over network +- Queue management improvements + +## License + +Same as ComfyUI - see ComfyUI repository for details. + +## Acknowledgments + +Built as an extension for the amazing [ComfyUI](https://github.com/comfyanonymous/ComfyUI) project. \ No newline at end of file diff --git a/__init__.py b/__init__.py new file mode 100644 index 0000000..2e88060 --- /dev/null +++ b/__init__.py @@ -0,0 +1,23 @@ +import os +import sys + +# Add the directory to Python path +sys.path.append(os.path.dirname(__file__)) + +# Import everything needed from the main module +from .multigpu import ( + NODE_CLASS_MAPPINGS, + NODE_DISPLAY_NAME_MAPPINGS, + ensure_config_exists, + CONFIG_FILE +) + +WEB_DIRECTORY = "./web" + +# Use the centralized config management from multigpu.py +ensure_config_exists() + +__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS'] + +print("[MultiGPU] Loaded Multi-GPU nodes.") +print(f"[MultiGPU] Config file: {CONFIG_FILE}") \ No newline at end of file diff --git a/gpu_config.json b/gpu_config.json new file mode 100644 index 0000000..ccfc23a --- /dev/null +++ b/gpu_config.json @@ -0,0 +1,35 @@ +{ + "master": { + "port": 8480, + "cuda_device": 0 + }, + "workers": [ + { + "id": 1, + "name": "GPU 1", + "cuda_device": 1, + "port": 8180, + "enabled": false, + "extra_args": "" + }, + { + "id": 2, + "name": "GPU 2", + "cuda_device": 2, + "port": 8280, + "enabled": false, + "extra_args": "" + }, + { + "id": 3, + "name": "GPU 3", + "cuda_device": 3, + "port": 8380, + "enabled": false, + "extra_args": "" + } + ], + "settings": { + "stagger_delay_ms": 15000 + } +} \ No newline at end of file diff --git a/launch_distributed.bat b/launch_distributed.bat new file mode 100644 index 0000000..77781bd --- /dev/null +++ b/launch_distributed.bat @@ -0,0 +1,119 @@ +@echo off +title ComfyUI Multi-GPU CLI Launcher +cd /d "%~dp0" +color 0B +cls + +echo ComfyUI Multi-GPU CLI Launcher +echo =============================== + +:: Check if gpu_config.json exists (relative path) +set "CONFIG_PATH=ComfyUI\custom_nodes\ComfyUI-Distributed\gpu_config.json" +if not exist "%CONFIG_PATH%" ( + color 0C + echo Error: Configuration file not found! + echo Expected location: %CONFIG_PATH% + pause + exit /b 1 +) + +:: Check if venv exists (relative path) +set "VENV_PATH=venv" +if not exist "%VENV_PATH%\Scripts\activate.bat" ( + color 0C + echo Error: Virtual environment not found! + echo Expected location: %CD%\%VENV_PATH% + echo Make sure this batch file is in the ComfyUI_Dev directory with venv folder + pause + exit /b 1 +) + +:: Check if ComfyUI CLI is available in venv +call "%VENV_PATH%\Scripts\activate.bat" && comfy --help >nul 2>&1 +if %ERRORLEVEL% NEQ 0 ( + color 0C + echo Error: ComfyUI CLI not found in virtual environment! + echo Make sure ComfyUI CLI is installed in: %VENV_PATH% + pause + exit /b 1 +) + +:: Read configuration with PowerShell and store in environment variables +echo Loading configuration... + +powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; $master = $json.master; $settings = $json.settings; $retryDelay = if ($settings.retry_delay_ms) { $settings.retry_delay_ms } else { 500 }; Write-Output \"RETRY_DELAY=$retryDelay\"; Write-Output \"MASTER_PORT=$($master.port)\"; Write-Output \"MASTER_CUDA=$($master.cuda_device)\"; foreach ($worker in $json.workers) { if ($worker.enabled) { Write-Output \"WORKER_$($worker.id)_NAME=$($worker.name)\"; Write-Output \"WORKER_$($worker.id)_PORT=$($worker.port)\"; Write-Output \"WORKER_$($worker.id)_CUDA=$($worker.cuda_device)\"; Write-Output \"WORKER_$($worker.id)_EXTRA=$($worker.extra_args)\" } } } catch { Write-Output \"PowerShell Error: $_\"; pause; exit 1 }" > %TEMP%\config_vars.txt + +if %ERRORLEVEL% NEQ 0 ( + color 0C + echo Error running PowerShell configuration reader! + pause + exit /b 1 +) + +for /f "tokens=*" %%i in (%TEMP%\config_vars.txt) do ( + set %%i +) + +del %TEMP%\config_vars.txt >nul 2>&1 + +:: Check Master +echo Starting Master instance... +:MASTER +powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %MASTER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" +if %ERRORLEVEL% == 1 ( + if not defined MASTER_STARTED ( + start wt -w 0 new-tab --title "Master - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title Master - ComfyUI CLI && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && comfy --here launch -- --port %MASTER_PORT%" + set MASTER_STARTED=1 + ) + %TEMP%\worker_ids.txt + +for /f %%w in (%TEMP%\worker_ids.txt) do ( + call :PROCESS_WORKER %%w +) + +del %TEMP%\worker_ids.txt >nul 2>&1 +goto :END_SCRIPT + +:PROCESS_WORKER +set "WORKER_ID=%1" +call set "WORKER_NAME=%%WORKER_%WORKER_ID%_NAME%%" +call set "WORKER_PORT=%%WORKER_%WORKER_ID%_PORT%%" +call set "WORKER_CUDA=%%WORKER_%WORKER_ID%_CUDA%%" +call set "WORKER_EXTRA=%%WORKER_%WORKER_ID%_EXTRA%%" + +echo Starting %WORKER_NAME%... + +:WORKER_CHECK_LOOP +powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %WORKER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" +if %ERRORLEVEL% == 1 ( + if not defined WORKER%WORKER_ID%_STARTED ( + start wt -w 0 new-tab --title "%WORKER_NAME% - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title %WORKER_NAME% - ComfyUI CLI && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && comfy --here launch -- --port %WORKER_PORT% %WORKER_EXTRA% --enable-cors-header" + set WORKER%WORKER_ID%_STARTED=1 + ) + nul +exit \ No newline at end of file diff --git a/multigpu.py b/multigpu.py new file mode 100644 index 0000000..34d2094 --- /dev/null +++ b/multigpu.py @@ -0,0 +1,286 @@ +import torch +import numpy as np +from PIL import Image +import folder_paths +import os +import json +import asyncio +import aiohttp +from aiohttp import web +import io +import server +from concurrent.futures import Future +import comfy.model_management as mm +import gc + +# --- Config Management --- +CONFIG_FILE = os.path.join(os.path.dirname(__file__), "gpu_config.json") + +def get_default_config(): + """Returns the default configuration dictionary. Single source of truth.""" + return { + "master": {"port": 8188, "cuda_device": 0}, + "workers": [], + "settings": { + "stagger_delay_ms": 15000 + } + } + +def load_config(): + """Loads the config, falling back to defaults if the file is missing or invalid.""" + if os.path.exists(CONFIG_FILE): + try: + with open(CONFIG_FILE, 'r') as f: + return json.load(f) + except Exception as e: + print(f"[MultiGPU Config] Error loading config, using defaults: {e}") + return get_default_config() + +def save_config(config): + """Saves the configuration to file.""" + try: + with open(CONFIG_FILE, 'w') as f: + json.dump(config, f, indent=2) + return True + except Exception as e: + print(f"[MultiGPU Config] Error saving config: {e}") + return False + +def ensure_config_exists(): + """Creates default config file if it doesn't exist. Used by __init__.py""" + if not os.path.exists(CONFIG_FILE): + default_config = get_default_config() + if save_config(default_config): + print("[MultiGPU] Created default config file") + else: + print("[MultiGPU] Could not create default config file") + +# --- Helper Functions --- +async def handle_api_error(request, error, status=500): + """Standardized error response handler""" + print(f"[MultiGPU] API Error: {error}") + return web.json_response({"status": "error", "message": str(error)}, status=status) + +# --- API Endpoints --- +@server.PromptServer.instance.routes.get("/multigpu/config") +async def get_config_endpoint(request): + config = load_config() + return web.json_response(config) + +@server.PromptServer.instance.routes.post("/multigpu/config/update_worker") +async def update_worker_endpoint(request): + try: + data = await request.json() + worker_id = data.get("worker_id") + enabled = data.get("enabled", False) + + if worker_id is None: + return await handle_api_error(request, "Missing worker_id", 400) + + config = load_config() + worker_found = False + + for worker in config.get("workers", []): + if worker["id"] == worker_id: + worker["enabled"] = enabled + worker_found = True + break + + if not worker_found: + return await handle_api_error(request, f"Worker {worker_id} not found", 404) + + if save_config(config): + return web.json_response({"status": "success"}) + else: + return await handle_api_error(request, "Failed to save config") + except Exception as e: + return await handle_api_error(request, e, 400) + +@server.PromptServer.instance.routes.post("/multigpu/config/update_setting") +async def update_setting_endpoint(request): + """Updates a specific key in the settings object.""" + try: + data = await request.json() + key = data.get("key") + value = data.get("value") + + if not key or value is None: + return await handle_api_error(request, "Missing 'key' or 'value' in request", 400) + + config = load_config() + if 'settings' not in config: + config['settings'] = {} + + config['settings'][key] = value + + if save_config(config): + return web.json_response({"status": "success", "message": f"Setting '{key}' updated."}) + else: + return await handle_api_error(request, "Failed to save config") + except Exception as e: + return await handle_api_error(request, e, 400) + + +@server.PromptServer.instance.routes.post("/multigpu/prepare_job") +async def prepare_job_endpoint(request): + try: + data = await request.json() + multi_job_id = data.get('multi_job_id') + if not multi_job_id: + return await handle_api_error(request, "Missing multi_job_id", 400) + + async with PENDING_JOBS_LOCK: + if multi_job_id not in PENDING_JOBS: + PENDING_JOBS[multi_job_id] = asyncio.Queue() + + print(f"[MultiGPU] Prepared queue for job {multi_job_id}") + return web.json_response({"status": "success"}) + except Exception as e: + return await handle_api_error(request, e) + +@server.PromptServer.instance.routes.post("/multigpu/clear_memory") +async def clear_memory_endpoint(request): + print("[MultiGPU] Received request to clear VRAM.") + try: + gc.collect() + mm.unload_all_models() + mm.soft_empty_cache() + print("[MultiGPU] VRAM cleared successfully.") + return web.json_response({"status": "success", "message": "GPU memory cleared."}) + except Exception as e: + return await handle_api_error(request, f"Error clearing VRAM: {e}") + +# --- Global State & Callback Endpoint --- +PENDING_JOBS = {} +PENDING_JOBS_LOCK = asyncio.Lock() + +@server.PromptServer.instance.routes.post("/multigpu/job_complete") +async def job_complete_endpoint(request): + try: + data = await request.post() + multi_job_id = data.get('multi_job_id') + image_file = data.get('image') + + if not all([multi_job_id, image_file]): + return await handle_api_error(request, "Missing job_id or image data", 400) + + # Process image + img_data = image_file.file.read() + img = Image.open(io.BytesIO(img_data)).convert("RGB") + img_np = np.array(img).astype(np.float32) / 255.0 + tensor = torch.from_numpy(img_np)[None,] + + async with PENDING_JOBS_LOCK: + if multi_job_id in PENDING_JOBS: + await PENDING_JOBS[multi_job_id].put(tensor) + print(f"[MultiGPU] Received result for job {multi_job_id}") + return web.json_response({"status": "success"}) + else: + return await handle_api_error(request, "Job not found or already complete", 404) + except Exception as e: + return await handle_api_error(request, e) + +# --- Collector Node --- +class MultiGPUCollectorNode: + @classmethod + def INPUT_TYPES(s): + return { + "required": { "images": ("IMAGE",) }, + "hidden": { + "multi_job_id": ("STRING", {"default": ""}), + "is_worker": ("BOOLEAN", {"default": False}), + "master_url": ("STRING", {"default": ""}), + "enabled_worker_ids": ("STRING", {"default": "[]"}), + # --- FIX [7 of 7]: Type Mismatch --- + # Changed from FLOAT to INT for type safety and clarity. + "worker_batch_size": ("INT", {"default": 1, "min": 1, "max": 1024}), + # Added a dynamic timeout for staggered mode + "stagger_timeout": ("FLOAT", {"default": 300.0, "min": 0.0}), + }, + } + + RETURN_TYPES = ("IMAGE",) + FUNCTION = "run" + CATEGORY = "image" + + def run(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1, stagger_timeout=300.0): + if not multi_job_id: + return (images,) + + main_loop = server.PromptServer.instance.loop + future = Future() + + async def run_and_set_future(): + try: + result = await self.execute(images, multi_job_id, is_worker, master_url, enabled_worker_ids, worker_batch_size, stagger_timeout) + future.set_result(result) + except Exception as e: + future.set_exception(e) + + asyncio.run_coroutine_threadsafe(run_and_set_future(), main_loop) + return future.result() + + async def send_image_to_master(self, image_tensor, multi_job_id, master_url, image_index): + """Helper method to send a single image to the master""" + img_np = (image_tensor.cpu().numpy() * 255.).astype(np.uint8) + img = Image.fromarray(img_np) + byte_io = io.BytesIO() + img.save(byte_io, format='PNG') + byte_io.seek(0) + + data = aiohttp.FormData() + data.add_field('multi_job_id', multi_job_id) + data.add_field('image', byte_io, filename=f'image_{image_index}.png', content_type='image/png') + + try: + async with aiohttp.ClientSession() as session: + async with session.post(f"{master_url}/multigpu/job_complete", data=data) as response: + response.raise_for_status() + except Exception as e: + print(f"[MultiGPU Worker] Failed to send image {image_index+1} to master: {e}") + + async def execute(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1, stagger_timeout=300.0): + if is_worker: + # Worker mode: send images to master + print(f"[MultiGPU Worker] Job {multi_job_id} complete. Sending {images.shape[0]} image(s) to master at {master_url}") + + tasks = [self.send_image_to_master(images[i], multi_job_id, master_url, i) for i in range(images.shape[0])] + await asyncio.gather(*tasks) + return (images,) + else: + # Master mode: collect images from workers + num_workers = len(json.loads(enabled_worker_ids)) + if num_workers == 0: + return (images,) + + images_on_cpu = images.cpu() + total_images_to_collect = int(num_workers * worker_batch_size) + print(f"[MultiGPU Master] Job {multi_job_id}: Waiting for {total_images_to_collect} image(s)...") + + all_tensors = [images_on_cpu] + + async with PENDING_JOBS_LOCK: + if multi_job_id not in PENDING_JOBS: + PENDING_JOBS[multi_job_id] = asyncio.Queue() + q = PENDING_JOBS[multi_job_id] + + try: + # Collect all images with a potentially dynamic timeout + for i in range(total_images_to_collect): + worker_tensor = await asyncio.wait_for(q.get(), timeout=stagger_timeout) + all_tensors.append(worker_tensor) + print(f"[MultiGPU Master] Collected image {i+1}/{total_images_to_collect}") + except asyncio.TimeoutError: + print(f"[MultiGPU Master] Timeout waiting for worker images (got {len(all_tensors)-1}/{total_images_to_collect})") + finally: + # Clean up job queue + async with PENDING_JOBS_LOCK: + if multi_job_id in PENDING_JOBS: + del PENDING_JOBS[multi_job_id] + + combined = torch.cat(all_tensors, dim=0) + print(f"[MultiGPU Master] Job {multi_job_id} complete. Combined {combined.shape[0]} images total.") + return (combined,) + +NODE_CLASS_MAPPINGS = { "MultiGPUCollector": MultiGPUCollectorNode } +NODE_DISPLAY_NAME_MAPPINGS = { "MultiGPUCollector": "Multi-GPU Collector" } \ No newline at end of file diff --git a/web/gpupanel.js b/web/gpupanel.js new file mode 100644 index 0000000..76e16ac --- /dev/null +++ b/web/gpupanel.js @@ -0,0 +1,452 @@ +import { app } from "../../scripts/app.js"; +import { api } from "../../scripts/api.js"; + +class MultiGPUExtension { + // Button styling constants + static BUTTON_STYLES = { + base: "width: 100%; padding: 8px; font-size: 12px; color: white; border: none; border-radius: 4px; cursor: pointer; transition: background-color 0.2s;", + clearMemory: "background-color: #555;", + stagger: "background-color: #224422; margin-top: 8px;", + success: "background-color: #3ca03c;", + error: "background-color: #c04c4c;" + }; + + constructor() { + this.config = null; + this.originalQueuePrompt = api.queuePrompt.bind(api); + this.isStaggeredRun = false; + + // --- FIX [1 of 7]: Constructor Race Condition --- + // The interceptor is now set up *after* the config is guaranteed to be loaded. + this.loadConfig().then(() => { + this.registerSidebarTab(); + this.setupInterceptor(); + }); + } + + // --- State & Config Management (Single Source of Truth) --- + + get enabledWorkers() { + return this.config?.workers?.filter(w => w.enabled) || []; + } + + get isEnabled() { + return this.enabledWorkers.length > 0; + } + + async loadConfig() { + try { + const response = await fetch(`${window.location.origin}/multigpu/config`); + if (response.ok) { + this.config = await response.json(); + console.log("[MultiGPU] Loaded config:", this.config); + } else { + console.error("[MultiGPU] Failed to load config"); + this.config = { workers: [], settings: {} }; + } + } catch (error) { + console.error("[MultiGPU] Error loading config:", error); + this.config = { workers: [], settings: {} }; + } + } + + async updateWorkerEnabled(workerId, enabled) { + const worker = this.config.workers.find(w => w.id === workerId); + if (worker) { + worker.enabled = enabled; + } + + try { + await fetch(`${window.location.origin}/multigpu/config/update_worker`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ worker_id: workerId, enabled: enabled }) + }); + } catch (error) { + console.error("[MultiGPU] Error updating worker:", error); + } + } + + async _updateSetting(key, value) { + try { + await fetch(`${window.location.origin}/multigpu/config/update_setting`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ key, value }) + }); + } catch (error) { + console.error(`[MultiGPU] Error updating setting '${key}':`, error); + } + } + + // --- UI Rendering --- + + registerSidebarTab() { + app.extensionManager.registerSidebarTab({ + id: "multiGPU", + icon: "pi pi-server", + title: "Multi-GPU", + tooltip: "Multi-GPU Control Panel", + type: "custom", + render: (el) => this.renderSidebarContent(el) + }); + } + + _createButton(text, onClick, style) { + const button = document.createElement("button"); + button.textContent = text; + button.style.cssText = MultiGPUExtension.BUTTON_STYLES.base + style; + button.onclick = onClick; + return button; + } + + async _handleClearMemory(button) { + const originalText = button.textContent; + button.textContent = "Clearing..."; + button.disabled = true; + + try { + const urlsToClear = [{ name: 'Master', url: window.location.origin }, ...this.enabledWorkers.map(w => ({ name: w.name, url: `http://localhost:${w.port}` }))]; + + const promises = urlsToClear.map(target => + fetch(`${target.url}/multigpu/clear_memory`, { method: 'POST', mode: 'cors' }) + .then(response => ({ ok: response.ok, name: target.name })) + .catch(() => ({ ok: false, name: target.name })) + ); + + const results = await Promise.all(promises); + const failures = results.filter(r => !r.ok); + + if (failures.length === 0) { + button.textContent = "Success!"; + button.style.backgroundColor = MultiGPUExtension.BUTTON_STYLES.success.split(':')[1].trim().replace(';', ''); + } else { + button.textContent = "Error! See Console"; + button.style.backgroundColor = MultiGPUExtension.BUTTON_STYLES.error.split(':')[1].trim().replace(';', ''); + } + + setTimeout(() => { + button.textContent = originalText; + button.style.cssText = MultiGPUExtension.BUTTON_STYLES.base + MultiGPUExtension.BUTTON_STYLES.clearMemory; + }, 3000); + } finally { + // --- FIX [2 of 7]: Button State --- + // Ensure the button is re-enabled even if an error occurs. + button.disabled = false; + } + } + + renderSidebarContent(el) { + el.innerHTML = ''; + const container = document.createElement("div"); + container.style.cssText = "padding: 15px; display: flex; flex-direction: column; height: 100%;"; + + const title = document.createElement("h3"); + title.textContent = "Multi-GPU Control"; + title.style.cssText = "margin: 0 0 15px 0; padding-bottom: 10px; border-bottom: 1px solid #444;"; + container.appendChild(title); + + const gpuSection = document.createElement("div"); + gpuSection.style.cssText = "flex: 1; overflow-y: auto; margin-bottom: 15px;"; + const gpuTitle = document.createElement("h4"); + gpuTitle.textContent = "Available Workers"; + gpuTitle.style.cssText = "margin: 0 0 10px 0; font-size: 14px;"; + gpuSection.appendChild(gpuTitle); + + const gpuList = document.createElement("div"); + (this.config?.workers || []).forEach(worker => { + const gpuDiv = document.createElement("div"); + gpuDiv.style.cssText = "margin-bottom: 8px; padding: 8px; background: #2a2a2a; border-radius: 4px;"; + const checkbox = document.createElement("input"); + checkbox.type = "checkbox"; + checkbox.id = `gpu-${worker.id}`; + checkbox.checked = worker.enabled; + checkbox.onchange = async (e) => { + await this.updateWorkerEnabled(worker.id, e.target.checked); + this.updateSummary(); + }; + const label = document.createElement("label"); + label.htmlFor = `gpu-${worker.id}`; + label.style.cssText = "cursor: pointer; display: flex; align-items: center; gap: 8px;"; + const gpuInfo = document.createElement("span"); + gpuInfo.innerHTML = `${worker.name}
CUDA ${worker.cuda_device} • Port ${worker.port}`; + label.appendChild(checkbox); + label.appendChild(gpuInfo); + gpuDiv.appendChild(label); + gpuList.appendChild(gpuDiv); + }); + gpuSection.appendChild(gpuList); + container.appendChild(gpuSection); + + const actionsSection = document.createElement("div"); + actionsSection.style.cssText = "padding-top: 10px; margin-bottom: 15px; border-top: 1px solid #444;"; + + const clearMemButton = this._createButton("Clear VRAM", (e) => this._handleClearMemory(e.target), MultiGPUExtension.BUTTON_STYLES.clearMemory); + actionsSection.appendChild(clearMemButton); + + const staggerButton = this._createButton("Staggered Start", (e) => { + // --- FIX [3 of 7]: Button State & Race Condition --- + const button = e.target; + button.disabled = true; + try { + this.isStaggeredRun = true; + app.queuePrompt(0, 1); + } finally { + // Re-enable the button after a short delay to allow the prompt to be processed. + setTimeout(() => { button.disabled = false; }, 1000); + } + }, MultiGPUExtension.BUTTON_STYLES.stagger); + staggerButton.id = "stagger-queue-button"; + staggerButton.title = "Run on Master immediately, then on each Worker with a delay."; + actionsSection.appendChild(staggerButton); + + const settingsSection = document.createElement("div"); + settingsSection.style.cssText = "margin-top: 10px;"; + + const delayDiv = document.createElement("div"); + delayDiv.style.cssText = "display: flex; align-items: center; justify-content: space-between;"; + const delayLabel = document.createElement("label"); + delayLabel.textContent = "Stagger Delay (s)"; + delayLabel.htmlFor = "stagger-delay-input"; + delayLabel.style.cssText = "font-size: 12px; color: #888;"; + const delayInput = document.createElement("input"); + delayInput.id = "stagger-delay-input"; + delayInput.type = "number"; + delayInput.min = "0"; + delayInput.step = "0.1"; + delayInput.style.cssText = "width: 60px; background-color: #222; border: 1px solid #444; border-radius: 4px; padding: 4px; color: white; font-size: 12px;"; + delayInput.value = (this.config?.settings?.stagger_delay_ms || 15000) / 1000; + delayInput.onchange = async (e) => { + const valueInSeconds = parseFloat(e.target.value); + if(isNaN(valueInSeconds) || valueInSeconds < 0) return; + const valueInMs = Math.round(valueInSeconds * 1000); + this.config.settings.stagger_delay_ms = valueInMs; + await this._updateSetting('stagger_delay_ms', valueInMs); + }; + delayDiv.appendChild(delayLabel); + delayDiv.appendChild(delayInput); + settingsSection.appendChild(delayDiv); + actionsSection.appendChild(settingsSection); + + container.appendChild(actionsSection); + + const summarySection = document.createElement("div"); + summarySection.style.cssText = "border-top: 1px solid #444; padding-top: 10px;"; + const summary = document.createElement("div"); + summary.id = "multigpu-summary"; + summary.style.cssText = "font-size: 11px; color: #888;"; + summarySection.appendChild(summary); + container.appendChild(summarySection); + el.appendChild(container); + this.updateSummary(); + } + + updateSummary() { + const summaryEl = document.getElementById('multigpu-summary'); + if (summaryEl) { + const totalGPUs = this.enabledWorkers.length + 1; + if (this.isEnabled) { + summaryEl.textContent = `If Collector node is present, total generation = (${totalGPUs} GPUs × Batch Size)`; + } else { + summaryEl.textContent = "Only the master GPU will be used."; + } + } + } + + // --- Core Logic & Execution --- + + setupInterceptor() { + api.queuePrompt = async (number, prompt) => { + // --- FIX [4 of 7]: isStaggeredRun Race Condition --- + // The check for the flag is now safer. It's read and reset in a way that + // prevents the race condition caused by the original 'finally' block. + if (this.isEnabled && this.findNodesByClass(prompt.output, "MultiGPUCollector").length > 0) { + const handleStaggered = this.isStaggeredRun; + this.isStaggeredRun = false; // Consume the flag immediately + + console.log(`[MultiGPU] Intercepting prompt. Staggered mode: ${handleStaggered}`); + if (handleStaggered) { + return await this.executeStaggeredMultiGPU(prompt); + } else { + return await this.executeParallelMultiGPU(prompt); + } + } + return this.originalQueuePrompt(number, prompt); + }; + } + + async executeParallelMultiGPU(promptWrapper) { + try { + console.log("[MultiGPU] Collector node found. Activating standard Multi-GPU execution."); + const multi_job_id = "mgpu_" + Date.now(); + const enabledWorkers = this.enabledWorkers; + const batchSize = this._getBatchSizeFromPrompt(promptWrapper); + await this._prepareMultiGpuJob(multi_job_id); + + const jobs = []; + const participants = ['master', ...enabledWorkers.map(w => w.id)]; + + for (const participantId of participants) { + const jobApiPrompt = this._prepareApiPromptForParticipant( + promptWrapper.output, multi_job_id, participantId, + { enabled_worker_ids: enabledWorkers.map(w => w.id), worker_batch_size: batchSize } + ); + if (participantId === 'master') { + jobs.push({ type: 'master', promptWrapper: { ...promptWrapper, output: jobApiPrompt } }); + } else { + const worker = this.config.workers.find(w => w.id === participantId); + if (worker) jobs.push({ type: 'worker', worker, prompt: jobApiPrompt, workflow: promptWrapper.workflow }); + } + } + await this._executeJobs(jobs); + return { "prompt_id": "multi-gpu-parallel-job-dispatched" }; + } catch (error) { + console.error("[MultiGPU] Parallel execution failed:", error); + alert(`[MultiGPU] Parallel execution failed: ${error.message}`); + throw error; + } + } + + async executeStaggeredMultiGPU(promptWrapper) { + try { + console.log("[MultiGPU] Staggered Start activated."); + const multi_job_id = "mgpu_" + Date.now(); + const enabledWorkers = this.enabledWorkers; + const batchSize = this._getBatchSizeFromPrompt(promptWrapper); + await this._prepareMultiGpuJob(multi_job_id); + + // --- FIX [5 of 7]: Synchronization --- + // Calculate a dynamic timeout for the master node. + // Baseline of 300s + stagger delay for each worker. + const staggerDelay = this.config.settings?.stagger_delay_ms || 15000; + const requiredTimeout = ((enabledWorkers.length * staggerDelay) / 1000) + 300; + console.log(`[MultiGPU] Master collector timeout set to ${requiredTimeout}s to accommodate staggered workers.`); + + const masterPrompt = this._prepareApiPromptForParticipant( + promptWrapper.output, multi_job_id, 'master', + { + enabled_worker_ids: enabledWorkers.map(w => w.id), + worker_batch_size: batchSize, + stagger_timeout: requiredTimeout // Pass the new timeout to the master + } + ); + this.originalQueuePrompt(0, { ...promptWrapper, output: masterPrompt }); + console.log("[MultiGPU] Dispatched to Master immediately."); + + console.log(`[MultiGPU] Beginning staggered dispatch to workers with a ${staggerDelay / 1000}s delay.`); + + for (const worker of enabledWorkers) { + await new Promise(resolve => setTimeout(resolve, staggerDelay)); + // --- FIX [6 of 7]: Missing Parameters --- + // The options object was missing here, which prevented the 'master_url' + // from being set correctly for the worker. + const workerPrompt = this._prepareApiPromptForParticipant( + promptWrapper.output, + multi_job_id, + worker.id, + { enabled_worker_ids: [], worker_batch_size: 0 } // This ensures master_url is set + ); + await this._dispatchToWorker(worker, workerPrompt, promptWrapper.workflow); + } + return { "prompt_id": "multi-gpu-staggered-job-dispatched" }; + } catch (error) { + console.error("[MultiGPU] Staggered execution failed:", error); + alert(`[MultiGPU] Staggered execution failed: ${error.message}`); + throw error; + } + } + + async _executeJobs(jobs) { + const promises = jobs.map(job => + job.type === 'master' + ? this.originalQueuePrompt(0, job.promptWrapper) + : this._dispatchToWorker(job.worker, job.prompt, job.workflow) + ); + await Promise.all(promises); + } + + // --- Helper Methods --- + + findNodesByClass(apiPrompt, className) { + return Object.entries(apiPrompt) + .filter(([, nodeData]) => nodeData.class_type === className) + .map(([nodeId, nodeData]) => ({ id: nodeId, data: nodeData })); + } + + findBatchNode(workflow) { + return workflow.nodes.find(node => { + const title = node.title?.trim().toUpperCase(); + const displayName = node.properties?.["Node name for S&R"]?.trim().toUpperCase(); + return title === "BATCH" || title === "BATCH SIZE" || displayName === "BATCH" || displayName === "BATCH SIZE"; + }); + } + + _getBatchSizeFromPrompt(promptWrapper) { + const batchNode = this.findBatchNode(promptWrapper.workflow); + if (!batchNode) throw new Error("MultiGPU: Could not find a 'Batch' node."); + return promptWrapper.output[String(batchNode.id)].inputs.batch_size || promptWrapper.output[String(batchNode.id)].inputs.value || 1; + } + + _prepareApiPromptForParticipant(baseApiPrompt, multi_job_id, participantId, options = {}) { + const jobApiPrompt = JSON.parse(JSON.stringify(baseApiPrompt)); + const collectorNodes = this.findNodesByClass(jobApiPrompt, "MultiGPUCollector"); + const isMaster = participantId === 'master'; + + for (const collector of collectorNodes) { + const { inputs } = jobApiPrompt[collector.id]; + inputs.multi_job_id = multi_job_id; + inputs.is_worker = !isMaster; + if (isMaster) { + inputs.enabled_worker_ids = JSON.stringify(options.enabled_worker_ids || []); + inputs.worker_batch_size = options.worker_batch_size || 1; + // Pass the dynamic timeout for staggered mode + if (options.stagger_timeout) { + inputs.stagger_timeout = options.stagger_timeout; + } + } else { + inputs.master_url = window.location.origin; + } + } + return jobApiPrompt; + } + + async _prepareMultiGpuJob(multi_job_id) { + try { + await fetch(`${window.location.origin}/multigpu/prepare_job`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ multi_job_id }) + }); + } catch (error) { + console.error("[MultiGPU] Error preparing job:", error); + throw error; + } + } + + async _dispatchToWorker(worker, prompt, workflow) { + const workerUrl = `http://localhost:${worker.port}`; + console.log(`[MultiGPU] Dispatching to worker: ${worker.name} (${workerUrl})`); + const promptToSend = { + prompt, + extra_data: { extra_pnginfo: { workflow } }, + client_id: api.clientId + }; + try { + await fetch(`${workerUrl}/prompt`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + mode: 'cors', + body: JSON.stringify(promptToSend) + }); + } catch (e) { + console.error(`[MultiGPU] Failed to connect to worker ${worker.name} at ${workerUrl}`, e); + } + } +} + +app.registerExtension({ + name: "MultiGPU.Panel", + async setup() { + new MultiGPUExtension(); + } +}); \ No newline at end of file From 593a276c7ca0a216beda16d203161eaf2ae46f63 Mon Sep 17 00:00:00 2001 From: Robert Date: Wed, 25 Jun 2025 12:04:28 +1000 Subject: [PATCH 02/41] Update README.md --- README.md | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 7b78a9e..2f3660e 100644 --- a/README.md +++ b/README.md @@ -52,17 +52,22 @@ For example, with 4 GPUs and a batch size of 2, you'll generate 8 images in para 1. Copy the `launch_distributed.bat` file to your ComfyUI root directory (alongside other .bat files like `run_nvidia_gpu.bat`) -2. Run the launcher script from the ComfyUI root directory: +2. Edit the `gpu_config.json` file in `ComfyUI/custom_nodes/ComfyUI-Distributed/` to match your GPU setup: + - Adjust the number of workers based on your available GPUs + - Update CUDA device numbers if needed + - Set appropriate ports (ensure they're not in use) + +3. Run the launcher script from the ComfyUI root directory: ```bash launch_distributed.bat ``` This will automatically start ComfyUI instances for each GPU -3. Open the master instance (usually http://localhost:8188) +4. Open the master instance (usually http://localhost:8188) -4. Add the "Multi-GPU Collector" node to your workflow +5. Add the "Multi-GPU Collector" node to your workflow -5. Use the GPU control panel in the sidebar to: +6. Use the GPU control panel in the sidebar to: - Select which GPUs to use - Choose between parallel or staggered execution - Clear VRAM when needed @@ -125,6 +130,11 @@ The Multi-GPU Collector node integrates seamlessly with existing ComfyUI workflo 3. All GPUs will process their assigned batches 4. Results are collected and passed to the next node as a single batch +### Important Node Requirements + +- **Batch Node**: Your workflow must include a node with the title "BATCH" or "BATCH SIZE" (case-insensitive). This is required for the Multi-GPU system to detect the batch size for distribution. +- The system will show an error if no batch node with the correct title is found. + ## Performance Considerations - **Parallel Mode**: Best for workflows with consistent VRAM usage @@ -142,12 +152,6 @@ The Multi-GPU Collector node integrates seamlessly with existing ComfyUI workflo This project is under active development. Contributions are welcome! -### Known Issues - -- Windows Terminal required for the launcher script -- Manual configuration needed for more than 4 GPUs -- Some custom nodes may not be compatible with distributed processing - ### Planned Features - Linux/Mac launcher scripts @@ -162,4 +166,4 @@ Same as ComfyUI - see ComfyUI repository for details. ## Acknowledgments -Built as an extension for the amazing [ComfyUI](https://github.com/comfyanonymous/ComfyUI) project. \ No newline at end of file +Built as an extension for the amazing [ComfyUI](https://github.com/comfyanonymous/ComfyUI) project. From ec3d4011853d519b6ae27f6f66affcaa5b59723d Mon Sep 17 00:00:00 2001 From: Robert Date: Wed, 25 Jun 2025 12:10:13 +1000 Subject: [PATCH 03/41] Update README.md --- README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 2f3660e..983dc3c 100644 --- a/README.md +++ b/README.md @@ -152,13 +152,14 @@ The Multi-GPU Collector node integrates seamlessly with existing ComfyUI workflo This project is under active development. Contributions are welcome! -### Planned Features +### Planned Features/Improvements -- Linux/Mac launcher scripts -- Automatic GPU detection and configuration -- Load balancing based on GPU capabilities - Support for remote workers over network -- Queue management improvements +- Better VRAM management for multi-GPU setups +- Better Seed handling +- Better Batch handling +- Handle worker failure +- Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut ## License From 1f630e6cb65e6f8a10561c0715ea360bb599c61b Mon Sep 17 00:00:00 2001 From: Robert Date: Wed, 25 Jun 2025 12:14:06 +1000 Subject: [PATCH 04/41] Update README.md --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 983dc3c..5458a71 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@ A custom node extension for ComfyUI that enables distributed image generation across multiple GPUs through a master-worker architecture. +![image](https://github.com/user-attachments/assets/1e95e9e6-5237-4b24-8500-ac3b2703a1ea) + ## Overview ComfyUI-Distributed extends ComfyUI with the ability to distribute batch processing across multiple GPUs, allowing for linear scaling of image generation performance. It achieves this by coordinating multiple ComfyUI instances, each running on a different GPU, with one instance acting as the master coordinator. From 79acf8022484e1ca032e8143dd45506d52ada248 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Thu, 26 Jun 2025 08:10:20 +1000 Subject: [PATCH 05/41] Update launch_distributed.bat --- launch_distributed.bat | 104 +++++++++++++++++++++++------------------ 1 file changed, 59 insertions(+), 45 deletions(-) diff --git a/launch_distributed.bat b/launch_distributed.bat index 77781bd..c62ac5a 100644 --- a/launch_distributed.bat +++ b/launch_distributed.bat @@ -1,11 +1,15 @@ @echo off -title ComfyUI Multi-GPU CLI Launcher +title ComfyUI Multi-GPU Fast Launcher (Check & Start) cd /d "%~dp0" color 0B cls -echo ComfyUI Multi-GPU CLI Launcher -echo =============================== +echo ComfyUI Multi-GPU Fast Launcher (Check ^& Start) +echo =============================================== +echo. +echo This launcher checks for running instances and starts only +echo those that aren't running, with a 2-second delay between launches. +echo. :: Check if gpu_config.json exists (relative path) set "CONFIG_PATH=ComfyUI\custom_nodes\ComfyUI-Distributed\gpu_config.json" @@ -38,10 +42,18 @@ if %ERRORLEVEL% NEQ 0 ( exit /b 1 ) +:: Configure Windows memory management for large model loading +echo Configuring memory management for large models... + +:: Configure PyTorch memory allocation for large models +set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + +echo Memory management configured. + :: Read configuration with PowerShell and store in environment variables echo Loading configuration... -powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; $master = $json.master; $settings = $json.settings; $retryDelay = if ($settings.retry_delay_ms) { $settings.retry_delay_ms } else { 500 }; Write-Output \"RETRY_DELAY=$retryDelay\"; Write-Output \"MASTER_PORT=$($master.port)\"; Write-Output \"MASTER_CUDA=$($master.cuda_device)\"; foreach ($worker in $json.workers) { if ($worker.enabled) { Write-Output \"WORKER_$($worker.id)_NAME=$($worker.name)\"; Write-Output \"WORKER_$($worker.id)_PORT=$($worker.port)\"; Write-Output \"WORKER_$($worker.id)_CUDA=$($worker.cuda_device)\"; Write-Output \"WORKER_$($worker.id)_EXTRA=$($worker.extra_args)\" } } } catch { Write-Output \"PowerShell Error: $_\"; pause; exit 1 }" > %TEMP%\config_vars.txt +powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; $master = $json.master; Write-Output \"MASTER_PORT=$($master.port)\"; Write-Output \"MASTER_CUDA=$($master.cuda_device)\"; Write-Output \"MASTER_EXTRA=$($master.extra_args)\"; foreach ($worker in $json.workers) { if ($worker.enabled) { Write-Output \"WORKER_$($worker.id)_NAME=$($worker.name)\"; Write-Output \"WORKER_$($worker.id)_PORT=$($worker.port)\"; Write-Output \"WORKER_$($worker.id)_CUDA=$($worker.cuda_device)\"; Write-Output \"WORKER_$($worker.id)_EXTRA=$($worker.extra_args)\" } } } catch { Write-Output \"PowerShell Error: $_\"; pause; exit 1 }" > %TEMP%\config_vars.txt if %ERRORLEVEL% NEQ 0 ( color 0C @@ -56,59 +68,41 @@ for /f "tokens=*" %%i in (%TEMP%\config_vars.txt) do ( del %TEMP%\config_vars.txt >nul 2>&1 -:: Check Master -echo Starting Master instance... -:MASTER +echo. +echo Checking running instances... +echo. + +set "INSTANCES_STARTED=0" + +:: Check and start Master if not running +echo Checking Master (port %MASTER_PORT%)... powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %MASTER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" if %ERRORLEVEL% == 1 ( - if not defined MASTER_STARTED ( - start wt -w 0 new-tab --title "Master - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title Master - ComfyUI CLI && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && comfy --here launch -- --port %MASTER_PORT%" - set MASTER_STARTED=1 - ) - nul +) else ( + echo Master already running on port %MASTER_PORT% ) -echo Master ready! -:: Process workers sequentially - wait for each to be ready before starting next +:: Check and start workers powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; foreach ($worker in $json.workers) { if ($worker.enabled) { Write-Output $worker.id } } } catch { exit 1 }" > %TEMP%\worker_ids.txt for /f %%w in (%TEMP%\worker_ids.txt) do ( - call :PROCESS_WORKER %%w + call :CHECK_AND_START_WORKER %%w ) del %TEMP%\worker_ids.txt >nul 2>&1 -goto :END_SCRIPT - -:PROCESS_WORKER -set "WORKER_ID=%1" -call set "WORKER_NAME=%%WORKER_%WORKER_ID%_NAME%%" -call set "WORKER_PORT=%%WORKER_%WORKER_ID%_PORT%%" -call set "WORKER_CUDA=%%WORKER_%WORKER_ID%_CUDA%%" -call set "WORKER_EXTRA=%%WORKER_%WORKER_ID%_EXTRA%%" - -echo Starting %WORKER_NAME%... - -:WORKER_CHECK_LOOP -powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %WORKER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" -if %ERRORLEVEL% == 1 ( - if not defined WORKER%WORKER_ID%_STARTED ( - start wt -w 0 new-tab --title "%WORKER_NAME% - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title %WORKER_NAME% - ComfyUI CLI && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && comfy --here launch -- --port %WORKER_PORT% %WORKER_EXTRA% --enable-cors-header" - set WORKER%WORKER_ID%_STARTED=1 - ) - nul -exit \ No newline at end of file +exit + +:CHECK_AND_START_WORKER +set "WORKER_ID=%1" +call set "WORKER_NAME=%%WORKER_%WORKER_ID%_NAME%%" +call set "WORKER_PORT=%%WORKER_%WORKER_ID%_PORT%%" +call set "WORKER_CUDA=%%WORKER_%WORKER_ID%_CUDA%%" +call set "WORKER_EXTRA=%%WORKER_%WORKER_ID%_EXTRA%%" + +echo Checking %WORKER_NAME% (port %WORKER_PORT%)... +powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %WORKER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" +if %ERRORLEVEL% == 1 ( + echo %WORKER_NAME% NOT running - Starting with CUDA_VISIBLE_DEVICES=%WORKER_CUDA% + start wt -w 0 new-tab --title "%WORKER_NAME% - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title %WORKER_NAME% - ComfyUI CLI && echo CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && comfy --here launch -- --port %WORKER_PORT% %WORKER_EXTRA% --enable-cors-header" + set /a INSTANCES_STARTED+=1 + echo Waiting 2 seconds before checking next instance... + timeout /t 2 /nobreak >nul +) else ( + echo %WORKER_NAME% already running on port %WORKER_PORT% +) +goto :eof From 539ce6fced38aa023ca064ccd41e88f24c05591b Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Thu, 26 Jun 2025 08:10:50 +1000 Subject: [PATCH 06/41] Update multigpu.py --- multigpu.py | 29 ++++++----------------------- 1 file changed, 6 insertions(+), 23 deletions(-) diff --git a/multigpu.py b/multigpu.py index 34d2094..580c289 100644 --- a/multigpu.py +++ b/multigpu.py @@ -10,8 +10,6 @@ from aiohttp import web import io import server from concurrent.futures import Future -import comfy.model_management as mm -import gc # --- Config Management --- CONFIG_FILE = os.path.join(os.path.dirname(__file__), "gpu_config.json") @@ -21,9 +19,7 @@ def get_default_config(): return { "master": {"port": 8188, "cuda_device": 0}, "workers": [], - "settings": { - "stagger_delay_ms": 15000 - } + "settings": {} } def load_config(): @@ -138,17 +134,6 @@ async def prepare_job_endpoint(request): except Exception as e: return await handle_api_error(request, e) -@server.PromptServer.instance.routes.post("/multigpu/clear_memory") -async def clear_memory_endpoint(request): - print("[MultiGPU] Received request to clear VRAM.") - try: - gc.collect() - mm.unload_all_models() - mm.soft_empty_cache() - print("[MultiGPU] VRAM cleared successfully.") - return web.json_response({"status": "success", "message": "GPU memory cleared."}) - except Exception as e: - return await handle_api_error(request, f"Error clearing VRAM: {e}") # --- Global State & Callback Endpoint --- PENDING_JOBS = {} @@ -194,8 +179,6 @@ class MultiGPUCollectorNode: # --- FIX [7 of 7]: Type Mismatch --- # Changed from FLOAT to INT for type safety and clarity. "worker_batch_size": ("INT", {"default": 1, "min": 1, "max": 1024}), - # Added a dynamic timeout for staggered mode - "stagger_timeout": ("FLOAT", {"default": 300.0, "min": 0.0}), }, } @@ -203,7 +186,7 @@ class MultiGPUCollectorNode: FUNCTION = "run" CATEGORY = "image" - def run(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1, stagger_timeout=300.0): + def run(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1): if not multi_job_id: return (images,) @@ -212,7 +195,7 @@ class MultiGPUCollectorNode: async def run_and_set_future(): try: - result = await self.execute(images, multi_job_id, is_worker, master_url, enabled_worker_ids, worker_batch_size, stagger_timeout) + result = await self.execute(images, multi_job_id, is_worker, master_url, enabled_worker_ids, worker_batch_size) future.set_result(result) except Exception as e: future.set_exception(e) @@ -239,7 +222,7 @@ class MultiGPUCollectorNode: except Exception as e: print(f"[MultiGPU Worker] Failed to send image {image_index+1} to master: {e}") - async def execute(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1, stagger_timeout=300.0): + async def execute(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1): if is_worker: # Worker mode: send images to master print(f"[MultiGPU Worker] Job {multi_job_id} complete. Sending {images.shape[0]} image(s) to master at {master_url}") @@ -267,7 +250,7 @@ class MultiGPUCollectorNode: try: # Collect all images with a potentially dynamic timeout for i in range(total_images_to_collect): - worker_tensor = await asyncio.wait_for(q.get(), timeout=stagger_timeout) + worker_tensor = await asyncio.wait_for(q.get(), timeout=300.0) all_tensors.append(worker_tensor) print(f"[MultiGPU Master] Collected image {i+1}/{total_images_to_collect}") except asyncio.TimeoutError: @@ -283,4 +266,4 @@ class MultiGPUCollectorNode: return (combined,) NODE_CLASS_MAPPINGS = { "MultiGPUCollector": MultiGPUCollectorNode } -NODE_DISPLAY_NAME_MAPPINGS = { "MultiGPUCollector": "Multi-GPU Collector" } \ No newline at end of file +NODE_DISPLAY_NAME_MAPPINGS = { "MultiGPUCollector": "Multi-GPU Collector" } From eea215bbd758abae082759cab1332aa161c1a352 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Thu, 26 Jun 2025 08:11:04 +1000 Subject: [PATCH 07/41] Update gpu_config.json --- gpu_config.json | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/gpu_config.json b/gpu_config.json index ccfc23a..fba68e9 100644 --- a/gpu_config.json +++ b/gpu_config.json @@ -1,7 +1,8 @@ { "master": { "port": 8480, - "cuda_device": 0 + "cuda_device": 0, + "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" }, "workers": [ { @@ -9,27 +10,25 @@ "name": "GPU 1", "cuda_device": 1, "port": 8180, - "enabled": false, - "extra_args": "" + "enabled": true, + "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" }, { "id": 2, "name": "GPU 2", "cuda_device": 2, "port": 8280, - "enabled": false, - "extra_args": "" + "enabled": true, + "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" }, { "id": 3, "name": "GPU 3", "cuda_device": 3, "port": 8380, - "enabled": false, - "extra_args": "" + "enabled": true, + "extra_args": "--mmap-torch-files --highvram --disable-smart-memory --reserve-vram 2" } ], - "settings": { - "stagger_delay_ms": 15000 - } -} \ No newline at end of file + "settings": {} +} From 675affa8e584ad950996c145526ea10b50a8ed40 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Thu, 26 Jun 2025 08:11:37 +1000 Subject: [PATCH 08/41] Update gpupanel.js --- web/gpupanel.js | 157 +----------------------------------------------- 1 file changed, 3 insertions(+), 154 deletions(-) diff --git a/web/gpupanel.js b/web/gpupanel.js index 76e16ac..44c674c 100644 --- a/web/gpupanel.js +++ b/web/gpupanel.js @@ -5,8 +5,6 @@ class MultiGPUExtension { // Button styling constants static BUTTON_STYLES = { base: "width: 100%; padding: 8px; font-size: 12px; color: white; border: none; border-radius: 4px; cursor: pointer; transition: background-color 0.2s;", - clearMemory: "background-color: #555;", - stagger: "background-color: #224422; margin-top: 8px;", success: "background-color: #3ca03c;", error: "background-color: #c04c4c;" }; @@ -14,7 +12,6 @@ class MultiGPUExtension { constructor() { this.config = null; this.originalQueuePrompt = api.queuePrompt.bind(api); - this.isStaggeredRun = false; // --- FIX [1 of 7]: Constructor Race Condition --- // The interceptor is now set up *after* the config is guaranteed to be loaded. @@ -100,41 +97,6 @@ class MultiGPUExtension { return button; } - async _handleClearMemory(button) { - const originalText = button.textContent; - button.textContent = "Clearing..."; - button.disabled = true; - - try { - const urlsToClear = [{ name: 'Master', url: window.location.origin }, ...this.enabledWorkers.map(w => ({ name: w.name, url: `http://localhost:${w.port}` }))]; - - const promises = urlsToClear.map(target => - fetch(`${target.url}/multigpu/clear_memory`, { method: 'POST', mode: 'cors' }) - .then(response => ({ ok: response.ok, name: target.name })) - .catch(() => ({ ok: false, name: target.name })) - ); - - const results = await Promise.all(promises); - const failures = results.filter(r => !r.ok); - - if (failures.length === 0) { - button.textContent = "Success!"; - button.style.backgroundColor = MultiGPUExtension.BUTTON_STYLES.success.split(':')[1].trim().replace(';', ''); - } else { - button.textContent = "Error! See Console"; - button.style.backgroundColor = MultiGPUExtension.BUTTON_STYLES.error.split(':')[1].trim().replace(';', ''); - } - - setTimeout(() => { - button.textContent = originalText; - button.style.cssText = MultiGPUExtension.BUTTON_STYLES.base + MultiGPUExtension.BUTTON_STYLES.clearMemory; - }, 3000); - } finally { - // --- FIX [2 of 7]: Button State --- - // Ensure the button is re-enabled even if an error occurs. - button.disabled = false; - } - } renderSidebarContent(el) { el.innerHTML = ''; @@ -178,57 +140,6 @@ class MultiGPUExtension { gpuSection.appendChild(gpuList); container.appendChild(gpuSection); - const actionsSection = document.createElement("div"); - actionsSection.style.cssText = "padding-top: 10px; margin-bottom: 15px; border-top: 1px solid #444;"; - - const clearMemButton = this._createButton("Clear VRAM", (e) => this._handleClearMemory(e.target), MultiGPUExtension.BUTTON_STYLES.clearMemory); - actionsSection.appendChild(clearMemButton); - - const staggerButton = this._createButton("Staggered Start", (e) => { - // --- FIX [3 of 7]: Button State & Race Condition --- - const button = e.target; - button.disabled = true; - try { - this.isStaggeredRun = true; - app.queuePrompt(0, 1); - } finally { - // Re-enable the button after a short delay to allow the prompt to be processed. - setTimeout(() => { button.disabled = false; }, 1000); - } - }, MultiGPUExtension.BUTTON_STYLES.stagger); - staggerButton.id = "stagger-queue-button"; - staggerButton.title = "Run on Master immediately, then on each Worker with a delay."; - actionsSection.appendChild(staggerButton); - - const settingsSection = document.createElement("div"); - settingsSection.style.cssText = "margin-top: 10px;"; - - const delayDiv = document.createElement("div"); - delayDiv.style.cssText = "display: flex; align-items: center; justify-content: space-between;"; - const delayLabel = document.createElement("label"); - delayLabel.textContent = "Stagger Delay (s)"; - delayLabel.htmlFor = "stagger-delay-input"; - delayLabel.style.cssText = "font-size: 12px; color: #888;"; - const delayInput = document.createElement("input"); - delayInput.id = "stagger-delay-input"; - delayInput.type = "number"; - delayInput.min = "0"; - delayInput.step = "0.1"; - delayInput.style.cssText = "width: 60px; background-color: #222; border: 1px solid #444; border-radius: 4px; padding: 4px; color: white; font-size: 12px;"; - delayInput.value = (this.config?.settings?.stagger_delay_ms || 15000) / 1000; - delayInput.onchange = async (e) => { - const valueInSeconds = parseFloat(e.target.value); - if(isNaN(valueInSeconds) || valueInSeconds < 0) return; - const valueInMs = Math.round(valueInSeconds * 1000); - this.config.settings.stagger_delay_ms = valueInMs; - await this._updateSetting('stagger_delay_ms', valueInMs); - }; - delayDiv.appendChild(delayLabel); - delayDiv.appendChild(delayInput); - settingsSection.appendChild(delayDiv); - actionsSection.appendChild(settingsSection); - - container.appendChild(actionsSection); const summarySection = document.createElement("div"); summarySection.style.cssText = "border-top: 1px solid #444; padding-top: 10px;"; @@ -257,19 +168,9 @@ class MultiGPUExtension { setupInterceptor() { api.queuePrompt = async (number, prompt) => { - // --- FIX [4 of 7]: isStaggeredRun Race Condition --- - // The check for the flag is now safer. It's read and reset in a way that - // prevents the race condition caused by the original 'finally' block. if (this.isEnabled && this.findNodesByClass(prompt.output, "MultiGPUCollector").length > 0) { - const handleStaggered = this.isStaggeredRun; - this.isStaggeredRun = false; // Consume the flag immediately - - console.log(`[MultiGPU] Intercepting prompt. Staggered mode: ${handleStaggered}`); - if (handleStaggered) { - return await this.executeStaggeredMultiGPU(prompt); - } else { - return await this.executeParallelMultiGPU(prompt); - } + console.log("[MultiGPU] Intercepting prompt. Executing parallel Multi-GPU."); + return await this.executeParallelMultiGPU(prompt); } return this.originalQueuePrompt(number, prompt); }; @@ -307,54 +208,6 @@ class MultiGPUExtension { } } - async executeStaggeredMultiGPU(promptWrapper) { - try { - console.log("[MultiGPU] Staggered Start activated."); - const multi_job_id = "mgpu_" + Date.now(); - const enabledWorkers = this.enabledWorkers; - const batchSize = this._getBatchSizeFromPrompt(promptWrapper); - await this._prepareMultiGpuJob(multi_job_id); - - // --- FIX [5 of 7]: Synchronization --- - // Calculate a dynamic timeout for the master node. - // Baseline of 300s + stagger delay for each worker. - const staggerDelay = this.config.settings?.stagger_delay_ms || 15000; - const requiredTimeout = ((enabledWorkers.length * staggerDelay) / 1000) + 300; - console.log(`[MultiGPU] Master collector timeout set to ${requiredTimeout}s to accommodate staggered workers.`); - - const masterPrompt = this._prepareApiPromptForParticipant( - promptWrapper.output, multi_job_id, 'master', - { - enabled_worker_ids: enabledWorkers.map(w => w.id), - worker_batch_size: batchSize, - stagger_timeout: requiredTimeout // Pass the new timeout to the master - } - ); - this.originalQueuePrompt(0, { ...promptWrapper, output: masterPrompt }); - console.log("[MultiGPU] Dispatched to Master immediately."); - - console.log(`[MultiGPU] Beginning staggered dispatch to workers with a ${staggerDelay / 1000}s delay.`); - - for (const worker of enabledWorkers) { - await new Promise(resolve => setTimeout(resolve, staggerDelay)); - // --- FIX [6 of 7]: Missing Parameters --- - // The options object was missing here, which prevented the 'master_url' - // from being set correctly for the worker. - const workerPrompt = this._prepareApiPromptForParticipant( - promptWrapper.output, - multi_job_id, - worker.id, - { enabled_worker_ids: [], worker_batch_size: 0 } // This ensures master_url is set - ); - await this._dispatchToWorker(worker, workerPrompt, promptWrapper.workflow); - } - return { "prompt_id": "multi-gpu-staggered-job-dispatched" }; - } catch (error) { - console.error("[MultiGPU] Staggered execution failed:", error); - alert(`[MultiGPU] Staggered execution failed: ${error.message}`); - throw error; - } - } async _executeJobs(jobs) { const promises = jobs.map(job => @@ -399,10 +252,6 @@ class MultiGPUExtension { if (isMaster) { inputs.enabled_worker_ids = JSON.stringify(options.enabled_worker_ids || []); inputs.worker_batch_size = options.worker_batch_size || 1; - // Pass the dynamic timeout for staggered mode - if (options.stagger_timeout) { - inputs.stagger_timeout = options.stagger_timeout; - } } else { inputs.master_url = window.location.origin; } @@ -449,4 +298,4 @@ app.registerExtension({ async setup() { new MultiGPUExtension(); } -}); \ No newline at end of file +}); From 7ddb5bc6871de80097c07b5ed07b786683faa133 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Thu, 26 Jun 2025 08:18:27 +1000 Subject: [PATCH 09/41] Update README.md --- README.md | 52 +++++++++++++++++++++++++++++----------------------- 1 file changed, 29 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 5458a71..24f1b58 100644 --- a/README.md +++ b/README.md @@ -15,11 +15,9 @@ ComfyUI-Distributed extends ComfyUI with the ability to distribute batch process - **Multi-GPU Distributed Processing**: Leverage all available GPUs simultaneously - **Master-Worker Architecture**: Automatic coordination between GPU instances - **Dynamic Worker Management**: Enable/disable GPUs on-the-fly through the UI -- **Execution Modes**: - - **Parallel Mode**: All GPUs process simultaneously for maximum speed - - **Staggered Mode**: Sequential GPU startup with configurable delays to manage VRAM usage -- **VRAM Management**: Built-in tools to clear GPU memory across all instances +- **Parallel Execution**: All enabled GPUs process batches simultaneously for maximum speed - **Web UI Integration**: Seamless control panel integrated into ComfyUI's interface +- **Smart Launch**: Launcher checks for running instances and starts only those that aren't running ## How It Works @@ -58,21 +56,20 @@ For example, with 4 GPUs and a batch size of 2, you'll generate 8 images in para - Adjust the number of workers based on your available GPUs - Update CUDA device numbers if needed - Set appropriate ports (ensure they're not in use) + - Add custom launch arguments in `extra_args` for master and workers 3. Run the launcher script from the ComfyUI root directory: ```bash launch_distributed.bat ``` - This will automatically start ComfyUI instances for each GPU 4. Open the master instance (usually http://localhost:8188) 5. Add the "Multi-GPU Collector" node to your workflow 6. Use the GPU control panel in the sidebar to: - - Select which GPUs to use - - Choose between parallel or staggered execution - - Clear VRAM when needed + - View GPU status and worker information + - Monitor active instances ### Manual Setup (All Platforms) @@ -98,31 +95,39 @@ The system uses a `gpu_config.json` file for configuration. Default configuratio ```json { "master": { - "port": 8188, - "device": 0, - "enabled": true + "port": 8480, + "cuda_device": 0, + "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" }, "workers": [ { - "port": 8189, - "device": 1, + "id": 1, + "name": "GPU 1", + "cuda_device": 1, + "port": 8180, "enabled": true, - "extra_args": "--enable-cors-header" + "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" }, { - "port": 8190, - "device": 2, + "id": 2, + "name": "GPU 2", + "cuda_device": 2, + "port": 8280, "enabled": true, - "extra_args": "--enable-cors-header" + "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" } ], "settings": { - "stagger_delay": 2.0, - "job_timeout": 120 + "retry_delay_ms": 500 } } ``` +**Configuration Options:** +- `extra_args`: Custom launch arguments for each instance (e.g., memory management flags) +- `retry_delay_ms`: Delay between connection attempts during startup +- Worker `enabled` flag: Control which GPUs are active + ## Workflow Integration The Multi-GPU Collector node integrates seamlessly with existing ComfyUI workflows: @@ -139,16 +144,17 @@ The Multi-GPU Collector node integrates seamlessly with existing ComfyUI workflo ## Performance Considerations -- **Parallel Mode**: Best for workflows with consistent VRAM usage -- **Staggered Mode**: Ideal for memory-intensive workflows or when GPUs have different VRAM capacities +- **Parallel Execution**: All enabled GPUs process simultaneously for maximum throughput - **Batch Size**: Total generation = batch_size × number_of_active_GPUs +- **Smart Launching**: The launcher only starts instances that aren't already running +- **Memory Management**: Use appropriate `extra_args` in configuration for memory optimization ## Troubleshooting - **Workers not connecting**: Ensure CORS headers are enabled (`--enable-cors-header`) -- **VRAM errors**: Use staggered mode with appropriate delays -- **Timeout errors**: Increase `job_timeout` in configuration +- **VRAM errors**: Add memory management flags in `extra_args` (e.g., `--highvram`, `--disable-smart-memory`) - **Port conflicts**: Ensure each instance uses a unique port +- **Partial startup**: The launcher will automatically start only missing instances without affecting running ones ## Development From bded8ac4169bfcd53ec9ade61065ed334e9f7e47 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Thu, 26 Jun 2025 08:24:48 +1000 Subject: [PATCH 10/41] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 24f1b58..0c96b7b 100644 --- a/README.md +++ b/README.md @@ -163,7 +163,7 @@ This project is under active development. Contributions are welcome! ### Planned Features/Improvements - Support for remote workers over network -- Better VRAM management for multi-GPU setups +- ~Better VRAM management for multi-GPU setups~ - Better Seed handling - Better Batch handling - Handle worker failure From 0eba92a3522809e92efcc432052503f3ac136f71 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Thu, 26 Jun 2025 08:55:00 +1000 Subject: [PATCH 11/41] Update README.md --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 0c96b7b..0fc128c 100644 --- a/README.md +++ b/README.md @@ -163,6 +163,8 @@ This project is under active development. Contributions are welcome! ### Planned Features/Improvements - Support for remote workers over network + - Worker status (read/in progress/offline) + - Restart worker button - ~Better VRAM management for multi-GPU setups~ - Better Seed handling - Better Batch handling From 522c7705fa2c640fc9504eb0af8b9ff37e8e1a9e Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Fri, 27 Jun 2025 09:24:09 +1000 Subject: [PATCH 12/41] Update README.md --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 0fc128c..a8c0997 100644 --- a/README.md +++ b/README.md @@ -165,9 +165,12 @@ This project is under active development. Contributions are welcome! - Support for remote workers over network - Worker status (read/in progress/offline) - Restart worker button + - Cold start workers from UI - ~Better VRAM management for multi-GPU setups~ - Better Seed handling - Better Batch handling + - Batch Size Override + - UI control to dynamically adjust batch size per generation - Handle worker failure - Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut From 1ce10e1aa68b3d0821f637175b762f683989e0a3 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sat, 28 Jun 2025 10:31:10 +1000 Subject: [PATCH 13/41] Update README.md --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index a8c0997..0b2890c 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,6 @@ This project is under active development. Contributions are welcome! - Support for remote workers over network - Worker status (read/in progress/offline) - Restart worker button - - Cold start workers from UI - ~Better VRAM management for multi-GPU setups~ - Better Seed handling - Better Batch handling From 22c3c843faab87ba9b57d99d244375e7dc17bebc Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:13:51 +1000 Subject: [PATCH 14/41] Update multigpu.py --- multigpu.py | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/multigpu.py b/multigpu.py index 580c289..f5896fa 100644 --- a/multigpu.py +++ b/multigpu.py @@ -134,6 +134,51 @@ async def prepare_job_endpoint(request): except Exception as e: return await handle_api_error(request, e) +@server.PromptServer.instance.routes.post("/multigpu/clear_memory") +async def clear_memory_endpoint(request): + print("[MultiGPU] Received request to clear VRAM.") + try: + # Use ComfyUI's prompt server queue system like the /free endpoint does + if hasattr(server.PromptServer.instance, 'prompt_queue'): + server.PromptServer.instance.prompt_queue.set_flag("unload_models", True) + server.PromptServer.instance.prompt_queue.set_flag("free_memory", True) + print("[MultiGPU] Set queue flags for memory clearing.") + + # Wait a bit for the queue to process + await asyncio.sleep(0.5) + + # Also do direct cleanup as backup, but with error handling + import gc + import comfy.model_management as mm + + try: + mm.unload_all_models() + except AttributeError as e: + print(f"[MultiGPU] Warning during model unload: {e}") + + try: + mm.soft_empty_cache() + except Exception as e: + print(f"[MultiGPU] Warning during cache clear: {e}") + + for _ in range(3): + gc.collect() + + if torch.cuda.is_available(): + torch.cuda.empty_cache() + torch.cuda.ipc_collect() + + print("[MultiGPU] VRAM cleared successfully.") + return web.json_response({"status": "success", "message": "GPU memory cleared."}) + except Exception as e: + # Even if there's an error, try to do basic cleanup + import gc + gc.collect() + if torch.cuda.is_available(): + torch.cuda.empty_cache() + print(f"[MultiGPU] Partial VRAM clear completed with warning: {e}") + return web.json_response({"status": "success", "message": "GPU memory cleared (with warnings)"}) + # --- Global State & Callback Endpoint --- PENDING_JOBS = {} From ed43acae5c9ebc0be3d5c5fa915e9789454e0c7b Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:14:03 +1000 Subject: [PATCH 15/41] Update gpupanel.js --- web/gpupanel.js | 66 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 64 insertions(+), 2 deletions(-) diff --git a/web/gpupanel.js b/web/gpupanel.js index 44c674c..261d5fd 100644 --- a/web/gpupanel.js +++ b/web/gpupanel.js @@ -5,6 +5,7 @@ class MultiGPUExtension { // Button styling constants static BUTTON_STYLES = { base: "width: 100%; padding: 8px; font-size: 12px; color: white; border: none; border-radius: 4px; cursor: pointer; transition: background-color 0.2s;", + clearMemory: "background-color: #555;", success: "background-color: #3ca03c;", error: "background-color: #c04c4c;" }; @@ -97,6 +98,57 @@ class MultiGPUExtension { return button; } + async _handleClearMemory(button) { + const originalText = button.textContent; + button.textContent = "Clearing..."; + button.disabled = true; + + try { + const urlsToClear = this.enabledWorkers.map(w => ({ + name: w.name, + url: `http://${w.host || window.location.hostname}:${w.port}` + })); + + if (urlsToClear.length === 0) { + button.textContent = "No Workers"; + button.style.backgroundColor = MultiGPUExtension.BUTTON_STYLES.error.split(':')[1].trim().replace(';', ''); + setTimeout(() => { + button.textContent = originalText; + button.style.cssText = MultiGPUExtension.BUTTON_STYLES.base + MultiGPUExtension.BUTTON_STYLES.clearMemory; + }, 3000); + return; + } + + const promises = urlsToClear.map(target => + fetch(`${target.url}/multigpu/clear_memory`, { + method: 'POST', + mode: 'cors' + }) + .then(response => ({ ok: response.ok, name: target.name })) + .catch(() => ({ ok: false, name: target.name })) + ); + + const results = await Promise.all(promises); + const failures = results.filter(r => !r.ok); + + if (failures.length === 0) { + button.textContent = "Success!"; + button.style.backgroundColor = MultiGPUExtension.BUTTON_STYLES.success.split(':')[1].trim().replace(';', ''); + } else { + button.textContent = "Error! See Console"; + button.style.backgroundColor = MultiGPUExtension.BUTTON_STYLES.error.split(':')[1].trim().replace(';', ''); + console.error("[MultiGPU] Failed to clear memory on:", failures.map(f => f.name).join(", ")); + } + + setTimeout(() => { + button.textContent = originalText; + button.style.cssText = MultiGPUExtension.BUTTON_STYLES.base + MultiGPUExtension.BUTTON_STYLES.clearMemory; + }, 3000); + } finally { + button.disabled = false; + } + } + renderSidebarContent(el) { el.innerHTML = ''; @@ -131,7 +183,9 @@ class MultiGPUExtension { label.htmlFor = `gpu-${worker.id}`; label.style.cssText = "cursor: pointer; display: flex; align-items: center; gap: 8px;"; const gpuInfo = document.createElement("span"); - gpuInfo.innerHTML = `${worker.name}
CUDA ${worker.cuda_device} • Port ${worker.port}`; + const hostInfo = worker.host ? ` • ${worker.host}` : ''; + const cudaInfo = worker.cuda_device !== undefined ? `CUDA ${worker.cuda_device} • ` : ''; + gpuInfo.innerHTML = `${worker.name}
${cudaInfo}Port ${worker.port}${hostInfo}`; label.appendChild(checkbox); label.appendChild(gpuInfo); gpuDiv.appendChild(label); @@ -140,6 +194,14 @@ class MultiGPUExtension { gpuSection.appendChild(gpuList); container.appendChild(gpuSection); + const actionsSection = document.createElement("div"); + actionsSection.style.cssText = "padding-top: 10px; margin-bottom: 15px; border-top: 1px solid #444;"; + + const clearMemButton = this._createButton("Clear Worker VRAM", (e) => this._handleClearMemory(e.target), MultiGPUExtension.BUTTON_STYLES.clearMemory); + clearMemButton.title = "Clear VRAM on all enabled worker GPUs (not master)"; + actionsSection.appendChild(clearMemButton); + + container.appendChild(actionsSection); const summarySection = document.createElement("div"); summarySection.style.cssText = "border-top: 1px solid #444; padding-top: 10px;"; @@ -273,7 +335,7 @@ class MultiGPUExtension { } async _dispatchToWorker(worker, prompt, workflow) { - const workerUrl = `http://localhost:${worker.port}`; + const workerUrl = `http://${worker.host || window.location.hostname}:${worker.port}`; console.log(`[MultiGPU] Dispatching to worker: ${worker.name} (${workerUrl})`); const promptToSend = { prompt, From f99bde17833ecc0348739de533c2ded26c6008b3 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:16:47 +1000 Subject: [PATCH 16/41] Update README.md --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0b2890c..b3c773b 100644 --- a/README.md +++ b/README.md @@ -162,8 +162,11 @@ This project is under active development. Contributions are welcome! ### Planned Features/Improvements -- Support for remote workers over network - - Worker status (read/in progress/offline) +- Custom launcher + - Launch local workers + - Launch remote workers (via SSH) +- ~Support for remote workers over network~ + - Worker status (ready/in progress/offline) - Restart worker button - ~Better VRAM management for multi-GPU setups~ - Better Seed handling From e753bd8a8d1a3838330fe9c409879d92751bb2f0 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:19:24 +1000 Subject: [PATCH 17/41] New launcher (supports remote workers) --- launch_comfyui.bat | 497 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 497 insertions(+) create mode 100644 launch_comfyui.bat diff --git a/launch_comfyui.bat b/launch_comfyui.bat new file mode 100644 index 0000000..3207f34 --- /dev/null +++ b/launch_comfyui.bat @@ -0,0 +1,497 @@ +@echo off +title ComfyUI Unified Launcher +cd /d "%~dp0" +color 0B +cls + +echo. +echo ================================================================ +echo ComfyUI Unified Launcher +echo ================================================================ +echo. + +:: Check command line parameter +set "MODE=%1" +if "%MODE%"=="" goto :SHOW_MENU +if /i "%MODE%"=="master" goto :SETUP +if /i "%MODE%"=="all-slow" goto :SETUP +if /i "%MODE%"=="all-fast" goto :SETUP +if /i "%MODE%"=="venv" goto :ACTIVATE_VENV +if /i "%MODE%"=="update" goto :UPDATE_COMFYUI +if /i "%MODE%"=="update-nodes" goto :UPDATE_NODES + +echo. +echo ERROR: Invalid parameter: %MODE% +echo. +timeout /t 2 /nobreak >nul +goto :SHOW_MENU + +:SHOW_MENU +color 0B +echo ---------------------------------------------------------------- +echo Launch Mode Selection +echo ---------------------------------------------------------------- +echo. +echo [1] Master only +echo ^> Launch only the master instance +echo. +echo [2] All instances (fast) +echo ^> Launch all instances with 2-second delays +echo. +echo [3] All instances (sequential) +echo ^> Launch all instances, wait for each to be ready +echo. +echo [4] Activate virtual environment +echo ^> Open command prompt with venv activated +echo. +echo [5] Update ComfyUI +echo ^> Update ComfyUI to latest version +echo. +echo [6] Update custom nodes +echo ^> Update all custom nodes to latest versions +echo. +echo [Q] Quit +echo. +echo ---------------------------------------------------------------- +echo TIP: You can also launch directly from command line: +echo launch_comfyui.bat master +echo launch_comfyui.bat all-slow +echo launch_comfyui.bat all-fast +echo launch_comfyui.bat venv +echo launch_comfyui.bat update +echo launch_comfyui.bat update-nodes +echo ---------------------------------------------------------------- +echo. +set /p "CHOICE= ^> Enter your choice (1, 2, 3, 4, 5, 6, or Q): " + +if /i "%CHOICE%"=="1" ( + set "MODE=master" + goto :SETUP +) +if /i "%CHOICE%"=="2" ( + set "MODE=all-fast" + goto :SETUP +) +if /i "%CHOICE%"=="3" ( + set "MODE=all-slow" + goto :SETUP +) +if /i "%CHOICE%"=="4" ( + goto :ACTIVATE_VENV +) +if /i "%CHOICE%"=="5" ( + goto :UPDATE_COMFYUI +) +if /i "%CHOICE%"=="6" ( + goto :UPDATE_NODES +) +if /i "%CHOICE%"=="q" ( + echo Exiting... + exit /b 0 +) + +echo. +echo ERROR: Invalid choice. Please enter 1, 2, 3, 4, 5, 6, or Q. +echo. +timeout /t 2 /nobreak >nul +goto :SHOW_MENU + +:SETUP +echo Mode: %MODE% +echo. + +:: Check if gpu_config.json exists (relative path) +set "CONFIG_PATH=ComfyUI\custom_nodes\ComfyUI-Distributed\gpu_config.json" +if not exist "%CONFIG_PATH%" ( + color 0C + echo Error: Configuration file not found! + echo Expected location: %CONFIG_PATH% + echo. + echo Make sure ComfyUI-Distributed custom node is installed. + pause + exit /b 1 +) + +:: Check if venv exists (relative path) +set "VENV_PATH=venv" +if not exist "%VENV_PATH%\Scripts\activate.bat" ( + color 0C + echo Error: Virtual environment not found! + echo Expected location: %CD%\%VENV_PATH% + echo Make sure this batch file is in the ComfyUI directory with venv folder + pause + exit /b 1 +) + +:: Check if ComfyUI CLI is available in venv +call "%VENV_PATH%\Scripts\activate.bat" && comfy --help >nul 2>&1 +if %ERRORLEVEL% NEQ 0 ( + color 0C + echo Error: ComfyUI CLI not found in virtual environment! + echo Make sure ComfyUI CLI is installed in: %VENV_PATH% + pause + exit /b 1 +) + +:: Jump to appropriate mode +if /i "%MODE%"=="master" goto :MODE_MASTER +if /i "%MODE%"=="all-slow" goto :MODE_ALL_SLOW +if /i "%MODE%"=="all-fast" goto :MODE_ALL_FAST + +::============================================================================= +:: MASTER MODE - Launch only the master instance +::============================================================================= +:MODE_MASTER +echo Configuring memory management for large models... +set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True +echo Memory management configured. + +echo Loading master configuration... + +:: Read master configuration +powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; $master = $json.master; Write-Output \"MASTER_PORT=$($master.port)\"; Write-Output \"MASTER_CUDA=$($master.cuda_device)\"; Write-Output \"MASTER_EXTRA=$($master.extra_args)\" } catch { Write-Output \"PowerShell Error: $_\"; pause; exit 1 }" > %TEMP%\master_config.txt + +if %ERRORLEVEL% NEQ 0 ( + color 0C + echo Error reading master configuration! + pause + exit /b 1 +) + +for /f "tokens=*" %%i in (%TEMP%\master_config.txt) do ( + set %%i +) + +del %TEMP%\master_config.txt >nul 2>&1 + +:: Display master configuration +echo. +echo Master Configuration: +echo Port: %MASTER_PORT% +echo CUDA Device: %MASTER_CUDA% +echo Extra Args: %MASTER_EXTRA% +echo. + +:: Check if master is already running +echo Checking if master is already running... +powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %MASTER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" + +if %ERRORLEVEL% == 0 ( + color 0E + echo Master is already running on port %MASTER_PORT%! + echo URL: http://localhost:%MASTER_PORT% + echo. + echo Press any key to exit... + pause >nul + exit /b 0 +) + +:: Launch master +echo Launching Master ComfyUI instance... + +:: Try Windows Terminal first, fall back to cmd +wt -w 0 new-tab --title "Master - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title Master - ComfyUI CLI && echo CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && echo Starting ComfyUI Master... && comfy --here launch -- --port %MASTER_PORT% %MASTER_EXTRA%" 2>nul + +if %ERRORLEVEL% NEQ 0 ( + echo Windows Terminal not available, using regular command prompt... + start "Master - ComfyUI CLI" cmd /k "title Master - ComfyUI CLI && cd /d "%CD%" && echo CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && echo Starting ComfyUI Master... && comfy --here launch -- --port %MASTER_PORT% %MASTER_EXTRA%" +) + +echo. +echo ====================================== +echo Master ComfyUI launcher started! +echo ====================================== +echo. +echo Expected URL: http://localhost:%MASTER_PORT% +echo CUDA Device: %MASTER_CUDA% +echo. +echo The ComfyUI window will open shortly. +echo This launcher will close in 1 second... + +timeout /t 1 /nobreak >nul +exit + +::============================================================================= +:: ALL-SLOW MODE - Launch all instances sequentially +::============================================================================= +:MODE_ALL_SLOW +echo Configuring memory management for large models... +set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True +echo Memory management configured. + +:: Read configuration with PowerShell and store in environment variables +echo Loading configuration... + +powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; $master = $json.master; $settings = $json.settings; $retryDelay = if ($settings.retry_delay_ms) { $settings.retry_delay_ms } else { 500 }; Write-Output \"RETRY_DELAY=$retryDelay\"; Write-Output \"MASTER_PORT=$($master.port)\"; Write-Output \"MASTER_CUDA=$($master.cuda_device)\"; Write-Output \"MASTER_EXTRA=$($master.extra_args)\"; foreach ($worker in $json.workers) { if ($worker.enabled -and -not $worker.host) { Write-Output \"WORKER_$($worker.id)_NAME=$($worker.name)\"; Write-Output \"WORKER_$($worker.id)_PORT=$($worker.port)\"; Write-Output \"WORKER_$($worker.id)_CUDA=$($worker.cuda_device)\"; Write-Output \"WORKER_$($worker.id)_EXTRA=$($worker.extra_args)\" } } } catch { Write-Output \"PowerShell Error: $_\"; pause; exit 1 }" > %TEMP%\config_vars.txt + +if %ERRORLEVEL% NEQ 0 ( + color 0C + echo Error running PowerShell configuration reader! + pause + exit /b 1 +) + +for /f "tokens=*" %%i in (%TEMP%\config_vars.txt) do ( + set %%i +) + +del %TEMP%\config_vars.txt >nul 2>&1 + +:: Check Master +echo Starting Master instance... +:MASTER_SLOW +powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %MASTER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" +if %ERRORLEVEL% == 1 ( + if not defined MASTER_STARTED ( + echo Starting Master with CUDA_VISIBLE_DEVICES=%MASTER_CUDA% + start wt -w 0 new-tab --title "Master - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title Master - ComfyUI CLI && echo CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && comfy --here launch -- --port %MASTER_PORT% %MASTER_EXTRA%" + set MASTER_STARTED=1 + ) + %TEMP%\worker_ids.txt + +for /f %%w in (%TEMP%\worker_ids.txt) do ( + call :PROCESS_WORKER_SLOW %%w +) + +del %TEMP%\worker_ids.txt >nul 2>&1 +goto :FINISH_ALL + +:PROCESS_WORKER_SLOW +set "WORKER_ID=%1" +call set "WORKER_NAME=%%WORKER_%WORKER_ID%_NAME%%" +call set "WORKER_PORT=%%WORKER_%WORKER_ID%_PORT%%" +call set "WORKER_CUDA=%%WORKER_%WORKER_ID%_CUDA%%" +call set "WORKER_EXTRA=%%WORKER_%WORKER_ID%_EXTRA%%" + +echo Starting %WORKER_NAME%... + +:WORKER_CHECK_LOOP_SLOW +powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %WORKER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" +if %ERRORLEVEL% == 1 ( + if not defined WORKER%WORKER_ID%_STARTED ( + echo Starting %WORKER_NAME% with CUDA_VISIBLE_DEVICES=%WORKER_CUDA% + start wt -w 0 new-tab --title "%WORKER_NAME% - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title %WORKER_NAME% - ComfyUI CLI && echo CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && comfy --here launch -- --port %WORKER_PORT% %WORKER_EXTRA% --enable-cors-header" + set WORKER%WORKER_ID%_STARTED=1 + ) + %TEMP%\config_vars.txt + +if %ERRORLEVEL% NEQ 0 ( + color 0C + echo Error running PowerShell configuration reader! + pause + exit /b 1 +) + +for /f "tokens=*" %%i in (%TEMP%\config_vars.txt) do ( + set %%i +) + +del %TEMP%\config_vars.txt >nul 2>&1 + +echo. +echo Checking running instances... +echo. + +set "INSTANCES_STARTED=0" + +:: Check and start Master if not running +echo Checking Master (port %MASTER_PORT%)... +powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %MASTER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" +if %ERRORLEVEL% == 1 ( + echo Master NOT running - Starting with CUDA_VISIBLE_DEVICES=%MASTER_CUDA% + start wt -w 0 new-tab --title "Master - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title Master - ComfyUI CLI && echo CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && comfy --here launch -- --port %MASTER_PORT% %MASTER_EXTRA%" + set /a INSTANCES_STARTED+=1 + echo Waiting 2 seconds before checking next instance... + timeout /t 2 /nobreak >nul +) else ( + echo Master already running on port %MASTER_PORT% +) + +:: Check and start workers +powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; foreach ($worker in $json.workers) { if ($worker.enabled -and -not $worker.host) { Write-Output $worker.id } } } catch { exit 1 }" > %TEMP%\worker_ids.txt + +for /f %%w in (%TEMP%\worker_ids.txt) do ( + call :CHECK_AND_START_WORKER_FAST %%w +) + +del %TEMP%\worker_ids.txt >nul 2>&1 + +echo. +echo ====================================== +if %INSTANCES_STARTED% == 0 ( + echo All instances were already running! +) else ( + echo Started %INSTANCES_STARTED% new instance^(s^)! +) +echo ====================================== +goto :FINISH_ALL + +:CHECK_AND_START_WORKER_FAST +set "WORKER_ID=%1" +call set "WORKER_NAME=%%WORKER_%WORKER_ID%_NAME%%" +call set "WORKER_PORT=%%WORKER_%WORKER_ID%_PORT%%" +call set "WORKER_CUDA=%%WORKER_%WORKER_ID%_CUDA%%" +call set "WORKER_EXTRA=%%WORKER_%WORKER_ID%_EXTRA%%" + +echo Checking %WORKER_NAME% (port %WORKER_PORT%)... +powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %WORKER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" +if %ERRORLEVEL% == 1 ( + echo %WORKER_NAME% NOT running - Starting with CUDA_VISIBLE_DEVICES=%WORKER_CUDA% + start wt -w 0 new-tab --title "%WORKER_NAME% - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title %WORKER_NAME% - ComfyUI CLI && echo CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && comfy --here launch -- --port %WORKER_PORT% %WORKER_EXTRA% --enable-cors-header" + set /a INSTANCES_STARTED+=1 + echo Waiting 2 seconds before checking next instance... + timeout /t 2 /nobreak >nul +) else ( + echo %WORKER_NAME% already running on port %WORKER_PORT% +) +goto :eof + +::============================================================================= +:: FINISH - Common ending for all modes +::============================================================================= +:FINISH_ALL +echo. +echo Master: http://localhost:%MASTER_PORT% +powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; foreach ($worker in $json.workers) { if ($worker.enabled -and -not $worker.host) { Write-Output \"$($worker.name): http://localhost:$($worker.port)\" } } } catch { }" +echo. +echo Launcher will close in 5 seconds... +timeout /t 5 /nobreak >nul +exit + +::============================================================================= +:: ACTIVATE VIRTUAL ENVIRONMENT - Open command prompt with venv activated +::============================================================================= +:ACTIVATE_VENV +echo. +echo ================================================================ +echo Activating Virtual Environment +echo ================================================================ +echo. + +:: Check if venv exists +set "VENV_PATH=venv" +if not exist "%VENV_PATH%\Scripts\activate.bat" ( + color 0C + echo Error: Virtual environment not found! + echo Expected location: %CD%\%VENV_PATH% + pause + exit /b 1 +) + +echo Activating virtual environment... +echo Project folder: %CD% +echo. +echo Opening new command prompt with activated environment... +echo (You can close this window) +echo. + +:: Open new command prompt with venv activated +start "ComfyUI (Active venv)" cmd /k "call "%CD%\%VENV_PATH%\Scripts\activate.bat" & cd /d "%CD%"" +exit + +::============================================================================= +:: UPDATE COMFYUI - Update ComfyUI to latest version +::============================================================================= +:UPDATE_COMFYUI +echo. +echo ================================================================ +echo Updating ComfyUI +echo ================================================================ +echo. + +:: Check if venv exists +set "VENV_PATH=venv" +if not exist "%VENV_PATH%\Scripts\activate.bat" ( + color 0C + echo Error: Virtual environment not found! + pause + exit /b 1 +) + +:: Check if ComfyUI CLI is available +call "%VENV_PATH%\Scripts\activate.bat" && comfy --help >nul 2>&1 +if %ERRORLEVEL% NEQ 0 ( + color 0C + echo Error: ComfyUI CLI not found in virtual environment! + pause + exit /b 1 +) + +echo Starting ComfyUI update... +echo This may take a few minutes... +echo. + +:: Activate venv and run update +call "%VENV_PATH%\Scripts\activate.bat" && comfy --here update + +echo. +echo Update completed! +echo. +pause +exit + +::============================================================================= +:: UPDATE CUSTOM NODES - Update all custom nodes +::============================================================================= +:UPDATE_NODES +echo. +echo ================================================================ +echo Updating Custom Nodes +echo ================================================================ +echo. + +:: Check if venv exists +set "VENV_PATH=venv" +if not exist "%VENV_PATH%\Scripts\activate.bat" ( + color 0C + echo Error: Virtual environment not found! + pause + exit /b 1 +) + +:: Check if ComfyUI CLI is available +call "%VENV_PATH%\Scripts\activate.bat" && comfy --help >nul 2>&1 +if %ERRORLEVEL% NEQ 0 ( + color 0C + echo Error: ComfyUI CLI not found in virtual environment! + pause + exit /b 1 +) + +echo Starting custom nodes update... +echo This may take several minutes... +echo. + +:: Activate venv and run node update +call "%VENV_PATH%\Scripts\activate.bat" && comfy --here node update-all + +echo. +echo Custom nodes update completed! +echo. +pause +exit \ No newline at end of file From c8a7ed8bc1338715f7c5004330cbf8f4f3456b57 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:19:40 +1000 Subject: [PATCH 18/41] Delete launch_distributed.bat --- launch_distributed.bat | 133 ----------------------------------------- 1 file changed, 133 deletions(-) delete mode 100644 launch_distributed.bat diff --git a/launch_distributed.bat b/launch_distributed.bat deleted file mode 100644 index c62ac5a..0000000 --- a/launch_distributed.bat +++ /dev/null @@ -1,133 +0,0 @@ -@echo off -title ComfyUI Multi-GPU Fast Launcher (Check & Start) -cd /d "%~dp0" -color 0B -cls - -echo ComfyUI Multi-GPU Fast Launcher (Check ^& Start) -echo =============================================== -echo. -echo This launcher checks for running instances and starts only -echo those that aren't running, with a 2-second delay between launches. -echo. - -:: Check if gpu_config.json exists (relative path) -set "CONFIG_PATH=ComfyUI\custom_nodes\ComfyUI-Distributed\gpu_config.json" -if not exist "%CONFIG_PATH%" ( - color 0C - echo Error: Configuration file not found! - echo Expected location: %CONFIG_PATH% - pause - exit /b 1 -) - -:: Check if venv exists (relative path) -set "VENV_PATH=venv" -if not exist "%VENV_PATH%\Scripts\activate.bat" ( - color 0C - echo Error: Virtual environment not found! - echo Expected location: %CD%\%VENV_PATH% - echo Make sure this batch file is in the ComfyUI_Dev directory with venv folder - pause - exit /b 1 -) - -:: Check if ComfyUI CLI is available in venv -call "%VENV_PATH%\Scripts\activate.bat" && comfy --help >nul 2>&1 -if %ERRORLEVEL% NEQ 0 ( - color 0C - echo Error: ComfyUI CLI not found in virtual environment! - echo Make sure ComfyUI CLI is installed in: %VENV_PATH% - pause - exit /b 1 -) - -:: Configure Windows memory management for large model loading -echo Configuring memory management for large models... - -:: Configure PyTorch memory allocation for large models -set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True - -echo Memory management configured. - -:: Read configuration with PowerShell and store in environment variables -echo Loading configuration... - -powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; $master = $json.master; Write-Output \"MASTER_PORT=$($master.port)\"; Write-Output \"MASTER_CUDA=$($master.cuda_device)\"; Write-Output \"MASTER_EXTRA=$($master.extra_args)\"; foreach ($worker in $json.workers) { if ($worker.enabled) { Write-Output \"WORKER_$($worker.id)_NAME=$($worker.name)\"; Write-Output \"WORKER_$($worker.id)_PORT=$($worker.port)\"; Write-Output \"WORKER_$($worker.id)_CUDA=$($worker.cuda_device)\"; Write-Output \"WORKER_$($worker.id)_EXTRA=$($worker.extra_args)\" } } } catch { Write-Output \"PowerShell Error: $_\"; pause; exit 1 }" > %TEMP%\config_vars.txt - -if %ERRORLEVEL% NEQ 0 ( - color 0C - echo Error running PowerShell configuration reader! - pause - exit /b 1 -) - -for /f "tokens=*" %%i in (%TEMP%\config_vars.txt) do ( - set %%i -) - -del %TEMP%\config_vars.txt >nul 2>&1 - -echo. -echo Checking running instances... -echo. - -set "INSTANCES_STARTED=0" - -:: Check and start Master if not running -echo Checking Master (port %MASTER_PORT%)... -powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %MASTER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" -if %ERRORLEVEL% == 1 ( - echo Master NOT running - Starting with CUDA_VISIBLE_DEVICES=%MASTER_CUDA% - start wt -w 0 new-tab --title "Master - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title Master - ComfyUI CLI && echo CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%MASTER_CUDA% && comfy --here launch -- --port %MASTER_PORT% %MASTER_EXTRA%" - set /a INSTANCES_STARTED+=1 - echo Waiting 2 seconds before checking next instance... - timeout /t 2 /nobreak >nul -) else ( - echo Master already running on port %MASTER_PORT% -) - -:: Check and start workers -powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; foreach ($worker in $json.workers) { if ($worker.enabled) { Write-Output $worker.id } } } catch { exit 1 }" > %TEMP%\worker_ids.txt - -for /f %%w in (%TEMP%\worker_ids.txt) do ( - call :CHECK_AND_START_WORKER %%w -) - -del %TEMP%\worker_ids.txt >nul 2>&1 - -echo. -echo ====================================== -if %INSTANCES_STARTED% == 0 ( - echo All instances were already running! -) else ( - echo Started %INSTANCES_STARTED% new instance(s)! -) -echo ====================================== -echo. -echo Master: http://localhost:%MASTER_PORT% -powershell -NoProfile -Command "try { $json = Get-Content '%CONFIG_PATH%' -Raw | ConvertFrom-Json; foreach ($worker in $json.workers) { if ($worker.enabled) { Write-Output \"$($worker.name): http://localhost:$($worker.port)\" } } } catch { }" -echo. -echo Launcher will close in 5 seconds... -timeout /t 5 /nobreak >nul -exit - -:CHECK_AND_START_WORKER -set "WORKER_ID=%1" -call set "WORKER_NAME=%%WORKER_%WORKER_ID%_NAME%%" -call set "WORKER_PORT=%%WORKER_%WORKER_ID%_PORT%%" -call set "WORKER_CUDA=%%WORKER_%WORKER_ID%_CUDA%%" -call set "WORKER_EXTRA=%%WORKER_%WORKER_ID%_EXTRA%%" - -echo Checking %WORKER_NAME% (port %WORKER_PORT%)... -powershell -NoProfile -Command "$ErrorActionPreference = 'SilentlyContinue'; $tcpClient = New-Object System.Net.Sockets.TcpClient; $connect = $tcpClient.BeginConnect('127.0.0.1', %WORKER_PORT%, $null, $null); $wait = $connect.AsyncWaitHandle.WaitOne(200, $false); if (!$wait) { exit 1 } else { $tcpClient.EndConnect($connect); $tcpClient.Close(); exit 0 }" -if %ERRORLEVEL% == 1 ( - echo %WORKER_NAME% NOT running - Starting with CUDA_VISIBLE_DEVICES=%WORKER_CUDA% - start wt -w 0 new-tab --title "%WORKER_NAME% - ComfyUI CLI" --startingDirectory "%CD%" cmd /k "title %WORKER_NAME% - ComfyUI CLI && echo CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && call venv\Scripts\activate.bat && set CUDA_VISIBLE_DEVICES=%WORKER_CUDA% && comfy --here launch -- --port %WORKER_PORT% %WORKER_EXTRA% --enable-cors-header" - set /a INSTANCES_STARTED+=1 - echo Waiting 2 seconds before checking next instance... - timeout /t 2 /nobreak >nul -) else ( - echo %WORKER_NAME% already running on port %WORKER_PORT% -) -goto :eof From 247fbf393d9e0bf4d2626e1454f7fd328b6e8867 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:20:18 +1000 Subject: [PATCH 19/41] Update and rename launch_comfyui.bat to launch_distributed.bat --- launch_comfyui.bat => launch_distributed.bat | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename launch_comfyui.bat => launch_distributed.bat (97%) diff --git a/launch_comfyui.bat b/launch_distributed.bat similarity index 97% rename from launch_comfyui.bat rename to launch_distributed.bat index 3207f34..a33186f 100644 --- a/launch_comfyui.bat +++ b/launch_distributed.bat @@ -494,4 +494,4 @@ echo. echo Custom nodes update completed! echo. pause -exit \ No newline at end of file +exit From 394ffc24f30ebaba48941516e22964ce43fa971b Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:22:22 +1000 Subject: [PATCH 20/41] Update gpupanel.js --- web/gpupanel.js | 2 -- 1 file changed, 2 deletions(-) diff --git a/web/gpupanel.js b/web/gpupanel.js index 261d5fd..29ae974 100644 --- a/web/gpupanel.js +++ b/web/gpupanel.js @@ -14,8 +14,6 @@ class MultiGPUExtension { this.config = null; this.originalQueuePrompt = api.queuePrompt.bind(api); - // --- FIX [1 of 7]: Constructor Race Condition --- - // The interceptor is now set up *after* the config is guaranteed to be loaded. this.loadConfig().then(() => { this.registerSidebarTab(); this.setupInterceptor(); From 398ee5a47af4aca7b0ebdb408db1ba43d0d900c2 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:26:44 +1000 Subject: [PATCH 21/41] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b3c773b..5610b03 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ For example, with 4 GPUs and a batch size of 2, you'll generate 8 images in para ## Requirements -- ComfyUI installation +- ComfyUI installation (venv, portable not supported yet) - Multiple NVIDIA GPUs - Python 3.8+ - Windows (for the provided launcher script) or Linux/Mac with manual setup From f247ea21ff548743bbb73bed2330558a9293ba93 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:38:01 +1000 Subject: [PATCH 22/41] Update README.md --- README.md | 114 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 74 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index 5610b03..b1ece13 100644 --- a/README.md +++ b/README.md @@ -16,12 +16,14 @@ ComfyUI-Distributed extends ComfyUI with the ability to distribute batch process - **Master-Worker Architecture**: Automatic coordination between GPU instances - **Dynamic Worker Management**: Enable/disable GPUs on-the-fly through the UI - **Parallel Execution**: All enabled GPUs process batches simultaneously for maximum speed -- **Web UI Integration**: Seamless control panel integrated into ComfyUI's interface -- **Smart Launch**: Launcher checks for running instances and starts only those that aren't running +- **Web UI Integration**: Seamless control panel integrated into ComfyUI's sidebar +- **Smart Launcher**: Unified launcher with multiple modes and instance detection +- **VRAM Management**: Clear worker VRAM remotely through the UI +- **Flexible Configuration**: JSON-based configuration with runtime updates ## How It Works -1. Launch multiple ComfyUI instances, each assigned to a different GPU +1. Launch multiple ComfyUI instances using the unified launcher 2. One instance acts as the master (typically on CUDA device 0) 3. Add the "Multi-GPU Collector" node to your workflow 4. The extension automatically distributes batches across available GPUs @@ -31,10 +33,10 @@ For example, with 4 GPUs and a batch size of 2, you'll generate 8 images in para ## Requirements -- ComfyUI installation (venv, portable not supported yet) +- Virtual environment with ComfyUI installed +- ComfyUI installation with ComfyUI CLI (`comfy` command) - Multiple NVIDIA GPUs - Python 3.8+ -- Windows (for the provided launcher script) or Linux/Mac with manual setup ## Installation @@ -50,47 +52,41 @@ For example, with 4 GPUs and a batch size of 2, you'll generate 8 images in para ### Quick Start (Windows) -1. Copy the `launch_distributed.bat` file to your ComfyUI root directory (alongside other .bat files like `run_nvidia_gpu.bat`) +1. Copy the `launch_distributed.bat` file to your ComfyUI root directory (alongside your `venv` folder) 2. Edit the `gpu_config.json` file in `ComfyUI/custom_nodes/ComfyUI-Distributed/` to match your GPU setup: - Adjust the number of workers based on your available GPUs - Update CUDA device numbers if needed - Set appropriate ports (ensure they're not in use) - - Add custom launch arguments in `extra_args` for master and workers + - Add custom launch arguments in `extra_args` -3. Run the launcher script from the ComfyUI root directory: +3. Run the launcher from the ComfyUI root directory: ```bash launch_distributed.bat ``` + + Or use command-line options: + ```bash + launch_distributed.bat master # Launch master only + launch_distributed.bat all-fast # Launch all with 2-second delays + launch_distributed.bat all-slow # Launch all, wait for each to be ready + launch_distributed.bat venv # Open command prompt with venv activated + launch_distributed.bat update # Update ComfyUI + launch_distributed.bat update-nodes # Update all custom nodes + ``` -4. Open the master instance (usually http://localhost:8188) +4. Open the master instance (check the launcher output for the URL) 5. Add the "Multi-GPU Collector" node to your workflow -6. Use the GPU control panel in the sidebar to: - - View GPU status and worker information - - Monitor active instances - -### Manual Setup (All Platforms) - -Start each ComfyUI instance with the appropriate CUDA device: - -```bash -# Master instance (GPU 0) -CUDA_VISIBLE_DEVICES=0 python main.py --port 8188 - -# Worker instance (GPU 1) -CUDA_VISIBLE_DEVICES=1 python main.py --port 8189 --enable-cors-header - -# Worker instance (GPU 2) -CUDA_VISIBLE_DEVICES=2 python main.py --port 8190 --enable-cors-header - -# Continue for additional GPUs... -``` +6. Use the Multi-GPU sidebar panel to: + - Enable/disable worker GPUs + - View worker status + - Clear worker VRAM ### Configuration -The system uses a `gpu_config.json` file for configuration. Default configuration: +The system uses a `gpu_config.json` file for configuration. Example configuration: ```json { @@ -117,24 +113,48 @@ The system uses a `gpu_config.json` file for configuration. Default configuratio "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" } ], - "settings": { - "retry_delay_ms": 500 - } + "settings": {} } ``` **Configuration Options:** -- `extra_args`: Custom launch arguments for each instance (e.g., memory management flags) -- `retry_delay_ms`: Delay between connection attempts during startup -- Worker `enabled` flag: Control which GPUs are active +- `master`: Master instance configuration + - `port`: Port number for the master instance + - `cuda_device`: CUDA device index + - `extra_args`: Custom launch arguments +- `workers`: Array of worker configurations + - `id`: Unique worker identifier + - `name`: Display name in UI + - `cuda_device`: CUDA device index + - `port`: Port number for the worker + - `enabled`: Whether the worker is active + - `extra_args`: Custom launch arguments + - `host` (optional): Remote host address +- `settings`: Additional settings (reserved for future use) +### Remote Workers + +Remote workers are fully supported! Add a `host` field to any worker configuration: + +```json +{ + "id": 3, + "name": "Remote GPU", + "host": "192.168.1.100", + "port": 8190, + "enabled": true, + "extra_args": "--enable-cors-header" +} +``` + +Ensure remote workers are started with `--enable-cors-header` and are accessible from the master instance. ## Workflow Integration The Multi-GPU Collector node integrates seamlessly with existing ComfyUI workflows: 1. Connect the node after your image generation pipeline 2. The node will automatically detect batch sizes and distribute work -3. All GPUs will process their assigned batches +3. All GPUs will process their assigned batches in parallel 4. Results are collected and passed to the next node as a single batch ### Important Node Requirements @@ -142,23 +162,37 @@ The Multi-GPU Collector node integrates seamlessly with existing ComfyUI workflo - **Batch Node**: Your workflow must include a node with the title "BATCH" or "BATCH SIZE" (case-insensitive). This is required for the Multi-GPU system to detect the batch size for distribution. - The system will show an error if no batch node with the correct title is found. +## API Endpoints + +The extension provides several REST API endpoints for integration: + +- `GET /multigpu/config` - Retrieve current configuration +- `POST /multigpu/config/update_worker` - Enable/disable a worker +- `POST /multigpu/config/update_setting` - Update configuration settings +- `POST /multigpu/prepare_job` - Prepare a multi-GPU job +- `POST /multigpu/clear_memory` - Clear VRAM on workers +- `POST /multigpu/job_complete` - Worker callback for job completion + ## Performance Considerations - **Parallel Execution**: All enabled GPUs process simultaneously for maximum throughput -- **Batch Size**: Total generation = batch_size × number_of_active_GPUs +- **Batch Size**: Total generation = batch_size × (1 master + number_of_enabled_workers) - **Smart Launching**: The launcher only starts instances that aren't already running - **Memory Management**: Use appropriate `extra_args` in configuration for memory optimization +- **Network Overhead**: Minimal - only final images are transferred between instances ## Troubleshooting - **Workers not connecting**: Ensure CORS headers are enabled (`--enable-cors-header`) - **VRAM errors**: Add memory management flags in `extra_args` (e.g., `--highvram`, `--disable-smart-memory`) - **Port conflicts**: Ensure each instance uses a unique port -- **Partial startup**: The launcher will automatically start only missing instances without affecting running ones +- **Launcher errors**: Ensure the batch file is in the ComfyUI root directory with the `venv` folder +- **Batch node not found**: Ensure your workflow has a node titled "BATCH" or "BATCH SIZE" +- **ComfyUI CLI not found**: Make sure ComfyUI CLI is installed (`pip install comfyui-cli`) ## Development -This project is under active development. Contributions are welcome! +This project is stable and ready for use. Contributions are welcome! ### Planned Features/Improvements From f3903b1fe5f62515938b5ed6196dde731a09e89e Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:40:05 +1000 Subject: [PATCH 23/41] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b1ece13..e395c29 100644 --- a/README.md +++ b/README.md @@ -196,8 +196,8 @@ This project is stable and ready for use. Contributions are welcome! ### Planned Features/Improvements -- Custom launcher - - Launch local workers +- ~Custom launcher~ + - ~Launch local workers~ - Launch remote workers (via SSH) - ~Support for remote workers over network~ - Worker status (ready/in progress/offline) From 5d9a279869e19279dc1d559cbc79c898b688f691 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 09:41:00 +1000 Subject: [PATCH 24/41] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e395c29..2ce081d 100644 --- a/README.md +++ b/README.md @@ -192,7 +192,7 @@ The extension provides several REST API endpoints for integration: ## Development -This project is stable and ready for use. Contributions are welcome! +This project is under active development. Contributions are welcome! ### Planned Features/Improvements From 50fd52596e482b0fa31b994c859f020acd6b87ea Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 10:29:59 +1000 Subject: [PATCH 25/41] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2ce081d..0dcae4e 100644 --- a/README.md +++ b/README.md @@ -208,7 +208,7 @@ This project is under active development. Contributions are welcome! - Batch Size Override - UI control to dynamically adjust batch size per generation - Handle worker failure -- Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut +- ~Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut~ 027629c753dd3aae1ceeff5214ceb701943dd3fe ## License From 6d02cb32bf30b47e1d5b93cb14311bff8d951654 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 10:30:26 +1000 Subject: [PATCH 26/41] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0dcae4e..c788d8c 100644 --- a/README.md +++ b/README.md @@ -208,7 +208,7 @@ This project is under active development. Contributions are welcome! - Batch Size Override - UI control to dynamically adjust batch size per generation - Handle worker failure -- ~Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut~ 027629c753dd3aae1ceeff5214ceb701943dd3fe +- ~Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut~ the commit 027629c753dd3aae1ceeff5214ceb701943dd3fe ## License From c9b98c177ed55372032e88400eb24dc1afe35b87 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Sun, 29 Jun 2025 10:32:54 +1000 Subject: [PATCH 27/41] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c788d8c..8bdb7fe 100644 --- a/README.md +++ b/README.md @@ -208,7 +208,7 @@ This project is under active development. Contributions are welcome! - Batch Size Override - UI control to dynamically adjust batch size per generation - Handle worker failure -- ~Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut~ the commit 027629c753dd3aae1ceeff5214ceb701943dd3fe +- ~Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut~ [Commit 027629c](https://github.com/robertvoy/ComfyUI-Flux-Continuum/commit/027629c753dd3aae1ceeff5214ceb701943dd3fe) ## License From 573e266b1e90e23c3f9a30b405c45b34a45bfc92 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 07:37:17 +1000 Subject: [PATCH 28/41] Update launch_distributed.bat --- launch_distributed.bat | 85 ++++++++++++++++++++++-------------------- 1 file changed, 45 insertions(+), 40 deletions(-) diff --git a/launch_distributed.bat b/launch_distributed.bat index a33186f..bce8c97 100644 --- a/launch_distributed.bat +++ b/launch_distributed.bat @@ -6,7 +6,7 @@ cls echo. echo ================================================================ -echo ComfyUI Unified Launcher +echo ComfyUI Unified Launcher echo ================================================================ echo. @@ -29,28 +29,28 @@ goto :SHOW_MENU :SHOW_MENU color 0B echo ---------------------------------------------------------------- -echo Launch Mode Selection +echo Launch Mode Selection echo ---------------------------------------------------------------- echo. -echo [1] Master only -echo ^> Launch only the master instance +echo [1] Master only +echo ^> Launch only the master instance echo. -echo [2] All instances (fast) -echo ^> Launch all instances with 2-second delays +echo [2] All instances (fast) +echo ^> Launch all instances with 2-second delays echo. -echo [3] All instances (sequential) -echo ^> Launch all instances, wait for each to be ready +echo [3] All instances (sequential) +echo ^> Launch all instances, wait for each to be ready echo. -echo [4] Activate virtual environment -echo ^> Open command prompt with venv activated +echo [4] Activate virtual environment +echo ^> Open command prompt with venv activated echo. -echo [5] Update ComfyUI -echo ^> Update ComfyUI to latest version +echo [5] Update ComfyUI +echo ^> Update ComfyUI to latest version echo. -echo [6] Update custom nodes -echo ^> Update all custom nodes to latest versions +echo [6] Update custom nodes +echo ^> Update all custom nodes to latest versions echo. -echo [Q] Quit +echo [Q] Quit echo. echo ---------------------------------------------------------------- echo TIP: You can also launch directly from command line: @@ -62,40 +62,45 @@ echo launch_comfyui.bat update echo launch_comfyui.bat update-nodes echo ---------------------------------------------------------------- echo. -set /p "CHOICE= ^> Enter your choice (1, 2, 3, 4, 5, 6, or Q): " -if /i "%CHOICE%"=="1" ( - set "MODE=master" - goto :SETUP -) -if /i "%CHOICE%"=="2" ( - set "MODE=all-fast" - goto :SETUP -) -if /i "%CHOICE%"=="3" ( +:: Use CHOICE to wait for a single keypress without needing Enter +:: /C specifies the allowed characters. /N hides the default "[1,2,3,4,5,6,Q]?" prompt. /M sets a custom message. +choice /c 123456Q /n /m " ^> Enter your choice: " +echo. + +:: Check the ERRORLEVEL. IMPORTANT: Must check from highest to lowest! +:: CHOICE sets ERRORLEVEL to the index of the character in /C +:: 1=1, 2=2, 3=3, 4=4, 5=5, 6=6, Q=7 + +if errorlevel 7 goto :QUIT_CHOICE +if errorlevel 6 goto :UPDATE_NODES +if errorlevel 5 goto :UPDATE_COMFYUI +if errorlevel 4 goto :ACTIVATE_VENV +if errorlevel 3 ( set "MODE=all-slow" goto :SETUP ) -if /i "%CHOICE%"=="4" ( - goto :ACTIVATE_VENV +if errorlevel 2 ( + set "MODE=all-fast" + goto :SETUP ) -if /i "%CHOICE%"=="5" ( - goto :UPDATE_COMFYUI -) -if /i "%CHOICE%"=="6" ( - goto :UPDATE_NODES -) -if /i "%CHOICE%"=="q" ( - echo Exiting... - exit /b 0 +if errorlevel 1 ( + set "MODE=master" + goto :SETUP ) +:: This part of the script should now be unreachable, but it's good practice to handle it. echo. -echo ERROR: Invalid choice. Please enter 1, 2, 3, 4, 5, 6, or Q. +echo ERROR: Invalid choice. echo. timeout /t 2 /nobreak >nul goto :SHOW_MENU +:QUIT_CHOICE +echo Exiting... +timeout /t 1 /nobreak >nul +exit /b 0 + :SETUP echo Mode: %MODE% echo. @@ -389,7 +394,7 @@ exit :ACTIVATE_VENV echo. echo ================================================================ -echo Activating Virtual Environment +echo Activating Virtual Environment echo ================================================================ echo. @@ -420,7 +425,7 @@ exit :UPDATE_COMFYUI echo. echo ================================================================ -echo Updating ComfyUI +echo Updating ComfyUI echo ================================================================ echo. @@ -461,7 +466,7 @@ exit :UPDATE_NODES echo. echo ================================================================ -echo Updating Custom Nodes +echo Updating Custom Nodes echo ================================================================ echo. From 691e1d028b222787476d6a1bbfc85a460c8aabe2 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 07:37:43 +1000 Subject: [PATCH 29/41] Update multigpu.py --- multigpu.py | 192 +++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 161 insertions(+), 31 deletions(-) diff --git a/multigpu.py b/multigpu.py index f5896fa..6abbd92 100644 --- a/multigpu.py +++ b/multigpu.py @@ -190,6 +190,9 @@ async def job_complete_endpoint(request): data = await request.post() multi_job_id = data.get('multi_job_id') image_file = data.get('image') + worker_id = data.get('worker_id') + image_index = data.get('image_index') + is_last = data.get('is_last', 'False').lower() == 'true' if not all([multi_job_id, image_file]): return await handle_api_error(request, "Missing job_id or image data", 400) @@ -202,8 +205,13 @@ async def job_complete_endpoint(request): async with PENDING_JOBS_LOCK: if multi_job_id in PENDING_JOBS: - await PENDING_JOBS[multi_job_id].put(tensor) - print(f"[MultiGPU] Received result for job {multi_job_id}") + await PENDING_JOBS[multi_job_id].put({ + 'tensor': tensor, + 'worker_id': worker_id, + 'image_index': int(image_index) if image_index else 0, + 'is_last': is_last + }) + print(f"[MultiGPU] Received result for job {multi_job_id} from worker {worker_id} (last: {is_last})") return web.json_response({"status": "success"}) else: return await handle_api_error(request, "Job not found or already complete", 404) @@ -221,9 +229,8 @@ class MultiGPUCollectorNode: "is_worker": ("BOOLEAN", {"default": False}), "master_url": ("STRING", {"default": ""}), "enabled_worker_ids": ("STRING", {"default": "[]"}), - # --- FIX [7 of 7]: Type Mismatch --- - # Changed from FLOAT to INT for type safety and clarity. "worker_batch_size": ("INT", {"default": 1, "min": 1, "max": 1024}), + "worker_id": ("STRING", {"default": ""}), }, } @@ -231,7 +238,7 @@ class MultiGPUCollectorNode: FUNCTION = "run" CATEGORY = "image" - def run(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1): + def run(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1, worker_id=""): if not multi_job_id: return (images,) @@ -240,7 +247,7 @@ class MultiGPUCollectorNode: async def run_and_set_future(): try: - result = await self.execute(images, multi_job_id, is_worker, master_url, enabled_worker_ids, worker_batch_size) + result = await self.execute(images, multi_job_id, is_worker, master_url, enabled_worker_ids, worker_batch_size, worker_id) future.set_result(result) except Exception as e: future.set_exception(e) @@ -248,7 +255,7 @@ class MultiGPUCollectorNode: asyncio.run_coroutine_threadsafe(run_and_set_future(), main_loop) return future.result() - async def send_image_to_master(self, image_tensor, multi_job_id, master_url, image_index): + async def send_image_to_master(self, image_tensor, multi_job_id, master_url, image_index, worker_id, is_last=False): """Helper method to send a single image to the master""" img_np = (image_tensor.cpu().numpy() * 255.).astype(np.uint8) img = Image.fromarray(img_np) @@ -258,6 +265,9 @@ class MultiGPUCollectorNode: data = aiohttp.FormData() data.add_field('multi_job_id', multi_job_id) + data.add_field('worker_id', str(worker_id)) + data.add_field('image_index', str(image_index)) + data.add_field('is_last', str(is_last)) data.add_field('image', byte_io, filename=f'image_{image_index}.png', content_type='image/png') try: @@ -267,48 +277,168 @@ class MultiGPUCollectorNode: except Exception as e: print(f"[MultiGPU Worker] Failed to send image {image_index+1} to master: {e}") - async def execute(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1): + async def execute(self, images, multi_job_id="", is_worker=False, master_url="", enabled_worker_ids="[]", worker_batch_size=1, worker_id=""): if is_worker: # Worker mode: send images to master - print(f"[MultiGPU Worker] Job {multi_job_id} complete. Sending {images.shape[0]} image(s) to master at {master_url}") + print(f"[MultiGPU Worker] Job {multi_job_id} complete. Sending {images.shape[0]} image(s) to master") + + # Send all images, marking the last one + for i in range(images.shape[0]): + is_last = (i == images.shape[0] - 1) + await self.send_image_to_master(images[i], multi_job_id, master_url, i, worker_id, is_last) - tasks = [self.send_image_to_master(images[i], multi_job_id, master_url, i) for i in range(images.shape[0])] - await asyncio.gather(*tasks) return (images,) else: # Master mode: collect images from workers - num_workers = len(json.loads(enabled_worker_ids)) + enabled_workers = json.loads(enabled_worker_ids) + num_workers = len(enabled_workers) if num_workers == 0: return (images,) images_on_cpu = images.cpu() - total_images_to_collect = int(num_workers * worker_batch_size) - print(f"[MultiGPU Master] Job {multi_job_id}: Waiting for {total_images_to_collect} image(s)...") + print(f"[MultiGPU Master] Job {multi_job_id}: Collecting images from {num_workers} workers...") - all_tensors = [images_on_cpu] + # Store master images first + master_batch_size = images.shape[0] + + # Initialize storage for collected images + worker_images = {} # Dict to store images by worker_id and index async with PENDING_JOBS_LOCK: if multi_job_id not in PENDING_JOBS: PENDING_JOBS[multi_job_id] = asyncio.Queue() q = PENDING_JOBS[multi_job_id] - try: - # Collect all images with a potentially dynamic timeout - for i in range(total_images_to_collect): - worker_tensor = await asyncio.wait_for(q.get(), timeout=300.0) - all_tensors.append(worker_tensor) - print(f"[MultiGPU Master] Collected image {i+1}/{total_images_to_collect}") - except asyncio.TimeoutError: - print(f"[MultiGPU Master] Timeout waiting for worker images (got {len(all_tensors)-1}/{total_images_to_collect})") - finally: - # Clean up job queue - async with PENDING_JOBS_LOCK: - if multi_job_id in PENDING_JOBS: - del PENDING_JOBS[multi_job_id] + # Collect images until all workers report they're done + collected_count = 0 + workers_done = set() + + # Use a reasonable timeout for the first image + timeout = 60.0 + + while len(workers_done) < num_workers: + try: + result = await asyncio.wait_for(q.get(), timeout=timeout) + worker_id = result['worker_id'] + image_index = result['image_index'] + tensor = result['tensor'] + is_last = result.get('is_last', False) + + if worker_id not in worker_images: + worker_images[worker_id] = {} + worker_images[worker_id][image_index] = tensor + + collected_count += 1 + + # Once we start receiving images, use shorter timeout + timeout = 10.0 + + if is_last: + workers_done.add(worker_id) + print(f"[MultiGPU Master] Worker {worker_id} done. Collected {len(worker_images[worker_id])} images") + else: + print(f"[MultiGPU Master] Collected image {image_index + 1} from worker {worker_id}") + + except asyncio.TimeoutError: + missing_workers = set(enabled_workers) - workers_done + print(f"[MultiGPU Master] Timeout. Still waiting for workers: {list(missing_workers)}") + break + + total_collected = sum(len(imgs) for imgs in worker_images.values()) + print(f"[MultiGPU Master] Collection complete. Received {total_collected} images from {len(workers_done)} workers") + + # Clean up job queue + async with PENDING_JOBS_LOCK: + if multi_job_id in PENDING_JOBS: + del PENDING_JOBS[multi_job_id] - combined = torch.cat(all_tensors, dim=0) + # Reorder images according to seed distribution pattern + # Pattern: master img 1, master img 2, worker 1 img 1, worker 1 img 2, worker 2 img 1, worker 2 img 2, etc. + ordered_tensors = [] + + # Add master images first + for i in range(master_batch_size): + ordered_tensors.append(images_on_cpu[i:i+1]) + + # Add worker images in order + for worker_id in enabled_workers: + # Convert worker_id to string since it comes as string from form data + worker_id_str = str(worker_id) + if worker_id_str in worker_images: + # Sort by image index for each worker + for idx in sorted(worker_images[worker_id_str].keys()): + ordered_tensors.append(worker_images[worker_id_str][idx]) + + combined = torch.cat(ordered_tensors, dim=0) print(f"[MultiGPU Master] Job {multi_job_id} complete. Combined {combined.shape[0]} images total.") return (combined,) -NODE_CLASS_MAPPINGS = { "MultiGPUCollector": MultiGPUCollectorNode } -NODE_DISPLAY_NAME_MAPPINGS = { "MultiGPUCollector": "Multi-GPU Collector" } +# --- Distributor Node --- +class MultiGPUDistributor: + """ + Distributes seed values across multiple GPUs. + On master: passes through the original seed. + On workers: adds offset based on worker ID. + """ + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "seed": ("INT", { + "default": 0, + "min": -2147483648, + "max": 2147483647, + "forceInput": False # Widget by default, can be converted to input + }), + "batch_size": ("INT", { + "default": 1, + "min": 1, + "max": 4096, + "forceInput": False # Widget by default, can be converted to input + }), + }, + "hidden": { + "is_worker": ("BOOLEAN", {"default": False}), + "worker_id": ("STRING", {"default": ""}), + }, + } + + RETURN_TYPES = ("INT", "INT") + RETURN_NAMES = ("seed", "batch_size") + FUNCTION = "distribute" + CATEGORY = "utils" + + def distribute(self, seed, batch_size, is_worker=False, worker_id=""): + if not is_worker: + # Master node: pass through original values + print(f"[MultiGPU Distributor] Master: seed={seed}, batch={batch_size}") + return (seed, batch_size) + else: + # Worker node: apply offset based on worker index + # Find worker index from enabled_worker_ids + try: + # Worker IDs are passed as "worker_0", "worker_1", etc. + if worker_id.startswith("worker_"): + worker_index = int(worker_id.split("_")[1]) + else: + # Fallback: try to parse as direct index + worker_index = int(worker_id) + + offset = worker_index + 1 + new_seed = seed + offset + print(f"[MultiGPU Distributor] Worker {worker_index}: seed={seed} → {new_seed}, batch={batch_size}") + return (new_seed, batch_size) + except (ValueError, IndexError) as e: + print(f"[MultiGPU Distributor] Error parsing worker_id '{worker_id}': {e}") + # Fallback: return original seed + return (seed, batch_size) + +NODE_CLASS_MAPPINGS = { + "MultiGPUCollector": MultiGPUCollectorNode, + "MultiGPUDistributor": MultiGPUDistributor +} +NODE_DISPLAY_NAME_MAPPINGS = { + "MultiGPUCollector": "Multi-GPU Collector", + "MultiGPUDistributor": "Multi-GPU Distributor" +} From 119496d8767266ff0e7415615cb207fa6ba6779d Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 07:37:53 +1000 Subject: [PATCH 30/41] Update gpupanel.js --- web/gpupanel.js | 97 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 69 insertions(+), 28 deletions(-) diff --git a/web/gpupanel.js b/web/gpupanel.js index 29ae974..2656223 100644 --- a/web/gpupanel.js +++ b/web/gpupanel.js @@ -224,12 +224,13 @@ class MultiGPUExtension { } } + // --- Helper Methods --- + // --- Core Logic & Execution --- setupInterceptor() { api.queuePrompt = async (number, prompt) => { if (this.isEnabled && this.findNodesByClass(prompt.output, "MultiGPUCollector").length > 0) { - console.log("[MultiGPU] Intercepting prompt. Executing parallel Multi-GPU."); return await this.executeParallelMultiGPU(prompt); } return this.originalQueuePrompt(number, prompt); @@ -238,20 +239,43 @@ class MultiGPUExtension { async executeParallelMultiGPU(promptWrapper) { try { - console.log("[MultiGPU] Collector node found. Activating standard Multi-GPU execution."); + const multi_job_id = "mgpu_" + Date.now(); const enabledWorkers = this.enabledWorkers; - const batchSize = this._getBatchSizeFromPrompt(promptWrapper); + await this._prepareMultiGpuJob(multi_job_id); const jobs = []; const participants = ['master', ...enabledWorkers.map(w => w.id)]; + + // Track seed distribution for final summary + const seedDistribution = []; for (const participantId of participants) { const jobApiPrompt = this._prepareApiPromptForParticipant( promptWrapper.output, multi_job_id, participantId, - { enabled_worker_ids: enabledWorkers.map(w => w.id), worker_batch_size: batchSize } + { + enabled_worker_ids: enabledWorkers.map(w => w.id), + workflow: promptWrapper.workflow + } ); + + // Use distributor nodes for seed display + const distributorNodes = this.findNodesByClass(jobApiPrompt, "MultiGPUDistributor"); + + if (distributorNodes.length > 0) { + // Get the first distributor node's seed value + const firstDistributor = distributorNodes[0]; + const seedValue = jobApiPrompt[firstDistributor.id].inputs.seed; + + if (typeof seedValue === 'number' && seedValue >= 0) { + seedDistribution.push({ + participant: participantId === 'master' ? 'Master' : `Worker ${enabledWorkers.findIndex(w => w.id === participantId)}`, + seed: seedValue + }); + } + } + if (participantId === 'master') { jobs.push({ type: 'master', promptWrapper: { ...promptWrapper, output: jobApiPrompt } }); } else { @@ -259,8 +283,9 @@ class MultiGPUExtension { if (worker) jobs.push({ type: 'worker', worker, prompt: jobApiPrompt, workflow: promptWrapper.workflow }); } } - await this._executeJobs(jobs); - return { "prompt_id": "multi-gpu-parallel-job-dispatched" }; + + const result = await this._executeJobs(jobs); + return result; } catch (error) { console.error("[MultiGPU] Parallel execution failed:", error); alert(`[MultiGPU] Parallel execution failed: ${error.message}`); @@ -270,12 +295,19 @@ class MultiGPUExtension { async _executeJobs(jobs) { - const promises = jobs.map(job => - job.type === 'master' - ? this.originalQueuePrompt(0, job.promptWrapper) - : this._dispatchToWorker(job.worker, job.prompt, job.workflow) - ); + let masterPromptId = null; + const promises = jobs.map(job => { + if (job.type === 'master') { + return this.originalQueuePrompt(0, job.promptWrapper).then(result => { + masterPromptId = result; + return result; + }); + } else { + return this._dispatchToWorker(job.worker, job.prompt, job.workflow); + } + }); await Promise.all(promises); + return masterPromptId || { "prompt_id": "multi-gpu-job-dispatched" }; } // --- Helper Methods --- @@ -286,36 +318,43 @@ class MultiGPUExtension { .map(([nodeId, nodeData]) => ({ id: nodeId, data: nodeData })); } - findBatchNode(workflow) { - return workflow.nodes.find(node => { - const title = node.title?.trim().toUpperCase(); - const displayName = node.properties?.["Node name for S&R"]?.trim().toUpperCase(); - return title === "BATCH" || title === "BATCH SIZE" || displayName === "BATCH" || displayName === "BATCH SIZE"; - }); - } - - _getBatchSizeFromPrompt(promptWrapper) { - const batchNode = this.findBatchNode(promptWrapper.workflow); - if (!batchNode) throw new Error("MultiGPU: Could not find a 'Batch' node."); - return promptWrapper.output[String(batchNode.id)].inputs.batch_size || promptWrapper.output[String(batchNode.id)].inputs.value || 1; - } _prepareApiPromptForParticipant(baseApiPrompt, multi_job_id, participantId, options = {}) { const jobApiPrompt = JSON.parse(JSON.stringify(baseApiPrompt)); - const collectorNodes = this.findNodesByClass(jobApiPrompt, "MultiGPUCollector"); const isMaster = participantId === 'master'; - + + // Handle MultiGPU distributor nodes + const distributorNodes = this.findNodesByClass(jobApiPrompt, "MultiGPUDistributor"); + if (distributorNodes.length > 0) { + console.log(`[MultiGPU] Found ${distributorNodes.length} distributor node(s)`); + } + + for (const distributor of distributorNodes) { + const { inputs } = jobApiPrompt[distributor.id]; + inputs.is_worker = !isMaster; + if (!isMaster) { + const workerIndex = options.enabled_worker_ids.indexOf(participantId); + inputs.worker_id = `worker_${workerIndex}`; + console.log(`[MultiGPU] Set distributor ${distributor.id} for worker ${workerIndex}`); + } + } + + // Handle MultiGPU collector nodes + const collectorNodes = this.findNodesByClass(jobApiPrompt, "MultiGPUCollector"); for (const collector of collectorNodes) { const { inputs } = jobApiPrompt[collector.id]; inputs.multi_job_id = multi_job_id; inputs.is_worker = !isMaster; if (isMaster) { inputs.enabled_worker_ids = JSON.stringify(options.enabled_worker_ids || []); - inputs.worker_batch_size = options.worker_batch_size || 1; } else { inputs.master_url = window.location.origin; + // Add a unique identifier to prevent caching issues + inputs.worker_job_id = `${multi_job_id}_worker_${participantId}`; + inputs.worker_id = participantId; } } + return jobApiPrompt; } @@ -334,12 +373,14 @@ class MultiGPUExtension { async _dispatchToWorker(worker, prompt, workflow) { const workerUrl = `http://${worker.host || window.location.hostname}:${worker.port}`; - console.log(`[MultiGPU] Dispatching to worker: ${worker.name} (${workerUrl})`); + const promptToSend = { prompt, extra_data: { extra_pnginfo: { workflow } }, client_id: api.clientId }; + + try { await fetch(`${workerUrl}/prompt`, { method: 'POST', From c6a9d38eb68af38d516e7467bba4b95af0e001b7 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 08:07:54 +1000 Subject: [PATCH 31/41] Update README.md --- README.md | 86 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 66 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 8bdb7fe..ccf30fd 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ ComfyUI-Distributed extends ComfyUI with the ability to distribute batch process - **Master-Worker Architecture**: Automatic coordination between GPU instances - **Dynamic Worker Management**: Enable/disable GPUs on-the-fly through the UI - **Parallel Execution**: All enabled GPUs process batches simultaneously for maximum speed +- **Automatic Seed Distribution**: Smart seed offsetting across workers - **Web UI Integration**: Seamless control panel integrated into ComfyUI's sidebar - **Smart Launcher**: Unified launcher with multiple modes and instance detection - **VRAM Management**: Clear worker VRAM remotely through the UI @@ -26,11 +27,26 @@ ComfyUI-Distributed extends ComfyUI with the ability to distribute batch process 1. Launch multiple ComfyUI instances using the unified launcher 2. One instance acts as the master (typically on CUDA device 0) 3. Add the "Multi-GPU Collector" node to your workflow -4. The extension automatically distributes batches across available GPUs -5. Results are collected and combined by the master instance +4. Optionally add "Multi-GPU Distributor" nodes for automatic seed distribution +5. The extension automatically distributes batches across available GPUs +6. Results are collected and combined by the master instance For example, with 4 GPUs and a batch size of 2, you'll generate 8 images in parallel. +## Nodes + +### Multi-GPU Distributor +Handles automatic seed distribution for diverse results: +- Master uses the original seed value +- Workers automatically get offset seeds (seed + worker_index + 1) +- Can connect to any node that accepts seed/batch_size inputs +- Ensures each GPU generates unique images + +### Multi-GPU Collector +The core node that handles job result collection: +- Automatically detects enabled workers +- Collects and combines results in a predictable order + ## Requirements - Virtual environment with ComfyUI installed @@ -91,9 +107,9 @@ The system uses a `gpu_config.json` file for configuration. Example configuratio ```json { "master": { - "port": 8480, + "port": 8080, "cuda_device": 0, - "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" + "extra_args": "--mmap-torch-files --listen" }, "workers": [ { @@ -102,7 +118,7 @@ The system uses a `gpu_config.json` file for configuration. Example configuratio "cuda_device": 1, "port": 8180, "enabled": true, - "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" + "extra_args": "--mmap-torch-files --listen" }, { "id": 2, @@ -110,7 +126,15 @@ The system uses a `gpu_config.json` file for configuration. Example configuratio "cuda_device": 2, "port": 8280, "enabled": true, - "extra_args": "--mmap-torch-files --highvram --disable-smart-memory" + "extra_args": "--mmap-torch-files --listen" + }, + { + "id": 3, + "name": "GPU 3", + "cuda_device": 3, + "port": 8380, + "enabled": false, + "extra_args": "--mmap-torch-files --listen --reserve-vram 2" } ], "settings": {} @@ -143,24 +167,49 @@ Remote workers are fully supported! Add a `host` field to any worker configurati "host": "192.168.1.100", "port": 8190, "enabled": true, - "extra_args": "--enable-cors-header" + "extra_args": "--enable-cors-header --listen" } ``` -Ensure remote workers are started with `--enable-cors-header` and are accessible from the master instance. +Ensure remote workers: +- Are started with `--listen --enable-cors-header` +- Are accessible from the master instance +- Have the same models and custom nodes installed + ## Workflow Integration -The Multi-GPU Collector node integrates seamlessly with existing ComfyUI workflows: +### Basic Setup +1. Add "Multi-GPU Collector" node after your image generation pipeline +2. The node automatically handles distribution and collection +3. Connect subsequent nodes normally - they'll receive the combined batch -1. Connect the node after your image generation pipeline -2. The node will automatically detect batch sizes and distribute work -3. All GPUs will process their assigned batches in parallel -4. Results are collected and passed to the next node as a single batch +### Seed Distribution +This node is optional, it will help you control seeds across workers. Without it, you will generate exactly the same workflow as the master (same seed): +1. Add "Multi-GPU Seed" node(s) before your samplers +2. Connect the seed output to your sampling node(s) +3. Each GPU will automatically use different seeds: + - Master: original seed + - Worker 0: seed + 1 + - Worker 1: seed + 2 + - etc. -### Important Node Requirements +### Result Ordering +Images are returned in a predictable order: +1. All images from master (in batch order) +2. All images from worker 1 (in batch order) +3. All images from worker 2 (in batch order) +4. etc. -- **Batch Node**: Your workflow must include a node with the title "BATCH" or "BATCH SIZE" (case-insensitive). This is required for the Multi-GPU system to detect the batch size for distribution. -- The system will show an error if no batch node with the correct title is found. +## API Endpoints + +The extension provides REST API endpoints for integration: + +- `GET /multigpu/config` - Retrieve current configuration +- `POST /multigpu/config/update_worker` - Enable/disable a worker +- `POST /multigpu/config/update_setting` - Update configuration settings +- `POST /multigpu/prepare_job` - Prepare a multi-GPU job +- `POST /multigpu/clear_memory` - Clear VRAM on workers +- `POST /multigpu/job_complete` - Worker callback for job completion ## API Endpoints @@ -203,10 +252,7 @@ This project is under active development. Contributions are welcome! - Worker status (ready/in progress/offline) - Restart worker button - ~Better VRAM management for multi-GPU setups~ -- Better Seed handling -- Better Batch handling - - Batch Size Override - - UI control to dynamically adjust batch size per generation +- ~Better Seed handling~ - Handle worker failure - ~Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut~ [Commit 027629c](https://github.com/robertvoy/ComfyUI-Flux-Continuum/commit/027629c753dd3aae1ceeff5214ceb701943dd3fe) From b21485db5e76b671ccab2c1b1a402117fd7af7b1 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 08:09:15 +1000 Subject: [PATCH 32/41] Update README.md --- README.md | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/README.md b/README.md index ccf30fd..c51d8c8 100644 --- a/README.md +++ b/README.md @@ -211,17 +211,6 @@ The extension provides REST API endpoints for integration: - `POST /multigpu/clear_memory` - Clear VRAM on workers - `POST /multigpu/job_complete` - Worker callback for job completion -## API Endpoints - -The extension provides several REST API endpoints for integration: - -- `GET /multigpu/config` - Retrieve current configuration -- `POST /multigpu/config/update_worker` - Enable/disable a worker -- `POST /multigpu/config/update_setting` - Update configuration settings -- `POST /multigpu/prepare_job` - Prepare a multi-GPU job -- `POST /multigpu/clear_memory` - Clear VRAM on workers -- `POST /multigpu/job_complete` - Worker callback for job completion - ## Performance Considerations - **Parallel Execution**: All enabled GPUs process simultaneously for maximum throughput @@ -254,6 +243,7 @@ This project is under active development. Contributions are welcome! - ~Better VRAM management for multi-GPU setups~ - ~Better Seed handling~ - Handle worker failure +- Handle multiple image output nodes within 1 workflow - ~Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut~ [Commit 027629c](https://github.com/robertvoy/ComfyUI-Flux-Continuum/commit/027629c753dd3aae1ceeff5214ceb701943dd3fe) ## License From efcfab748bf232840bc5af8341e674ca12f8dd65 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 08:10:26 +1000 Subject: [PATCH 33/41] Update gpupanel.js From 9181c9ff53c7ac5c24a37800bf02285d8b64a901 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 08:10:34 +1000 Subject: [PATCH 34/41] Update multigpu.py --- multigpu.py | 28 +++++++++++----------------- 1 file changed, 11 insertions(+), 17 deletions(-) diff --git a/multigpu.py b/multigpu.py index 6abbd92..aa52112 100644 --- a/multigpu.py +++ b/multigpu.py @@ -387,14 +387,8 @@ class MultiGPUDistributor: "required": { "seed": ("INT", { "default": 0, - "min": -2147483648, - "max": 2147483647, - "forceInput": False # Widget by default, can be converted to input - }), - "batch_size": ("INT", { - "default": 1, - "min": 1, - "max": 4096, + "min": 0, + "max": 1125899906842624, "forceInput": False # Widget by default, can be converted to input }), }, @@ -404,16 +398,16 @@ class MultiGPUDistributor: }, } - RETURN_TYPES = ("INT", "INT") - RETURN_NAMES = ("seed", "batch_size") + RETURN_TYPES = ("INT",) + RETURN_NAMES = ("seed",) FUNCTION = "distribute" CATEGORY = "utils" - def distribute(self, seed, batch_size, is_worker=False, worker_id=""): + def distribute(self, seed, is_worker=False, worker_id=""): if not is_worker: # Master node: pass through original values - print(f"[MultiGPU Distributor] Master: seed={seed}, batch={batch_size}") - return (seed, batch_size) + print(f"[MultiGPU Distributor] Master: seed={seed}") + return (seed,) else: # Worker node: apply offset based on worker index # Find worker index from enabled_worker_ids @@ -427,12 +421,12 @@ class MultiGPUDistributor: offset = worker_index + 1 new_seed = seed + offset - print(f"[MultiGPU Distributor] Worker {worker_index}: seed={seed} → {new_seed}, batch={batch_size}") - return (new_seed, batch_size) + print(f"[MultiGPU Distributor] Worker {worker_index}: seed={seed} → {new_seed}") + return (new_seed,) except (ValueError, IndexError) as e: print(f"[MultiGPU Distributor] Error parsing worker_id '{worker_id}': {e}") # Fallback: return original seed - return (seed, batch_size) + return (seed,) NODE_CLASS_MAPPINGS = { "MultiGPUCollector": MultiGPUCollectorNode, @@ -440,5 +434,5 @@ NODE_CLASS_MAPPINGS = { } NODE_DISPLAY_NAME_MAPPINGS = { "MultiGPUCollector": "Multi-GPU Collector", - "MultiGPUDistributor": "Multi-GPU Distributor" + "MultiGPUDistributor": "Multi-GPU Seed" } From e8f253443386ddede7d69bb01021c00db884dea6 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 08:43:02 +1000 Subject: [PATCH 35/41] Update multigpu.py From 3e388311b359cad0a5f79f234af902df97daacd8 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 08:43:29 +1000 Subject: [PATCH 36/41] Update gpupanel.js --- web/gpupanel.js | 39 ++++++++++++--------------------------- 1 file changed, 12 insertions(+), 27 deletions(-) diff --git a/web/gpupanel.js b/web/gpupanel.js index 2656223..6d8bc58 100644 --- a/web/gpupanel.js +++ b/web/gpupanel.js @@ -239,43 +239,27 @@ class MultiGPUExtension { async executeParallelMultiGPU(promptWrapper) { try { - - const multi_job_id = "mgpu_" + Date.now(); const enabledWorkers = this.enabledWorkers; - await this._prepareMultiGpuJob(multi_job_id); + // Find all collector nodes in the workflow + const collectorNodes = this.findNodesByClass(promptWrapper.output, "MultiGPUCollector"); + + // Prepare a separate job queue on the backend for each collector instance using its unique node ID + const preparePromises = collectorNodes.map(node => this._prepareMultiGpuJob(node.id)); + await Promise.all(preparePromises); const jobs = []; const participants = ['master', ...enabledWorkers.map(w => w.id)]; - - // Track seed distribution for final summary - const seedDistribution = []; for (const participantId of participants) { const jobApiPrompt = this._prepareApiPromptForParticipant( - promptWrapper.output, multi_job_id, participantId, + promptWrapper.output, participantId, { enabled_worker_ids: enabledWorkers.map(w => w.id), workflow: promptWrapper.workflow } ); - // Use distributor nodes for seed display - const distributorNodes = this.findNodesByClass(jobApiPrompt, "MultiGPUDistributor"); - - if (distributorNodes.length > 0) { - // Get the first distributor node's seed value - const firstDistributor = distributorNodes[0]; - const seedValue = jobApiPrompt[firstDistributor.id].inputs.seed; - - if (typeof seedValue === 'number' && seedValue >= 0) { - seedDistribution.push({ - participant: participantId === 'master' ? 'Master' : `Worker ${enabledWorkers.findIndex(w => w.id === participantId)}`, - seed: seedValue - }); - } - } - if (participantId === 'master') { jobs.push({ type: 'master', promptWrapper: { ...promptWrapper, output: jobApiPrompt } }); } else { @@ -319,7 +303,7 @@ class MultiGPUExtension { } - _prepareApiPromptForParticipant(baseApiPrompt, multi_job_id, participantId, options = {}) { + _prepareApiPromptForParticipant(baseApiPrompt, participantId, options = {}) { const jobApiPrompt = JSON.parse(JSON.stringify(baseApiPrompt)); const isMaster = participantId === 'master'; @@ -343,14 +327,15 @@ class MultiGPUExtension { const collectorNodes = this.findNodesByClass(jobApiPrompt, "MultiGPUCollector"); for (const collector of collectorNodes) { const { inputs } = jobApiPrompt[collector.id]; - inputs.multi_job_id = multi_job_id; + // KEY CHANGE: Use the node's own ID as the job ID + inputs.multi_job_id = collector.id; inputs.is_worker = !isMaster; if (isMaster) { inputs.enabled_worker_ids = JSON.stringify(options.enabled_worker_ids || []); } else { inputs.master_url = window.location.origin; - // Add a unique identifier to prevent caching issues - inputs.worker_job_id = `${multi_job_id}_worker_${participantId}`; + // Add a unique identifier per-collector to prevent caching issues + inputs.worker_job_id = `${collector.id}_worker_${participantId}`; inputs.worker_id = participantId; } } From 300ccaaf0cad41c4f232e3d84ede32738980300c Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 08:43:55 +1000 Subject: [PATCH 37/41] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c51d8c8..7d33b4a 100644 --- a/README.md +++ b/README.md @@ -243,7 +243,7 @@ This project is under active development. Contributions are welcome! - ~Better VRAM management for multi-GPU setups~ - ~Better Seed handling~ - Handle worker failure -- Handle multiple image output nodes within 1 workflow +- ~Handle multiple collector nodes~ - ~Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut~ [Commit 027629c](https://github.com/robertvoy/ComfyUI-Flux-Continuum/commit/027629c753dd3aae1ceeff5214ceb701943dd3fe) ## License From 36f746f01e3685df48042678b5f86021b2e2d7c8 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 08:50:59 +1000 Subject: [PATCH 38/41] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7d33b4a..71a7bd5 100644 --- a/README.md +++ b/README.md @@ -242,7 +242,7 @@ This project is under active development. Contributions are welcome! - Restart worker button - ~Better VRAM management for multi-GPU setups~ - ~Better Seed handling~ -- Handle worker failure +- ~Handle worker failure~ - ~Handle multiple collector nodes~ - ~Compatibility with FLUX Continuum `CTRL+SHIFT+C` shortcut~ [Commit 027629c](https://github.com/robertvoy/ComfyUI-Flux-Continuum/commit/027629c753dd3aae1ceeff5214ceb701943dd3fe) From c38f2fe7a6fc72d3e7a9da200181483b14f04b1c Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 08:51:30 +1000 Subject: [PATCH 39/41] Update gpupanel.js --- web/gpupanel.js | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/web/gpupanel.js b/web/gpupanel.js index 6d8bc58..fd73478 100644 --- a/web/gpupanel.js +++ b/web/gpupanel.js @@ -239,13 +239,17 @@ class MultiGPUExtension { async executeParallelMultiGPU(promptWrapper) { try { + const executionPrefix = "exec_" + Date.now(); // Unique ID for this specific execution const enabledWorkers = this.enabledWorkers; // Find all collector nodes in the workflow const collectorNodes = this.findNodesByClass(promptWrapper.output, "MultiGPUCollector"); - // Prepare a separate job queue on the backend for each collector instance using its unique node ID - const preparePromises = collectorNodes.map(node => this._prepareMultiGpuJob(node.id)); + // Map original node IDs to truly unique job IDs for this specific run + const job_id_map = new Map(collectorNodes.map(node => [node.id, `${executionPrefix}_${node.id}`])); + + // Prepare a separate job queue on the backend for each unique job ID + const preparePromises = Array.from(job_id_map.values()).map(uniqueId => this._prepareMultiGpuJob(uniqueId)); await Promise.all(preparePromises); const jobs = []; @@ -256,7 +260,8 @@ class MultiGPUExtension { promptWrapper.output, participantId, { enabled_worker_ids: enabledWorkers.map(w => w.id), - workflow: promptWrapper.workflow + workflow: promptWrapper.workflow, + job_id_map: job_id_map // Pass the map of unique IDs } ); @@ -327,15 +332,19 @@ class MultiGPUExtension { const collectorNodes = this.findNodesByClass(jobApiPrompt, "MultiGPUCollector"); for (const collector of collectorNodes) { const { inputs } = jobApiPrompt[collector.id]; - // KEY CHANGE: Use the node's own ID as the job ID - inputs.multi_job_id = collector.id; + + // Get the unique job ID from the map created for this execution + const uniqueJobId = options.job_id_map ? options.job_id_map.get(collector.id) : collector.id; + + // Use the truly unique ID for this execution + inputs.multi_job_id = uniqueJobId; inputs.is_worker = !isMaster; if (isMaster) { inputs.enabled_worker_ids = JSON.stringify(options.enabled_worker_ids || []); } else { inputs.master_url = window.location.origin; - // Add a unique identifier per-collector to prevent caching issues - inputs.worker_job_id = `${collector.id}_worker_${participantId}`; + // Also make the worker_job_id unique to prevent potential caching issues + inputs.worker_job_id = `${uniqueJobId}_worker_${participantId}`; inputs.worker_id = participantId; } } From 474f85eb20ad86688b2c397c6858d970f4234392 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 10:46:06 +1000 Subject: [PATCH 40/41] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 71a7bd5..3ce1585 100644 --- a/README.md +++ b/README.md @@ -240,6 +240,7 @@ This project is under active development. Contributions are welcome! - ~Support for remote workers over network~ - Worker status (ready/in progress/offline) - Restart worker button +- Improve worker timeout logic - ~Better VRAM management for multi-GPU setups~ - ~Better Seed handling~ - ~Handle worker failure~ From 08076490667304aaaef7ae2a342634b8dcdb4c92 Mon Sep 17 00:00:00 2001 From: Robert Wojciechowski Date: Mon, 30 Jun 2025 11:54:40 +1000 Subject: [PATCH 41/41] Update multigpu.py --- multigpu.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/multigpu.py b/multigpu.py index aa52112..bfeedae 100644 --- a/multigpu.py +++ b/multigpu.py @@ -314,7 +314,7 @@ class MultiGPUCollectorNode: workers_done = set() # Use a reasonable timeout for the first image - timeout = 60.0 + timeout = 10.0 while len(workers_done) < num_workers: try: