Update worker-setup-guides.md

README.md

@@ -1,5 +1,5 @@
 <div align="center">
-<img width="320" src="https://github.com/user-attachments/assets/537d38cc-2e34-4527-bca7-0d79f4120cce" />
+<img width="320" src="https://github.com/user-attachments/assets/533bb98d-0c4a-499f-9bca-5c937e361087" />
 <br><br>
 <a href="https://www.youtube.com/watch?v=p6eE3IlAbOs"><img src="https://img.shields.io/badge/Video_Tutorial-grey?style=flat&logo=youtube&logoColor=white" alt="Video Tutorial"></a>
 <a href="/docs/worker-setup-guides.md"><img src="https://img.shields.io/badge/Setup_Guides-grey?style=flat&logo=gitbook&logoColor=white" alt="Setup Guides"></a>
@@ -9,7 +9,7 @@
 <br><br>
 </div>
 
-> **A powerful extension for ComfyUI that enables distributed and parallel processing across multiple GPUs and machines. Generate more images and videos and accelerate your upscaling workflows by leveraging all available GPU resources in your network.**
+> **A powerful extension for ComfyUI that enables distributed and parallel processing across multiple GPUs and machines. Generate more images and videos and accelerate your upscaling workflows by leveraging all available GPU resources in your network and cloud.**
 
 ![Clipboard Image (7)](https://github.com/user-attachments/assets/66aaadef-f195-48a1-a368-17dd0dae477d)
 
@@ -18,7 +18,7 @@
 ## Key Features
 
 #### Parallel Workflow Processing
-- Run workflows on multiple GPUs simultaneously with varied seeds
+- Run your workflow on multiple GPUs simultaneously with varied seeds, collect results on the master
 - Scale output with more workers
 - Supports images and videos
 
@@ -34,11 +34,26 @@
 
 ---
 
+## Worker Types
+
+<img width="200" align="right" alt="ComfyUI_temp_khvcc_00034_@0 25x" src="https://github.com/user-attachments/assets/651e4912-7c23-4e32-bd88-250f5175e129" />
+
+ComfyUI Distributed supports three types of workers:
+
+- **Local Workers** - Additional GPUs on the same machine (auto-configured on first launch)
+- **Remote Workers** - GPUs on other computers in your network
+- **Cloud Workers** - GPUs hosted on a cloud service like Runpod, accessible via secure tunnels
+
+> For detailed setup instructions, see the [setup guide](/docs/worker-setup-guides.md)
+
+---
+
 ## Requirements
 
 - ComfyUI
 > Note: Desktop app not currently supported
-- Multiple GPUs
+- Multiple NVIDIA GPUs
+> No additional GPUs? Use [Cloud Workers](https://github.com/robertvoy/ComfyUI-Distributed/blob/main/docs/worker-setup-guides.md#cloud-workers)
 - That's it
 
 ---
@@ -50,9 +65,10 @@
    git clone https://github.com/robertvoy/ComfyUI-Distributed.git
    ```
    
-2. **Restart ComfyUI** - If you'll be using remote/cloud workers, add `--enable-cors-header` to your launch arguments on the master
+2. **Restart ComfyUI**
+   - If you'll be using remote/cloud workers, add `--enable-cors-header` to your launch arguments on the master
 
-3. Read the [setup guide](https://github.com/robertvoy/ComfyUI-Distributed/blob/feature/cloud-workers/docs/worker-setup-guides.md) for adding workers
+3. Read the [setup guide](/docs/worker-setup-guides.md) for adding workers
 
 ---
 
@@ -88,7 +104,7 @@ Generate multiple videos in the time it takes to generate one. Each worker uses
 8. Run the workflow!
 
 ### Distributed Upscaling
-Accelerate the process by distributing tiles across multiple workers, with speed scaling as you add more GPUs.
+Accelerate Ultimate SD Upscaler by distributing tiles across multiple workers, with speed scaling as you add more GPUs.
 
 ![Clipboard Image (3)](https://github.com/user-attachments/assets/ffb57a0d-7b75-4497-96d2-875d60865a1a)
 
@@ -98,6 +114,7 @@ Accelerate the process by distributing tiles across multiple workers, with speed
 2. Upscale with ESRGAN or similar
 3. Connect to **Ultimate SD Upscale Distributed**
 4. Configure tile settings
+> If your GPUs are similar, set `static_distribution` to true; otherwise, false
 5. Enable workers for faster processing
 
 ---
@@ -143,4 +160,7 @@ This software is provided "as is" without any warranties, express or implied, in
 ---
 
 ## Support the Project
+
+<img width="200"  src="https://github.com/user-attachments/assets/84291921-c44e-4556-94f2-a3b16500f4f9" />
+
 If my custom nodes have added value to your workflow, consider fueling future development with a coffee! Your support helps keep this project thriving. Buy me a coffee at: https://buymeacoffee.com/robertvoy

distributed.py

@@ -1740,6 +1740,9 @@ async def job_complete_endpoint(request):
             image_data_list.sort(key=lambda x: x[0])
             tensors = [tensor for _, tensor in image_data_list]
             
+            # Keep the indices for later use
+            indices = [idx for idx, _ in image_data_list]
+            
             # Validate final order
             if metadata:
                 debug_log(f"Reordered {len(tensors)} images based on metadata indices")
@@ -1768,10 +1771,11 @@ async def job_complete_endpoint(request):
             debug_log(f"Current pending jobs: {list(prompt_server.distributed_pending_jobs.keys())}")
             if multi_job_id in prompt_server.distributed_pending_jobs:
                 if batch_size > 0:
-                    # Put batch as single item
+                    # Put batch as single item with indices
                     await prompt_server.distributed_pending_jobs[multi_job_id].put({
                         'worker_id': worker_id,
                         'tensors': tensors,
+                        'indices': indices,
                         'is_last': is_last
                     })
                     debug_log(f"Received batch result for job {multi_job_id} from worker {worker_id}, size={len(tensors)}")
@@ -1934,6 +1938,7 @@ class DistributedCollectorNode:
                     
                     # Check if batch mode
                     tensors = result.get('tensors', [])
+                    indices = result.get('indices', [])  # Get the indices
                     if tensors:
                         # Batch mode
                         debug_log(f"Master - Got batch from worker {worker_id}, size={len(tensors)}, is_last={is_last}")
@@ -1941,9 +1946,15 @@ class DistributedCollectorNode:
                         if worker_id not in worker_images:
                             worker_images[worker_id] = {}
                         
-                        # Add all tensors with sequential indices
-                        for idx, tensor in enumerate(tensors):
-                            worker_images[worker_id][idx] = tensor
+                        # Use actual indices if available, otherwise fall back to sequential
+                        if indices:
+                            for i, tensor in enumerate(tensors):
+                                actual_idx = indices[i]
+                                worker_images[worker_id][actual_idx] = tensor
+                        else:
+                            # Fallback for backward compatibility
+                            for idx, tensor in enumerate(tensors):
+                                worker_images[worker_id][idx] = tensor
                             
                         
                         collected_count += len(tensors)

distributed_upscale.py

@@ -1076,7 +1076,7 @@ class UltimateSDUpscaleDistributed:
         progress_interval = 5.0
         last_heartbeat_check = time.time()
         stall_count = 0
-        max_stall_count = 10  # Allow 10 iterations with no progress before returning
+        max_stall_count = 120  # Allow 120 iterations with no progress before returning
         last_completed_count = 0
         
         while True:

docs/model-download-script.md

@@ -1,15 +1,14 @@
 ## Automating ComfyUI Model Downloads
+> This guide will walk you through creating a shell script to automatically download the necessary models for your ComfyUI workflow, leveraging an advanced Large Language Model (LLM).
 
-This guide will walk you through creating a shell script to automatically download the necessary models for your ComfyUI workflow, leveraging an advanced Large Language Model (LLM).
+1. In ComfyUI (on your local machine), export your workflow as an API workflow
+2. Copy the below prompt and upload the API workflow to an LLM **that has access to the internet**
+
+<details>
+<summary><strong>📋 Click to expand the full prompt</strong></summary>
 
-1. Launched the Pod with these Environment Variables:
-  - CIVITAI_API_TOKEN ([get your token here](https://civitai.com/user/account))
-  - HF_API_TOKEN ([get your token here](https://huggingface.co/settings/tokens))
-2. In ComfyUI, export your workflow as an API workflow
-3. Copy the below prompt and upload the API workflow to a LLM **that has access to the internet**
 ```
 Create a sh script that will download the models from this workflow into the correct folders. For reference, these are the paths:
-
 base_path: /workspace/ComfyUI
 checkpoints: models/checkpoints/
 clip: models/clip/
@@ -25,32 +24,30 @@ text_encoders: models/text_encoders/
 unet: models/unet/
 upscale_models: models/upscale_models/
 vae: models/vae/
-
 ---
-
 Important:
 Make sure you find the correct URLs for the models online.
 Use comfy cli to download the models: `comfy model download --url <URL> [--relative-path <PATH>] [--set-civitai-api-token <TOKEN>] [--set-hf-api-token <TOKEN>]`
 Make sure you add `--set-civitai-api-token $CIVITAI_API_TOKEN` for CivitAI download and `--set-hf-api-token $HF_API_TOKEN` for Hugging Face downloads.
-
 ---
-
 Example:
-
 #!/bin/bash
-
 # Download from CivitAI
 comfy model download --url https://civitai.com/api/download/models/1759168 --relative-path /workspace/ComfyUI/models/checkpoints --set-civitai-api-token $CIVITAI_API_TOKEN
-
 # Download model from Hugging Face
 comfy model download --url https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/flux1-dev.safetensors --relative-path /workspace/ComfyUI/models/unet --set-hf-api-token $HF_API_TOKEN
-
 # If a model in the workflow was in a subfolder
 comfy model download --url https://civitai.com/api/download/models/1759168 --relative-path /workspace/ComfyUI/models/checkpoints/SDXL --set-civitai-api-token $CIVITAI_API_TOKEN
-
 ```
-4. Review the LLMs output to make sure all download links are correct and save it as a .sh file, for example `download_models.sh`
-5. Upload it onto your Runpod instance, into `/workspace`
+
+</details>
+
+3. Review the LLMs output to make sure all download links are correct and save it as a .sh file, for example `download_models.sh`
+4. Launch the [ComfyUI Distributed Pod](https://console.runpod.io/deploy?template=m21ynvo8yo&ref=ak218p52) with these Environment Variables:
+   - `CIVITAI_API_TOKEN`: [get your token here](https://civitai.com/user/account)
+   - `HF_API_TOKEN`: [get your token here](https://huggingface.co/settings/tokens)
+5. Upload the .sh file to your Runpod instance, into `/workspace`
 6. Then run these commands:
    - `chmod 755 /workspace/download_models.sh`
    - `/workspace/download_models.sh`
+7. Confirm each model name (sometimes you might need to rename them to match the name on your local machine)

docs/worker-setup-guides.md

@@ -1,11 +1,11 @@
-## ComfyUI Distributed Worker Setup Guide
-
-ComfyUI-Distributed allows you to distribute your workflows across multiple GPUs, whether they're in the same machine, on your local network, or in the cloud.
+## Worker Setup Guide
 
 **Master**: The main ComfyUI instance that coordinates and distributes work. This is where you load workflows, manage the queue, and view results.
 
 **Worker**: A ComfyUI instance that receives and processes tasks from the master. Workers handle just the GPU computation and send results back to the master. You can have multiple workers connected to a single master, each utilizing their own GPU.
 
+<img width="600" src="https://github.com/user-attachments/assets/609c42aa-8a1c-4a3f-939e-f3552fa1d54f" />
+
 ### Types of Workers
 
 - **Local workers**: Additional GPUs on the same machine as the master
@@ -13,8 +13,14 @@ ComfyUI-Distributed allows you to distribute your workflows across multiple GPUs
 - **Cloud workers**: GPUs hosted on cloud services like Runpod
 
 ## Local workers
+
+<img align="right" width="200" src="https://github.com/user-attachments/assets/651e4912-7c23-4e32-bd88-250f5175e129" />
+
 > These are added automatically on first launch, but you can add them manually if you need to.
 
+
+📺 [Watch Tutorial](https://youtu.be/p6eE3IlAbOs?si=K7Km0_flmPHwRQwz&t=43)
+
 1. **Open** the Distributed GPU panel.
 2. **Click** "Add Worker" in the UI.
 3. **Configure** your local worker:
@@ -25,8 +31,14 @@ ComfyUI-Distributed allows you to distribute your workflows across multiple GPUs
 4. **Save** and  launch the local worker.
 
 ## Remote workers
+
+<img align="right" width="200"  src="https://github.com/user-attachments/assets/84291921-c44e-4556-94f2-a3b16500f4f9" />
+
+
 > ComfyUI instances running on completely different computers on your network. These allow you to harness GPU power from other machines. Remote workers must be manually started on their respective computers and are connected via IP address.
 
+📺 [Watch Tutorial](https://youtu.be/p6eE3IlAbOs?si=Oxj3EzPyf4jKDvfG&t=140)
+
 **On the Remote Worker Machine:**
 1. **Launch** ComfyUI with the `--listen --enable-cors-header` arguments. ⚠️ **Required!**
    - This ComfyUI instance will serve as a worker for your main master.
@@ -38,47 +50,66 @@ ComfyUI-Distributed allows you to distribute your workflows across multiple GPUs
 3. **Open** the ComfyUI port (e.g., 8188) and any additional worker ports (e.g., 8189, 8190) in the firewall.
   
 **On the Main Machine:**
-1. **Open** the Distributed GPU panel (sidebar on the left).
-2. **Click** "Add Worker."
-3. **Choose** "Remote".
-4. **Configure** your remote worker:
+1. **Launch** ComfyUI with `--enable-cors-header` launch argument.
+2. **Open** the Distributed GPU panel (sidebar on the left).
+3. **Click** "Add Worker."
+4. **Choose** "Remote".
+5. **Configure** your remote worker:
    - **Name**: A descriptive name for the worker (e.g., "Server Rack GPU 0")
    - **Host**: The remote worker's IP address.
    - **Port**: The port number used when launching ComfyUI on the remote master/worker (e.g., 8188).
-5. **Save** the remote worker configuration.
+6. **Save** the remote worker configuration.
   
 ## Cloud workers
-> ComfyUI instances running on a cloud service like [Runpod](https://get.runpod.io/0bw29uf3ug0p). 
+
+<img align="right" width="200"  src="https://github.com/user-attachments/assets/a053f3ae-22f0-4e1c-8f2e-f26a1f660adf" />
+
+> ComfyUI instances running on a cloud service like Runpod. 
 
 ### Deploy Cloud Worker on Runpod
 
+📺 [Watch Tutorial](https://www.youtube.com/watch?v=wxKKWMQhYTk)
+
 **On Runpod:**
+> If using your own template, make sure you launch ComfyUI with the `--enable-cors-header` argument and you `git clone ComfyUI-Distributed` into custom_nodes. ⚠️ **Required!**
+
 1. Register a [Runpod](https://get.runpod.io/0bw29uf3ug0p) account.
-2. On Runpod, go to Storage > New Network Volume and create a volume that will store the models you need. Start with 40 GB, you can always add more later.
-3. Now go to Pods and find a suitable GPU for your workflows. 
-4. Choose the [ComfyUI Distributed Pod](https://console.runpod.io/deploy?template=m21ynvo8yo&ref=ak218p52) template and make sure your network drive is mounted.
-> To use the ComfyUI Distributed Pod template, you will need to filter instances by CUDA 12.8.
-6. Launch your pod.
-7. Access your pod using JupyterLabs.
-8. Download models into /workspaces/ComfyUI/models/ (these will remain on your network drive even after you terminate the pod).
-> You can use [this guide](model-download-script.md) to make this process easy for you. It will generate a shell script that will automatically download the models you need for a given workflow.
-8. If using your own template, make sure you launch ComfyUI with the `--enable-cors-header` argument and you git clone ComfyUI-Distributed into custom_nodes. ⚠️ **Required!**
-9. Download any additional custom nodes using the ComfyUI Manager.
+2. On Runpod, go to Storage > New Network Volume and create a volume that will store the models you need. Start with 40 GB, you can always add more later. Learn more [about Network Volumes](https://docs.runpod.io/pods/storage/create-network-volumes).
+3. Use the [ComfyUI Distributed Pod](https://console.runpod.io/deploy?template=m21ynvo8yo&ref=ak218p52) template.
+4. Make sure your Network Volume is mounted and choose a suitable GPU.
+> ⚠️ To use the ComfyUI Distributed Pod template, you will need to filter instances by CUDA 12.8 (add filter in Additional Filters).
+6. Press Edit Template to configure the pod's Environment Variables:
+	- CIVITAI_API_TOKEN: [get your token here](https://civitai.com/user/account)
+	- HF_API_TOKEN: [get your token here](https://huggingface.co/settings/tokens)
+	- SAGE_ATTENTION: optional optimisation (set to true/false)
+5. Deploy your pod.
+6. Connect to your pod using JupyterLabs. This gives us access to the pod's file system.
+7. Download models into /workspaces/ComfyUI/models/ (these will remain on your network drive even after you terminate the pod). Example commands below:
+```
+# Download from CivitAI
+comfy model download --url https://civitai.com/api/download/models/1759168 --relative-path /workspace/ComfyUI/models/checkpoints --set-civitai-api-token $CIVITAI_API_TOKEN
+# Download model from Hugging Face
+comfy model download --url https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/flux1-dev.safetensors --relative-path /workspace/ComfyUI/models/unet --set-hf-api-token $HF_API_TOKEN
+```
+> ℹ️ Use [this guide](model-download-script.md) to make this process easy. It will generate a shell script that automatically downloads the models for a given workflow.
+9. Access ComfyUI through the Runpod URL.
+10. Download any additional custom nodes you need using the ComfyUI Manager.
 
 **On the Main Machine:**
-1. Launch a Cloudflare tunnel
+1. **Launch** a Cloudflare tunnel.
    - Download from here: [https://github.com/cloudflare/cloudflared/releases](https://github.com/cloudflare/cloudflared/releases)
 	- Then run, for example: `cloudflared-windows-amd64.exe tunnel --url http://localhost:8188`
-> Cloudflare tunnels create secure connections without exposing ports directly to the internet
-2. Copy the Cloudflare address
-3. **Open** the Distributed GPU panel (sidebar on the left).
-4. **Edit** the Master's host address and replace it with the Cloudflare address.
-   - **Click** "Add Worker."
-   - **Choose** "Cloud".
-   - **Configure** your cloud worker:
-     - **Host**: The Runpod address
-     - **Port**: 443
-   - **Save** the remote worker configuration.
+> ℹ️ Cloudflare tunnels create secure connections without exposing ports directly to the internet and are required for Cloud Workers.
+2. **Copy** the Cloudflare address
+3. **Launch** ComfyUI with `--enable-cors-header` launch argument.
+4. **Open** the Distributed GPU panel (sidebar on the left).
+5. **Edit** the Master's settings to change the host address to the Cloudflare address.
+6. **Click** "Add Worker."
+7. **Choose** "Cloud".
+8. **Configure** your cloud worker:
+	- **Host**: The ComfyUI Runpod address. For example: `wcegfo9tbbml9l-8188.proxy.runpod.net`
+	- **Port**: 443
+9. **Save** the remote worker configuration.
 
 ---
 
@@ -89,7 +120,7 @@ ComfyUI-Distributed allows you to distribute your workflows across multiple GPUs
    - If your cloud platform doesn't provide a secure connection, use Cloudflare to create a tunnel for the worker. Each GPU needs their own tunnel for their respective port.
 	- For example: `./cloudflared tunnel --url http://localhost:8188`
 1. **Launch** ComfyUI with the `--listen --enable-cors-header` arguments. ⚠️ **Required!**
-2. **Add** workers in the UI panel if the remote machine has more than one GPU.
+2. **Add** workers in the UI panel if the cloud machine has more than one GPU.
    - Make sure that they also have `--listen` set in `Extra Args`.
    - Then launch them.
   
@@ -98,11 +129,12 @@ ComfyUI-Distributed allows you to distribute your workflows across multiple GPUs
    - Download from here: [https://github.com/cloudflare/cloudflared/releases](https://github.com/cloudflare/cloudflared/releases)
    - Then run, for example: `cloudflared-windows-amd64.exe tunnel --url http://localhost:8188`
 2. **Copy** the Cloudflare address
-3. **Open** the Distributed GPU panel (sidebar on the left).
-4. **Edit** the Master's host address and replace it with the Cloudflare address.
-5. **Click** "Add Worker."
-6. **Choose** "Cloud".
-7. **Configure** your cloud worker:
+3. **Launch** ComfyUI with `--enable-cors-header` launch argument.
+4. **Open** the Distributed GPU panel (sidebar on the left).
+5. **Edit** the Master's host address and replace it with the Cloudflare address.
+6. **Click** "Add Worker."
+7. **Choose** "Cloud".
+8. **Configure** your cloud worker:
    - **Host**: The remote worker's IP address/domain
    - **Port**: 443
-8. **Save** the remote worker configuration.
+9. **Save** the remote worker configuration.

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "ComfyUI-Distributed"
 description = "ComfyUI extension that enables multi-GPU processing locally and remotely "
-version = "1.0.1"
+version = "1.0.5"
 license = {file = "LICENSE"}
 dependencies  = []
 

web/ui.js

@@ -697,7 +697,7 @@ export class DistributedUI {
         // Create powered by Runpod text (initially hidden)
         const runpodText = document.createElement("a");
         runpodText.id = `runpod-text-${worker.id}`;
-        runpodText.href = "https://get.runpod.io/0bw29uf3ug0p";
+        runpodText.href = "https://github.com/robertvoy/ComfyUI-Distributed/blob/main/docs/worker-setup-guides.md#cloud-workers";
         runpodText.target = "_blank";
         runpodText.textContent = "Deploy Cloud Worker with Runpod";
         runpodText.style.cssText = "font-size: 12px; color: #4a90e2; text-decoration: none; margin-top: 4px; display: none; cursor: pointer;";
@@ -1233,4 +1233,4 @@ export class DistributedUI {
 
         return card;
     }
-}
+}