nodes/pollockjj-ComfyUI-MultiGPU
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Bump version to 1.1.0 and update README to reflect CPU offloading capabilities; rename example JSON for clarity

Browse Source
This commit is contained in:
John Pollock
2024-12-27 04:54:21 -06:00
parent d099ecd497
commit d13f821547
4 changed files with 722 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+21 -16
View File
@@ -1,10 +1,10 @@
# ComfyUI-MultiGPU
## Experimental nodes for using multiple GPUs in a single ComfyUI workflow
## Experimental nodes for using multiple GPUs and/or offload to the CPU in a single ComfyUI workflow
This extension adds device selection capabilities to model loading nodes in ComfyUI. It monkey patches the memory management of ComfyUI in a hacky way and is neither a comprehensive solution is nor is it well-tested on any edge-case VRAM solutions. **Use at your own risk.**
This extension adds device selection capabilities to model loading nodes in ComfyUI. It monkey patches the memory management of ComfyUI in a hacky way and is neither a comprehensive solution is nor is it well-tested on any edge-case CUDA/CPU solutions. **Use at your own risk.**
*Note: This does not add parallelism. The workflow steps are still executed sequentially just with model components loaded on different GPUs where allowed. Any potential speedup comes from not having to constantly load and unload models from VRAM.*
*Note: This does not add parallelism. The workflow steps are still executed sequentially just with model components loaded on different GPUs or offloaded to the CPU where allowed. Any potential speedup comes from not having to constantly load and unload models from VRAM.*
## Installation
@@ -48,20 +48,15 @@ All MultiGPU nodes can be found in the "multigpu" category in the node menu.
All workflows have been tested on a 2x 3090 setup.
### Loading two SDXL checkpoints on different GPUs
- [examples/sdxl_2gpu.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/sdxl_2gpu.json)
This workflow loads two SDXL checkpoints on two different GPUs. The first checkpoint is loaded on GPU 0, and the second checkpoint is loaded on GPU 1.
### Split FLUX.1-dev across two GPUs
- [examples/flux1dev_2gpu.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/flux1dev_2gpu.json)
This workflow loads a FLUX.1-dev model and splits its components across two GPUs. The UNet model is loaded on GPU 1 while the text encoders and VAE are loaded on GPU 0.
### FLUX.1-dev and SDXL in the same workflow
### Split FLUX.1-dev between the CPU and a single GPU
- [examples/flux1dev_sdxl_2gpu.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/flux1dev_sdxl_2gpu.json)
This workflow loads a FLUX.1-dev model and an SDXL model in the same workflow. The FLUX.1-dev model has its UNet on GPU 1 with VAE and text encoders on GPU 0, while the SDXL model uses separate allocations on GPU 0.
- [examples/flux1dev_cpugpu.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/flux1dev_cpu_1gpu_GGUF.json)
This workflow demonstrates splitting a quantized, GGUF FLUX.1-dev model between a CPU and a single GPU. The UNet model is loaded on the GPU, while the VAE and text encoders are handled by the CPU.
### Using GGUF quantized models across GPUs
@@ -73,20 +68,30 @@ This workflow demonstrates using quantized GGUF models split across multiple GPU
- [examples/hunyuan_2gpu_GGUF.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/hunyuan_2gpu_GGUF.json)
This workflow demonstrates using quantized GGUF models for Hunyan Video split across multiple GPUs with the FastVideo LoRA. In this instance, the video model is on GPU 0 whereas the VAE and text encoders are on GPU 1.
### Loading two SDXL checkpoints on different GPUs
- [examples/sdxl_2gpu.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/sdxl_2gpu.json)
This workflow loads two SDXL checkpoints on two different GPUs. The first checkpoint is loaded on GPU 0, and the second checkpoint is loaded on GPU 1.
### FLUX.1-dev and SDXL in the same workflow
- [examples/flux1dev_sdxl_2gpu.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/flux1dev_sdxl_2gpu.json)
This workflow loads a FLUX.1-dev model and an SDXL model in the same workflow. The FLUX.1-dev model has its UNet on GPU 1 with VAE and text encoders on GPU 0, while the SDXL model uses separate allocations on GPU 0.
### EXPERIMENTAL - USE AT YOUR OWN RISK
These workflows combine multiple features and non-core loaders types and, as provided, require significant VRAM to execute. They are provided as examples of what's possible but may require adjustment for your specific setup.
#### Image to Prompt to Image to Video Generation Pipeline
- [examples/florence2_flux1dev_ltxv_2gpu_GGUF.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/florence2_flux1dev_ltxv_2gpu_GGUF.json)
- [examples/florence2_flux1dev_ltxv_cpu_2gpu_GGUF.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/florence2_flux1dev_ltxv_cpu_2gpu_GGUF.json)
This workflow creates an img2txt2img2vid video generation pipeline by:
1. Loading the Florence2 model and providing a starting image for analysis and generating a text response
2. Loading FLUX.1 Dev UNET, Clip, and VAE and generate an image using the Florence2 text as a prompt
3. Loading the LTX Video Taking the resulting FLUX.1 image and provide it as the starting image for an LTX Video image-to-video generation
1. Loading the Florence2 model on the CPU and providing a starting image for analysis and generating a text response
2. Loading FLUX.1 Dev UNET on GPU 1, with CLIP and VAE on the CPU and generating an image using the Florence2 text as a prompt
3. Loading the LTX Video UNet and VAE on GPU 2, and LTX-encoded CLIP on the CPU, and taking the resulting FLUX.1 image and provide it as the starting image for an LTX Video image-to-video generation
4. Generate a 5 second video based on the provided image
All models are distributed across available GPUs with no model reloading on dual 3090s
All models are distributed across available the available CPU and GPUs with no model reloading on dual 3090s
#### LLM-Guided Video Generation
examples/florence2_flux1dev_ltxv_2gpu_GGUF.json → examples/florence2_flux1dev_ltxv_cpu_2gpu_GGUF.json
+4 -4
View File
@@ -258,7 +258,7 @@
"microsoft/Florence-2-large",
"fp16",
"sdpa",
"cuda:0"
"cpu"
],
"color": "#ff9414",
"bgcolor": "#ff8000"
@@ -331,7 +331,7 @@
"t5xxl_fp16.safetensors",
"clip_l.safetensors",
"flux",
"cuda:0"
"cpu"
]
},
{
@@ -534,7 +534,7 @@
},
"widgets_values": [
"ae.safetensors",
"cuda:0"
"cpu"
]
},
{
@@ -1509,7 +1509,7 @@
"widgets_values": [
"t5-v1_1-xxl-encoder-Q8_0.gguf",
"ltxv",
"cuda:0"
"cpu"
]
},
{
examples/flux1dev_cpu_1gpu_GGUF.json
+696
View File
@@ -0,0 +1,696 @@
{
"last_node_id": 34,
"last_link_id": 51,
"nodes": [
{
"id": 22,
"type": "BasicGuider",
"pos": [
430,
260
],
"size": [
310,
50
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 50,
"slot_index": 0
},
{
"name": "conditioning",
"type": "CONDITIONING",
"link": 40,
"slot_index": 1
}
],
"outputs": [
{
"name": "GUIDER",
"type": "GUIDER",
"links": [
30
],
"slot_index": 0,
"shape": 3
}
],
"properties": {
"Node name for S&R": "BasicGuider"
},
"widgets_values": []
},
{
"id": 16,
"type": "KSamplerSelect",
"pos": [
430,
350
],
"size": [
315,
58
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "SAMPLER",
"type": "SAMPLER",
"links": [
19
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "KSamplerSelect"
},
"widgets_values": [
"euler"
]
},
{
"id": 13,
"type": "SamplerCustomAdvanced",
"pos": [
801,
125
],
"size": [
398.79998779296875,
326
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "noise",
"type": "NOISE",
"link": 37,
"slot_index": 0
},
{
"name": "guider",
"type": "GUIDER",
"link": 30,
"slot_index": 1
},
{
"name": "sampler",
"type": "SAMPLER",
"link": 19,
"slot_index": 2
},
{
"name": "sigmas",
"type": "SIGMAS",
"link": 20,
"slot_index": 3
},
{
"name": "latent_image",
"type": "LATENT",
"link": 44,
"slot_index": 4
}
],
"outputs": [
{
"name": "output",
"type": "LATENT",
"links": [
24
],
"slot_index": 0,
"shape": 3
},
{
"name": "denoised_output",
"type": "LATENT",
"links": null,
"shape": 3
}
],
"properties": {
"Node name for S&R": "SamplerCustomAdvanced"
},
"widgets_values": []
},
{
"id": 8,
"type": "VAEDecode",
"pos": [
1296,
136
],
"size": [
210,
46
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 24
},
{
"name": "vae",
"type": "VAE",
"link": 48
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
9
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecode"
},
"widgets_values": []
},
{
"id": 25,
"type": "RandomNoise",
"pos": [
433,
136
],
"size": [
315,
82
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "NOISE",
"type": "NOISE",
"links": [
37
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "RandomNoise"
},
"widgets_values": [
469973742106324,
"randomize"
]
},
{
"id": 9,
"type": "SaveImage",
"pos": [
1390.503662109375,
262.9825744628906
],
"size": [
610.9039306640625,
935.4635620117188
],
"flags": {},
"order": 13,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 9
}
],
"outputs": [],
"properties": {
"Node name for S&R": "SaveImage"
},
"widgets_values": [
"ComfyUI"
]
},
{
"id": 32,
"type": "DualCLIPLoaderMultiGPU",
"pos": [
-145,
230
],
"size": [
456.3470458984375,
130
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"links": [
47
],
"slot_index": 0,
"shape": 3
}
],
"properties": {
"Node name for S&R": "DualCLIPLoaderMultiGPU"
},
"widgets_values": [
"t5xxl_fp16.safetensors",
"clip_l.safetensors",
"flux",
"cpu"
]
},
{
"id": 33,
"type": "VAELoaderMultiGPU",
"pos": [
-141,
406
],
"size": [
450.23980712890625,
82
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
48,
49
],
"slot_index": 0,
"shape": 3
}
],
"properties": {
"Node name for S&R": "VAELoaderMultiGPU"
},
"widgets_values": [
"ae.safetensors",
"cpu"
]
},
{
"id": 26,
"type": "LoadImage",
"pos": [
832.099609375,
276.4795837402344
],
"size": [
557.5253295898438,
885.68408203125
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
41
],
"slot_index": 0,
"shape": 3
},
{
"name": "MASK",
"type": "MASK",
"links": null,
"shape": 3
}
],
"properties": {
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"43877497.jpeg",
"image"
]
},
{
"id": 6,
"type": "CLIPTextEncode",
"pos": [
428,
456
],
"size": [
320.41754150390625,
77.88488006591797
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 47
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
40
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"steampunk style"
]
},
{
"id": 17,
"type": "BasicScheduler",
"pos": [
415,
588
],
"size": [
341.3883056640625,
106
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 51,
"slot_index": 0
}
],
"outputs": [
{
"name": "SIGMAS",
"type": "SIGMAS",
"links": [
20
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "BasicScheduler"
},
"widgets_values": [
"simple",
20,
0.75
]
},
{
"id": 29,
"type": "ImageScaleToTotalPixels",
"pos": [
411,
746
],
"size": [
345.02252197265625,
83.25662231445312
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "image",
"type": "IMAGE",
"link": 41
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
42
],
"slot_index": 0,
"shape": 3
}
],
"properties": {
"Node name for S&R": "ImageScaleToTotalPixels"
},
"widgets_values": [
"lanczos",
1
]
},
{
"id": 30,
"type": "VAEEncode",
"pos": [
414,
877
],
"size": [
343.8751525878906,
54.39801025390625
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "pixels",
"type": "IMAGE",
"link": 42
},
{
"name": "vae",
"type": "VAE",
"link": 49
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
44
],
"slot_index": 0,
"shape": 3
}
],
"properties": {
"Node name for S&R": "VAEEncode"
},
"widgets_values": []
},
{
"id": 34,
"type": "UnetLoaderGGUFMultiGPU",
"pos": [
-146.0491180419922,
81.0137710571289
],
"size": [
460.7639465332031,
87.65463256835938
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
50,
51
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UnetLoaderGGUFMultiGPU"
},
"widgets_values": [
"flux1-dev-Q4_0.gguf",
"cuda:0"
]
}
],
"links": [
[
9,
8,
0,
9,
0,
"IMAGE"
],
[
19,
16,
0,
13,
2,
"SAMPLER"
],
[
20,
17,
0,
13,
3,
"SIGMAS"
],
[
24,
13,
0,
8,
0,
"LATENT"
],
[
30,
22,
0,
13,
1,
"GUIDER"
],
[
37,
25,
0,
13,
0,
"NOISE"
],
[
40,
6,
0,
22,
1,
"CONDITIONING"
],
[
41,
26,
0,
29,
0,
"IMAGE"
],
[
42,
29,
0,
30,
0,
"IMAGE"
],
[
44,
30,
0,
13,
4,
"LATENT"
],
[
47,
32,
0,
6,
0,
"CLIP"
],
[
48,
33,
0,
8,
1,
"VAE"
],
[
49,
33,
0,
30,
1,
"VAE"
],
[
50,
34,
0,
22,
0,
"MODEL"
],
[
51,
34,
0,
17,
0,
"MODEL"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 0.7247295000000716,
"offset": [
900.0643503440059,
214.25137987084307
]
},
"ue_links": [],
"VHS_latentpreview": false,
"VHS_latentpreviewrate": 0
},
"version": 0.4
}
pyproject.toml
+1 -1
View File
@@ -1,7 +1,7 @@
[project]
name = "comfyui-multigpu"
description = "This extension adds CUDA device selection to supported loader nodes in ComfyUI. By monkey-patching ComfyUI’s memory management, each model component (like UNet, Clip, or VAE) can be loaded on a specific GPU. Examples included are multi-GPU workflows for SDXL, FLUX, LTXVideo, and Hunyuan Video for both standard and GGUF loader nodes."
version = "1.0.2"
version = "1.1.0"
license = {file = "LICENSE"}
[project.urls]