nodes/pollockjj-ComfyUI-MultiGPU
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

This release includes an embeddings adapter for the IP2V part of kijai's CLIP loader for HunyuanVideo. See examples. Bump version to 1.4.3 and update category for HunyuanVideoEmbeddingsAdapter to multigpu; enhance README with new workflow examples for HunyuanVideo GGUF-quantized models.

Browse Source
This commit is contained in:
John Pollock
2025-01-29 11:55:16 -06:00
parent 3260b7e38e
commit 005b5b1882
4 changed files with 929 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+10 -6
View File
@@ -6,11 +6,11 @@ This extension empowers users to spread model components across multiple GPUs or
- Full control over model placement
- Support for all major model loader nodes
- NEW! Hunyuan Video UNet model splitting across two devices using DiffSynth block-swapping
- NEW! GGUF UNet and CLIP model-splitting across cpu/gpus using new DisTorch nodes.
*EXPERIMENTAL: This extension modifies ComfyUI's memory management behavior. While functional for standard GPU and CPU configurations, edge cases may exist. Use at your own risk.*
**Note:** This enhances memory management, not parallelism. Workflow steps execute sequentially but with components loaded across your specified devices. Performance gains come from avoiding repeated model loading/unloading when VRAM is constrained.
**Note:** This enhances memory management, not parallelism. Workflow steps execute sequentially but with components, or in the case of GGUF files `GGML` layers, loaded across your specified devices. *Performance gains* come from avoiding repeated model loading/unloading when VRAM is constrained. *Capability gains* come from offloading as much of the model (VAE/CLIP/UNet) off of your main `compute` device, allowing you to maximize the amount of latent space available for `compute`
# NEW: DisTorch - Advanced GGUF-Quantized Model Layer Distribution
@@ -20,7 +20,7 @@ DisTorch nodes are now available, allowing fine-grained control over model layer
- Utilizing CPU memory alongside GPU VRAM for handling memory-intensive models
- Optimizing layer placement based on your specific hardware configuration
Check out the updated examples `hunyuan_gguf_distorch.json` and `flux1dev_gguf_distorch.json` to see DisTorch in action, demonstrating advanced layer distribution across multiple devices.
Check out the updated examples `hunyuan_ip2v_distorch_gguf.json`, `hunyuan_gguf_distorch.json`, and `flux1dev_gguf_distorch.json` to see DisTorch in action, demonstrating advanced layer distribution across multiple devices including adapting kijai's `HunyuanVideo TextImageEncode (IP2V)` node for the native HunyuanVideo sampler in Comfy Core - meaning HunyuanVideo with IP2V has come to GGUFs!
## Installation
@@ -64,8 +64,9 @@ Currently supported nodes (automatically detected if available):
- HyVideoModelLoaderDiffSynthMultiGPU (**NEW** - MultiGPU-specific node for offloading to an `offload_device` using MultiGPU's device selectors)
- HyVideoVAELoaderMultiGPU
- DownloadAndLoadHyVideoTextEncoderMultiGPU
- Native to ComfyUI-MultiGPU
- DeviceSelectorMultiGPU (Allows user to link loaders together to use the same selected device)
- **Native to ComfyUI-MultiGPU**
- DeviceSelectorMultiGPU - Allows user to link loaders together to use the same selected device
- HunyuanVideoEmbeddingsAdapter - Allows Kijai's excellent IP2V CLIP for HunyuanVideo to be used with Comfy Core sampler.
All MultiGPU nodes available for your install can be found in the "multigpu" category in the node menu.
@@ -75,8 +76,11 @@ All workflows have been tested on a 2x 3090 linux setup, a 4070 win 11 setup, an
### Split GGUF-quantized UNet and CLIP models across multiple devices using DisTorch
- [hunyuan_ip2v_distorch_gguf.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/hunyuan_ip2v_distorch_gguf.json)
This workflow attaches a HunyuanVideo GGUF-quantized model on `cuda:0` for compute and distrubutes its UNet across itself and the `cpu`. Kijai's IP2V clip loader is loaded on the `cpu`. While the VAE is loaded on GPU 1 directly and uses `cuda:1` for compute.
- [examples/hunyuan_gguf_distorch.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/hunyuan_gguf_distorch.json)
This workflow attaches a HunyuanVideo GGUF-quantized model on `cuda:0` for compute and distrubutes its UNet across itself, a secondary CUDA device, and the system's main memory (`cpu`) using a new DisTorch distributed-load methodology. The text encoder now attaches itself to `cuda:1` and splits iteself between `cuda:1` amd `cpu` layers. While the VAE is loaded on GPU 1 directly and use `cuda:1` for compute.
This workflow attaches a HunyuanVideo GGUF-quantized model on `cuda:0` for compute and distrubutes its UNet across itself, a secondary CUDA device, and the system's main memory (`cpu`) using a new DisTorch distributed-load methodology. The text encoder now attaches itself to `cuda:1` and splits iteself between `cuda:1` amd `cpu` layers. While the VAE is loaded on GPU 1 directly and uses `cuda:1` for compute.
- [examples/flux1dev_gguf_distorch.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/flux1dev_gguf_distorch.json)
This workflow loads a FLUX.1-dev model on `cuda:0` for compute and distrubutes its UNet across multiple CUDA devices using new DisTorch distributed-load methodology. While the text encoders and VAE are loaded on GPU 1 and use `cuda:1` for compute.
__init__.py
+2 -2
View File
@@ -280,7 +280,7 @@ class HunyuanVideoEmbeddingsAdapter:
RETURN_TYPES = ("CONDITIONING",)
FUNCTION = "adapt_embeddings"
CATEGORY = "conditioning/hunyuan_video"
CATEGORY = "multigpu"
def adapt_embeddings(self, hyvid_embeds):
# Create main conditioning tensor
@@ -371,7 +371,7 @@ def override_class_with_distorch(cls):
default_device = devices[1] if len(devices) > 1 else devices[0]
inputs["optional"] = inputs.get("optional", {})
inputs["optional"]["device"] = (devices, {"default": default_device})
inputs["optional"]["allocations"] = ("STRING", {"multiline": False, "default": "{cuda:0,0.15;cpu,0.5}"})
inputs["optional"]["allocations"] = ("STRING", {"multiline": False, "default": "cuda:0,0.15;cpu,0.5"})
return inputs
CATEGORY = "multigpu"
examples/hunyuan_ip2v_distorch_gguf.json
+916
View File
@@ -0,0 +1,916 @@
{
"last_node_id": 115,
"last_link_id": 277,
"nodes": [
{
"id": 13,
"type": "SamplerCustomAdvanced",
"pos": [
815.8301391601562,
241.12867736816406
],
"size": [
292.4319763183594,
479.03521728515625
],
"flags": {
"collapsed": false
},
"order": 14,
"mode": 0,
"inputs": [
{
"name": "noise",
"type": "NOISE",
"link": 37,
"slot_index": 0
},
{
"name": "guider",
"type": "GUIDER",
"link": 30,
"slot_index": 1
},
{
"name": "sampler",
"type": "SAMPLER",
"link": 19,
"slot_index": 2
},
{
"name": "sigmas",
"type": "SIGMAS",
"link": 20,
"slot_index": 3
},
{
"name": "latent_image",
"type": "LATENT",
"link": 180,
"slot_index": 4
}
],
"outputs": [
{
"name": "output",
"type": "LATENT",
"links": [
210
],
"slot_index": 0,
"shape": 3
},
{
"name": "denoised_output",
"type": "LATENT",
"links": null,
"shape": 3
}
],
"properties": {
"Node name for S&R": "SamplerCustomAdvanced"
},
"widgets_values": []
},
{
"id": 111,
"type": "DownloadAndLoadHyVideoTextEncoderMultiGPU",
"pos": [
-821.001220703125,
504.2577209472656
],
"size": [
371.9022521972656,
202
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "hyvid_text_encoder",
"type": "HYVIDTEXTENCODER",
"links": [
269
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "DownloadAndLoadHyVideoTextEncoderMultiGPU"
},
"widgets_values": [
"xtuner/llava-llama-3-8b-v1_1-transformers",
"openai/clip-vit-large-patch14",
"bf16",
false,
2,
"disabled",
"cpu"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 88,
"type": "VAELoaderMultiGPU",
"pos": [
-805.2030639648438,
373.4107360839844
],
"size": [
322.5263366699219,
82
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
275
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAELoaderMultiGPU"
},
"widgets_values": [
"hunyuan_video_vae_bf16.safetensors",
"cuda:1"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 109,
"type": "HyVideoTextImageEncode",
"pos": [
-374.6673278808594,
532.1463012695312
],
"size": [
295.6000061035156,
452.87860107421875
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "text_encoders",
"type": "HYVIDTEXTENCODER",
"link": 269
},
{
"name": "custom_prompt_template",
"type": "PROMPT_TEMPLATE",
"link": null,
"shape": 7
},
{
"name": "clip_l",
"type": "CLIP",
"link": null,
"shape": 7
},
{
"name": "image1",
"type": "IMAGE",
"link": 272,
"shape": 7
},
{
"name": "image2",
"type": "IMAGE",
"link": null,
"shape": 7
},
{
"name": "hyvid_cfg",
"type": "HYVID_CFG",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "hyvid_embeds",
"type": "HYVIDEMBEDS",
"links": [
270
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "HyVideoTextImageEncode"
},
"widgets_values": [
"The animal shown in <image> appears within its own natural setting, moving calmly or resting in place as a soft light casts delicate shadows across its form. Over the course of five seconds, it makes subtle shifts in posture or position, revealing small details of its features, such as the texture of its skin or fur, and the quiet rhythm of its breathing.",
"::4",
false,
"video",
""
]
},
{
"id": 115,
"type": "UnetLoaderGGUFDisTorchMultiGPU",
"pos": [
-807.809814453125,
227.3965301513672
],
"size": [
342.1245422363281,
106
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
276,
277
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UnetLoaderGGUFDisTorchMultiGPU"
},
"widgets_values": [
"hunyuan-video-t2v-720p-Q8_0.gguf",
"cuda:0",
"cuda:0,0.15;cpu,0.5"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 112,
"type": "LoadImage",
"pos": [
-787.3131713867188,
795.4229736328125
],
"size": [
315,
314
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
272
],
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": null
}
],
"properties": {
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"pasted/image (224).png",
"image"
]
},
{
"id": 103,
"type": "Note",
"pos": [
-835.8104248046875,
-79.16381072998047
],
"size": [
353.56494140625,
190.77996826171875
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [],
"title": "This workflow requires ComfyUI-GGUF",
"properties": {},
"widgets_values": [
"**⚠️ Dependency Alert! ⚠️**\n\nThis workflow relies on nodes from the [ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF) custom node repository to function correctly. \n\nSpecifically:\n\n*\"CLIPLoaderGGUFMultiGPU\" \n\nwill not work without this dependency installed. Please install ComfyUI-GGUF before attempting to run this workflow."
],
"color": "#332922",
"bgcolor": "#593930"
},
{
"id": 67,
"type": "ModelSamplingSD3",
"pos": [
-364.1572265625,
168.46791076660156
],
"size": [
210,
58
],
"flags": {
"collapsed": true
},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 276
}
],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
252
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ModelSamplingSD3"
},
"widgets_values": [
7
]
},
{
"id": 17,
"type": "BasicScheduler",
"pos": [
-367.9955749511719,
236.91629028320312
],
"size": [
210,
109.8011474609375
],
"flags": {
"collapsed": true
},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 277,
"slot_index": 0
}
],
"outputs": [
{
"name": "SIGMAS",
"type": "SIGMAS",
"links": [
20
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "BasicScheduler"
},
"widgets_values": [
"simple",
20,
1
]
},
{
"id": 113,
"type": "HunyuanVideoEmbeddingsAdapter",
"pos": [
-369.37286376953125,
363.2193603515625
],
"size": [
283.43841552734375,
34.09494400024414
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "hyvid_embeds",
"type": "HYVIDEMBEDS",
"link": 270
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
271
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "HunyuanVideoEmbeddingsAdapter"
},
"widgets_values": []
},
{
"id": 26,
"type": "FluxGuidance",
"pos": [
-25.9213809967041,
325.2367858886719
],
"size": [
211.60000610351562,
58
],
"flags": {
"collapsed": true
},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "conditioning",
"type": "CONDITIONING",
"link": 271
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
129
],
"slot_index": 0,
"shape": 3
}
],
"properties": {
"Node name for S&R": "FluxGuidance"
},
"widgets_values": [
6
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 22,
"type": "BasicGuider",
"pos": [
206.57337951660156,
209.95970153808594
],
"size": [
222.3482666015625,
46
],
"flags": {
"collapsed": true
},
"order": 13,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 252,
"slot_index": 0
},
{
"name": "conditioning",
"type": "CONDITIONING",
"link": 129,
"slot_index": 1
}
],
"outputs": [
{
"name": "GUIDER",
"type": "GUIDER",
"links": [
30
],
"slot_index": 0,
"shape": 3
}
],
"properties": {
"Node name for S&R": "BasicGuider"
},
"widgets_values": []
},
{
"id": 16,
"type": "KSamplerSelect",
"pos": [
334.6855163574219,
327.40887451171875
],
"size": [
210,
58
],
"flags": {
"collapsed": true
},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "SAMPLER",
"type": "SAMPLER",
"links": [
19
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "KSamplerSelect"
},
"widgets_values": [
"euler"
]
},
{
"id": 45,
"type": "EmptyHunyuanLatentVideo",
"pos": [
11.676980018615723,
448.76055908203125
],
"size": [
210,
130
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
180
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "EmptyHunyuanLatentVideo"
},
"widgets_values": [
848,
480,
73,
1
]
},
{
"id": 25,
"type": "RandomNoise",
"pos": [
389.7550354003906,
-21.38304328918457
],
"size": [
250.37998962402344,
82
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "NOISE",
"type": "NOISE",
"links": [
37
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "RandomNoise"
},
"widgets_values": [
5770521,
"fixed"
],
"color": "#2a363b",
"bgcolor": "#3f5159"
},
{
"id": 73,
"type": "VAEDecodeTiled",
"pos": [
503.8817138671875,
509.217529296875
],
"size": [
210,
150
],
"flags": {},
"order": 15,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 210
},
{
"name": "vae",
"type": "VAE",
"link": 275
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
268
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecodeTiled"
},
"widgets_values": [
256,
64,
64,
8
]
},
{
"id": 102,
"type": "VHS_VideoCombine",
"pos": [
9.632485389709473,
698.3294677734375
],
"size": [
451.07391357421875,
592.0040893554688
],
"flags": {},
"order": 16,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 268
},
{
"name": "audio",
"type": "AUDIO",
"link": null,
"shape": 7
},
{
"name": "meta_batch",
"type": "VHS_BatchManager",
"link": null,
"shape": 7
},
{
"name": "vae",
"type": "VAE",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "Filenames",
"type": "VHS_FILENAMES",
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 24,
"loop_count": 0,
"filename_prefix": "HunyuanVideo",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 19,
"save_metadata": true,
"trim_to_audio": false,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "HunyuanVideo_00323.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 24,
"workflow": "HunyuanVideo_00323.png",
"fullpath": "/home/johnj/ComfyUI/output/HunyuanVideo_00323.mp4"
},
"muted": false
}
}
}
],
"links": [
[
19,
16,
0,
13,
2,
"SAMPLER"
],
[
20,
17,
0,
13,
3,
"SIGMAS"
],
[
30,
22,
0,
13,
1,
"GUIDER"
],
[
37,
25,
0,
13,
0,
"NOISE"
],
[
129,
26,
0,
22,
1,
"CONDITIONING"
],
[
180,
45,
0,
13,
4,
"LATENT"
],
[
210,
13,
0,
73,
0,
"LATENT"
],
[
252,
67,
0,
22,
0,
"MODEL"
],
[
268,
73,
0,
102,
0,
"IMAGE"
],
[
269,
111,
0,
109,
0,
"HYVIDTEXTENCODER"
],
[
270,
109,
0,
113,
0,
"HYVIDEMBEDS"
],
[
271,
113,
0,
26,
0,
"CONDITIONING"
],
[
272,
112,
0,
109,
3,
"IMAGE"
],
[
275,
88,
0,
73,
1,
"VAE"
],
[
276,
115,
0,
67,
0,
"MODEL"
],
[
277,
115,
0,
17,
0,
"MODEL"
]
],
"groups": [
{
"id": 2,
"title": "GGUFMultiGPU",
"bounding": [
-836.7138671875,
144.54360961914062,
403.62188720703125,
584.6732788085938
],
"color": "#8AA",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.7247295000001047,
"offset": [
1743.9442159363227,
215.31117435648653
]
},
"groupNodes": {},
"ue_links": [],
"VHS_latentpreview": false,
"VHS_latentpreviewrate": 0
},
"version": 0.4
}
pyproject.toml
+1 -1
View File
@@ -1,7 +1,7 @@
[project]
name = "comfyui-multigpu"
description = "This extension adds CUDA/CPU device selection to supported loader nodes in ComfyUI. Using these nodes, each model component (like UNet, Clip, or VAE) can be loaded on a specific GPU and GGUF UNet/CLIPs can be layer-split for storage across non-compute devices using DisTorch. Examples included are multi-GPU workflows for SDXL, FLUX, LTXVideo, and Hunyuan Video for both standard and GGUF loader nodes."
version = "1.4.2"
version = "1.4.3"
license = {file = "LICENSE"}
[project.urls]