nodes/pollockjj-ComfyUI-MultiGPU
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge dev into main, taking dev version of __init__.py

Browse Source
This commit is contained in:
John Pollock
2025-02-07 21:52:58 -06:00
parent 375276a485 c01ef265a8
commit 04882505cb
16 changed files with 4007 additions and 1665 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+12
View File
@@ -1,2 +1,14 @@
# Python and IDE
__pycache__/
.vscode/settings.json
# Binary management
binaries/*
binaries/win64/*
binaries/linux/*
# Keep directory structure and rename file for Linux
!binaries/
!binaries/win64/
!binaries/linux/
!binaries/linux/rename_this_to_keep_user_binary.txt
README.md
+59 -11
View File
@@ -1,5 +1,12 @@
# ComfyUI-MultiGPU
<p align="center">
<img src="https://raw.githubusercontent.com/pollockjj/ComfyUI-MultiGPU/main/assets/distorch_average.png" width="600">
<br>
<em>Add Virtual VRAM and unleash the power of all of your latent space</em>
</p>
## Device selection and model offloading tools for ComfyUI workloads that exceed single GPU capacity
This extension empowers users to spread model components across multiple GPUs or offload to CPU in a single ComfyUI workflow. Aimed at complex workloads that would normally require sequential model loading/unloading:
@@ -12,15 +19,61 @@ This extension empowers users to spread model components across multiple GPUs or
**Note:** This enhances memory management, not parallelism. Workflow steps execute sequentially but with components, or in the case of GGUF files `GGML` layers, loaded across your specified devices. *Performance gains* come from avoiding repeated model loading/unloading when VRAM is constrained. *Capability gains* come from offloading as much of the model (VAE/CLIP/UNet) off of your main `compute` device, allowing you to maximize the amount of latent space available for `compute`
# NEW: DisTorch - Advanced GGUF-Quantized Model Layer Distribution
<h1 align="center">**NEW** DisTorch 2.0: Virtual VRAM Made Simple</h1>
DisTorch nodes are now available, allowing fine-grained control over model layer distribution across multiple devices for GGUF quantized models. Using a simple allocation string (e.g., "cuda:0,0.025;cuda:1,0.05;cpu,0.10"), you can precisely specify how much memory each device should contribute to hosting model layers. This enables sophisticated memory management strategies like:
<p align="center">
<img src="https://raw.githubusercontent.com/pollockjj/ComfyUI-MultiGPU/main/assets/distorch2_0.gif" width="800">
<br>
<em>DisTorch 2.0 in Action</em>
</p>
- Splitting large models across multiple GPUs with different VRAM capacities
- Utilizing CPU memory alongside GPU VRAM for handling memory-intensive models
- Optimizing layer placement based on your specific hardware configuration
Check out the updated examples `hunyuan_ip2v_distorch_gguf.json`, `hunyuan_gguf_distorch.json`, and `flux1dev_gguf_distorch.json` to see DisTorch in action, demonstrating advanced layer distribution across multiple devices including adapting kijai's `HunyuanVideo TextImageEncode (IP2V)` node for the native HunyuanVideo sampler in Comfy Core - meaning HunyuanVideo with IP2V has come to GGUFs!
## What's New?
DisTorch now features simple Virtual VRAM control that lets you offload model layers from your GPU with zero configuration. Just set how much VRAM you want to free up, and DisTorch handles the rest.
## How It Works
- **Virtual VRAM**: Defaults to 4GB - just adjust it based on your needs
- **Two Modes**:
- **Default**: Offloads to system RAM
- **Multi-GPU**: Distributes across other GPUs (optional)
## 🎯 Key Benefits
- Free up GPU VRAM instantly without complex settings
- Run larger models by offloading layers to other system RAM
- Use all your main GPU's VRAM for actual `compute` / latent processing
- Seamlessly distribute GGML layers across multiple GPUs if available
- Allows **you** to easily shift from ___on-device speed___ to ___open-device capability___ with a simple one-number change
<p align="center">
<img src="https://raw.githubusercontent.com/pollockjj/ComfyUI-MultiGPU/main/assets/distorch_node.png" width="400">
<br>
<em>DisTorch 2.0 Node with one simple number to tune its Vitual VRAM to your needs</em>
</p>
## 💡 Quick Start
1. Load any GGUF model using a DisTorch node
2. Set your Virtual VRAM amount (default: 4GB, in this example we chose 8GB)
3. Toggle "Use Other VRAM" if you have multiple GPUs[^1]
4. That's it!
[^1]: DisTorch's Virtual VRAM aims to span as few a devices as possible. I recommend users try both ways (VRAM/DRAM) and see which works best for you.
## 🔄 Real-World Example
With a 12GB GPU running an 8GB model:
- Set Virtual VRAM to 4GB
- DisTorch moves 4GB of model layers to RAM
- Your GPU now has extra VRAM for larger batches, higher resolutions, or longer video
## 🚀 Compatibility
Works with all GGUF-quantized ComfyUI/ComfyUI-GGUF-supported UNet/CLIP models.
⚙️ Expert users: For those of you who were here for the 1.0 release of DisTorch, manual allocation strings still available for advanced configurations. Each log will contain the allocation string for the run so it can be easily recreated and/or manipulated for more sophisticated setups.
<p align="center">
<img src="https://raw.githubusercontent.com/pollockjj/ComfyUI-MultiGPU/main/assets/distorch2_0.png" width="300">
<br>
<em>The new Virtual VRAM even lets you offload ALL of the model and still run compute on your CUDA device!</em>
</p>
## Installation
@@ -85,11 +138,6 @@ This workflow attaches a HunyuanVideo GGUF-quantized model on `cuda:0` for compu
- [examples/flux1dev_gguf_distorch.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/flux1dev_gguf_distorch.json)
This workflow loads a FLUX.1-dev model on `cuda:0` for compute and distrubutes its UNet across multiple CUDA devices using new DisTorch distributed-load methodology. While the text encoders and VAE are loaded on GPU 1 and use `cuda:1` for compute.
### Split Hunyuan Video UNet across two devices and use DiffSynth Just-in-Time loading
- [examples/hunyuanvideowrapper_diffsynth.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/hunyuanvideowrapper_diffsynth.json)
This workflow demonstrates DiffSynth's memory optimization strategy enabled in kijai's `ComfyUI-HunyuanVideoWrapper` UNet loader, splitting the UNet model across two CUDA devices using block-swapping. The main device handles active computations while blocks are swapped to and from the offload device as needed. As written, the CLIP loads on cuda:0 and then offloads, and the VAE is loaded after the UNet model has been cleared from memory after generation. This approach enables processing of higher resolution or longer duration videos that would exceed a single GPU's memory capacity, though at the cost of additional processing time. Note that an initial OOM error is expected as the workflow calibrates its memory management strategy - simply run the generation again with the same parameters.
### Split Hunyuan Video generation across multiple resources
- [examples/hunyuanvideowrapper_native_vae.json](https://github.com/pollockjj/ComfyUI-MultiGPU/blob/main/examples/hunyuanvideowrapper_native_vae.json)
__init__.py
+344 -655
View File
File diff suppressed because it is too large Load Diff
assets/distorch2_0.gif
Regular → Executable
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.4 MiB

After

Width:  |  Height:  |  Size: 1.4 MiB

assets/distorch2_0.png
Regular → Executable
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 51 KiB

After

Width:  |  Height:  |  Size: 51 KiB

assets/distorch_average.png
Regular → Executable
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 104 KiB

After

Width:  |  Height:  |  Size: 104 KiB

assets/distorch_node.png
Regular → Executable
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 66 KiB

After

Width:  |  Height:  |  Size: 66 KiB

binaries/linux/rename_this_to_keep_user_binary.txt
View File
examples/flux1dev_gguf_distorch.json
Regular → Executable
+104 -157
View File
@@ -14,7 +14,7 @@
46
],
"flags": {},
"order": 10,
"order": 8,
"mode": 0,
"inputs": [
{
@@ -90,7 +90,7 @@
387.28668212890625
],
"flags": {},
"order": 11,
"order": 9,
"mode": 0,
"inputs": [
{
@@ -158,7 +158,7 @@
692.112548828125
],
"flags": {},
"order": 13,
"order": 11,
"mode": 0,
"inputs": [
{
@@ -187,7 +187,7 @@
200
],
"flags": {},
"order": 8,
"order": 7,
"mode": 0,
"inputs": [
{
@@ -213,79 +213,6 @@
"A towering technological monolith in a cyberpunk cityscape at night, with \"DisTorch\" emblazoned across its surface in massive neon red letters that illuminate the surrounding buildings. The text occupies the central third of the frame, crafted from glowing plasma tubes and crackling energy. Rain-slicked streets below reflect the brilliant signage, while holographic advertisements and flying vehicles populate the background. Moody atmospheric lighting, heavy contrast, photorealistic textures, cinematic color grading."
]
},
{
"id": 35,
"type": "DualCLIPLoaderGGUFMultiGPU",
"pos": [
-74.29531860351562,
272.3004455566406
],
"size": [
435.3804016113281,
130
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"links": [
99
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "DualCLIPLoaderGGUFMultiGPU"
},
"widgets_values": [
"t5-v1_1-xxl-encoder-Q4_K_M.gguf",
"clip_l.safetensors",
"flux",
"cuda:1"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 33,
"type": "VAELoaderMultiGPU",
"pos": [
-62.57144546508789,
460.42584228515625
],
"size": [
450.23980712890625,
82
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
84
],
"slot_index": 0,
"shape": 3
}
],
"properties": {
"Node name for S&R": "VAELoaderMultiGPU"
},
"widgets_values": [
"ae.safetensors",
"cuda:1"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 17,
"type": "BasicScheduler",
@@ -298,7 +225,7 @@
106
],
"flags": {},
"order": 9,
"order": 6,
"mode": 0,
"inputs": [
{
@@ -339,7 +266,7 @@
46
],
"flags": {},
"order": 12,
"order": 10,
"mode": 0,
"inputs": [
{
@@ -368,43 +295,6 @@
},
"widgets_values": []
},
{
"id": 37,
"type": "UnetLoaderGGUFDisTorchMultiGPU",
"pos": [
-71.38706970214844,
90.67648315429688
],
"size": [
417.6407775878906,
124
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
102,
103
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UnetLoaderGGUFDisTorchMultiGPU"
},
"widgets_values": [
"flux1-dev-Q8_0.gguf",
"cuda:0",
"cuda:0,0.4608;cuda:1,0.0271;cpu,0.0136"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 39,
"type": "EmptySD3LatentImage",
@@ -417,7 +307,7 @@
106
],
"flags": {},
"order": 4,
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
@@ -439,30 +329,6 @@
1
]
},
{
"id": 52,
"type": "Note",
"pos": [
-577.845458984375,
64.50007629394531
],
"size": [
495.7899169921875,
484.21075439453125
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [],
"title": "How the DisTorch allocation string works",
"properties": {},
"widgets_values": [
"The allocation string uses a semicolon (;) to separate device allocations, and each device allocation has two parts separated by a comma:\n\nDevice name (e.g., \"cuda:0\", \"cuda:1\", \"cpu\")\nMemory fraction as a decimal (e.g., 0.025 = 2.5%, 0.05 = 5%, 0.10 = 10%)\n\n\nFor my defaults:\n\nCUDA GPU 0: Will allocate 2.5% of its total available memory\nCUDA GPU 1: Will allocate 5% of its total available memory\nCPU: Will allocate 10% of available system memory\n\n\nDisTorch then:\n\nCalculates how much actual memory this means for each device\nDistributes model layers across devices proportionally based on these allocations\nAssigns layers to minimize memory usage while maintaining model functionality\n\n\n\nSo if you have a model with 100 layers and your GPUs have the following memory:\n\nCUDA:0 has 24GB\nCUDA:1 has 24GB\nSystem has 64GB RAM\n\nThe allocation would be:\n\nCUDA:0: 600MB (2.5% of 24GB)\nCUDA:1: 1.2GB (5% of 24GB)\nCPU: 6.4GB (10% of 64GB)\n\nThis would help manage memory usage across your devices while keeping the model functional. The semicolon-separated format makes it easy to specify different allocations for different devices in a single string."
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 25,
"type": "RandomNoise",
@@ -475,7 +341,7 @@
88.84210968017578
],
"flags": {},
"order": 6,
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
@@ -497,34 +363,113 @@
]
},
{
"id": 34,
"type": "UnetLoaderGGUFMultiGPU",
"id": 37,
"type": "UnetLoaderGGUFDisTorchMultiGPU",
"pos": [
-79.20733642578125,
-116.61892700195312
-71.38706970214844,
90.67648315429688
],
"size": [
417.5072937011719,
97.88888549804688
417.6407775878906,
154
],
"flags": {},
"order": 7,
"mode": 4,
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [],
"links": [
102,
103
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UnetLoaderGGUFMultiGPU"
"Node name for S&R": "UnetLoaderGGUFDisTorchMultiGPU"
},
"widgets_values": [
"flux1-dev-Q8_0.gguf",
"cuda:0"
"cuda:0",
6,
false,
""
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 33,
"type": "VAELoaderMultiGPU",
"pos": [
-66.57144165039062,
479.7591552734375
],
"size": [
450.23980712890625,
82
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
84
],
"slot_index": 0,
"shape": 3
}
],
"properties": {
"Node name for S&R": "VAELoaderMultiGPU"
},
"widgets_values": [
"ae.safetensors",
"cuda:1"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 35,
"type": "DualCLIPLoaderGGUFMultiGPU",
"pos": [
-69.62869262695312,
285.6338195800781
],
"size": [
435.3804016113281,
130
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"links": [
99
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "DualCLIPLoaderGGUFMultiGPU"
},
"widgets_values": [
"t5-v1_1-xxl-encoder-Q4_K_M.gguf",
"clip_l.safetensors",
"flux",
"cuda:1"
],
"color": "#233",
"bgcolor": "#355"
@@ -632,10 +577,10 @@
"config": {},
"extra": {
"ds": {
"scale": 0.8769226950001245,
"scale": 1,
"offset": [
955.3126283315189,
498.64012539888034
574.0015045555798,
267.12195585438565
]
},
"ue_links": [],
@@ -644,7 +589,9 @@
"node_versions": {
"comfy-core": "0.3.12",
"ComfyUI-MultiGPU": "3dbfcc713552b0560a471c10d9b29520e8b1981e"
}
},
"VHS_MetadataImage": true,
"VHS_KeepIntermediate": true
},
"version": 0.4
}
examples/hunyuan_gguf_distorch.json
Regular → Executable
+160 -181
View File
@@ -1,6 +1,6 @@
{
"last_node_id": 119,
"last_link_id": 155,
"last_node_id": 120,
"last_link_id": 156,
"nodes": [
{
"id": 72,
@@ -14,7 +14,7 @@
58
],
"flags": {},
"order": 11,
"order": 10,
"mode": 0,
"inputs": [
{
@@ -55,7 +55,7 @@
58
],
"flags": {},
"order": 9,
"order": 7,
"mode": 0,
"inputs": [
{
@@ -93,7 +93,7 @@
150
],
"flags": {},
"order": 14,
"order": 13,
"mode": 0,
"inputs": [
{
@@ -127,76 +127,6 @@
8
]
},
{
"id": 75,
"type": "SamplerCustomAdvanced",
"pos": [
3488.497314453125,
-150.8239288330078
],
"size": [
292.4319763183594,
479.03521728515625
],
"flags": {
"collapsed": false
},
"order": 13,
"mode": 0,
"inputs": [
{
"name": "noise",
"type": "NOISE",
"link": 87,
"slot_index": 0
},
{
"name": "guider",
"type": "GUIDER",
"link": 88,
"slot_index": 1
},
{
"name": "sampler",
"type": "SAMPLER",
"link": 89,
"slot_index": 2
},
{
"name": "sigmas",
"type": "SIGMAS",
"link": 90,
"slot_index": 3
},
{
"name": "latent_image",
"type": "LATENT",
"link": 155,
"slot_index": 4
}
],
"outputs": [
{
"name": "output",
"type": "LATENT",
"links": [
85
],
"slot_index": 0,
"shape": 3
},
{
"name": "denoised_output",
"type": "LATENT",
"links": null,
"shape": 3
}
],
"properties": {
"Node name for S&R": "SamplerCustomAdvanced"
},
"widgets_values": []
},
{
"id": 66,
"type": "BasicScheduler",
@@ -209,7 +139,7 @@
109.8011474609375
],
"flags": {},
"order": 10,
"order": 8,
"mode": 0,
"inputs": [
{
@@ -270,41 +200,6 @@
"euler"
]
},
{
"id": 76,
"type": "VAELoaderMultiGPU",
"pos": [
2629.39599609375,
275.80596923828125
],
"size": [
341.07464599609375,
87.14057922363281
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
99
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAELoaderMultiGPU"
},
"widgets_values": [
"hunyuan_video_vae_bf16.safetensors",
"cuda:1"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 115,
"type": "EmptyHunyuanLatentVideo",
@@ -319,7 +214,7 @@
"flags": {
"collapsed": false
},
"order": 2,
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
@@ -354,7 +249,7 @@
190.77996826171875
],
"flags": {},
"order": 3,
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
@@ -378,7 +273,7 @@
82
],
"flags": {},
"order": 4,
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
@@ -413,7 +308,7 @@
902.6787109375
],
"flags": {},
"order": 15,
"order": 14,
"mode": 0,
"inputs": [
{
@@ -465,13 +360,13 @@
"hidden": false,
"paused": false,
"params": {
"filename": "HunyuanVideo_00315.mp4",
"filename": "HunyuanVideo_00336.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 24,
"workflow": "HunyuanVideo_00315.png",
"fullpath": "/home/johnj/ComfyUI/output/HunyuanVideo_00315.mp4"
"workflow": "HunyuanVideo_00336.png",
"fullpath": "/home/johnj/ComfyUI/output/HunyuanVideo_00336.mp4"
},
"muted": false
}
@@ -491,7 +386,7 @@
"flags": {
"collapsed": true
},
"order": 12,
"order": 11,
"mode": 0,
"inputs": [
{
@@ -523,30 +418,6 @@
},
"widgets_values": []
},
{
"id": 119,
"type": "Note",
"pos": [
2050.9873046875,
-145.64498901367188
],
"size": [
549.7151489257812,
528.065185546875
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [],
"title": "How the DisTorch allocation string works",
"properties": {},
"widgets_values": [
"The allocation string uses a semicolon (;) to separate device allocations, and each device allocation has two parts separated by a comma:\n\nDevice name (e.g., \"cuda:0\", \"cuda:1\", \"cpu\")\nMemory fraction as a decimal (e.g., 0.025 = 2.5%, 0.05 = 5%, 0.10 = 10%)\n\n\nFor my defaults:\n\nCUDA GPU 0: Will allocate 2.5% of its total available memory\nCUDA GPU 1: Will allocate 5% of its total available memory\nCPU: Will allocate 10% of available system memory\n\n\nDisTorch then:\n\nCalculates how much actual memory this means for each device\nDistributes model layers across devices proportionally based on these allocations\nAssigns layers to minimize memory usage while maintaining model functionality\n\n\n\nSo if you have a model with 100 layers and your GPUs have the following memory:\n\nCUDA:0 has 24GB\nCUDA:1 has 24GB\nSystem has 64GB RAM\n\nThe allocation would be:\n\nCUDA:0: 600MB (2.5% of 24GB)\nCUDA:1: 1.2GB (5% of 24GB)\nCPU: 6.4GB (10% of 64GB)\n\nThis would help manage memory usage across your devices while keeping the model functional. The semicolon-separated format makes it easy to specify different allocations for different devices in a single string."
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 67,
"type": "CLIPTextEncode",
@@ -559,13 +430,13 @@
164.31304931640625
],
"flags": {},
"order": 8,
"order": 9,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 154
"link": 156
}
],
"outputs": [
@@ -589,56 +460,123 @@
"bgcolor": "#353"
},
{
"id": 100,
"type": "DualCLIPLoaderGGUFDisTorchMultiGPU",
"id": 76,
"type": "VAELoaderMultiGPU",
"pos": [
2623.7119140625,
64.39619445800781
2632.252685546875,
327.2115783691406
],
"size": [
348.5331726074219,
154
341.07464599609375,
87.14057922363281
],
"flags": {},
"order": 6,
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"name": "VAE",
"type": "VAE",
"links": [
154
99
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "DualCLIPLoaderGGUFDisTorchMultiGPU"
"Node name for S&R": "VAELoaderMultiGPU"
},
"widgets_values": [
"clip_l.safetensors",
"llava-llama-3-8B-v1_1-Q4_K_M.gguf",
"hunyuan_video",
"cuda:1",
"cuda:1,0.33;cpu,0.15"
"hunyuan_video_vae_bf16.safetensors",
"cuda:1"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 75,
"type": "SamplerCustomAdvanced",
"pos": [
3517.626953125,
-117.69574737548828
],
"size": [
292.4319763183594,
479.03521728515625
],
"flags": {
"collapsed": false
},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "noise",
"type": "NOISE",
"link": 87,
"slot_index": 0
},
{
"name": "guider",
"type": "GUIDER",
"link": 88,
"slot_index": 1
},
{
"name": "sampler",
"type": "SAMPLER",
"link": 89,
"slot_index": 2
},
{
"name": "sigmas",
"type": "SIGMAS",
"link": 90,
"slot_index": 3
},
{
"name": "latent_image",
"type": "LATENT",
"link": 155,
"slot_index": 4
}
],
"outputs": [
{
"name": "output",
"type": "LATENT",
"links": [
85
],
"slot_index": 0,
"shape": 3
},
{
"name": "denoised_output",
"type": "LATENT",
"links": null,
"shape": 3
}
],
"properties": {
"Node name for S&R": "SamplerCustomAdvanced"
},
"widgets_values": []
},
{
"id": 88,
"type": "UnetLoaderGGUFDisTorchMultiGPU",
"pos": [
2622.552734375,
-105.02156829833984
2624.267822265625,
-66.1816177368164
],
"size": [
358.005859375,
128.5614013671875
154
],
"flags": {},
"order": 7,
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
@@ -656,9 +594,48 @@
"Node name for S&R": "UnetLoaderGGUFDisTorchMultiGPU"
},
"widgets_values": [
"flux1-dev-Q8_0.gguf",
"hunyuan-video-t2v-720p-Q8_0.gguf",
"cuda:0",
"cuda:0,0.33;cuda:1,0.33;cpu,0.0825"
8,
false,
""
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 120,
"type": "DualCLIPLoaderGGUFMultiGPU",
"pos": [
2625.07763671875,
132.1048126220703
],
"size": [
348.16229248046875,
130
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"links": [
156
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "DualCLIPLoaderGGUFMultiGPU"
},
"widgets_values": [
"clip_l.safetensors",
"llava-llama-3-8B-v1_1-Q4_K_M.gguf",
"hunyuan_video",
"cpu"
],
"color": "#233",
"bgcolor": "#355"
@@ -761,14 +738,6 @@
0,
"MODEL"
],
[
154,
100,
0,
67,
0,
"CLIP"
],
[
155,
115,
@@ -776,6 +745,14 @@
75,
4,
"LATENT"
],
[
156,
120,
0,
67,
0,
"CLIP"
]
],
"groups": [
@@ -783,8 +760,8 @@
"id": 1,
"title": "GGUFDisTorchMultiGPU",
"bounding": [
2617.71240234375,
-173.0873565673828,
2620.569091796875,
-121.6815185546875,
360.4816589355469,
537.903076171875
],
@@ -796,15 +773,17 @@
"config": {},
"extra": {
"ds": {
"scale": 1.283902517749697,
"scale": 1.167184107045179,
"offset": [
-2107.057728768135,
344.093734201536
-2360.0515813693187,
465.0591039206934
]
},
"ue_links": [],
"VHS_latentpreview": false,
"VHS_latentpreviewrate": 0
"VHS_latentpreviewrate": 0,
"VHS_MetadataImage": true,
"VHS_KeepIntermediate": true
},
"version": 0.4
}
examples/hunyuan_ip2v_distorch_gguf.json
Regular → Executable
+90 -86
View File
@@ -159,7 +159,7 @@
452.87860107421875
],
"flags": {},
"order": 10,
"order": 8,
"mode": 0,
"inputs": [
{
@@ -219,43 +219,6 @@
""
]
},
{
"id": 115,
"type": "UnetLoaderGGUFDisTorchMultiGPU",
"pos": [
-807.809814453125,
227.3965301513672
],
"size": [
342.1245422363281,
106
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
276,
277
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UnetLoaderGGUFDisTorchMultiGPU"
},
"widgets_values": [
"hunyuan-video-t2v-720p-Q8_0.gguf",
"cuda:0",
"cuda:0,0.15;cpu,0.5"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 112,
"type": "LoadImage",
@@ -268,7 +231,7 @@
314
],
"flags": {},
"order": 3,
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
@@ -306,7 +269,7 @@
190.77996826171875
],
"flags": {},
"order": 4,
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [],
@@ -332,7 +295,7 @@
"flags": {
"collapsed": true
},
"order": 8,
"order": 9,
"mode": 0,
"inputs": [
{
@@ -372,7 +335,7 @@
"flags": {
"collapsed": true
},
"order": 9,
"order": 10,
"mode": 0,
"inputs": [
{
@@ -540,7 +503,7 @@
"flags": {
"collapsed": true
},
"order": 5,
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
@@ -572,7 +535,7 @@
130
],
"flags": {},
"order": 6,
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
@@ -595,41 +558,6 @@
1
]
},
{
"id": 25,
"type": "RandomNoise",
"pos": [
389.7550354003906,
-21.38304328918457
],
"size": [
250.37998962402344,
82
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "NOISE",
"type": "NOISE",
"links": [
37
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "RandomNoise"
},
"widgets_values": [
5770521,
"fixed"
],
"color": "#2a363b",
"bgcolor": "#3f5159"
},
{
"id": 73,
"type": "VAEDecodeTiled",
@@ -685,7 +613,7 @@
],
"size": [
451.07391357421875,
592.0040893554688
334
],
"flags": {},
"order": 16,
@@ -751,6 +679,80 @@
"muted": false
}
}
},
{
"id": 115,
"type": "UnetLoaderGGUFDisTorchMultiGPU",
"pos": [
-810.4765014648438,
219.3965301513672
],
"size": [
342.1245422363281,
154
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
276,
277
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UnetLoaderGGUFDisTorchMultiGPU"
},
"widgets_values": [
"hunyuan-video-t2v-720p-Q4_K_M.gguf",
"cuda:0",
4,
false,
""
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 25,
"type": "RandomNoise",
"pos": [
368.4217834472656,
72.61695861816406
],
"size": [
250.37998962402344,
82
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "NOISE",
"type": "NOISE",
"links": [
37
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "RandomNoise"
},
"widgets_values": [
5770521,
"fixed"
],
"color": "#2a363b",
"bgcolor": "#3f5159"
}
],
"links": [
@@ -901,16 +903,18 @@
"config": {},
"extra": {
"ds": {
"scale": 0.7247295000001047,
"offset": [
1743.9442159363227,
215.31117435648653
]
"scale": 1,
"offset": {
"0": 1075.0079345703125,
"1": 156.88380432128906
}
},
"groupNodes": {},
"ue_links": [],
"VHS_latentpreview": false,
"VHS_latentpreviewrate": 0
"VHS_latentpreviewrate": 0,
"VHS_MetadataImage": true,
"VHS_KeepIntermediate": true
},
"version": 0.4
}
examples/hunyuanvideowrapper_diffsynth.json
-575
View File
@@ -1,575 +0,0 @@
{
"last_node_id": 57,
"last_link_id": 75,
"nodes": [
{
"id": 34,
"type": "VHS_VideoCombine",
"pos": [
847.0758666992188,
-415.1882629394531
],
"size": [
512.8807373046875,
1128.39453125
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 69
},
{
"name": "audio",
"type": "AUDIO",
"link": null,
"shape": 7
},
{
"name": "meta_batch",
"type": "VHS_BatchManager",
"link": null,
"shape": 7
},
{
"name": "vae",
"type": "VAE",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "Filenames",
"type": "VHS_FILENAMES",
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 24,
"loop_count": 0,
"filename_prefix": "HunyuanVideo",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 19,
"save_metadata": true,
"trim_to_audio": false,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "HunyuanVideo_00182.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 24,
"workflow": "HunyuanVideo_00182.png",
"fullpath": "/home/johnj/ComfyUI/output/HunyuanVideo_00182.mp4"
},
"muted": false
}
}
},
{
"id": 45,
"type": "VAEDecodeTiled",
"pos": [
604.5225830078125,
-401.70220947265625
],
"size": [
210,
150
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 70
},
{
"name": "vae",
"type": "VAE",
"link": 56
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
69
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecodeTiled"
},
"widgets_values": [
256,
64,
64,
8
]
},
{
"id": 44,
"type": "VAELoaderMultiGPU",
"pos": [
208.21218872070312,
-586.37841796875
],
"size": [
353.3572692871094,
84.02234649658203
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
56
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAELoaderMultiGPU"
},
"widgets_values": [
"hunyuan_video_vae_bf16.safetensors",
"cuda:1"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 53,
"type": "HyVideoTextImageEncode",
"pos": [
-163.92579650878906,
-51.38969421386719
],
"size": [
367.4625244140625,
528.5474853515625
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "text_encoders",
"type": "HYVIDTEXTENCODER",
"link": 74
},
{
"name": "custom_prompt_template",
"type": "PROMPT_TEMPLATE",
"link": null,
"shape": 7
},
{
"name": "clip_l",
"type": "CLIP",
"link": null,
"shape": 7
},
{
"name": "image1",
"type": "IMAGE",
"link": 73,
"shape": 7
},
{
"name": "image2",
"type": "IMAGE",
"link": null,
"shape": 7
},
{
"name": "hyvid_cfg",
"type": "HYVID_CFG",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "hyvid_embeds",
"type": "HYVIDEMBEDS",
"links": [
75
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "HyVideoTextImageEncode"
},
"widgets_values": [
"The animal shown in <image> appears within its own natural setting, moving calmly or resting in place as a soft light casts delicate shadows across its form. Over the course of five seconds, it makes subtle shifts in posture or position, revealing small details of its features, such as the texture of its skin or fur, and the quiet rhythm of its breathing.",
"::3",
true,
"video",
""
]
},
{
"id": 54,
"type": "LoadImage",
"pos": [
-526.0797119140625,
233.78311157226562
],
"size": [
315,
314
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
73
],
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": null
}
],
"properties": {
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"pasted/image (165).png",
"image"
]
},
{
"id": 49,
"type": "DownloadAndLoadHyVideoTextEncoderMultiGPU",
"pos": [
-591.7539672851562,
-37.14900588989258
],
"size": [
380.8912353515625,
202
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "hyvid_text_encoder",
"type": "HYVIDTEXTENCODER",
"links": [
74
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "DownloadAndLoadHyVideoTextEncoderMultiGPU"
},
"widgets_values": [
"xtuner/llava-llama-3-8b-v1_1-transformers",
"openai/clip-vit-large-patch14",
"bf16",
false,
2,
"disabled",
"cuda:0"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 3,
"type": "HyVideoSampler",
"pos": [
255.96482849121094,
-403.58502197265625
],
"size": [
315,
630
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "HYVIDEOMODEL",
"link": 71
},
{
"name": "hyvid_embeds",
"type": "HYVIDEMBEDS",
"link": 75
},
{
"name": "samples",
"type": "LATENT",
"link": null,
"shape": 7
},
{
"name": "stg_args",
"type": "STGARGS",
"link": null,
"shape": 7
},
{
"name": "context_options",
"type": "HYVIDCONTEXT",
"link": null,
"shape": 7
},
{
"name": "feta_args",
"type": "FETAARGS",
"link": null,
"shape": 7
},
{
"name": "teacache_args",
"type": "TEACACHEARGS",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "samples",
"type": "LATENT",
"links": [
70
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "HyVideoSampler"
},
"widgets_values": [
768,
1216,
101,
30,
7.5,
7.5,
5770521,
"fixed",
true,
1,
"FlowMatchDiscreteScheduler"
]
},
{
"id": 52,
"type": "HyVideoModelLoaderDiffSynthMultiGPU",
"pos": [
-217.43194580078125,
-401.6243896484375
],
"size": [
441,
218
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "compile_args",
"type": "COMPILEARGS",
"link": null,
"shape": 7
},
{
"name": "block_swap_args",
"type": "BLOCKSWAPARGS",
"link": null,
"shape": 7
},
{
"name": "lora",
"type": "HYVIDLORA",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "model",
"type": "HYVIDEOMODEL",
"links": [
71
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "HyVideoModelLoaderDiffSynthMultiGPU"
},
"widgets_values": [
"hunyuan_video_FastVideo_720_fp8_e4m3fn.safetensors",
"bf16",
"fp8_e4m3fn_fast",
"sageattn_varlen",
"cuda:0",
"cuda:1"
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 57,
"type": "Note",
"pos": [
256.481689453125,
293.84710693359375
],
"size": [
389.5359191894531,
349.2815856933594
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [],
"title": "Explanation of Memory Constraints and OOM Behavior",
"properties": {},
"widgets_values": [
"The settings in this workflow deliberately push memory usage beyond what a single 24GB RTX 3090 can handle. As a result, UNet blocks must be offloaded to an offload_device, which is assumed to be a second 24GB RTX 3090. The chosen resolution and number of frames illustrate that one-megapixel image size videos over four seconds long can be generated—even though it exceeds the primary GPU's native memory capacity using the \"just-in-time\" DiffSynth block-swapping approach. The DiffSynth JiT algorithm is straightforward, but not optimized for speed.\n\n**Expected OOM Behavior**\n\n* One Initial \"Out of Memory\"\nSeeing a single OOM error at the start is normal for this workflow. It happens because kijai's HunyuanVideo Sampler node initially attempts a full load on \"device\", even if it will surpass available VRAM. A second attempt using the same generation parameters will correctly trigger the offload process to \"offload_device\".\n\n* Repeated OOM Errors \nIf you keep getting OOM errors with the same settings (beyond the first one), then the requirements of your generation parameters still exceed what your system—even with offloading—can manage. You may need to lower resolution, shorten video length, or upgrade hardware resources."
],
"color": "#233",
"bgcolor": "#355"
},
{
"id": 51,
"type": "Note",
"pos": [
-793.3005981445312,
-466.9315490722656
],
"size": [
564.3460693359375,
355.50164794921875
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [],
"title": "****FIRST ATTEMPT LOADS ONLY ON PRIMARY DEVICE, POSSIBLE OOM****",
"properties": {},
"widgets_values": [
"The MultiGPU implementation for HunyuanVideo now offers two approaches:\n1. The original workflow (examples/hunyuanvideowrapper_native_vae.json)\n2. A new experimental DiffSynth-enabled workflow that enables more efficient multi-GPU usage (examples/hunyuanvideowrapper_DiffSynth.json)\n\nNew DiffSynth-Based Workflow (Experimental - THIS WORKFLOW):\n* Uses the \"HyVideoModelLoaderDiffSynthMultiGPU\" node which implements DiffSynth's memory management approach with TWO device selections\n* Adds a second device selector \"offload_device\" specifically for model offloading\n* Automatically enables efficient block swapping between GPUs via kijai's adaptation of the DiffSynth methodology\n* Provides better memory utilization by keeping actively computed blocks in VRAM\n* Assumes a 2-device solution and sets \"force_offload\" to \"true\" to avoid VAE loader OOM issues. A third 3090, for instance could be used and thus the VAE would stay in memory as well\n\nKnown Behaviors:\n* Initial OOM errors are expected and consistent with the underlying implementation, occurring in the attention mechanism of the first double block\n* Second attempts typically succeed regardless of offload device selection (CPU or GPU)\n* When running, the secondary device shows active utilization, confirming proper block swapping\n\n***** Will require a second attempt if first generation fails with OOM****\n\nNOTE: The original workflow remains the recommended stable approach. The DiffSynth-based version is provided as an experimental alternative for users who want to explore more aggressive memory optimization strategies to extend either resolution or duration options at the expense of speed."
],
"color": "#233",
"bgcolor": "#355"
}
],
"links": [
[
56,
44,
0,
45,
1,
"VAE"
],
[
69,
45,
0,
34,
0,
"IMAGE"
],
[
70,
3,
0,
45,
0,
"LATENT"
],
[
71,
52,
0,
3,
0,
"HYVIDEOMODEL"
],
[
73,
54,
0,
53,
3,
"IMAGE"
],
[
74,
49,
0,
53,
0,
"HYVIDTEXTENCODER"
],
[
75,
53,
0,
3,
1,
"HYVIDEMBEDS"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 0.797202450000112,
"offset": [
1219.7255813472686,
777.2824223551936
]
},
"ue_links": [],
"VHS_latentpreview": false,
"VHS_latentpreviewrate": 0
},
"version": 0.4
}
examples/ltxvideo_gguf_distorch_lowvram_highdram.json
Executable
+2762
View File
File diff suppressed because it is too large Load Diff
nodes.py
+476
View File
@@ -0,0 +1,476 @@
import os
import torch
import folder_paths
from pathlib import Path
from nodes import NODE_CLASS_MAPPINGS
class UnetLoaderGGUF:
@classmethod
def INPUT_TYPES(s):
unet_names = [x for x in folder_paths.get_filename_list("unet_gguf")]
return {
"required": {
"unet_name": (unet_names,),
}
}
RETURN_TYPES = ("MODEL",)
FUNCTION = "load_unet"
CATEGORY = "bootleg"
TITLE = "Unet Loader (GGUF)"
def load_unet(self, unet_name, dequant_dtype=None, patch_dtype=None, patch_on_device=None):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["UnetLoaderGGUF"]()
return original_loader.load_unet(unet_name, dequant_dtype, patch_dtype, patch_on_device)
class UnetLoaderGGUFAdvanced(UnetLoaderGGUF):
@classmethod
def INPUT_TYPES(s):
unet_names = [x for x in folder_paths.get_filename_list("unet_gguf")]
return {
"required": {
"unet_name": (unet_names,),
"dequant_dtype": (["default", "target", "float32", "float16", "bfloat16"], {"default": "default"}),
"patch_dtype": (["default", "target", "float32", "float16", "bfloat16"], {"default": "default"}),
"patch_on_device": ("BOOLEAN", {"default": False}),
}
}
TITLE = "Unet Loader (GGUF/Advanced)"
class CLIPLoaderGGUF:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"clip_name": (s.get_filename_list(),),
"type": (["stable_diffusion", "stable_cascade", "sd3", "stable_audio", "mochi", "ltxv"],),
}
}
RETURN_TYPES = ("CLIP",)
FUNCTION = "load_clip"
CATEGORY = "bootleg"
TITLE = "CLIPLoader (GGUF)"
@classmethod
def get_filename_list(s):
files = []
files += folder_paths.get_filename_list("clip")
files += folder_paths.get_filename_list("clip_gguf")
return sorted(files)
def load_data(self, ckpt_paths):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["CLIPLoaderGGUF"]()
return original_loader.load_data(ckpt_paths)
def load_patcher(self, clip_paths, clip_type, clip_data):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["CLIPLoaderGGUF"]()
return original_loader.load_patcher(clip_paths, clip_type, clip_data)
def load_clip(self, clip_name, type="stable_diffusion"):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["CLIPLoaderGGUF"]()
return original_loader.load_clip(clip_name, type)
class DualCLIPLoaderGGUF(CLIPLoaderGGUF):
@classmethod
def INPUT_TYPES(s):
file_options = (s.get_filename_list(), )
return {
"required": {
"clip_name1": file_options,
"clip_name2": file_options,
"type": (("sdxl", "sd3", "flux", "hunyuan_video"),),
}
}
TITLE = "DualCLIPLoader (GGUF)"
def load_clip(self, clip_name1, clip_name2, type):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["DualCLIPLoaderGGUF"]()
clip = original_loader.load_clip(clip_name1, clip_name2, type)
clip[0].patcher.load(force_patch_weights=True)
return clip
class TripleCLIPLoaderGGUF(CLIPLoaderGGUF):
@classmethod
def INPUT_TYPES(s):
file_options = (s.get_filename_list(), )
return {
"required": {
"clip_name1": file_options,
"clip_name2": file_options,
"clip_name3": file_options,
}
}
TITLE = "TripleCLIPLoader (GGUF)"
def load_clip(self, clip_name1, clip_name2, clip_name3, type="sd3"):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["TripleCLIPLoaderGGUF"]()
return original_loader.load_clip(clip_name1, clip_name2, clip_name3, type)
class LTXVLoader:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"ckpt_name": (folder_paths.get_filename_list("checkpoints"),
{"tooltip": "The name of the checkpoint (model) to load."}),
"dtype": (["bfloat16", "float32"], {"default": "bfloat16"})
}
}
RETURN_TYPES = ("MODEL", "VAE")
RETURN_NAMES = ("model", "vae")
FUNCTION = "load"
CATEGORY = "lightricks/LTXV"
TITLE = "LTXV Loader"
OUTPUT_NODE = False
def load(self, ckpt_name, dtype):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["LTXVLoader"]()
return original_loader.load(ckpt_name, dtype)
def _load_unet(self, load_device, offload_device, weights, num_latent_channels, dtype, config=None ):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["LTXVLoader"]()
return original_loader._load_unet(load_device, offload_device, weights, num_latent_channels, dtype, config=None )
def _load_vae(self, weights, config=None):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["LTXVLoader"]()
return original_loader._load_vae(weights, config=None)
class Florence2ModelLoader:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"model": ([item.name for item in Path(folder_paths.models_dir, "LLM").iterdir() if item.is_dir()], {"tooltip": "models are expected to be in Comfyui/models/LLM folder"}),
"precision": (['fp16','bf16','fp32'],),
"attention": (
[ 'flash_attention_2', 'sdpa', 'eager'],
{
"default": 'sdpa'
}),
},
"optional": {
"lora": ("PEFTLORA",),
}
}
RETURN_TYPES = ("FL2MODEL",)
RETURN_NAMES = ("florence2_model",)
FUNCTION = "loadmodel"
CATEGORY = "Florence2"
def loadmodel(self, model, precision, attention, lora=None):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["Florence2ModelLoader"]()
return original_loader.loadmodel(model, precision, attention, lora)
class DownloadAndLoadFlorence2Model:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"model": (
[
'microsoft/Florence-2-base',
'microsoft/Florence-2-base-ft',
'microsoft/Florence-2-large',
'microsoft/Florence-2-large-ft',
'HuggingFaceM4/Florence-2-DocVQA',
'thwri/CogFlorence-2.1-Large',
'thwri/CogFlorence-2.2-Large',
'gokaygokay/Florence-2-SD3-Captioner',
'gokaygokay/Florence-2-Flux-Large',
'MiaoshouAI/Florence-2-base-PromptGen-v1.5',
'MiaoshouAI/Florence-2-large-PromptGen-v1.5',
'MiaoshouAI/Florence-2-base-PromptGen-v2.0',
'MiaoshouAI/Florence-2-large-PromptGen-v2.0'
],
{
"default": 'microsoft/Florence-2-base'
}),
"precision": ([ 'fp16','bf16','fp32'],
{
"default": 'fp16'
}),
"attention": (
[ 'flash_attention_2', 'sdpa', 'eager'],
{
"default": 'sdpa'
}),
},
"optional": {
"lora": ("PEFTLORA",),
}
}
RETURN_TYPES = ("FL2MODEL",)
RETURN_NAMES = ("florence2_model",)
FUNCTION = "loadmodel"
CATEGORY = "Florence2"
def loadmodel(self, model, precision, attention, lora=None):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["DownloadAndLoadFlorence2Model"]()
return original_loader.loadmodel(model, precision, attention, lora)
class CheckpointLoaderNF4:
@classmethod
def INPUT_TYPES(s):
return {"required": { "ckpt_name": (folder_paths.get_filename_list("checkpoints"), ),
}}
RETURN_TYPES = ("MODEL", "CLIP", "VAE")
FUNCTION = "load_checkpoint"
CATEGORY = "loaders"
def load_checkpoint(self, ckpt_name):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["CheckpointLoaderNF4"]()
return original_loader.load_checkpoint(ckpt_name)
class LoadFluxControlNet:
@classmethod
def INPUT_TYPES(s):
return {"required": {"model_name": (["flux-dev", "flux-dev-fp8", "flux-schnell"],),
"controlnet_path": (folder_paths.get_filename_list("xlabs_controlnets"), ),
}}
RETURN_TYPES = ("FluxControlNet",)
RETURN_NAMES = ("ControlNet",)
FUNCTION = "loadmodel"
CATEGORY = "XLabsNodes"
def loadmodel(self, model_name, controlnet_path):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["LoadFluxControlNet"]()
return original_loader.loadmodel(model_name, controlnet_path)
class MMAudioModelLoader:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"mmaudio_model": (folder_paths.get_filename_list("mmaudio"), {"tooltip": "These models are loaded from the 'ComfyUI/models/mmaudio' -folder",}),
"base_precision": (["fp16", "fp32", "bf16"], {"default": "fp16"}),
},
}
RETURN_TYPES = ("MMAUDIO_MODEL",)
RETURN_NAMES = ("mmaudio_model", )
FUNCTION = "loadmodel"
CATEGORY = "MMAudio"
def loadmodel(self, mmaudio_model, base_precision):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["MMAudioModelLoader"]()
return original_loader.loadmodel(mmaudio_model, base_precision)
class MMAudioFeatureUtilsLoader:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"vae_model": (folder_paths.get_filename_list("mmaudio"), {"tooltip": "These models are loaded from 'ComfyUI/models/mmaudio'"}),
"synchformer_model": (folder_paths.get_filename_list("mmaudio"), {"tooltip": "These models are loaded from 'ComfyUI/models/mmaudio'"}),
"clip_model": (folder_paths.get_filename_list("mmaudio"), {"tooltip": "These models are loaded from 'ComfyUI/models/mmaudio'"}),
},
"optional": {
"bigvgan_vocoder_model": ("VOCODER_MODEL", {"tooltip": "These models are loaded from 'ComfyUI/models/mmaudio'"}),
"mode": (["16k", "44k"], {"default": "44k"}),
"precision": (["fp16", "fp32", "bf16"],
{"default": "fp16"}
),
}
}
RETURN_TYPES = ("MMAUDIO_FEATUREUTILS",)
RETURN_NAMES = ("mmaudio_featureutils", )
FUNCTION = "loadmodel"
CATEGORY = "MMAudio"
def loadmodel(self, vae_model, precision, synchformer_model, clip_model, mode, bigvgan_vocoder_model=None):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["MMAudioFeatureUtilsLoader"]()
return original_loader.loadmodel(vae_model, precision, synchformer_model, clip_model, mode, bigvgan_vocoder_model)
class MMAudioSampler:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"mmaudio_model": ("MMAUDIO_MODEL",),
"feature_utils": ("MMAUDIO_FEATUREUTILS",),
"duration": ("FLOAT", {"default": 8, "step": 0.01, "tooltip": "Duration of the audio in seconds"}),
"steps": ("INT", {"default": 25, "step": 1, "tooltip": "Number of steps to interpolate"}),
"cfg": ("FLOAT", {"default": 4.5, "step": 0.1, "tooltip": "Strength of the conditioning"}),
"seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}),
"prompt": ("STRING", {"default": "", "multiline": True} ),
"negative_prompt": ("STRING", {"default": "", "multiline": True} ),
"mask_away_clip": ("BOOLEAN", {"default": False, "tooltip": "If true, the clip video will be masked away"}),
"force_offload": ("BOOLEAN", {"default": True, "tooltip": "If true, the model will be offloaded to the offload device"}),
},
"optional": {
"images": ("IMAGE",),
},
}
RETURN_TYPES = ("AUDIO",)
RETURN_NAMES = ("audio", )
FUNCTION = "sample"
CATEGORY = "MMAudio"
def sample(self, mmaudio_model, seed, feature_utils, duration, steps, cfg, prompt, negative_prompt, mask_away_clip, force_offload, images=None):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["MMAudioSampler"]()
return original_loader.sample(mmaudio_model, seed, feature_utils, duration, steps, cfg, prompt, negative_prompt, mask_away_clip, force_offload, images)
class PulidModelLoader:
@classmethod
def INPUT_TYPES(s):
return {"required": { "pulid_file": (folder_paths.get_filename_list("pulid"), )}}
RETURN_TYPES = ("PULID",)
FUNCTION = "load_model"
CATEGORY = "pulid"
def load_model(self, pulid_file):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["PulidModelLoader"]()
return original_loader.load_model(pulid_file)
class PulidInsightFaceLoader:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"provider": (["CPU", "CUDA", "ROCM", "CoreML"], ),
},
}
RETURN_TYPES = ("FACEANALYSIS",)
FUNCTION = "load_insightface"
CATEGORY = "pulid"
def load_insightface(self, provider):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["PulidInsightFaceLoader"]()
return original_loader.load_insightface(provider)
class PulidEvaClipLoader:
@classmethod
def INPUT_TYPES(s):
return {
"required": {},
}
RETURN_TYPES = ("EVA_CLIP",)
FUNCTION = "load_eva_clip"
CATEGORY = "pulid"
def load_eva_clip(self):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["PulidEvaClipLoader"]()
return original_loader.load_eva_clip()
class HyVideoModelLoader:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"model": (folder_paths.get_filename_list("diffusion_models"), {"tooltip": "These models are loaded from the 'ComfyUI/models/diffusion_models' -folder",}),
"base_precision": (["fp32", "bf16"], {"default": "bf16"}),
"quantization": (['disabled', 'fp8_e4m3fn', 'fp8_e4m3fn_fast', 'fp8_scaled', 'torchao_fp8dq', "torchao_fp8dqrow", "torchao_int8dq", "torchao_fp6", "torchao_int4", "torchao_int8"], {"default": 'disabled', "tooltip": "optional quantization method"}),
"load_device": (["main_device"], {"default": "main_device"}),
},
"optional": {
"attention_mode": ([
"sdpa",
"flash_attn_varlen",
"sageattn_varlen",
"comfy",
], {"default": "flash_attn"}),
"compile_args": ("COMPILEARGS", ),
"block_swap_args": ("BLOCKSWAPARGS", ),
"lora": ("HYVIDLORA", {"default": None}),
"auto_cpu_offload": ("BOOLEAN", {"default": False, "tooltip": "Enable auto offloading for reduced VRAM usage, implementation from DiffSynth-Studio, slightly different from block swapping and uses even less VRAM, but can be slower as you can't define how much VRAM to use"}),
}
}
RETURN_TYPES = ("HYVIDEOMODEL",)
RETURN_NAMES = ("model", )
FUNCTION = "loadmodel"
CATEGORY = "HunyuanVideoWrapper"
def loadmodel(self, model, base_precision, load_device, quantization, compile_args=None, attention_mode="sdpa", block_swap_args=None, lora=None, auto_cpu_offload=False):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["HyVideoModelLoader"]()
return original_loader.loadmodel(model, base_precision, load_device, quantization, compile_args, attention_mode, block_swap_args, lora, auto_cpu_offload)
class HyVideoVAELoader:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"model_name": (folder_paths.get_filename_list("vae"), {"tooltip": "These models are loaded from 'ComfyUI/models/vae'"}),
},
"optional": {
"precision": (["fp16", "fp32", "bf16"],
{"default": "bf16"}
),
"compile_args":("COMPILEARGS", ),
}
}
RETURN_TYPES = ("VAE",)
RETURN_NAMES = ("vae", )
FUNCTION = "loadmodel"
CATEGORY = "HunyuanVideoWrapper"
DESCRIPTION = "Loads Hunyuan VAE model from 'ComfyUI/models/vae'"
def loadmodel(self, model_name, precision, compile_args=None):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["HyVideoVAELoader"]()
return original_loader.loadmodel(model_name, precision, compile_args)
class DownloadAndLoadHyVideoTextEncoder:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"llm_model": (["Kijai/llava-llama-3-8b-text-encoder-tokenizer","xtuner/llava-llama-3-8b-v1_1-transformers"],),
"clip_model": (["disabled","openai/clip-vit-large-patch14",],),
"precision": (["fp16", "fp32", "bf16"],
{"default": "bf16"}
),
},
"optional": {
"apply_final_norm": ("BOOLEAN", {"default": False}),
"hidden_state_skip_layer": ("INT", {"default": 2}),
"quantization": (['disabled', 'bnb_nf4', "fp8_e4m3fn"], {"default": 'disabled'}),
}
}
RETURN_TYPES = ("HYVIDTEXTENCODER",)
RETURN_NAMES = ("hyvid_text_encoder", )
FUNCTION = "loadmodel"
CATEGORY = "HunyuanVideoWrapper"
DESCRIPTION = "Loads Hunyuan text_encoder model from 'ComfyUI/models/LLM'"
def loadmodel(self, llm_model, clip_model, precision, apply_final_norm=False, hidden_state_skip_layer=2, quantization="disabled"):
from nodes import NODE_CLASS_MAPPINGS
original_loader = NODE_CLASS_MAPPINGS["DownloadAndLoadHyVideoTextEncoder"]()
return original_loader.loadmodel(llm_model, clip_model, precision, apply_final_norm, hidden_state_skip_layer, quantization)
precompiled_binaries/linux/llama-quantize
Executable
BIN
View File
Binary file not shown.
precompiled_binaries/win64/llama-quantize.exe
Executable
BIN
View File
Binary file not shown.