nodes/maepopi-Diffusers-in-ComfyUI
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Added more info and workflow jsons to help users

Browse Source
This commit is contained in:
Dr. Jusseaux
2024-12-06 14:54:02 +01:00
parent 35f414a6ab
commit 4234d29328
14 changed files with 1324 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+37 -4
View File
@@ -53,10 +53,12 @@ If you want to do everything from scratch and don't even have Comfy UI install,
## Pipeline nodes
Under "Diffusers-in-Comfy/Pipelines", you will find one node per Diffuser pipeline. **Text2ImgStableDiffusionPipeline**, **Img2ImgStableDiffusionPipeline**, and **InpaintingStableDiffusionPipeline**. Inside each one, you can precise :
- Whether you want to use SDXL
- Whether you want to use a VAE (write the Hugging Face path to your VAE, for example : "stabilityai/sd-vae-ft-mse")
- Whether you want to use a ControlNet (write the Hugging Face path to your Controlnet model, for example : "XLabs-AI/flux-controlnet-canny")
- Whether you want to activate low VRAM options to avoid low GPU memory issues (this will notably send your pipeline to CPU and activate xformers memory efficient attention)
- Whether you want to use **SDXL.**
- Whether you want to use a **VAE** (write the **Hugging Face path** to your VAE, for example : **"stabilityai/sd-vae-ft-mse"**)
- Whether you want to use a **ControlNet** (write the **Hugging Face path** to your Controlnet model, for example : **"XLabs-AI/flux-controlnet-canny"**)
- Whether you want to activate **low VRAM** options to avoid low GPU memory issues (this will notably send your **pipeline to CPU** and activate xformers memory efficient attention)
## Inference nodes
Similarly as for pipelines, under "Diffusers-in-Comfy/Inference", you will find one node per type of inference pipeline you want to use : **GenerateTxt2Image**, **GenerateInpaintImage**, or **GenerateImg2Image**. The three of them have common options you can tweak:
@@ -76,6 +78,25 @@ Then you have specific inputs per node.
**GenerateImg2Image** : In this node, you must write the path to an input image to use as a prompt. Be careful to delete all quotes in the path.
### Some visual examples of the entire workflows
**Txt2Image Workflow:**
![screenshot](examples/Txt2Image_1.5_Generic_Screenshot.png)
**Inpainting Workflow:**
![screenshot](examples/Inpainting_SDXL_Generic_Screenshot.png)
**Img2Img Workflow:**
![screenshot](examples/Img2Img_1.5_Generic_Screenshot.png)
## Utils nodes
Under "Diffusers-in-Comfy/Utils", you will find nodes that will allow you to make different operations, such as processing images. For now, only one is available : **Make Canny**. It takes in an image, transforms it into a canny, and then you can connect the output canny to the "controlnet_image" input of one of the Inference nodes. You also have a second output "Preview" that you can connect to Comfy UI's "Preview Image" to see how your canny looks.
@@ -84,8 +105,20 @@ Under "Diffusers-in-Comfy/Components", you will find Diffusers components and ad
**LoraLoader** : This node allows you to load a LoRA into the UNET and define its weight. Don't forget to write the triggerword into the prompt of the Inference node.
**Example of LoRA workflow**
![screenshot](examples/Txt2Image_1.5_LoRA_Screenshot.png)
**Blora** : This node is the reason why I originally created this project. It implements a technique that allows for separating an image into its style and content. It acts as a LoRA but works a bit differently (it updates only two blocks of the Unet). Here's the [original repo](https://github.com/yardenfren1996/B-LoRA). Keep in mind that although a B-LoRA works well with a ControlNet, it cannot be used with a classic LoRA. Indeed, the classic LoRA will completely dilute the B-LoRA, supplanting it into the Unet. Also, B-LoRAS work **only** with SDXL models, so make sure you load a SDXL Pipeline and put the is_sdxl flag of the Pipeline node to "True"!
To load your B-LoRA:
- Put it inside the "lora" folder of your ComfyUI installation
- Inside the node, in **style_lora_name** and/or **content_lora_scale**, write the name + extension of your B-LoRA (ex: vangogh.safetensors).
**Example of B-LoRA workflow**
![screenshot](examples/BLoRA_Screenshot.png)
# To do
There are quite a few things I'd like to develop further, so here's a very basic roadmap
examples/BLoRA.json
+199
View File
@@ -0,0 +1,199 @@
{
"last_node_id": 6,
"last_link_id": 4,
"nodes": [
{
"id": 5,
"type": "PreviewImage",
"pos": [
-2041.0313720703125,
733.3960571289062
],
"size": [
210,
246
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 2
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 1,
"type": "Text2ImgStableDiffusionPipeline",
"pos": [
-3376.4130859375,
578.3900756835938
],
"size": [
390.5999755859375,
154
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
3
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Text2ImgStableDiffusionPipeline"
},
"widgets_values": [
true,
true,
"sd_xl_base_1.0.safetensors",
"",
""
]
},
{
"id": 4,
"type": "GenerateTxt2Image",
"pos": [
-2533.815185546875,
681.9490356445312
],
"size": [
400,
314
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 4
},
{
"name": "controlnet_image",
"type": "IMAGE",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "Result",
"type": "IMAGE",
"links": [
2
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "GenerateTxt2Image"
},
"widgets_values": [
42,
"fixed",
"[v2] style, a cat sleeping on a pillow",
"",
49,
1024,
1024,
5,
0.5
]
},
{
"id": 6,
"type": "BLoRALoader",
"pos": [
-2905.48681640625,
634.6033325195312
],
"size": [
352.2727355957031,
130
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 3
}
],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
4
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "BLoRALoader"
},
"widgets_values": [
"vangogh.safetensors",
1.1,
"",
0
]
}
],
"links": [
[
2,
4,
0,
5,
0,
"IMAGE"
],
[
3,
1,
0,
6,
0,
"PIPELINE"
],
[
4,
6,
0,
4,
0,
"PIPELINE"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.1,
"offset": [
3448.213506348067,
-454.2056709686145
]
}
},
"version": 0.4
}
examples/BLoRA_ControlNet.json
+284
View File
@@ -0,0 +1,284 @@
{
"last_node_id": 9,
"last_link_id": 6,
"nodes": [
{
"id": 5,
"type": "PreviewImage",
"pos": [
-2041.0313720703125,
733.3960571289062
],
"size": [
210,
246
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 2
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewImage"
},
"widgets_values": []
},
{
"id": 9,
"type": "PreviewImage",
"pos": [
-3078.213623046875,
1055.512451171875
],
"size": [
210,
246
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 5
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 7,
"type": "MakeCanny",
"pos": [
-3590.941650390625,
848.2393798828125
],
"size": [
400,
200
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "Canny",
"type": "IMAGE",
"links": [
6
],
"slot_index": 0
},
{
"name": "Canny Preview",
"type": "IMAGE",
"links": [
5
],
"slot_index": 1
}
],
"properties": {
"Node name for S&R": "MakeCanny"
},
"widgets_values": [
"/home/maelys/Downloads/hf-logo.png",
200,
100
]
},
{
"id": 6,
"type": "BLoRALoader",
"pos": [
-3020.941162109375,
602.78515625
],
"size": [
315,
130
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 3
}
],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
4
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "BLoRALoader"
},
"widgets_values": [
"vangogh.safetensors",
1.1,
"",
0
]
},
{
"id": 1,
"type": "Text2ImgStableDiffusionPipeline",
"pos": [
-3585.504150390625,
575.6626586914062
],
"size": [
390.5999755859375,
154
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
3
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Text2ImgStableDiffusionPipeline"
},
"widgets_values": [
true,
true,
"sd_xl_base_1.0.safetensors",
"",
"diffusers/controlnet-canny-sdxl-1.0"
]
},
{
"id": 4,
"type": "GenerateTxt2Image",
"pos": [
-2533.815185546875,
681.9490356445312
],
"size": [
400,
314
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 4
},
{
"name": "controlnet_image",
"type": "IMAGE",
"link": 6,
"shape": 7
}
],
"outputs": [
{
"name": "Result",
"type": "IMAGE",
"links": [
2
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "GenerateTxt2Image"
},
"widgets_values": [
42,
"fixed",
"A cat sleeping on a pillow",
"",
49,
1024,
1024,
5,
0.5
]
}
],
"links": [
[
2,
4,
0,
5,
0,
"IMAGE"
],
[
3,
1,
0,
6,
0,
"PIPELINE"
],
[
4,
6,
0,
4,
0,
"PIPELINE"
],
[
5,
7,
1,
9,
0,
"IMAGE"
],
[
6,
7,
0,
4,
1,
"IMAGE"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 0.9090909090909091,
"offset": [
3705.3316881662513,
-443.53294369588775
]
}
},
"version": 0.4
}
examples/BLoRA_Controlnet_Screenshot.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 283 KiB

examples/BLoRA_Screenshot.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 232 KiB

examples/Img2Img_1.5_Generic.json
+152
View File
@@ -0,0 +1,152 @@
{
"last_node_id": 13,
"last_link_id": 6,
"nodes": [
{
"id": 5,
"type": "PreviewImage",
"pos": [
-2207.395263671875,
736.1233520507812
],
"size": [
210,
246
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 6
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewImage"
},
"widgets_values": []
},
{
"id": 12,
"type": "Img2ImgStableDiffusionPipeline",
"pos": [
-3260.472900390625,
657.1368408203125
],
"size": [
378,
154
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
5
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Img2ImgStableDiffusionPipeline"
},
"widgets_values": [
false,
true,
"sd-v1-5-inpainting.ckpt",
"",
""
]
},
{
"id": 13,
"type": "GenerateImg2Image",
"pos": [
-2733.19921875,
645.3186645507812
],
"size": [
400,
364
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 5
},
{
"name": "controlnet_image",
"type": "IMAGE",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "Result",
"type": "IMAGE",
"links": [
6
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "GenerateImg2Image"
},
"widgets_values": [
0,
"randomize",
"a cat in a sofa",
"",
50,
512,
512,
5,
"Path to your input image (ex : home/downloads/image.jpeg)",
0.5
]
}
],
"links": [
[
5,
12,
0,
13,
0,
"PIPELINE"
],
[
6,
13,
0,
5,
0,
"IMAGE"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.1,
"offset": [
3371.381914332155,
-498.557334779669
]
}
},
"version": 0.4
}
examples/Img2Img_1.5_Generic_Screenshot.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

examples/Inpainting_SDXL_Generic.json
+154
View File
@@ -0,0 +1,154 @@
{
"last_node_id": 11,
"last_link_id": 4,
"nodes": [
{
"id": 5,
"type": "PreviewImage",
"pos": [
-2207.395263671875,
736.1233520507812
],
"size": [
210,
246
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 4
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewImage"
},
"widgets_values": []
},
{
"id": 6,
"type": "InpaintingStableDiffusionPipeline",
"pos": [
-3286.837158203125,
615.3182373046875
],
"size": [
415.8000183105469,
154
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
3
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "InpaintingStableDiffusionPipeline"
},
"widgets_values": [
true,
true,
"Juggernaut-XL_v9_RunDiffusionPhoto_v2.safetensors",
"",
""
]
},
{
"id": 7,
"type": "GenerateInpaintImage",
"pos": [
-2749.56396484375,
593.50048828125
],
"size": [
400,
438
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 3
},
{
"name": "controlnet_image",
"type": "IMAGE",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "Result",
"type": "IMAGE",
"links": [
4
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "GenerateInpaintImage"
},
"widgets_values": [
42,
"fixed",
"a cat sleeping on a cushion",
"",
50,
1024,
1024,
5,
"/home/maelys/Downloads/pexels-photo-1170986.jpeg",
"/home/maelys/Downloads/gettyimages-1193097080-640x640.jpg",
true,
0.5
]
}
],
"links": [
[
3,
6,
0,
7,
0,
"PIPELINE"
],
[
4,
7,
0,
5,
0,
"IMAGE"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.1,
"offset": [
3430.4728234230647,
-480.3755165978507
]
}
},
"version": 0.4
}
examples/Inpainting_SDXL_Generic_Screenshot.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 95 KiB

examples/Txt2Image_1.5_Generic.json
+151
View File
@@ -0,0 +1,151 @@
{
"last_node_id": 5,
"last_link_id": 2,
"nodes": [
{
"id": 5,
"type": "PreviewImage",
"pos": [
-2207.395263671875,
736.1233520507812
],
"size": [
210,
246
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 2
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewImage"
},
"widgets_values": []
},
{
"id": 1,
"type": "Text2ImgStableDiffusionPipeline",
"pos": [
-3180.95703125,
694.753662109375
],
"size": [
390.5999755859375,
154
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
1
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Text2ImgStableDiffusionPipeline"
},
"widgets_values": [
false,
true,
"v1-5-pruned-emaonly.safetensors",
"",
""
]
},
{
"id": 4,
"type": "GenerateTxt2Image",
"pos": [
-2693.815185546875,
697.4036865234375
],
"size": [
400,
314
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 1
},
{
"name": "controlnet_image",
"type": "IMAGE",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "Result",
"type": "IMAGE",
"links": [
2
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "GenerateTxt2Image"
},
"widgets_values": [
42,
"fixed",
"A cat sleeping on a pillow",
"",
20,
512,
512,
5,
0.5
]
}
],
"links": [
[
1,
1,
0,
4,
0,
"PIPELINE"
],
[
2,
4,
0,
5,
0,
"IMAGE"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.1,
"offset": [
3369.563732513973,
-497.6482438705781
]
}
},
"version": 0.4
}
examples/Txt2Image_1.5_Generic_Screenshot.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

examples/Txt2Image_1.5_LoRA.json
+197
View File
@@ -0,0 +1,197 @@
{
"last_node_id": 7,
"last_link_id": 7,
"nodes": [
{
"id": 5,
"type": "PreviewImage",
"pos": [
-2041.0313720703125,
733.3960571289062
],
"size": [
210,
246
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 2
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 4,
"type": "GenerateTxt2Image",
"pos": [
-2533.815185546875,
681.9490356445312
],
"size": [
400,
314
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 7
},
{
"name": "controlnet_image",
"type": "IMAGE",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "Result",
"type": "IMAGE",
"links": [
2
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "GenerateTxt2Image"
},
"widgets_values": [
42,
"fixed",
"emiregiw3, a man looking right at the camera, handsome, wise",
"",
20,
512,
512,
5,
0.5
]
},
{
"id": 7,
"type": "LoRALoader",
"pos": [
-2922.759033203125,
620.9671630859375
],
"size": [
315,
82
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 6
}
],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
7
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "LoRALoader"
},
"widgets_values": [
"emiregiw3.safetensors",
0.8
]
},
{
"id": 1,
"type": "Text2ImgStableDiffusionPipeline",
"pos": [
-3376.4130859375,
578.3900756835938
],
"size": [
390.5999755859375,
154
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
6
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Text2ImgStableDiffusionPipeline"
},
"widgets_values": [
false,
true,
"v1-5-pruned-emaonly.safetensors",
"",
""
]
}
],
"links": [
[
2,
4,
0,
5,
0,
"IMAGE"
],
[
6,
1,
0,
7,
0,
"PIPELINE"
],
[
7,
7,
0,
4,
0,
"PIPELINE"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.1,
"offset": [
3412.7589608935236,
-460.56930733225107
]
}
},
"version": 0.4
}
examples/Txt2Image_1.5_LoRA_Screenshot.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 225 KiB

examples/Txt2Image_SDXL_Generic.json
+150
View File
@@ -0,0 +1,150 @@
{
"last_node_id": 5,
"last_link_id": 2,
"nodes": [
{
"id": 5,
"type": "PreviewImage",
"pos": [
-2207.395263671875,
736.1233520507812
],
"size": [
210,
246
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 2
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 1,
"type": "Text2ImgStableDiffusionPipeline",
"pos": [
-3180.95703125,
694.753662109375
],
"size": [
390.5999755859375,
154
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "PIPELINE",
"type": "PIPELINE",
"links": [
1
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Text2ImgStableDiffusionPipeline"
},
"widgets_values": [
true,
true,
"sd_xl_base_1.0.safetensors",
"",
""
]
},
{
"id": 4,
"type": "GenerateTxt2Image",
"pos": [
-2693.815185546875,
697.4036865234375
],
"size": [
400,
314
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "pipeline",
"type": "PIPELINE",
"link": 1
},
{
"name": "controlnet_image",
"type": "IMAGE",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "Result",
"type": "IMAGE",
"links": [
2
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "GenerateTxt2Image"
},
"widgets_values": [
42,
"fixed",
"A cat sleeping on a pillow",
"",
49,
1024,
1024,
5,
0.5
]
}
],
"links": [
[
1,
1,
0,
4,
0,
"PIPELINE"
],
[
2,
4,
0,
5,
0,
"IMAGE"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.1,
"offset": [
3347.3044154389777,
-452.3874891504327
]
}
},
"version": 0.4
}