Adding more demonstration workflows
This commit is contained in:
Yahweasel
@@ -1,9 +1,10 @@
|
||||
# HuggingFace Diffusers for ComfyUI
|
||||
# HuggingFace Diffusers (and Transformers) for ComfyUI
|
||||
|
||||
A lot of custom nodes for ComfyUI are really just bindings to HuggingFace
|
||||
Diffusers, but overly constrained to use them in very particular ways. This set
|
||||
of custom nodes is intended to be a generic set of bindings for HuggingFace
|
||||
Diffusers, in theory allowing you to use any pipeline supported by HuggingFace.
|
||||
Diffusers and/or Transformers, but overly constrained to use them in very
|
||||
particular ways. This set of custom nodes is intended to be a generic set of
|
||||
bindings for HuggingFace Diffusers and Transformers, in theory allowing you to
|
||||
use any pipeline supported by HuggingFace.
|
||||
|
||||
It also supports bitsandbytes quantization, automatic device mapping, and the
|
||||
other advantages unique to HuggingFace.
|
||||
@@ -17,7 +18,9 @@ I recommend
|
||||
## Philosophy
|
||||
|
||||
I've tried to make everything work fairly generically, as unopinionated as I
|
||||
can manage.
|
||||
can manage. There are no nodes for specialized tasks. For example, while this
|
||||
node set is capable of being used for prompt enhancement, there is no “prompt
|
||||
enhancement” node, because that's simply built from a text-to-text pipeline.
|
||||
|
||||
Every node that loads a component has some default class that it loads, but you
|
||||
can replace the class simply by its name. For instance, the pipeline loader
|
||||
@@ -42,23 +45,18 @@ set them to `null` (`None`).
|
||||
These are some example workflows, in simple and exploded forms (where by
|
||||
“exploded” I mean “each step done separately”).
|
||||
|
||||
[GLM-Image](workflows/hf_glm_image.json) ([exploded])(workflows/hf_glm_image_exploded.json))
|
||||
[GLM-Image](workflows/hf_glm_image.json) ([exploded](workflows/hf_glm_image_exploded.json))
|
||||
(Note: As of the writing of this README, requires the git version of both Diffusers and Transformers.)
|
||||
|
||||
|
||||
|
||||
|
||||
[LongCat Image](workflows/hf_longcat_image.json) ([exploded](workflows/hf_longcat_image_exploded.json))
|
||||
(Note: Requires [my PR](https://github.com/huggingface/diffusers/pull/12963) to
|
||||
* [Prompt enhancer](workflows/hf_prompt_enhancer.json) ([image](workflows/hf_prompt_enhancer.webp))
|
||||
* [LongCat Image](workflows/hf_longcat_image.json) ([exploded](workflows/hf_longcat_image_exploded.json), [image](workflows/hf_longcat_image.webp), [image exploded](workflows/hf_longcat_image_exploded.webp))
|
||||
* (Note: Requires [my PR](https://github.com/huggingface/diffusers/pull/12963) to
|
||||
use quantization.)
|
||||
|
||||
|
||||
|
||||
|
||||
[SDXL](workflows/hf_sdxl.json) ([exploded](workflows/hf_sdxl_exploded.json))
|
||||
|
||||
|
||||
|
||||
* [SDXL](workflows/hf_sdxl.json) ([exploded](workflows/hf_sdxl_exploded.json), [image](workflows/hf_sdxl.webp), [image exploded](workflows/hf_sdxl_exploded.webp))
|
||||
* [AI trash converter](workflows/hf_make_ai_trash.json) ([image](workflows/hf_make_ai_trash.webp))
|
||||
|
||||
|
||||
## Nodes
|
||||
@@ -131,3 +129,28 @@ Some examples:
|
||||
|
||||
Use these to decode/encode HuggingFace latents using HuggingFace VAEs.
|
||||
Automatic if pipelines are run in PIL mode.
|
||||
|
||||
### HF Transformers load pipeline
|
||||
|
||||
Load a HuggingFace *Transformers* (not Diffusers) pipeline. Useful for using an
|
||||
LLM or VLM as part of a larger workflow, such as prompt enhancement or image
|
||||
description.
|
||||
|
||||
### HF Transformers create conversation
|
||||
|
||||
Create or extend a conversation in the style used by HuggingFace Transformers.
|
||||
Can include text or images. To create a multi-turn conversation, create
|
||||
multiple “create conversation” nodes and chain them together.
|
||||
|
||||
### HF Transformers unpack conversation
|
||||
|
||||
Unpack a part of a conversation in HuggingFace Transformers format. Mostly
|
||||
useful for unpacking the output of a Transformers pipeline.
|
||||
|
||||
### HF Transformers run pipeline
|
||||
|
||||
Run a HuggingFace Transformers pipeline. Takes anything as input, but for most
|
||||
pipelines, you want a conversation as input. Outputs the raw result, and if a
|
||||
conversation was generated, also outputs the extracted conversation. In most
|
||||
cases, you will want to pass that conversation into an “unpack conversation”
|
||||
node.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 47 KiB |
@@ -0,0 +1,690 @@
|
||||
{
|
||||
"id": "ec755963-7a99-4853-a631-e26edbd62658",
|
||||
"revision": 0,
|
||||
"last_node_id": 14,
|
||||
"last_link_id": 15,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 6,
|
||||
"type": "HFTUnpackConversation",
|
||||
"pos": [
|
||||
1750,
|
||||
290
|
||||
],
|
||||
"size": [
|
||||
353.3333343505859,
|
||||
98
|
||||
],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "conversation",
|
||||
"type": "HFT_CONVERSATION",
|
||||
"link": 4
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "role",
|
||||
"type": "STRING",
|
||||
"links": null
|
||||
},
|
||||
{
|
||||
"name": "text",
|
||||
"type": "STRING",
|
||||
"links": [
|
||||
5,
|
||||
6
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "image",
|
||||
"type": "PIL_IMAGE",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "HFTUnpackConversation"
|
||||
},
|
||||
"widgets_values": [
|
||||
-1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"type": "ConditioningZeroOut",
|
||||
"pos": [
|
||||
2180,
|
||||
520
|
||||
],
|
||||
"size": [
|
||||
204.0999984741211,
|
||||
26
|
||||
],
|
||||
"flags": {},
|
||||
"order": 9,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "conditioning",
|
||||
"type": "CONDITIONING",
|
||||
"link": 10
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
11
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "ConditioningZeroOut"
|
||||
},
|
||||
"widgets_values": []
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "EmptyLatentImage",
|
||||
"pos": [
|
||||
2110,
|
||||
590
|
||||
],
|
||||
"size": [
|
||||
270,
|
||||
106
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
12
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "EmptyLatentImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
1152,
|
||||
896,
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"type": "SaveImage",
|
||||
"pos": [
|
||||
2840,
|
||||
280
|
||||
],
|
||||
"size": [
|
||||
270,
|
||||
270
|
||||
],
|
||||
"flags": {},
|
||||
"order": 12,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 15
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"ComfyUI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "VAEDecode",
|
||||
"pos": [
|
||||
2690,
|
||||
280
|
||||
],
|
||||
"size": [
|
||||
140,
|
||||
46
|
||||
],
|
||||
"flags": {},
|
||||
"order": 11,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "samples",
|
||||
"type": "LATENT",
|
||||
"link": 13
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": 14
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
15
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VAEDecode"
|
||||
},
|
||||
"widgets_values": []
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"type": "KSampler",
|
||||
"pos": [
|
||||
2410,
|
||||
280
|
||||
],
|
||||
"size": [
|
||||
270,
|
||||
474
|
||||
],
|
||||
"flags": {},
|
||||
"order": 10,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "model",
|
||||
"type": "MODEL",
|
||||
"link": 8
|
||||
},
|
||||
{
|
||||
"name": "positive",
|
||||
"type": "CONDITIONING",
|
||||
"link": 9
|
||||
},
|
||||
{
|
||||
"name": "negative",
|
||||
"type": "CONDITIONING",
|
||||
"link": 11
|
||||
},
|
||||
{
|
||||
"name": "latent_image",
|
||||
"type": "LATENT",
|
||||
"link": 12
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
13
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "KSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
1,
|
||||
"fixed",
|
||||
20,
|
||||
7,
|
||||
"euler",
|
||||
"simple",
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "CheckpointLoaderSimple",
|
||||
"pos": [
|
||||
1820,
|
||||
500
|
||||
],
|
||||
"size": [
|
||||
270,
|
||||
98
|
||||
],
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MODEL",
|
||||
"type": "MODEL",
|
||||
"links": [
|
||||
8
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "CLIP",
|
||||
"type": "CLIP",
|
||||
"links": [
|
||||
7
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "VAE",
|
||||
"type": "VAE",
|
||||
"links": [
|
||||
14
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CheckpointLoaderSimple"
|
||||
},
|
||||
"widgets_values": [
|
||||
"juggernautXL_juggXIByRundiffusion.safetensors"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "HFTCreateConversation",
|
||||
"pos": [
|
||||
990,
|
||||
530
|
||||
],
|
||||
"size": [
|
||||
400,
|
||||
200
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "previous",
|
||||
"shape": 7,
|
||||
"type": "HFT_CONVERSATION",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "image",
|
||||
"shape": 7,
|
||||
"type": "PIL_IMAGE",
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "HFT_CONVERSATION",
|
||||
"type": "HFT_CONVERSATION",
|
||||
"links": [
|
||||
1
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "HFTCreateConversation"
|
||||
},
|
||||
"widgets_values": [
|
||||
"system",
|
||||
"You are an expert in writing prompts for AI image generation. Given a user's raw input prompt, expand it into a detailed image generation prompt with specific details to guide a text-to-image model.\n\n## Requirements\n\nStrictly follow all aspects of the user's raw input. Include every element requested (style and content). If the input is vague, invent concrete details, such as lighting, textures, materials, surrounding scene, etc. For characters, describe gender, clothing, hair, expressions, etc. Do not invent unrequested characters.\n\nInclude an overall visual style at the end, such as \"<rest of prompt>. Style: <style>\". If the prompt already describes the style, use it. Otherwise, infer the style from the prompt. Omit if unclear.\n\nDo not include any non-visual cues, such as sound, music, smell, taste, touch, etc.\n\nAvoid dramatic or exaggerated terms. Use mild, natural phrasing.\n\nIf the user's raw input prompt is already highly detailed and in the requested format, do not make major edits or introduce new elements.\n\n## Format\n\nA single, continuous paragraph in natural language (English). No titles, headings, prefaces, code fences, Markdown, etc.\n\nDo not use phrases like \"The image includes...\". Start directly with the description.\n\n## Example\n\nInput: \"Women laughing and eating salad\"\n\nOutput:\nTwo women are sitting at a rustic wooden table in a sunlit backyard garden, both smiling and laughing. The woman on the left has long, wavy brown hair, and is wearing a casual light gray t-shirt with a subtle floral pattern and loose-fitting jeans. Her expression is joyful, with her head tilted slightly as she laughs, her eyes crinkling at the corners. The woman on the right has short, curly red hair, and is dressed in a bright pink t-shirt and a pair of denim shorts. She is holding a fork and knife, cutting a piece of crisp green lettuce, and is laughing heartily, her mouth open in a wide smile. The table is covered with a white cloth and has a few colorful salad ingredients: a large bunch of fresh green lettuce, a small red tomato, and a few slices of cucumber. The background features a wooden fence, a few potted plants, and a wooden bench. The lighting is soft and natural, with sunlight filtering through the trees, creating gentle shadows. The overall scene is bright and cheerful, with a sense of relaxed happiness. Style: Warm, natural lighting with soft shadows, realistic textures, and a slightly vintage feel."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "CLIPTextEncode",
|
||||
"pos": [
|
||||
2180,
|
||||
290
|
||||
],
|
||||
"size": [
|
||||
400,
|
||||
200
|
||||
],
|
||||
"flags": {
|
||||
"collapsed": true
|
||||
},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "clip",
|
||||
"type": "CLIP",
|
||||
"link": 7
|
||||
},
|
||||
{
|
||||
"name": "text",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "text"
|
||||
},
|
||||
"link": 6
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
9,
|
||||
10
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CLIPTextEncode"
|
||||
},
|
||||
"widgets_values": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "PreviewAny",
|
||||
"pos": [
|
||||
1890,
|
||||
80
|
||||
],
|
||||
"size": [
|
||||
210,
|
||||
166
|
||||
],
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "source",
|
||||
"type": "*",
|
||||
"link": 5
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"properties": {
|
||||
"Node name for S&R": "PreviewAny"
|
||||
},
|
||||
"widgets_values": [
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "HFTCreateConversation",
|
||||
"pos": [
|
||||
1400,
|
||||
530
|
||||
],
|
||||
"size": [
|
||||
400,
|
||||
200
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "previous",
|
||||
"shape": 7,
|
||||
"type": "HFT_CONVERSATION",
|
||||
"link": 1
|
||||
},
|
||||
{
|
||||
"name": "image",
|
||||
"shape": 7,
|
||||
"type": "PIL_IMAGE",
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "HFT_CONVERSATION",
|
||||
"type": "HFT_CONVERSATION",
|
||||
"links": [
|
||||
3
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "HFTCreateConversation"
|
||||
},
|
||||
"widgets_values": [
|
||||
"user",
|
||||
"Women laughing and eating salad"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "HFTLoadPipeline",
|
||||
"pos": [
|
||||
990,
|
||||
290
|
||||
],
|
||||
"size": [
|
||||
299.3666595458984,
|
||||
202
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "HFT_PIPELINE",
|
||||
"type": "HFT_PIPELINE",
|
||||
"links": [
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "HFTLoadPipeline"
|
||||
},
|
||||
"widgets_values": [
|
||||
"text-to-text",
|
||||
"",
|
||||
"Qwen/Qwen3-VL-2B-Instruct",
|
||||
"default",
|
||||
"default",
|
||||
"{\"attn_implementation\": \"flash_attention_2\"}",
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "HFTRunPipeline",
|
||||
"pos": [
|
||||
1440,
|
||||
290
|
||||
],
|
||||
"size": [
|
||||
292.8333343505859,
|
||||
126
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "pipeline",
|
||||
"type": "HFT_PIPELINE",
|
||||
"link": 2
|
||||
},
|
||||
{
|
||||
"name": "input",
|
||||
"shape": 7,
|
||||
"type": "*",
|
||||
"link": 3
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "raw_result",
|
||||
"type": "HFT_RESULT",
|
||||
"links": null
|
||||
},
|
||||
{
|
||||
"name": "conversation",
|
||||
"type": "HFT_CONVERSATION",
|
||||
"links": [
|
||||
4
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "HFTRunPipeline"
|
||||
},
|
||||
"widgets_values": [
|
||||
1,
|
||||
"fixed",
|
||||
""
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
3,
|
||||
0,
|
||||
4,
|
||||
0,
|
||||
"HFT_CONVERSATION"
|
||||
],
|
||||
[
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
5,
|
||||
0,
|
||||
"HFT_PIPELINE"
|
||||
],
|
||||
[
|
||||
3,
|
||||
4,
|
||||
0,
|
||||
5,
|
||||
1,
|
||||
"HFT_CONVERSATION"
|
||||
],
|
||||
[
|
||||
4,
|
||||
5,
|
||||
1,
|
||||
6,
|
||||
0,
|
||||
"HFT_CONVERSATION"
|
||||
],
|
||||
[
|
||||
5,
|
||||
6,
|
||||
1,
|
||||
7,
|
||||
0,
|
||||
"STRING"
|
||||
],
|
||||
[
|
||||
6,
|
||||
6,
|
||||
1,
|
||||
9,
|
||||
1,
|
||||
"STRING"
|
||||
],
|
||||
[
|
||||
7,
|
||||
8,
|
||||
1,
|
||||
9,
|
||||
0,
|
||||
"CLIP"
|
||||
],
|
||||
[
|
||||
8,
|
||||
8,
|
||||
0,
|
||||
10,
|
||||
0,
|
||||
"MODEL"
|
||||
],
|
||||
[
|
||||
9,
|
||||
9,
|
||||
0,
|
||||
10,
|
||||
1,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
10,
|
||||
9,
|
||||
0,
|
||||
11,
|
||||
0,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
11,
|
||||
11,
|
||||
0,
|
||||
10,
|
||||
2,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
12,
|
||||
12,
|
||||
0,
|
||||
10,
|
||||
3,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
13,
|
||||
10,
|
||||
0,
|
||||
13,
|
||||
0,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
14,
|
||||
8,
|
||||
2,
|
||||
13,
|
||||
1,
|
||||
"VAE"
|
||||
],
|
||||
[
|
||||
15,
|
||||
13,
|
||||
0,
|
||||
14,
|
||||
0,
|
||||
"IMAGE"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"ds": {
|
||||
"scale": 0.8142056074766354,
|
||||
"offset": [
|
||||
-870.9366391184572,
|
||||
156.21280991735534
|
||||
]
|
||||
},
|
||||
"workflowRendererVersion": "LG",
|
||||
"frontendVersion": "1.35.9",
|
||||
"VHS_latentpreview": false,
|
||||
"VHS_latentpreviewrate": 0,
|
||||
"VHS_MetadataImage": true,
|
||||
"VHS_KeepIntermediate": true
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 60 KiB |
Reference in New Issue
Block a user