nodes/asagi4-comfyui-prompt-control
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Reorganize docs

Browse Source
This commit is contained in:
asagi4
2024-12-12 00:27:33 +02:00
parent eab2cc09dd
commit d71ec8d86a
2 changed files with 87 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+47 -84
View File
@@ -4,33 +4,37 @@ Nodes for LoRA and prompt scheduling that make basic operations in ComfyUI compl
LoRA and prompt scheduling should produce identical output to the equivalent ComfyUI workflow using multiple samplers or the various conditioning manipulation nodes. If you find situations where this is not the case, please report a bug.
## Note about stability
## Prompt Control v2
The nodes are currently undergoing some refactoring and rewriting.
Prompt control has been almost completely rewritten.
Interfaces for nodes marked "v2" are subject to breaking change at any time, though at worst you'll likely just have to recreate a few nodes.
It now uses ComfyUI's lazy execution to build graphs from the text prompt at runtime; this means that its output can be exactly equivalent to using native ComfyUI nodes (because that's what it does)
Nodes marked "experimental" are very likely to change or disappear completely, or might not even work completely.
In addition to building graphs, `PCTextEncode` provides an advanced text encoding node with many additional features compared to ComfyUI's base `CLIPTextEncode`.
## What can it do?
See [features](#features) below. Things you can control via the prompt:
- Weight interpretation types (comfy, A1111, etc.)
- Prompt editing and filtering without multiple samplers
- Prompt editing and filtering without noodle soup
- LoRA loading and scheduling via ComfyUI's hook system
- Masking, compositions and area control, combining prompts (`AND`), `BREAK`
- Prompt masking with cutoff
- SDXL parameters
- Masking, composition and area control (regional prompting)
- Prompt operations like `BREAK` and `AND`
- Weight interpretation types (comfy, A1111, etc.)
- Prompt masking with [cutoff](#cutoff)
- And a bunch more
If you find prompt scheduling inconvenient, `PCTextEncode` can be used as a drop-in replacement for `CLIPTextEncode` to get everything else.
See the [syntax documentation](doc/syntax.md)
If you find prompt scheduling inconvenient for some reason, `PCTextEncode` can be used as a drop-in replacement for `CLIPTextEncode` to get everything else.
[This example workflow](workflows/example.json?raw=1) implements a two-pass workflow illustrating most scheduling features.
The tools in this repository combine well with the macro and wildcard functionality in [comfyui-utility-nodes](https://github.com/asagi4/comfyui-utility-nodes)
## Requirements
For `PCEncodeSchedule` and `PCLoraHooksFromSchedule`, you'll need at least version 0.3.7 of ComfyUI (0.3.36 of ComfyUI desktop).
For LoRA scheduling to work, you'll need at least version 0.3.7 of ComfyUI (0.3.36 of ComfyUI desktop).
You need to have `lark` installed in your Python environment for parsing to work (If you reuse A1111's venv, it'll already be there).
@@ -41,26 +45,47 @@ If you use the portable version of ComfyUI on Windows with its embedded Python,
Then restart ComfyUI afterwards.
## Notable changes
# Core nodes
I try to avoid behavioural changes that break old prompts, but they may happen occasionally.
## PCLazyTextEncode
- 2024-12-03 ComfyUI merged support for model/conditioning hooks. There are two new nodes, `PCEncodeSchedule` and `PCLoraHooksFromSchedule` that can be used in combination with the hook nodes. Some functionality is still missing from them, but going forward, these nodes will be the only nodes supported; **I will not spend significant time fixing bugs in the old monkeypatched nodes anymore.**
`PCLazyTextEncode` uses ComfyUI's lazy graph execution mechanism to generate a graph of `PCTextEncode` and `SetConditioningTimestepRange` nodes from a prompt with schedules. This has the advantage that if a part of the schedule doesn't change, ComfyUI's caching mechanism allows you to avoid re-encoding the non-changed part.
## Note on how schedules work
for example, if you first encode `[cat:dog:0.1]` and later change that to `[cat:dog:0.5]`, no re-encoding takes place.
for added fun, put `NODE(NodeClassName, paramname)` in a prompt to generate a graph using **any other node** that's compatible. The node can't have required parameters besides a single CLIP parameter (which must be named `clip`) and the text prompt, and it must return a `CONDITIONING` as its first return value.
## PCLazyLoraLoader
This node reads LoRA expressions from the scheduled prompt and constructs a graph of `LoraLoader`s and `CreateHookLora`s as necessary to provide the necessary LoRA scheduling.
If you have `apply_hooks` set to true, you **do not** need to apply the `HOOKS` output to a CLIP model separately; it's provided in case you want to use it elsewhere.
## PCTextEncode
Encodes a single prompt with advanced (non-scheduling) syntax enabled. This is what actually does most of the work under the hood.
Note: `PCTextEncode` **does not** ignore `<lora:...:1>` and will treat it as part of the prompt. To use a combined prompt for LoRAs and your input, use `PCLazyTextEncode` and `PCLazyLoraLoader`
For the rest, see [schedule nodes](doc/schedule.md)
ComfyUI does not use the step number to determine whether to apply conds; instead, it uses the sampler's timestep value which is affected by the scheduler you're using. This means that when the sampler scheduler isn't linear, the schedules generated by prompt control will not be either.
# Features
## Scheduling and LoRA loading
Prompt control provides a way to easily schedule different prompts and control LoRA loading.
See the [syntax documentation](doc/syntax.md)
### Note on how schedules work
ComfyUI does not use the step number to determine whether to apply conds; instead, it uses the sampler's timestep value which is affected by the scheduler you're using. This means that when the sampler scheduler isn't linear, the schedules generated by prompt control will not be either.
## Advanced CLIP encoding
If you use `PCEncodeSchedule` or `PCTextEncode`. advanced encodings are available automatically. Thanks to BlenderNeko for the original code.
If you use `PCTextEncode`. advanced encodings are available automatically. Thanks to BlenderNeko for the original code.
You can use the syntax `STYLE(weight_interpretation, normalization)` in a prompt to affect how prompts are interpreted.
Use the syntax `STYLE(weight_interpretation, normalization)` in a prompt to affect how prompts are interpreted.
The weight interpretations available are:
- comfy (default)
@@ -92,7 +117,9 @@ For things (ie. the code imports) to work, the nodes must be cloned in a directo
## Cutoff
`PCEncodeSchedule` reimplements cutoff from [ComfyUI Cutoff](https://github.com/BlenderNeko/ComfyUI_Cutoff). You do not need to have the nodes installed.
NOTE: Cutoff syntax might change at some point; it's pretty clunky.
`PCTextEncode` reimplements cutoff from [ComfyUI Cutoff](https://github.com/BlenderNeko/ComfyUI_Cutoff).
The syntax is
```
@@ -112,70 +139,6 @@ If `strict_mask`, `start_from_masked` or `padding_token` are specified in more t
The parameters affect how the masked and unmasked prompts are combined to produce the final embedding. Just play around with them.
# Nodes
## PCLoraHooksFromSchedule
Creates a ComfyUI `HOOKS` object from a prompt schedule. Can be attached to a CLIP model to perform encoding and LoRA switching
The hooks can be a bit slow sometimes, especially in cases where the LoRA spec doesn't actually require reloading the patches every time you sample. Try `PCHooksFromScheduleWithOptimizationTest`. (Though that node is likely to go away eventually).
## PCEncodeSchedule
Encodes all prompts in a schedule. Pass in a `CLIP` object with hooks attached for LoRA scheduling, then use the resulting `CONDITIONING` normally
## PCTextEncode
Encodes a single prompt *without* scheduling or LoRA loading features (but including everything else).
Note: This does not currently ignore `<lora:...:1>` and will treat it as part of the prompt.
## PromptToSchedule
Parses a schedule from a text prompt. A schedule is essentially an array of `(valid_until, prompt)` pairs that the other nodes can use.
## FilterSchedule
Filters a schedule according to its parameters, removing any *changes* that do not occur within `[start, end)`.
The node also does tag filtering if any tags are specified.
Always returns at least the last prompt in the schedule if everything would otherwise be filtered.
`start=0, end=0` returns the prompt at the start and `start=1.0, end=1.0` returns the prompt at the end.
## PCScheduleSettings
Returns an object representing **default values** for the `SDXL` function and allows configuring `MASK_SIZE` outside the prompt. You need to apply them to a schedule with `PCApplySettings`. Note that for the SDXL settings to apply, you still need to have `SDXL()` in the prompt.
The "steps" parameter currently does nothing; it's for future features.
## PCApplySettings
Applies the give default values from `PCScheduleSettings` to a schedule
## PCPromptFromSchedule
Extracts a text prompt from a schedule; also logs it to the console.
LoRAs are *not* included in the text prompt, though they are logged.
## PCScheduleAddMasks
Add masks to a schedule object, for use with IMASK (see [syntax documentation](doc/syntax.md))
# Experimental nodes
## PCLazyEncode
`PCLazyEncode` is an experiment that uses ComfyUI's lazy graph execution mechanism to dynamically generate a graph of `PCTextEncode` and `SetConditioningTimestepRange` nodes from a prompt with schedules. This has the advantage that if a part of the schedule doesn't change, ComfyUI's caching mechanism allows you to avoid re-encoding the non-changed part.
for example, if you first encode `[cat:dog:0.1]` and later change that to `[cat:dog:0.5]`, no re-encoding takes place.
for added fun, put `NODE(NodeClassName, paramname)` in a prompt to encode the text using any other node (the default is to use `PCTextEncode` and `text`)
Note that the node can't have required parameters besides a single CLIP parameter (which must be named `clip`) and the text prompt, and it must return a `CONDITIONING` as its first return value.
## PCLazyLoraLoader
Another lazy node that will construct a graph of `LoraLoader`s and `CreateHookLora`s as necessary to provide the necessary LoRA scheduling.
If you have `apply_hooks` set to true, you **do not** need to apply the `HOOKS` output to a CLIP model separately.
# Known issues
- ComfyUI's LoRA hooks are a bit slower than LoRALoader currently when the LoRA doesn't actually require scheduling. Hopefully this will improve upstream.
- None at the moment
doc/schedule.md
+40
View File
@@ -0,0 +1,40 @@
# Nodes operating on schedules
These nodes operate on special "schedule" objects. They don't benefit from laziness like `PCLazyTextEncode`, but allow attaching custom masks.
## PCLoraHooksFromSchedule
Creates a ComfyUI `HOOKS` object from a prompt schedule. Can be attached to a CLIP model to perform encoding and LoRA switching.
## PCEncodeSchedule
Encodes all prompts in a schedule. Pass in a `CLIP` object with hooks attached for LoRA scheduling, then use the resulting `CONDITIONING` normally
## PCPromptToSchedule
Parses a schedule from a text prompt. A schedule is essentially an array of `(valid_until, prompt)` pairs that the other nodes can use.
## PCFilterSchedule
Filters a schedule according to its parameters, removing any *changes* that do not occur within `[start, end)`.
The node also does tag filtering if any tags are specified.
Always returns at least the last prompt in the schedule if everything would otherwise be filtered.
`start=0, end=0` returns the prompt at the start and `start=1.0, end=1.0` returns the prompt at the end.
## PCSettings
Returns an object representing **default values** for the `SDXL` function and allows configuring `MASK_SIZE` outside the prompt. You need to apply them to a schedule with `PCApplySettings`. Note that for the SDXL settings to apply, you still need to have `SDXL()` in the prompt.
The "steps" parameter currently does nothing; it's for future features.
## PCApplySettings
Applies the given default values from `PCSettings` to a schedule
## PCPromptFromSchedule
Extracts a text prompt from a schedule; also logs it to the console.
LoRAs are *not* included in the text prompt, though they are logged.
## PCScheduleAddMasks
Add masks to a schedule object, for use with IMASK