Compare commits

...
47 Commits
Author SHA1 Message Date
asagi4 8aec0d8f46 I can't run tests if this exists... 2025-12-18 23:51:43 +02:00
asagi4 013586904b Appease typing 2025-12-18 23:51:43 +02:00
asagi4 85ff8466ad Appease typing 2025-12-18 23:51:43 +02:00
asagi4 97d17d2bfd Minor refactor to appease typing 2025-12-18 23:51:43 +02:00
asagi4 c0e671b2de More typing 2025-12-18 23:51:43 +02:00
asagi4 7ff55c6717 Make split_quotable an iterator 2025-12-16 18:08:35 +02:00
asagi4 524738f21d Add some more typing 2025-12-16 18:01:50 +02:00
asagi4 5c7d507e91 Refactor get_function to makes its use consistent
Add some typing, just for fun
2025-12-16 17:47:10 +02:00
asagi4 a0ab709f50 v2.1.1 2025-12-14 17:57:53 +02:00
asagi4 66fef1ffe8 Avoid splitting AND, CAT and others when inside quotes
See #132
2025-12-05 21:25:28 +02:00
asagi4 3341e9f81e Fix TE_WEIGHT failing with some encoders
See #131

Prompt weighting and attention couple will probably not work, but
this should prevent exceptions.
2025-12-03 16:07:31 +02:00
asagi4 aa6b4608f0 Clarify attention couple docs a bit and add a warning if IMASK is used without attaching custom masks
See #108
2025-12-01 13:19:29 +02:00
asagi4 d1cc60b00a v2.1.0 2025-11-20 20:20:22 +02:00
asagi4 efe8939250 Fix issue with T5 encoding sometimes returning NaNs in test 2025-11-20 20:14:40 +02:00
asagi4 39b353b916 Fix tests for #130 2025-11-20 20:14:37 +02:00
asagi4 485bc7f2ab Prepare for v3 conversion of imported nodes, see #12
This should prevent things from breaking, but it needs a bit of testing.
2025-11-20 15:29:33 +02:00
asagi4 1d03ded9dd Find LoRAs with partial match 2025-11-20 15:23:51 +02:00
asagi4 94a4d076e0 Remove unused attributes 2025-08-30 13:46:00 +03:00
asagi4 228dc4b22b Remove the old advanced encoding implementation 2025-08-27 20:27:30 +03:00
asagi4 db523e1f16 Remove dead code 2025-08-27 20:23:41 +03:00
asagi4 6b08c7a90e v2.0.1 2025-08-27 20:20:13 +03:00
asagi4 76142c4b7e Fix #127 2025-08-27 20:15:30 +03:00
asagi4 1d84fdaf9e Release 2.0.0 2025-08-19 19:53:34 +03:00
asagi4 1c50ae5297 Disable the cache hack for now 2025-08-19 19:53:23 +03:00
asagi4 51618289e7 v2.0.0-rc.9 2025-06-21 13:16:57 +03:00
asagi4 cea1e5b30f Initial embedded documentation 2025-06-21 13:15:58 +03:00
asagi4 55f0574ac7 Clarification 2025-06-21 12:22:05 +03:00
asagi4 167689cb8b Add some explanations, see #121 2025-06-21 12:18:01 +03:00
asagi4 2be3abed44 Fix graph expansion node 2025-06-21 11:32:51 +03:00
asagi4 f86abb0816 Add tool for expanding lazy graphs 2025-06-21 11:19:40 +03:00
asagi4 a3537a5b2f Some progress... 2025-06-13 21:55:17 +03:00
asagi4 af7e4542d1 Let's just bruteforce it 2025-06-13 21:49:59 +03:00
asagi4 f37f14b2a2 Does this work? 2025-06-13 21:33:51 +03:00
asagi4 7b8231d36b ... 2025-06-13 21:16:50 +03:00
asagi4 9b5e15fde3 Forgot to import mock 2025-06-13 21:06:14 +03:00
asagi4 d97d30074f Encoder tests need a CPU mock too for CI 2025-06-13 21:03:52 +03:00
asagi4 b2eb9b88ba Try running encoder tests in CI 2025-06-13 20:59:37 +03:00
asagi4 c9459e39f9 Fix #120 and add a test 2025-06-13 09:08:04 +03:00
asagi4 7507b2b55f Also strip comments if they're at the start of a line 2025-06-12 23:30:54 +03:00
asagi4 cf1efecf4c Fix minor mistake in doc 2025-06-12 23:12:13 +03:00
asagi4 ffcf94bcaa Syntax 2025-06-12 23:05:44 +03:00
asagi4 ec8c40355c Split documentation 2025-06-12 23:03:39 +03:00
asagi4 6e538e0abc Fix markdown syntax 2025-06-12 22:42:45 +03:00
asagi4 25c44a1fbb Documentation 2025-06-12 22:41:31 +03:00
asagi4 72d5490498 Add support for commenting out things with #
You can escape it with \#

Fixes #105
2025-06-12 22:26:34 +03:00
asagi4 3de4538326 Test cleanup 2025-06-12 21:38:56 +03:00
asagi4 f61af15d52 Update the description a bit 2025-06-10 19:05:52 +03:00
38 changed files with 788 additions and 797 deletions
+18 -4
View File
@@ -4,13 +4,17 @@ on:
workflow_dispatch:
push:
paths:
- prompt_control/adv_encode.py
- prompt_control/attention_couple_ppm.py
- prompt_control/nodes_lazy.py
- prompt_control/prompts.py
- prompt_control/parser.py
- prompt_control/utils.py
jobs:
run-graph-tests:
name: Run graph tests
name: Run tests requiring ComfyUI
runs-on: ubuntu-latest
steps:
- name: Check out code
@@ -23,6 +27,16 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- run: pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
- run: pip install -r requirements.txt -r ComfyUI/requirements.txt
- run: PYTHONPATH=ComfyUI python -m prompt_control.test_graph
cache: pip
- name: install-torch
run: pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
- name: install ComfyUI
run: pip install -r requirements.txt -r ComfyUI/requirements.txt
- name: Download clip_l.safetensors
run: curl -LO https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors
- name: Force Comfy to use the CPU
run: sed -i "s/^cpu_state = CPUState.GPU/cpu_state = CPUState.CPU/g" ComfyUI/comfy/model_management.py
- name: Run graph tests
run: PYTHONPATH=ComfyUI python -m prompt_control.test_graph
- name: Run encoder tests (clip_l only)
run: PYTHONPATH=ComfyUI python -m prompt_control.test_encode
+1
View File
@@ -1 +1,2 @@
__pycache__
.pyre
+2 -2
View File
@@ -12,10 +12,10 @@ test_graph:
PYTHONPATH=../../ python -m prompt_control.test_graph
test_encode:
PYTHONPATH=../../ python -m prompt_control.test_encode
PYTHONPATH=../../ python -m prompt_control.test_encode --verbose
test_encode_both:
TEST_TE="clip_l t5" PYTHONPATH=../../ python -m prompt_control.test_encode
TEST_TE="clip_l t5" PYTHONPATH=../../ python -m prompt_control.test_encode --verbose
test_heavy: test_graph test_encode_both
+9 -28
View File
@@ -11,16 +11,16 @@ A `Basic Text to Image` template is included with the extension, and can be load
You can use text prompts to control the following:
- A1111-style prompt scheduling and filtering without noodle soup.
- LoRA loading and scheduling via ComfyUI's hook system
- Masking, composition and area control (regional prompting) with an implementation of [Attention Couple](/doc/attention_couple.md), also fully schedulable.
- Per-encoder prompts for models with multiple text encoders, such as SDXL and Flux
- Prompt combinators like `BREAK`, as well as `CAT`, `AVG()` and `AND` corresponding to ComfyUI's `ConditioningConcat`, `ConditioningAverage` and `ConditioningCombine` nodes.
- Different weight interpretation types (ComfyUI, A1111, compel, etc.)
- Prompt masking with an implementation of [cutoff](https://github.com/BlenderNeko/ComfyUI_Cutoff)
- Simple prompt macros with `DEF`
- And a bunch more
- LoRA loading and [scheduling](/doc/schedules.md) via the prompt, using ComfyUI's hook system
- Masking, composition and area control ([regional prompting](/doc/regional_prompts.md)) with an implementation of [Attention Couple](/doc/attention_couple.md), also fully schedulable.
- [Advanced prompt encoding](/doc/basic.md)
- Per-encoder prompts for models with multiple text encoders, such as SDXL and Flux
- Prompt combinators like `BREAK`, as well as `CAT`, `AVG()` and `AND` corresponding to ComfyUI's `ConditioningConcat`, `ConditioningAverage` and `ConditioningCombine` nodes.
- Different weight interpretation types (ComfyUI, A1111, compel, etc.)
- Prompt masking with an implementation of [cutoff](https://github.com/BlenderNeko/ComfyUI_Cutoff)
- Simple [prompt macros](/doc/macros.md) with `DEF`
All features are fully schedulable unless otherwise stated. See the [syntax documentation](doc/syntax.md) for details on how to use each feature.
All features are fully schedulable unless otherwise stated. See the [scheduling syntax documentation](doc/schedules.md) to get started.
If you find prompt scheduling inconvenient for some reason, `PCTextEncode` can be used as a drop-in replacement for `CLIPTextEncode` to get everything else.
@@ -32,21 +32,6 @@ Prompt Control uses graph generation, and tries to delegate functionality to co
If you encounter issues as a user or if you're a node developer and Prompt Control somehow breaks something, feel free to file a bug report.
## Prompt Control v2
Prompt control has been almost completely rewritten. It now uses ComfyUI's lazy execution to build graphs from the text prompt at runtime. The generated graph is often exactly equivalent to a manually built workflow using native ComfyUI nodes. There are no more weird sampling hooks that could cause problems with other nodes
### Removed features
- Prompt interpolation syntax; it was too cumbersome to maintain
- LoRA block weight integration; ditto, for now.
### Everything broke, where are the old nodes?
If you really need them, you can install the [legacy nodes](https://github.com/asagi4/comfyui-prompt-control-legacy). However, I will not fix bugs in those nodes, and I strongly recommend just migrating your workflows to the new nodes.
You can have both installed at the same time; none of the nodes conflict.
## Requirements
For LoRA scheduling to work, you'll need at least version 0.3.7 of ComfyUI (0.3.36 of ComfyUI desktop).
@@ -100,8 +85,4 @@ This node configures `PCTextEncode` default values for some functions by attachi
- ComfyUI's caching mechanism has an issue that makes it unnecessarily invalidate caches for certain inputs; you'll still get some benefit from the lazy nodes, but changing inputs that shouldn't affect downstream nodes (especially if using filtering) will still cause them to be recomputed because ComfyUI doesn't realize the inputs haven't changed.
If you want to enable a hack to fix this, set `PROMPTCONTROL_ENABLE_CACHE_HACK=1` in your environment. Unset it to disable.
It's a purely optional performance optimization that allows Prompt Control nodes to override their cache keys in a way that should not interfere with other nodes. Note that the optimization only works if the text input to the lazy nodes is a constant (so either directly on the node or from a primitive); outputs from other nodes can't be optimized.
- Cutoff does not work with models that use non-CLIP text encoders, like Flux. This might be fixable, but it's uncertain if cutoff even makes sense for those models.
+2 -3
View File
@@ -23,12 +23,11 @@ if os.environ.get("PROMPTCONTROL_DEBUG"):
else:
log.setLevel(logging.INFO)
cache_hack = importlib.import_module(".prompt_control.cache_hack", package=__name__)
cache_hack.init()
NODE_CLASS_MAPPINGS = {}
NODE_DISPLAY_NAME_MAPPINGS = {}
WEB_DIRECTORY = "web"
nodes = ["base", "lazy", "tools", "hooks"]
for node in nodes:
+3 -1
View File
@@ -15,7 +15,7 @@ To enable batching negative prompts, run your positive and negative prompt throu
## Syntax
See also the main syntax documentation for `MASK` etc.
See also the [regional prompting documentation](/doc/regional_prompts.md) for information about `MASK` etc.
### COUPLE: Trigger Attention Couple
@@ -38,3 +38,5 @@ For example:
```
dog FILL() COUPLE(0.5 1) cat
```
Note that because the generation still sees and diffuses the full latent, attention coupling is not guaranteed to perfectly limit the effect of your prompt to the masked area.
+9 -203
View File
@@ -1,101 +1,6 @@
# Prompt Control Syntax
# Basic Prompt Syntax
If you're viewing this on GitHub, I recommend opening the outline by clicking the button in the top right corner of the text view (it is annoyingly easy to miss).
Scheduling syntax is similar to A1111, but only fractions are supported for steps. LoRAs are scheduled by including them in a scheduling expression.
```
a [large::0.1] [cat|dog:0.05] [<lora:somelora:0.5:0.6>::0.5]
[in a park:in space:0.4]
```
## Scheduled prompts
There are two forms of scheduled prompts.
### Basic scheduling expressions
Basic expressions take the form `[before:after:X]` where `X` is the switch point, a decimal number between 0.0 and 1.0 inclusive, representing 0 to 100% of timesteps. Either prompt can also be empty.
For example:
```
a [red:blue:0.5] cat
```
switches from `a red cat` to `a blue cat` at 0.5. `before` and `after` can be arbitrary prompts (`after` can also be empty), including other scheduling expressions, allowing nesting:
```
a [red:[blue::0.7]:0.5] cat
```
switches from `a red cat` to `a blue cat` at 0.5 and to `a cat` at 0.7
For convenience `[cat:0.5]` is equivalent to `[:cat:0.5]` meaning it switches from empty to `cat` at 0.5.
### Range expressions
The most general form of a schedule is a range expression: For example, in `[before:during:after:0.3,0.7]`, The prompt be `a before` until 0.3, `a during` until 0.7, and then `a after`. This form is equivalent to `[before:[during:after:0.7]:0.3]`
For convenience, `[during:0.1,0.4]` is equivalent to `[:during::0.1,0.4]` and `[during:after:0.1,0.4]` is equivalent to `[:during:after:0.1,0.4]`.
`[before:during:after:0.1]` is the same as `[before:during:after:0.1,1.0]` which is same as `[before:during:0.1]`
### Using step numbers with the Advanced nodes
If you provide a non-zero value to `num_steps` to the `Advanced` versions of the scheduling nodes, you will be able to use step numbers in prompts.
For now, a value between 0 and 1.0 will be interpreted as a percentage if it contains a ., and as an absolute step otherwise.
This is just syntactic sugar. Behind the scenes, the values are converted to percentages and have normal ComfyUI scheduling behaviour.
## Tag selection
Using the `FilterSchedule` node, in addition to step percentages, you can use a *tag* to select part of an input:
```
a large [dog:cat<lora:catlora:0.5>:SECOND_PASS]
```
Set the `tags` parameter in the `FilterSchedule` node to filter the prompt. If the tag matches any tag `tags` (comma-separated), the second option is returned (`cat`, in this case, with the LoRA). Otherwise, the first option is chosen (`dog`, without LoRA).
the values in `tags` are case-insensitive, but the tags in the input **must** be uppercase A-Z and underscores only, or they won't be recognized. That is, `[dog:cat:hr]` will not work.
For example, a prompt
```
a [black:blue:X] [cat:dog:Y] [walking:running:Z] in space
```
with `tags` `x,z` would result in the prompt `a blue cat running in space`
The three prompt form `[a:b:c:TAG]` is parsed, but ignores `b` and is equivalent to `[a:c:TAG]`.
## LoRA Scheduling
When using the lazy graph building nodes, LoRAs can be scheduled by referring to them in a scheduling expression, like so:
`<lora:fulllora:1> [<lora:partialora:1>::0.5]`
This will schedule `fulllora` for the entire duration of the prompt and `partiallora` until half of sampling is complete.
You can refer to LoRAs by using the filename without extension and subdirectories will also be searched. For example, `<lora:cats:1>`. will match both `cats.safetensors` and `sd15/animals/cats.safetensors`. If there are multiple LoRAs with the same name, the first match will be loaded.
Alternatively, the name can include the full directory path relative to ComfyUI's search paths, without extension: `<lora:XL/sdxllora:0.5>`. In this case, the *full* path must match.
If no match is found, the node will try to replace spaces with underscores and search again. That is, `<lora:cats and dogs:1>` will find `cats_and_dogs.safetensors`. This helps with some autocompletion scripts that replace underscores with spaces.
Finally, you can give the exact path (including the extension) as shown in `LoRALoader`.
## Alternating
Alternating syntax is `[a|b:pct_steps]`, causing the prompt to alternate every `pct_steps`. `pct_steps` defaults to 0.1 if not specified. You can also have more than two options.
## Sequences
The syntax `[SEQ:a:N1:b:N2:c:N3]` is shorthand for `[a:[b:[c::N3]:N2]:N1]` ie. it switches from `a` to `b` to `c` to nothing at the specified points in sequence.
Might be useful with Jinja templating (see https://github.com/asagi4/comfyui-utility-nodes). For example:
```
[SEQ<% for x in steps(0.1, 0.9, 0.1) %>:<lora:test:<= sin(x*pi) + 0.1 =>>:<= x =><% endfor %>]
```
generates a LoRA schedule based on a sinewave
# Basic prompt syntax
This syntax is also available in outside scheduled with the `PCTextEncode` node, where applicable.
The syntax below documents the features of `PCTextEncode`
## Combining prompts
@@ -118,6 +23,8 @@ cat [\:0::0.5] AND dog
```
Note that the `:` needs to be escaped with a `\` or it will be interpreted as scheduling syntax.
If `AND` is placed inside quotes (eg. `Text saying "CAT AND DOG"`) it will be treated as regular text.
## Note about processing order
Prompt operators are processed in the following order, meaning that all features "below" another can be affected by the feature above it. That is, `BREAK` can go inside a `TE()` call, but not `AND` or `CAT`.
@@ -148,6 +55,8 @@ Note: Whitespace is usually *not* stripped from string parameters by default. Co
Like `AND`, functions are parsed after regular scheduling syntax has been expanded, allowing things like `[AREA:MASK:0.3](...)`, in case that's somehow useful.
like AND, if any function is placed inside quotes, it will *not* activate and is instead treated as regular text.
### BREAK
The keyword `BREAK` causes the prompt to be tokenized in separate chunks, padding each chunk to the text encoder's maximum size before encoding.
@@ -252,52 +161,9 @@ The function `NOISE(weight, seed)` adds some random noise into the cond tensor.
The usefulness of this is questionable, but it wasn't difficult to implement, so here it is.
## Regional prompting
See also [Attention Couple](#attention-couple) below
### MASK, IMASK and AREA
You can use `MASK(x1 x2, y1 y2, weight, op)` to specify a region mask for a prompt. The values are specified as a percentage with a float between `0` and `1`, or as absolute pixel values (these can't be mixed). `1` will be interpreted as a percentage instead of a pixel value.
Multiple `MASK` or `IMASK` calls will be composited together using ComfyUI's `MaskComposite` node, using `op` as the `operation` parameter (defaulting to `multiply`).
Similarly, you can use `AREA(x1 x2, y1 y2, weight)` to specify an area for the prompt (see ComfyUI's area composition examples). The area is calculated by ComfyUI relative to your latent size.
### Custom masks: IMASK and `PCAddMaskToCLIP`
You can attach custom masks to a `CLIP` with the `PC: Attach Mask` nodes and then refer to those masks in the prompt using `IMASK(index, weight, op)`. Indexing starts from zero, so 0 is the first attached mask etc. `PCSCheduleAddMasks` ignores empty inputs, so if you only add a mask to the `mask4` input, it will still have index 0.
Applying the nodes multiple times *appends* masks rather than overriding existing ones, so if you need more than 4, you can just use it more than once.
### Behaviour of masks
If multiple `MASK`s are specified, they are combined together with ComfyUI's `MaskComposite` node, with `op` specifying the operation to use (default `multiply`). In this case, the combined mask weight can be set with `MASKW(weight)` (defaults to 1.0).
Masks assume a size of `(512, 512)`, unless overridden with `PC: Configure PCTextEncode` and pixel values will be relative to that. ComfyUI will scale the mask to match the image resolution. You can change it manually by using `MASK_SIZE(width, height)` anywhere in the prompt,
These are handled per `AND`-ed prompt, so in `prompt1 AND MASK(...) prompt2`, the mask will only affect prompt2.
The default values are `MASK(0 1, 0 1, 1)` and you can omit unnecessary ones, that is, `MASK(0 0.5, 0.3)` is `MASK(0 0.5, 0.3 1, 1)`
Note that because the default values are percentages, `MASK(0 256, 64 512)` is valid, but `MASK(0 200)` will raise an error.
Masking does not affect LoRA scheduling unless you set unet weights to 0 for a LoRA.
### FEATHER: Mask operations
When you use `MASK` or `IMASK`, you can also call `FEATHER(left top right bottom)` to apply feathering using ComfyUI's `FeatherMask` node. The values are in pixels and default to `0`.
If multiple masks are used, `FEATHER` is applied *before compositing* in the order they appear in the prompt, and any leftovers are applied to the combined mask. If you want to skip feathering a mask while compositing, just use `FEATHER()` with no arguments.
For example:
```
MASK(1) MASK(2) MASK(3) FEATHER(1) FEATHER() FEATHER(3) weirdmask FEATHER(4)
```
gives you a mask that is a combination of 1, 2 and 3, where 1 and 3 are feathered before compositing and then `FEATHER(4)` is applied to the composite.
The order of the `FEATHER` and `MASK` calls doesn't matter; you can have `FEATHER` before `MASK` or even interleave them.
See [Regional prompting](/doc/regional_prompting.md)
## Cutoff
@@ -328,68 +194,8 @@ The parameters affect how the masked and unmasked prompts are combined to produc
# Experimental features
Experimental features are unstable and may disappear or change without warning.
## DEF: Lightweight prompt macros
You can define "prompt macros" by using `DEF`. Macros are expanded before any other parsing takes place. The expansion continues until no further changes occur. Recursion will raise an error.
`PCLazyTextEncode` and `PCLazyLoraLoader` expand macros, but `PCTextEncode` **does not**. If you need to expand macros for a single prompt, use `PCMacroExpand`
```
DEF(MYMACRO=this is a prompt)
[(MYMACRO:0.6):(MYMACRO:1.1):0.5]
```
is equivalent to
```
[(this is a prompt:0.5):(this is a prompt:1.1):0.5]
```
### Macro parameters
It's also possible to give parameters to a macro:
```
DEF(MYMACRO=[(prompt $1:$2):(prompt $1:$3):$4])
MYMACRO(test; 1.1; 0.7; 0.2)
```
gives
```
[(prompt test:1.1):(prompt test:0.7):0.2]
```
in this form, the variables $N (where N is any number corresponding to a positional parameter) will be replaced with the given parameter. The parameters must be separated with a semicolon, and can be empty.
You can also optionally specify default values:
```
DEF(MACRO(example; 0; 1)=[$1:$2,$3])
MACRO MACRO(test; 0.2)
```
gives
```
[example:0,1] [test:0.2,1]
```
```
DEF(MACRO() = [a:$1:0.5])
```
sets the default value of `$1` to an empty string.
### Unspecified parameters in macros
Unspecified parameters (either via defaults or explicitly given) will not be substituted. Compare:
```
DEF(mything=a "$1" b "$2")
mything
mything()
mything(A)
```
gives
```
a "$1" b "$2"
a "" b "$2"
a "A" b "$2"
```
> [!WARN]
> These features are may change or disappear without warning
## COUPLE: Attention couple
+60
View File
@@ -0,0 +1,60 @@
## DEF: Lightweight prompt macros
You can define "prompt macros" by using `DEF`. Macros are expanded before any other parsing takes place. The expansion continues until no further changes occur. Recursion will raise an error.
`PCLazyTextEncode` and `PCLazyLoraLoader` expand macros, but `PCTextEncode` **does not**. If you need to expand macros for a single prompt, use `PCMacroExpand`
```
DEF(MYMACRO=this is a prompt)
[(MYMACRO:0.6):(MYMACRO:1.1):0.5]
```
is equivalent to
```
[(this is a prompt:0.5):(this is a prompt:1.1):0.5]
```
### Macro parameters
It's also possible to give parameters to a macro:
```
DEF(MYMACRO=[(prompt $1:$2):(prompt $1:$3):$4])
MYMACRO(test; 1.1; 0.7; 0.2)
```
gives
```
[(prompt test:1.1):(prompt test:0.7):0.2]
```
in this form, the variables $N (where N is any number corresponding to a positional parameter) will be replaced with the given parameter. The parameters must be separated with a semicolon, and can be empty.
You can also optionally specify default values:
```
DEF(MACRO(example; 0; 1)=[$1:$2,$3])
MACRO MACRO(test; 0.2)
```
gives
```
[example:0,1] [test:0.2,1]
```
```
DEF(MACRO() = [a:$1:0.5])
```
sets the default value of `$1` to an empty string.
### Unspecified parameters in macros
Unspecified parameters (either via defaults or explicitly given) will not be substituted. Compare:
```
DEF(mything=a "$1" b "$2")
mything
mything()
mything(A)
```
gives
```
a "$1" b "$2"
a "" b "$2"
a "A" b "$2"
```
+63
View File
@@ -0,0 +1,63 @@
# Regional prompting
This section documents the masking functionality of `PCTextEncode`
See also [Attention Couple](/doc/attention_couple.md)
Remember that when using the lazy nodes, prompt scheduling applies to masks as well, so you can change or enable/disable regional prompts at any point during sampling.
## Behaviour
For each prompt separated by `AND`, you can specify either latent masks or an area.
- When masked, ComfyUI generates the model output using the **full latent** as the input, and then applies the mask to the output before adding it to your latent for the next step.
- When an area is specified, ComfyUI generates a separate model output using the **part of the latent specified by the area** and then composites it into the full latent afterwards.
- You can have *both* an AREA and a MASK specified, in which case the mask is applied to the latent specified by the AREA.
For example, consider a 1024 by 1024 (width x height) generation:
- `cat MASK(0 0.5, 0 1) AND dog MASK(0.5 1, 0 1)` generates two outputs at 1024x1024 for "dog" and "cat", then masks half of them off and adds the results together. The following step still see both the dog and the cat from the previous step, so they may blend slightly.
- `cat AREA(0 0.5, 0 1) AND dog AREA(0.5 1, 0 1)` generates two completely separate outputs at **512**x1024 and then composites them together into the 1024x1024 latent. Because the areas do not overlap, the generation for `cat` will not see the output of `dog` and vice versa in subsequent steps as long as the area restriction is in effect.
## MASK, IMASK and AREA
You can use `MASK(x1 x2, y1 y2, weight, op)` to specify a region mask for a prompt. The values are specified as a percentage with a float between `0` and `1`, or as absolute pixel values (these can't be mixed). `1` will be interpreted as a percentage instead of a pixel value.
Multiple `MASK` or `IMASK` calls will be composited together using ComfyUI's `MaskComposite` node, using `op` as the `operation` parameter (defaulting to `multiply`).
Similarly, you can use `AREA(x1 x2, y1 y2, weight)` to specify an area for the prompt (see ComfyUI's area composition examples). The area is calculated by ComfyUI relative to your latent size.
### Custom masks: IMASK and `PCAddMaskToCLIP`
You can attach custom masks to a `CLIP` with the `PC: Attach Mask` nodes and then refer to those masks in the prompt using `IMASK(index, weight, op)`. Indexing starts from zero, so 0 is the first attached mask etc. `PCSCheduleAddMasks` ignores empty inputs, so if you only add a mask to the `mask4` input, it will still have index 0.
Applying the nodes multiple times *appends* masks rather than overriding existing ones, so if you need more than 4, you can just use it more than once.
### Behaviour of multiple masks
If multiple `MASK`s are specified, they are combined together with ComfyUI's `MaskComposite` node, with `op` specifying the operation to use (default `multiply`). In this case, the combined mask weight can be set with `MASKW(weight)` (defaults to 1.0).
Masks assume a size of `(512, 512)`, unless overridden with `PC: Configure PCTextEncode` and pixel values will be relative to that. ComfyUI will scale the mask to match the image resolution. You can change it manually by using `MASK_SIZE(width, height)` anywhere in the prompt,
These are handled per `AND`-ed prompt, so in `prompt1 AND MASK(...) prompt2`, the mask will only affect prompt2.
The default values are `MASK(0 1, 0 1, 1)` and you can omit unnecessary ones, that is, `MASK(0 0.5, 0.3)` is `MASK(0 0.5, 0.3 1, 1)`
Note that because the default values are percentages, `MASK(0 256, 64 512)` is valid, but `MASK(0 200)` will raise an error.
Masking does not affect LoRA scheduling unless you set unet weights to 0 for a LoRA.
## FEATHER: Mask operations
When you use `MASK` or `IMASK`, you can also call `FEATHER(left top right bottom)` to apply feathering using ComfyUI's `FeatherMask` node. The values are in pixels and default to `0`.
If multiple masks are used, `FEATHER` is applied *before compositing* in the order they appear in the prompt, and any leftovers are applied to the combined mask. If you want to skip feathering a mask while compositing, just use `FEATHER()` with no arguments.
For example:
```
MASK(1) MASK(2) MASK(3) FEATHER(1) FEATHER() FEATHER(3) weirdmask FEATHER(4)
```
gives you a mask that is a combination of 1, 2 and 3, where 1 and 3 are feathered before compositing and then `FEATHER(4)` is applied to the composite.
The order of the `FEATHER` and `MASK` calls doesn't matter; you can have `FEATHER` before `MASK` or even interleave them.
+117
View File
@@ -0,0 +1,117 @@
# Prompt Schedule Syntax
> [!TIP]
> If you're viewing this on GitHub, I recommend opening the outline by clicking the button in the top right corner of the text view (it is annoyingly easy to miss).
> [!NOTE]
> The syntax documented in this section is only available with the `PC: Schedule Prompt` and `PC: Schedule LoRAs` nodes and their advanced variants.
Scheduling syntax is available with is similar to A1111, but only fractions are supported for steps. LoRAs are scheduled by including them in a scheduling expression.
Besides the syntax documented below, the [basic syntax](/doc/basic.md) and [prompt macro](/doc/macros.md) features are also automatically available.
```
a [large::0.1] [cat|dog:0.05] [<lora:somelora:0.5:0.6>::0.5]
[in a park:in space:0.4]
```
## Comments and escaping
In schedules, any text on a line following a `#` is considered a comment and removed, including the `#` character.
You can escape the following characters in places where they would otherwise conflict with syntax:
- `#` with `\#`
- `:` with `\:`
- `\` with `\\`
Escaping is only required if it would otherwise be considered syntax, that is `\o/` will be interpreted literally and the `\` does not need to be escaped, but in `[embedding:a:0.5]` you would need to escape the `:`.
## Scheduled prompts
There are two forms of scheduled prompts.
### Basic scheduling expressions
Basic expressions take the form `[before:after:X]` where `X` is the switch point, a decimal number between 0.0 and 1.0 inclusive, representing 0 to 100% of timesteps. Either prompt can also be empty.
For example:
```
a [red:blue:0.5] cat
```
switches from `a red cat` to `a blue cat` at 0.5. `before` and `after` can be arbitrary prompts (`after` can also be empty), including other scheduling expressions, allowing nesting:
```
a [red:[blue::0.7]:0.5] cat
```
switches from `a red cat` to `a blue cat` at 0.5 and to `a cat` at 0.7
For convenience `[cat:0.5]` is equivalent to `[:cat:0.5]` meaning it switches from empty to `cat` at 0.5.
### Range expressions
The most general form of a schedule is a range expression: For example, in `prompt [before:during:after:0.3,0.7]`, The prompt be `prompt before` until 0.3, `prompt during` until 0.7, and then `prompt after`. This form is equivalent to `prompt [before:[during:after:0.7]:0.3]`
For convenience, `[during:0.1,0.4]` is equivalent to `[:during::0.1,0.4]` and `[during:after:0.1,0.4]` is equivalent to `[:during:after:0.1,0.4]`.
`[before:during:after:0.1]` is the same as `[before:during:after:0.1,1.0]` which is same as `[before:during:0.1]`
### Using step numbers with the Advanced nodes
If you provide a non-zero value to `num_steps` to the `Advanced` versions of the scheduling nodes, you will be able to use step numbers in prompts.
For now, a value between 0 and 1.0 will be interpreted as a percentage if it contains a ., and as an absolute step otherwise.
This is just syntactic sugar. Behind the scenes, the values are converted to percentages and have normal ComfyUI scheduling behaviour.
## Tag selection
Using the `FilterSchedule` node, in addition to step percentages, you can use a *tag* to select part of an input:
```
a large [dog:cat<lora:catlora:0.5>:SECOND_PASS]
```
Set the `tags` parameter in the `FilterSchedule` node to filter the prompt. If the tag matches any tag `tags` (comma-separated), the second option is returned (`cat`, in this case, with the LoRA). Otherwise, the first option is chosen (`dog`, without LoRA).
the values in `tags` are case-insensitive, but the tags in the input **must** be uppercase A-Z and underscores only, or they won't be recognized. That is, `[dog:cat:hr]` will not work.
For example, a prompt
```
a [black:blue:X] [cat:dog:Y] [walking:running:Z] in space
```
with `tags` `x,z` would result in the prompt `a blue cat running in space`
The three prompt form `[a:b:c:TAG]` is parsed, but ignores `b` and is equivalent to `[a:c:TAG]`.
## LoRA Scheduling
When using the lazy graph building nodes, LoRAs can be scheduled by referring to them in a scheduling expression, like so:
`<lora:fulllora:1> [<lora:partialora:1>::0.5]`
This will schedule `fulllora` for the entire duration of the prompt and `partiallora` until half of sampling is complete.
You can refer to LoRAs by using the filename without extension and subdirectories will also be searched. For example, `<lora:cats:1>`. will match both `cats.safetensors` and `sd15/animals/cats.safetensors`. If there are multiple LoRAs with the same name, the first match will be loaded.
Alternatively, the name can include the full directory path relative to ComfyUI's search paths, without extension: `<lora:XL/sdxllora:0.5>`. In this case, the *full* path must match.
You can also give the exact path (including the extension) as shown in `LoRALoader`.
If no match is found, the node will try to replace spaces with underscores and search again. That is, `<lora:cats and dogs:1>` will find `cats_and_dogs.safetensors`. This helps with some autocompletion scripts that replace underscores with spaces.
Finally, if none of the above produce a match, the search term will be split by whitespace and files that contain all of the parts in any order will be considered. If this returns only a single match, it will be loaded. For example, consider LoRAs:
- `xl/red_cats.safetensors`
- `flux/blue_cats.safetensors`
- `flux/red_cats.safetensors`
Then `<lora:cats xl:1>` would match the red cats LoRA, but `cats flux` would be ambiguous and not match.
## Alternating
Alternating syntax is `[a|b:pct_steps]`, causing the prompt to alternate every `pct_steps`. `pct_steps` defaults to 0.1 if not specified. You can also have more than two options.
## Sequences
The syntax `[SEQ:a:N1:b:N2:c:N3]` is shorthand for `[a:[b:[c::N3]:N2]:N1]` ie. it switches from `a` to `b` to `c` to nothing at the specified points in sequence.
Might be useful with Jinja templating (see https://github.com/asagi4/comfyui-utility-nodes). For example:
```
[SEQ<% for x in steps(0.1, 0.9, 0.1) %>:<lora:test:<= sin(x*pi) + 0.1 =>>:<= x =><% endfor %>]
```
generates a LoRA schedule based on a sinewave
+29 -41
View File
@@ -3,7 +3,6 @@ import numpy as np
from math import copysign
import logging
import itertools
from .adv_encode_old import old_advanced_encode_from_tokens
log = logging.getLogger("comfyui-prompt-control")
@@ -26,7 +25,7 @@ def _grouper(n, iterable):
def batched_clip_encode(tokens, length, encode_func, num_chunks):
embs = []
for e in _grouper(32, tokens):
enc, pooled = encode_func(e)
enc, pooled, *_ = encode_func(e)
enc = enc.reshape((len(e), length, -1))
embs.append(enc)
@@ -101,24 +100,24 @@ def style_comfy(encoder, tokens, **kwargs):
def style_a1111(encoder, tokens, **kwargs):
base_emb, pooled = encoder.base_emb(tokens)
base_emb, pooled, *extra = encoder.base_emb(tokens)
weighted_emb = base_emb * weights_like(encoder.weights(tokens), base_emb)
weighted_emb = (base_emb.mean() / weighted_emb.mean()) * weighted_emb # renormalize
return weighted_emb, pooled
return (weighted_emb, pooled) + tuple(extra)
def style_compel(encoder, tokens, **kwargs):
pos_tokens = encoder.weighted_with(tokens, lambda w: w if w > 1.0 else 1.0)
weighted_emb, pooled = encoder.encode_fn(pos_tokens)
weighted_emb, pooled, *extra = encoder.encode_fn(pos_tokens)
weighted_emb, _, pooled = encoder.down_weight(
pos_tokens, encoder.weights(tokens), encoder.word_ids(tokens), weighted_emb, pooled
)
return weighted_emb, pooled
return (weighted_emb, pooled) + tuple(extra)
def style_comfypp(encoder, tokens, **kwargs):
unweighted_tokens = encoder.unweighted(tokens)
base_emb, pooled_base = encoder.base_emb(tokens)
base_emb, pooled_base, *extra = encoder.base_emb(tokens)
weighted_emb, tokens_down, _ = encoder.down_weight(
unweighted_tokens, encoder.weights(tokens), encoder.word_ids(tokens), base_emb, pooled_base
)
@@ -132,23 +131,23 @@ def style_comfypp(encoder, tokens, **kwargs):
)
weighted_emb += embs
return weighted_emb, pooled
return (weighted_emb, pooled) + tuple(extra)
def style_downweight(encoder, tokens, **kwargs):
weights = scale_to_norm(encoder.weights(tokens), encoder.word_ids(tokens), encoder.w_max)
base_emb, pooled_base = encoder.base_emb(tokens)
base_emb, pooled_base, *extra = encoder.base_emb(tokens)
weighted_emb, _, pooled = encoder.down_weight(
encoder.unweighted(tokens), weights, encoder.word_ids(tokens), base_emb, pooled_base
)
return weighted_emb, pooled
return (weighted_emb, pooled) + tuple(extra)
def style_perp(encoder, tokens, **kwargs):
zero_emb, zero_pooled = encoder.encode_fn(encoder.tokenizer.tokenize_with_weights(""))
base_emb, pooled = encoder.base_emb(tokens)
return perp_weight(encoder.weights(tokens), (base_emb, pooled), (zero_emb, zero_pooled))
zero_emb, zero_pooled, *_ = encoder.encode_fn(encoder.tokenizer.tokenize_with_weights(""))
base_emb, pooled, *extra = encoder.base_emb(tokens)
return perp_weight(encoder.weights(tokens), (base_emb, pooled), (zero_emb, zero_pooled)) + tuple(extra)
def apply_negpip(encoder, emb, pooled, **kwargs):
@@ -198,6 +197,7 @@ class AdvancedEncoder:
def add_encoder(cls, name, fn):
cls.STYLES[name] = fn
@classmethod
def add_normalization_op(cls, name, fn):
cls.NORMALIZATION_OPS[name] = fn
@@ -258,8 +258,8 @@ class AdvancedEncoder:
if negpip:
def _encode(t):
emb, pooled = encode_fn(t)
return emb[:, 0::2, :], pooled
emb, pooled, *extra = encode_fn(t)
return (emb[:, 0::2, :], pooled) + tuple(extra)
self.encode_fn = _encode
self.preprocessors.insert(0, lambda encoder, tokens, **kwargs: encoder.weighted_with(tokens, abs))
@@ -289,7 +289,7 @@ class AdvancedEncoder:
if w[i] >= 1:
continue
masked_current = mask_inds(masked_current, np.where(w_inv == i)[0], self.m_token)
masked, _ = self.encode_fn(masked_current)
masked, _, *extra = self.encode_fn(masked_current)
emblist.append(masked)
embs = torch.cat(emblist)
@@ -297,7 +297,7 @@ class AdvancedEncoder:
w_mix = np.diff([0] + w.tolist())
w_mix = torch.tensor(w_mix, dtype=embs.dtype, device=embs.device).reshape((-1, 1, 1))
weighted_emb = (w_mix * embs).sum(axis=0, keepdim=True)
weighted_emb = (w_mix * embs).sum(dim=0, keepdim=True)
pooled = pooled_base
if pooled is not None and self.max_length:
pooled = weighted_emb[0, self.max_length - 1 : self.max_length, :]
@@ -329,12 +329,13 @@ class AdvancedEncoder:
masks = torch.cat(masks)
embs = base_emb.expand(embs.shape) - embs
pooled = None
if pooled_base is not None and self.max_length:
pooled = embs[0, self.max_length - 1 : self.max_length, :]
pooled_start = pooled_base.expand(len(ws), -1)
ws = torch.tensor(ws).reshape(-1, 1).expand(pooled_start.shape)
pooled = (pooled - pooled_start) * (ws - 1)
pooled = pooled.mean(axis=0, keepdim=True)
pooled = pooled.mean(dim=0, keepdim=True)
pooled = pooled_base + pooled
if embs.shape[0] != masks.shape[0]:
@@ -349,16 +350,16 @@ class AdvancedEncoder:
for op in self.preprocessors:
normalized_tokens = op(self, normalized_tokens)
emb, pooled = self.weight_fn(self, normalized_tokens, original_tokens=tokens)
emb, pooled, *extra = self.weight_fn(self, normalized_tokens, original_tokens=tokens)
for fn in self.postprocessors:
emb, pooled = fn(self, emb, pooled, tokens=tokens, original_tokens=tokens)
if return_pooled:
if not apply_to_pooled:
_, pooled = self.base_emb(tokens)
return emb, pooled
return emb, None
if not return_pooled:
pooled = None
elif not apply_to_pooled:
_, pooled, *_ = self.base_emb(tokens)
return (emb, pooled) + tuple(extra)
def advanced_encode_from_tokens(
@@ -373,20 +374,7 @@ def advanced_encode_from_tokens(
tokenizer=None,
**extra_args,
):
if "old+" not in weight_interpretation:
enc = AdvancedEncoder(
encode_func, weight_interpretation, token_normalization, tokenizer, m_token, w_max, **extra_args
)
return enc(tokenized, return_pooled=return_pooled, apply_to_pooled=apply_to_pooled)
else:
weight_interpretation = weight_interpretation.replace("old+", "")
log.warning("Using old implementation of %s", weight_interpretation)
return old_advanced_encode_from_tokens(
tokenized,
token_normalization,
weight_interpretation,
encode_func,
266,
return_pooled=return_pooled,
apply_to_pooled=apply_to_pooled,
)
enc = AdvancedEncoder(
encode_func, weight_interpretation, token_normalization, tokenizer, m_token, w_max, **extra_args
)
return enc(tokenized, return_pooled=return_pooled, apply_to_pooled=apply_to_pooled)
-235
View File
@@ -1,235 +0,0 @@
import torch
import numpy as np
import logging
import itertools
log = logging.getLogger("comfyui-prompt-control")
def _norm_mag(w, n):
d = w - 1
return 1 + np.sign(d) * np.sqrt(np.abs(d) ** 2 / n)
# return np.sign(w) * np.sqrt(np.abs(w)**2 / n)
def _grouper(n, iterable):
it = iter(iterable)
while True:
chunk = list(itertools.islice(it, n))
if not chunk:
return
yield chunk
def batched_clip_encode(tokens, length, encode_func, num_chunks):
embs = []
for e in _grouper(32, tokens):
enc, pooled = encode_func(e)
enc = enc.reshape((len(e), length, -1))
embs.append(enc)
embs = torch.cat(embs)
embs = embs.reshape((len(tokens) // num_chunks, length * num_chunks, -1))
return embs
def weights_like(weights, emb):
return torch.tensor(weights, dtype=emb.dtype, device=emb.device).reshape(1, -1, 1).expand(emb.shape)
def divide_length(word_ids, weights):
sums = dict(zip(*np.unique(word_ids, return_counts=True)))
sums[0] = 1
weights = [[_norm_mag(w, sums[id]) if id != 0 else 1.0 for w, id in zip(x, y)] for x, y in zip(weights, word_ids)]
return weights
def shift_mean_weight(word_ids, weights):
delta = 1 - np.mean([w for x, y in zip(weights, word_ids) for w, id in zip(x, y) if id != 0])
weights = [[w if id == 0 else w + delta for w, id in zip(x, y)] for x, y in zip(weights, word_ids)]
return weights
def scale_to_norm(weights, word_ids, w_max):
top = np.max(weights)
w_max = min(top, w_max)
weights = [[w_max if id == 0 else (w / top) * w_max for w, id in zip(x, y)] for x, y in zip(weights, word_ids)]
return weights
def mask_word_id(tokens, word_ids, target_id, mask_token):
new_tokens = [[mask_token if wid == target_id else t for t, wid in zip(x, y)] for x, y in zip(tokens, word_ids)]
mask = np.array(word_ids) == target_id
return (new_tokens, mask)
def from_masked(tokens, weights, word_ids, base_emb, length, encode_func, m_token=266):
pooled_base = base_emb[0, length - 1 : length, :]
wids, inds = np.unique(np.array(word_ids).reshape(-1), return_index=True)
weight_dict = dict((id, w) for id, w in zip(wids, np.array(weights).reshape(-1)[inds]) if w != 1.0)
if len(weight_dict) == 0:
return torch.zeros_like(base_emb), base_emb[0, length - 1 : length, :]
weight_tensor = torch.tensor(weights, dtype=base_emb.dtype, device=base_emb.device)
weight_tensor = weight_tensor.reshape(1, -1, 1).expand(base_emb.shape)
# m_token = (clip.tokenizer.end_token, 1.0) if clip.tokenizer.pad_with_end else (0,1.0)
# TODO: find most suitable masking token here
m_token = (m_token, 1.0)
ws = []
masked_tokens = []
masks = []
# create prompts
for id, w in weight_dict.items():
masked, m = mask_word_id(tokens, word_ids, id, m_token)
masked_tokens.extend(masked)
m = torch.tensor(m, dtype=base_emb.dtype, device=base_emb.device)
m = m.reshape(1, -1, 1).expand(base_emb.shape)
masks.append(m)
ws.append(w)
# batch process prompts
embs = batched_clip_encode(masked_tokens, length, encode_func, len(tokens))
masks = torch.cat(masks)
embs = base_emb.expand(embs.shape) - embs
pooled = embs[0, length - 1 : length, :]
embs *= masks
embs = embs.sum(axis=0, keepdim=True)
pooled_start = pooled_base.expand(len(ws), -1)
ws = torch.tensor(ws).reshape(-1, 1).expand(pooled_start.shape)
pooled = (pooled - pooled_start) * (ws - 1)
pooled = pooled.mean(axis=0, keepdim=True)
return ((weight_tensor - 1) * embs), pooled_base + pooled
def mask_inds(tokens, inds, mask_token):
clip_len = len(tokens[0])
inds_set = set(inds)
new_tokens = [
[mask_token if i * clip_len + j in inds_set else t for j, t in enumerate(x)] for i, x in enumerate(tokens)
]
return new_tokens
def down_weight(tokens, weights, word_ids, base_emb, length, encode_func):
w, w_inv = np.unique(weights, return_inverse=True)
if np.sum(w < 1) == 0:
return base_emb, tokens, base_emb[0, length - 1 : length, :]
# m_token = (clip.tokenizer.end_token, 1.0) if clip.tokenizer.pad_with_end else (0,1.0)
# using the comma token as a masking token seems to work better than aos tokens for SD 1.x
m_token = (266, 1.0)
masked_tokens = []
masked_current = tokens
for i in range(len(w)):
if w[i] >= 1:
continue
masked_current = mask_inds(masked_current, np.where(w_inv == i)[0], m_token)
masked_tokens.extend(masked_current)
embs = batched_clip_encode(masked_tokens, length, encode_func, len(tokens))
embs = torch.cat([base_emb, embs])
w = w[w <= 1.0]
w_mix = np.diff([0] + w.tolist())
w_mix = torch.tensor(w_mix, dtype=embs.dtype, device=embs.device).reshape((-1, 1, 1))
weighted_emb = (w_mix * embs).sum(axis=0, keepdim=True)
return weighted_emb, masked_current, weighted_emb[0, length - 1 : length, :]
def scale_emb_to_mag(base_emb, weighted_emb):
norm_base = torch.linalg.norm(base_emb)
norm_weighted = torch.linalg.norm(weighted_emb)
embeddings_final = (norm_base / norm_weighted) * weighted_emb
return embeddings_final
# For verification
def A1111_renorm(base_emb, weighted_emb):
embeddings_final = (base_emb.mean() / weighted_emb.mean()) * weighted_emb
return embeddings_final
def from_zero(weights, base_emb):
weight_tensor = torch.tensor(weights, dtype=base_emb.dtype, device=base_emb.device)
weight_tensor = weight_tensor.reshape(1, -1, 1).expand(base_emb.shape)
return base_emb * weight_tensor
def old_advanced_encode_from_tokens(
tokenized,
token_normalization,
weight_interpretation,
encode_func,
m_token=266,
w_max=1.0,
return_pooled=False,
apply_to_pooled=False,
**extra_args,
):
length = 77
tokens = [[t for t, _, _ in x] for x in tokenized]
weights = [[w for _, w, _ in x] for x in tokenized]
word_ids = [[wid for _, _, wid in x] for x in tokenized]
# weight normalization
# ====================
# distribute down/up weights over word lengths
if token_normalization.startswith("length"):
weights = divide_length(word_ids, weights)
# make mean of word tokens 1
if token_normalization.endswith("mean"):
weights = shift_mean_weight(word_ids, weights)
# weight interpretation
# =====================
pooled = None
if weight_interpretation in ["comfy", "perp"]:
weighted_tokens = [[(t, w) for t, w in zip(x, y)] for x, y in zip(tokens, weights)]
weighted_emb, pooled_base = encode_func(weighted_tokens)
pooled = pooled_base
else:
unweighted_tokens = [[(t, 1.0) for t, _, _ in x] for x in tokenized]
base_emb, pooled_base = encode_func(unweighted_tokens)
if weight_interpretation == "A1111":
weighted_emb = from_zero(weights, base_emb)
weighted_emb = A1111_renorm(base_emb, weighted_emb)
pooled = pooled_base
if weight_interpretation == "compel":
pos_tokens = [[(t, w) if w >= 1.0 else (t, 1.0) for t, w in zip(x, y)] for x, y in zip(tokens, weights)]
weighted_emb, _ = encode_func(pos_tokens)
weighted_emb, _, pooled = down_weight(pos_tokens, weights, word_ids, weighted_emb, length, encode_func)
if weight_interpretation == "comfy++":
weighted_emb, tokens_down, _ = down_weight(unweighted_tokens, weights, word_ids, base_emb, length, encode_func)
weights = [[w if w > 1.0 else 1.0 for w in x] for x in weights]
# unweighted_tokens = [[(t,1.0) for t, _,_ in x] for x in tokens_down]
embs, pooled = from_masked(unweighted_tokens, weights, word_ids, base_emb, length, encode_func)
weighted_emb += embs
if weight_interpretation == "down_weight":
weights = scale_to_norm(weights, word_ids, w_max)
weighted_emb, _, pooled = down_weight(unweighted_tokens, weights, word_ids, base_emb, length, encode_func)
if return_pooled:
if apply_to_pooled:
return weighted_emb, pooled
else:
return weighted_emb, pooled_base
return weighted_emb, None
+3 -7
View File
@@ -71,18 +71,13 @@ class AttentionCoupleHook(TransformerOptionsHook):
}
}
self.has_negpip = False
# calculate later. All clones must refer to the same kv dict
self.kv = {"k": None, "v": None}
self.kv = {}
def initialize_regions(self, base_cond, conds, fill):
self._base_cond = base_cond
self._conds = conds
self._fill = fill
self.num_conds = len(conds) + 1
self.base_strength = base_cond[1].get("strength", 1.0)
self.strengths = [cond[1].get("strength", 1.0) for cond in conds]
self.strengths: list[float] = [cond[1].get("strength", 1.0) for cond in conds]
self.conds: list[torch.Tensor] = [base_cond[0]] + [cond[0] for cond in conds]
base_mask = base_cond[1].get("mask", None)
masks = [cond[1].get("mask") * cond[1].get("mask_strength") for cond in conds]
@@ -219,6 +214,7 @@ class AttentionCoupleHook(TransformerOptionsHook):
dim=0,
)
)
assert self.num_conds is not None, "this is a bug"
cond_or_uncond_couple.extend(itertools.repeat(self.COND, self.num_conds))
q = torch.cat(qs, dim=0)
-47
View File
@@ -1,47 +0,0 @@
import comfy_execution.caching
from comfy_execution.graph_utils import is_link
import nodes
from os import environ
import logging
log = logging.getLogger("comfyui-prompt-control")
include_unique_id_in_input = comfy_execution.caching.include_unique_id_in_input
def promptcontrol_get_immediate_node_signature(self, dynprompt, node_id, ancestor_order_mapping):
if not dynprompt.has_node(node_id):
# This node doesn't exist -- we can't cache it.
return [float("NaN")]
node = dynprompt.get_node(node_id)
class_type = node["class_type"]
class_def = nodes.NODE_CLASS_MAPPINGS[class_type]
inputs = node["inputs"]
if hasattr(class_def, "CACHE_KEY"):
inputs = getattr(class_def, "CACHE_KEY")(inputs)
signature = [class_type, self.is_changed_cache.get(node_id)]
if (
self.include_node_id_in_input()
or (hasattr(class_def, "NOT_IDEMPOTENT") and class_def.NOT_IDEMPOTENT)
or include_unique_id_in_input(class_type)
):
signature.append(node_id)
for key in sorted(inputs.keys()):
if is_link(inputs[key]):
(ancestor_id, ancestor_socket) = inputs[key]
ancestor_index = ancestor_order_mapping[ancestor_id]
signature.append((key, ("ANCESTOR", ancestor_index, ancestor_socket)))
else:
signature.append((key, inputs[key]))
return signature
def init():
if environ.get("PROMPTCONTROL_ENABLE_CACHE_HACK") != "1":
return
log.warning("Enabling Prompt Control cache hack")
comfy_execution.caching.CacheKeySetInputSignature.get_immediate_node_signature = (
promptcontrol_get_immediate_node_signature
)
+1 -1
View File
@@ -215,7 +215,7 @@ def encode_regions(clip_regions, encode, tokenizer):
region_emb *= region_masking
region_embeddings.append(region_emb)
region_embeddings = torch.stack(region_embeddings).sum(axis=0)
region_embeddings = torch.stack(region_embeddings).sum(dim=0)
embeddings_final_mask = torch.tensor(
global_region_mask, dtype=base_embedding_full.dtype, device=base_embedding_full.device
+2 -1
View File
@@ -1,3 +1,4 @@
# pyright: reportSelfClsParameterName=false
import logging
from .prompts import encode_prompt
@@ -40,7 +41,7 @@ class PCTextEncode:
DESCRIPTION = "Encodes a prompt with extra goodies from Prompt Control. This node does *not* support scheduling"
def apply(self, clip, text):
return PCTextEncodeWithRange.apply(self, clip, text, 0.0, 1.0)
return PCTextEncodeWithRange().apply(clip, text, 0.0, 1.0)
NODE_CLASS_MAPPINGS = {"PCTextEncode": PCTextEncode, "PCTextEncodeWithRange": PCTextEncodeWithRange}
+4 -5
View File
@@ -1,3 +1,4 @@
# pyright: reportSelfClsParameterName=false
import logging
import comfy.hooks
@@ -38,7 +39,6 @@ def lora_hooks_from_schedule(schedules, non_scheduled):
all_hooks = []
def create_hook(loraspec, start_pct, end_pct, non_scheduled):
nonlocal lora_cache
hooks = []
hook_kf = comfy.hooks.HookKeyframeGroup()
for path, info in loras.items():
@@ -53,7 +53,8 @@ def lora_hooks_from_schedule(schedules, non_scheduled):
lora_cache[path], strength_model=info["weight"], strength_clip=info["weight_clip"]
)
# Set hook_ref so that identical hooks compare equal
new_hook.hooks[0].hook_ref = f"pc-{path}-{info['weight']}-{info['weight_clip']}"
ref = f"pc-{path}-{info['weight']}-{info['weight_clip']}"
new_hook.hooks[0].hook_ref = ref # pyright: ignore[reportAttributeAccessIssue]
hooks.append(new_hook)
if start_pct > 0.0:
kf = comfy.hooks.HookKeyframe(strength=0.0, start_percent=0.0)
@@ -74,8 +75,6 @@ def lora_hooks_from_schedule(schedules, non_scheduled):
all_hooks.append(hook)
start_pct = end_pct
del lora_cache
all_hooks = [x for x in all_hooks if x]
if all_hooks:
@@ -85,7 +84,7 @@ def lora_hooks_from_schedule(schedules, non_scheduled):
class PCAttentionCoupleBatchNegative(ComfyNodeABC):
@classmethod
def INPUT_TYPES(cls) -> InputTypeDict:
def INPUT_TYPES(s) -> InputTypeDict:
return {
"required": {
"positive": (IO.CONDITIONING, {}),
+4 -3
View File
@@ -1,3 +1,5 @@
# pyright: reportSelfClsParameterName=false
from __future__ import annotations
import logging
from .parser import parse_prompt_schedules
from comfy_execution.graph_utils import GraphBuilder, is_link
@@ -167,7 +169,7 @@ class PCLazyLoraLoaderAdvanced:
"hidden": {"unique_id": "UNIQUE_ID"},
}
RETURN_TYPES = ("MODEL", "CLIP", "HOOKS")
RETURN_TYPES: tuple[str, ...] = ("MODEL", "CLIP", "HOOKS")
OUTPUT_TOOLTIPS = ("Returns a model and clip with LoRAs scheduled",)
CATEGORY = "promptcontrol"
FUNCTION = "apply"
@@ -214,8 +216,7 @@ def build_scheduled_prompts(graph, schedules, clip):
classname = "PCTextEncode"
paramname = "text"
if classnames:
classname = classnames[0][0]
paramname = classnames[0][1]
classname, paramname = classnames[0].args
node = graph.node(classname)
node.set_input("clip", clip)
node.set_input(paramname, p)
+6 -22
View File
@@ -1,10 +1,11 @@
# pyright: reportSelfClsParameterName=false
import logging
from .parser import parse_prompt_schedules, expand_macros
from .nodes_lazy import NODE_CLASS_MAPPINGS as LAZY_NODES
from .utils import expand_graph
import json
import folder_paths
from pathlib import Path
from comfy_execution.graph_utils import is_link
log = logging.getLogger("comfyui-prompt-control")
@@ -20,7 +21,7 @@ class PCSaveExpandedWorkflow:
"any": ("*", {}),
},
"hidden": {
"prompt": "DYNPROMPT",
"prompt": "PROMPT",
},
}
@@ -31,7 +32,7 @@ class PCSaveExpandedWorkflow:
OUTPUT_NODE = True
RETURN_TYPES = ()
CATEGORY = "promptcontrol/tools"
DESCRIPTION = "Saves the current expanded dynamic prompt into a JSON file"
DESCRIPTION = "Expands lazy prompt control nodes in the prompt and saves the expanded prompt into a JSON file"
FUNCTION = "apply"
@@ -39,29 +40,12 @@ class PCSaveExpandedWorkflow:
full_output_folder, filename, counter, subfolder, prefix = folder_paths.get_save_image_path(
"pc_workflow_debug", self.output_dir
)
p = {}
input_replace_map = {}
for node in prompt.all_node_ids():
n = prompt.get_node(node)
t = n["class_type"]
if t in LAZY_NODES:
expanded_prompt = LAZY_NODES[t]().apply(**n["inputs"], unique_id=node)
for k in expanded_prompt["expand"]:
p[k] = expanded_prompt["expand"][k]
for i, _ in enumerate(expanded_prompt["result"]):
input_replace_map[(node, i)] = [k, i]
else:
p[node] = n
for k in p:
for ik in p[k]["inputs"]:
x = p[k]["inputs"][ik]
if is_link(x) and tuple(x) in input_replace_map:
p[k]["inputs"][ik] = input_replace_map[tuple(x)]
expanded = expand_graph(LAZY_NODES, prompt)
file = f"{filename}_{counter:05}_.json"
full_path = Path(full_output_folder) / file
with open(full_path, "w") as f:
log.info(f"Saving workflow to {full_path}")
json.dump(p, f)
json.dump(expanded, f)
return ()
+59 -21
View File
@@ -1,4 +1,5 @@
# vim: sw=4 ts=4
from __future__ import annotations
import lark
import logging
from math import ceil
@@ -24,10 +25,40 @@ if lark.__version__ == "0.12.0":
raise ImportError(x)
ESCAPES = [
("XxPCBackslashESCAPExX", "\\"),
("XxPCColonESCAPExX", ":"),
("XxPCCommentESCAPExX", "#"),
]
def escape_specials(string):
for ph, c in ESCAPES:
string = string.replace(rf"\{c}", ph)
return string
def restore_escaped(string):
for ph, c in ESCAPES:
string = string.replace(ph, c)
return string
def remove_comments(string):
r = []
for line in string.split("\n"):
comment = line.find("#")
if comment >= 0:
r.append(line[:comment])
else:
r.append(line)
return "\n".join(r)
prompt_parser = lark.Lark(
r"""
!start: (prompt | /[][():|]/+)*
prompt: (emphasized | embedding | scheduled | alternate | sequence | loraspec | PLAIN | /</ | />/ | WHITESPACE)+
prompt: (emphasized | embedding | scheduled | alternate | sequence | loraspec | PLAIN | | /\\:/ | /</ | />/ | WHITESPACE)+
!emphasized: "(" prompt? ")"
| "(" prompt ":" prompt ")"
| "[" prompt "]"
@@ -70,7 +101,15 @@ class CutTransform(lark.Transformer):
def cut(self, args):
prompt, cutout, weight, strict_mask, start_from_masked, mask_token = args
return ("".join(flatten(prompt)), "".join(flatten(cutout)), weight, strict_mask, start_from_masked, mask_token)
# prompts and cutouts are always sequences of str
return (
"".join(flatten(prompt)), # pyright: ignore
"".join(flatten(cutout)), # pyright: ignore
weight,
strict_mask,
start_from_masked,
mask_token,
) # pyright: ignore
def start(self, args):
prompt = []
@@ -241,7 +280,7 @@ def at_step(step, filters, tree):
return {"prompt": p, "loras": loraspecs}
def PLAIN(self, args):
return args.replace("\\:", ":")
return restore_escaped(args)
def FILENAME(self, value):
return str(value)
@@ -271,8 +310,7 @@ def at_step(step, filters, tree):
return name, params, lbw
def __default__(self, data, children, meta):
for child in children:
yield child
return children
return AtStep().transform(tree)
@@ -284,7 +322,8 @@ class PromptSchedule(object):
self.start = start
self.end = end
self.num_steps = num_steps
self.prompt = prompt.strip()
# placeholder is restored on parse
self.prompt = remove_comments(escape_specials(prompt.strip()))
self.defaults = {}
self.loaded_loras = {}
@@ -373,9 +412,11 @@ def parse_search(search):
args = ""
name = search.strip()
if arg_start > 0:
arg_end = find_closing_paren(search, arg_start)
arg_end = find_closing_paren(search, arg_start + 1)
if arg_end < 0:
arg_end = len(search)
name = search[:arg_start].strip()
args = search[arg_start + 1 : arg_end - 1]
args = search[arg_start + 1 : arg_end]
if not name:
return None
@@ -394,7 +435,9 @@ def expand_macros(text):
prevres = text
replacements = []
for d in defs:
r = d.split("=", 1)
if not d.args:
continue
r = d.args[0].split("=", 1)
search = parse_search(r[0].strip())
if not search or len(r) != 2:
log.warning("Ignoring invalid DEF(%s)", d)
@@ -417,21 +460,16 @@ def expand_macros(text):
return res
def substitute_def(text, search, replace):
search, default_args = search
for i, v in enumerate(default_args):
replace = re.sub(rf"\${i+1}\b", v, replace)
return re.sub(rf"\b{re.escape(search)}\b", replace, text)
def substitute_defcall(text, search, replace):
name, default_args = search
text, defns = get_function(text, name, defaults=None, placeholder=f"DEFNCALL{name}")
for i, parameters in enumerate(defns):
ph = f"\0DEFNCALL{name}{i}\0"
text, defns = get_function(text, name, defaults=None, placeholder=f"DEFNCALL{name}", require_args=False)
for i, d in enumerate(defns):
ph = d.placeholder
assert ph is not None, "This is a bug"
parameters = d.args
paramvals = []
if parameters is not None:
paramvals = [x.strip() for x in parameters.split(";")]
if parameters:
paramvals = [x.strip() for x in parameters[0].split(";")]
r = replace
for i, v in enumerate(paramvals):
r = re.sub(rf"\${i+1}\b", v, r)
+71 -41
View File
@@ -1,11 +1,23 @@
from __future__ import annotations
import logging
import re
import torch
import math
from functools import partial
from comfy_extras.nodes_mask import FeatherMask, MaskComposite
from nodes import ConditioningAverage
from .utils import safe_float, get_function, split_by_function, parse_floats, smarter_split
from .utils import (
safe_float,
get_function,
split_by_function,
parse_floats,
smarter_split,
call_node,
split_quotable,
FunctionSpec,
ComfyConditioning,
)
from .adv_encode import advanced_encode_from_tokens
from .cutoff import process_cuts
from .parser import parse_cuts
@@ -25,7 +37,7 @@ def get_sdxl(text, defaults):
text, sdxl = get_function(text, "SDXL", ["none", "none", "none"])
if not sdxl:
return text, {}
args = sdxl[0]
args = sdxl[0].args
d = defaults
w, h = parse_floats(args[0], [d.get("sdxl_width", 1024), d.get("sdxl_height", 1024)], split_re="\\s+")
tw, th = parse_floats(args[1], [d.get("sdxl_twidth", 1024), d.get("sdxl_theight", 1024)], split_re="\\s+")
@@ -46,7 +58,7 @@ def get_clipweights(text, existing_spec=None):
text, spec = get_function(text, "TE_WEIGHT", defaults=None)
if not spec:
return existing_spec or {}, text
args = spec[0].strip()
args = spec[0].args[0].strip()
res = {}
for arg in args.split(","):
try:
@@ -62,7 +74,7 @@ def get_style(text, default_style="comfy", default_normalization="none"):
text, styles = get_function(text, "STYLE", [default_style, default_normalization])
if not styles:
return default_style, default_normalization, text
style, normalization = styles[0]
style, normalization = styles[0].args
style = style.strip()
normalization = normalization.strip()
if style.replace("old+", "") not in AVAILABLE_STYLES:
@@ -77,8 +89,9 @@ def get_style(text, default_style="comfy", default_normalization="none"):
return style, normalization, text
def shuffle_chunk(shuffle, c):
func, shuffle = shuffle
def shuffle_chunk(func_spec: FunctionSpec, c: str) -> str:
func = func_spec.name
shuffle = func_spec.args
shuffle_count = int(safe_float(shuffle[0], 0))
_, separator, joiner = shuffle
if separator == "default":
@@ -128,11 +141,11 @@ def fix_word_ids(tokens):
def tokenize_chunks(clip, text, need_word_ids, can_break):
chunks = re.split(r"\bBREAK\b", text)
chunks = list(split_quotable(text, r"\bBREAK\b"))
token_chunks = []
shuffled_chunks = []
for c in chunks:
c, shuffles = get_function(c.strip(), "(SHIFT|SHUFFLE)", ["0", "default", "default"], return_func_name=True)
c, shuffles = get_function(c.strip(), "(SHIFT|SHUFFLE)", ["0", "default", "default"])
r = c
for s in shuffles:
r = shuffle_chunk(s, r)
@@ -168,9 +181,10 @@ def tokenize(clip, text, can_break, empty_tokens):
per_te_prompts = {}
if l_prompts:
log.warning("Note: CLIP_L is deprecated. Use TE(l=prompt) instead")
per_te_prompts["l"] = l_prompts
per_te_prompts["l"] = [x.args for x in l_prompts]
for prompt in te_prompts:
prompt = prompt.args[0]
if prompt.strip() == "help":
log.info("Encoders available for TE: %s", ", ".join(tokens.keys()))
continue
@@ -211,7 +225,7 @@ def encode_prompt_segment(
default_style="comfy",
default_normalization="none",
clip_weights=None,
) -> list[tuple[torch.Tensor, dict[str]]]:
) -> list[ComfyConditioning]:
style, normalization, text = get_style(text, default_style, default_normalization)
clip_weights, text = get_clipweights(text, clip_weights)
text, cuts = parse_cuts(text)
@@ -231,19 +245,18 @@ def encode_prompt_segment(
# Chunks to ConditioningAverage:
text, averages = split_by_function(text, "AVG", ["0.5"])
text, averages = split_by_function(text, "AVG", ["0.5"], require_args=False)
prompts_to_avg = []
for avg in averages:
w = safe_float(avg["args"][0], 0.5)
prompts_to_avg.append(text, w)
text = avg["text"]
for chunk, avg in averages:
w = safe_float(avg.args[0], 0.5)
prompts_to_avg.append((text, w))
text = chunk
prompts_to_avg.append((text, 1.0))
conds_to_avg = []
for prompt, weight in prompts_to_avg:
conds_to_cat = []
chunks = re.split(r"\bCAT\b", prompt)
for c in chunks:
for c in split_quotable(prompt, r"\bCAT\b"):
tokens = tokenize(clip, c, can_break, empty)
conds_to_cat.append(clip.encode_from_tokens_scheduled(tokens, add_dict=settings))
@@ -264,13 +277,22 @@ def encode_prompt_segment(
w = next_w
continue
for i in range(len(base)):
(cond,) = ConditioningAverage.addWeighted(None, [base[i]], [cond[i]], w)
(cond,) = call_node(ConditioningAverage, [base[i]], [cond[i]], w)
base[i] = cond[0]
w = next_w
return base
def calc_w(tensor, w):
if math.isclose(w, 0):
return torch.zeros_like(tensor)
elif math.isclose(w, 1.0):
return tensor
else:
return tensor * w
def apply_weights(output, te_name, spec):
"""Applies weights to TE outputs"""
if not spec:
@@ -282,7 +304,7 @@ def apply_weights(output, te_name, spec):
default = spec.get("all", None)
if isinstance(output, tuple):
out, pooled = output
out, pooled, *extra = output
pkey = te_name + "_pooled"
if te_name in spec or pkey in spec or default is not None:
w = spec.get(te_name, default)
@@ -292,16 +314,16 @@ def apply_weights(output, te_name, spec):
if pooled_w is None:
pooled_w = 1.0
log.info("Weighting %s output by %s, pooled by %s", te_name, w, pooled_w)
out = out * w
out = calc_w(out, w)
if pooled is not None:
pooled = pooled * pooled_w
pooled = calc_w(pooled, pooled_w)
return out, pooled
return (out, pooled) + tuple(extra)
else:
if te_name in spec or default is not None:
w = spec.get(te_name, default)
log.info("Weighting %s output by %s", te_name, w)
output = output * w
output = calc_w(output, w)
return output
@@ -356,7 +378,7 @@ def get_area(text):
if not areas:
return text, None
args = areas[0]
args = areas[0].args
x, w = parse_floats(args[0], [0.0, 1.0], split_re="\\s+")
y, h = parse_floats(args[1], [0.0, 1.0], split_re="\\s+")
weight = safe_float(args[2], 1.0)
@@ -383,7 +405,7 @@ def get_mask_size(text, defaults):
text, sizes = get_function(text, "MASK_SIZE", ["512", "512"])
if not sizes:
return text, (defaults.get("mask_width", 512), defaults.get("mask_height", 512))
w, h = sizes[0]
w, h = sizes[0].args
return text, (int(w), int(h))
@@ -429,46 +451,53 @@ def get_mask(text, size, input_masks):
def feather(f, mask):
l, t, r, b, *_ = [int(x) for x in parse_floats(f[0], [0, 0, 0, 0], split_re="\\s+")]
mask = FeatherMask().feather(mask, l, t, r, b)[0]
mask = call_node(FeatherMask, mask, l, t, r, b)[0]
log.info("FeatherMask l=%s, t=%s, r=%s, b=%s", l, t, r, b)
return mask
mask = None
totalweight = 1.0
if maskw:
totalweight = safe_float(maskw[0][0], 1.0)
totalweight = safe_float(maskw[0].args[0], 1.0)
i = 0
for m in masks:
weight = safe_float(m[2], 1.0)
op = m[3]
nextmask = make_mask(m, size, weight)
weight = safe_float(m.args[2], 1.0)
op = m.args[3]
nextmask = make_mask(m.args, size, weight)
if i < len(feathers):
nextmask = feather(feathers[i], nextmask)
nextmask = feather(feathers[i].args, nextmask)
i += 1
if mask is not None:
log.info("MaskComposite op=%s", op)
mask = MaskComposite().combine(mask, nextmask, 0, 0, op)[0]
mask = call_node(MaskComposite, mask, nextmask, 0, 0, op)[0]
else:
mask = nextmask
for idx, w, op in imasks:
for im in imasks:
idx, w, op = im.args
idx = int(safe_float(idx, 0.0))
w = safe_float(w, 1.0)
if input_masks is None:
log.warn(
"IMASK requires you to attach custom masks to the CLIP object using PCAddMasksToClIP before using it"
)
input_masks = []
if len(input_masks) < idx + 1:
log.warn("IMASK index %s not found, ignoring...", idx)
continue
nextmask = input_masks[idx] * w
if i < len(feathers):
nextmask = feather(feathers[i], nextmask)
nextmask = feather(feathers[i].args, nextmask)
i += 1
if mask is not None:
mask = MaskComposite().combine(mask, nextmask, 0, 0, op)[0]
mask = call_node(MaskComposite, mask, nextmask, 0, 0, op)[0]
else:
mask = nextmask
# apply leftover FEATHER() specs to the whole
for f in feathers[i:]:
mask = feather(f, mask)
mask = feather(f.args, mask)
return text, mask, totalweight
@@ -483,14 +512,15 @@ def get_noise(text):
return text, None, None
w = 0
# Only take seed from first noise spec, for simplicity
seed = safe_float(noises[0][1], "none")
seed = noises[0].args[0].strip()
if seed == "none":
gen = None
else:
seed = safe_float(seed, 0)
gen = torch.Generator()
gen.manual_seed(int(seed))
for n in noises:
w += safe_float(n[0], 0.0)
w += safe_float(n.args[0], 0.0)
return text, max(min(w, 1.0), 0.0), gen
@@ -551,7 +581,7 @@ def encode_prompt(clip, text, start_pct, end_pct, defaults, masks):
style, normalization, text = get_style(text)
text, mask_size = get_mask_size(text, defaults)
prompts = [p.strip() for p in re.split(r"\bAND\b", text)]
prompts = list(split_quotable(text, r"\bAND\b"))
p, sdxl_opts = get_sdxl(prompts[0], defaults)
prompts[0] = p
@@ -573,9 +603,9 @@ def encode_prompt(clip, text, start_pct, end_pct, defaults, masks):
return f"MASK({args})"
for prompt in prompts:
base_prompt, attn_couple_prompts = split_by_function(prompt, "COUPLE", defaults=None)
base_prompt, attn_couple_prompts = split_by_function(prompt, "COUPLE", defaults=None, require_args=False)
prompts = [base_prompt] + [couple_mask(p["args"]) + p["text"] for p in attn_couple_prompts]
prompts = [base_prompt] + [couple_mask(f.args) + chunk for (chunk, f) in attn_couple_prompts]
encoded = []
for p in prompts:
p, settings = process_settings(p, defaults, masks, mask_size, sdxl_opts)
+66 -28
View File
@@ -1,6 +1,10 @@
import unittest
import unittest.mock as mock
import numpy.testing as npt
from os import environ
import nodes
import comfy_extras.nodes_mask
from .nodes_base import PCTextEncode
clips = []
@@ -10,10 +14,40 @@ logging.basicConfig()
def run(f, *args):
return getattr(f, f.FUNCTION)(*args)
if hasattr(f, "execute"):
return f.execute(*args)
else:
return getattr(f, f.FUNCTION)(*args)
class TestEncode(unittest.TestCase):
@classmethod
def setUpClass(cls):
print("Loading ComfyUI")
from comfy.sd import load_clip
from pathlib import Path
to_test = environ.get("TEST_TE", "clip_l").split()
model_dir = environ.get("COMFYUI_TE_DIR", ".")
te_root = Path(model_dir).resolve()
if "clip_l" in to_test:
clip_l = load_clip(
ckpt_paths=[str(te_root / "clip_l.safetensors")], clip_type="stable_diffusion", model_options={}
)
clips.append(("clip_l", clip_l))
if "t5" in to_test:
dual = load_clip(
[str(te_root / "clip_l.safetensors"), str(te_root / "t5xxl_fp16.safetensors")],
clip_type="flux",
model_options={},
)
clips.append(("clip_l+t5", dual))
print("Starting tests")
def tensorsEqual(self, t1, t2):
npt.assert_equal(t1.detach().numpy(), t2.detach().numpy())
@@ -30,6 +64,7 @@ class TestEncode(unittest.TestCase):
pc = PCTextEncode()
comfy = nodes.CLIPTextEncode()
combine = nodes.ConditioningCombine()
average = nodes.ConditioningAverage()
concat = nodes.ConditioningConcat()
zeroout = nodes.ConditioningZeroOut()
for k, clip in clips:
@@ -46,6 +81,17 @@ class TestEncode(unittest.TestCase):
c = c2 # Used in later tests
self.condEqual(c1, c2)
with self.subTest("Quotes"):
(c1,) = run(pc, clip, 'Text saying "DOG MASK AND CAT COUPLE MASK(X)"')
(c2,) = run(comfy, clip, 'Text saying "DOG MASK AND CAT COUPLE MASK(X)"')
self.condEqual(c1, c2)
with self.subTest("Function cornercase"):
(c1,) = run(pc, clip, "test SDXL function")
(c2,) = run(comfy, clip, "test SDXL function")
(c3,) = run(pc, clip, "test SDXL() function")
self.condEqual(c1, c2)
with self.subTest("Weights"):
(c1,) = run(pc, clip, "(test:1.2) (test:0.6)")
(c2,) = run(comfy, clip, "(test:1.2) (test:0.6)")
@@ -66,6 +112,25 @@ class TestEncode(unittest.TestCase):
(c2,) = run(zeroout, c)
self.condEqual(c1, c2)
with self.subTest("Average"):
(c1,) = run(comfy, clip, "test1")
(c2,) = run(comfy, clip, "test2")
(c3,) = run(pc, clip, "test1 AVG() test2")
(c4,) = run(pc, clip, "test1 AVG test2")
(avg,) = run(average, c1, c2, 0.5)
self.condEqual(avg, c3)
self.condEqual(avg, c4)
@unittest.expectedFailure
def test_failure(self):
pc = PCTextEncode()
comfy = nodes.CLIPTextEncode()
for k, clip in clips:
with self.subTest(k):
(c1,) = run(comfy, clip, "test SDXL function")
(c2,) = run(pc, clip, "test SDXL() function")
self.condEqual(c1, c2)
def test_weight(self):
pc = PCTextEncode()
comfy = nodes.CLIPTextEncode()
@@ -127,31 +192,4 @@ class TestEncode(unittest.TestCase):
if __name__ == "__main__":
print("Loading ComfyUI")
from comfy.sd import load_clip
import nodes
import comfy_extras.nodes_mask
from .nodes_base import PCTextEncode
from pathlib import Path
to_test = environ.get("TEST_TE", "clip_l").split()
model_path = environ.get("COMFYUI_MODEL_ROOT", ".")
te_root = (Path(model_path) / "text_encoders").resolve()
if "clip_l" in to_test:
clip_l = load_clip(
ckpt_paths=[str(te_root / "clip_l.safetensors")], clip_type="stable_diffusion", model_options={}
)
clips.append(("clip_l", clip_l))
if "t5" in to_test:
dual = load_clip(
[str(te_root / "clip_l.safetensors"), str(te_root / "t5xxl_fp16.safetensors")],
clip_type="flux",
model_options={},
)
clips.append(("clip_l+t5", dual))
print("Starting tests")
unittest.main()
-56
View File
@@ -1,56 +0,0 @@
import unittest
import numpy.testing as npt
clip_l = None
dual = None
def run(f, *args):
return getattr(f, f.FUNCTION)(*args)
class TestEncode(unittest.TestCase):
def tensorsEqual(self, t1, t2):
npt.assert_equal(t1.detach().numpy(), t2.detach().numpy())
def condEqual(self, c1, c2, key=None, key_assert=None):
self.assertEqual(len(c1), len(c2))
for i in range(len(c1)):
a, b = c1[i], c2[i]
if key:
(key_assert or self.assertEqual)(a[1][key], b[1][key])
else:
self.tensorsEqual(a[0], b[0])
def test_styles(self):
pc = PCTextEncode()
for k, clip in [("l", clip_l), ("dual", dual)]:
for style in ["comfy++", "A1111", "comfy++", "compel", "down_weight"]:
with self.subTest(f"TE {k} style {style} does not fail when encoding weights"):
for normalization in ["none", "mean", "length", "length+mean"]:
with self.subTest(f"TE {k} style {style} normalization {normalization}"):
(c,) = run(
pc,
clip,
f"STYLE(old+{style}, {normalization}) this prompt has weights, (a:1.2) (b:1.2)",
)
(c2,) = run(
pc,
clip,
f"STYLE({style}, {normalization}) this prompt has weights, (a:1.2) (b:1.2)",
)
self.condEqual(c, c2)
if __name__ == "__main__":
print("Loading ComfyUI")
import main
id(main) # get rid of flake warning
import nodes
from .nodes_base import PCTextEncode
(clip_l,) = nodes.CLIPLoader().load_clip("clip_l.safetensors")
(dual,) = nodes.DualCLIPLoader().load_clip("clip_l.safetensors", "clip_g.safetensors", "sdxl")
print("Starting tests")
unittest.main()
+40
View File
@@ -18,6 +18,13 @@ class TestParser(unittest.TestCase):
self.assertEqual(p.at_step(0.5), expected)
self.assertEqual(p.at_step(1), expected)
def test_quote(self):
p = parse('This is a text with a "QUOTED DEF(X=Y)"')
expected = prompt(1.0, 'This is a text with a "QUOTED DEF(X=Y)"')
self.assertEqual(p.at_step(0), expected)
self.assertEqual(p.at_step(0.5), expected)
self.assertEqual(p.at_step(1), expected)
def test_equivalences(self):
eqs = [
[parse(p) for p in ["[a:0.1]", "[:a:0.1]", "[:a:0,0.1]", "[:a::0.1,1.0]", "[:a::0.1]"]],
@@ -156,6 +163,39 @@ class TestParser(unittest.TestCase):
expand_macros("DEF(X=recurse Y) DEF(Y=recurse X) X")
self.assertTrue("Unable to resolve DEFs" in str(c.exception))
def test_escapes(self):
p = parse(r"[a:\:a:0.5] :\[a:b:0.5]")
self.assertPrompt(p, 0, 0.5, r"a :\[a:b:0.5]")
self.assertPrompt(p, 0.55, 1, r":a :\[a:b:0.5]")
p = parse(r"[embedding\:a:embedding\:b:0.1,0.5]")
self.assertPrompt(p, 0.15, 0.5, r"embedding:a")
self.assertPrompt(p, 0.55, 1, r"embedding:b")
p = parse(r"[embedding\:a:embedding\:b:embedding\:c:0.1,0.5]")
self.assertPrompt(p, 0.0, 0.1, r"embedding:a")
self.assertPrompt(p, 0.15, 0.5, r"embedding:b")
self.assertPrompt(p, 0.55, 1, r"embedding:c")
p = parse(r"[a\:b\\:c:0.5]")
self.assertPrompt(p, 0.0, 0.5, "a:b\\")
self.assertPrompt(p, 0.55, 1, r"c")
p = parse(r"[a:\#b:0.5]")
self.assertPrompt(p, 0.0, 0.5, "a")
self.assertPrompt(p, 0.55, 1, "#b")
def test_comments(self):
p = parse("this is a # comment")
self.assertPrompt(p, 0, 1.0, "this is a ")
p = parse("this is a [comment#:scheduled:0.6]")
self.assertPrompt(p, 0, 1.0, "this is a [comment")
p = parse(r"this is a [comment\#:scheduled:0.6]")
self.assertPrompt(p, 0, 0.6, "this is a comment#")
self.assertPrompt(p, 0.65, 1.0, "this is a scheduled")
p = parse("#this is a comment\nthis is a prompt")
self.assertPrompt(p, 0, 1.0, "\nthis is a prompt")
def test_misc(self):
p = parse("[[a:c:0.5]:0.7]")
p2 = parse("[:[a:c:0.5]:0.7]")
+151 -46
View File
@@ -1,19 +1,48 @@
from __future__ import annotations
from pathlib import Path
import re
import logging
import copy
from dataclasses import dataclass
from typing import Any, TypeAlias, Iterator, TypeVar, TYPE_CHECKING
if TYPE_CHECKING:
import torch # flakes8: noqa
FunctionArgs: TypeAlias = list[str]
ComfyConditioning: TypeAlias = tuple["torch.Tensor", dict[str, Any]]
@dataclass
class FunctionSpec:
name: str
args: FunctionArgs
position: int
placeholder: str | None
# Allow testing
try:
from folder_paths import get_filename_list
except ImportError:
def get_filename_list(x):
raise NotImplementedError("How did you get here?")
def get_filename_list(folder_name) -> list[str]:
return []
log = logging.getLogger("comfyui-prompt-control")
def call_node(cls, *args, **kwargs):
if hasattr(cls, "execute"):
# v3 node
return cls.execute(*args, **kwargs)
else:
func = getattr(cls(), cls.FUNCTION)
return func(*args, **kwargs)
def consolidate_schedule(prompt_schedule):
prev_loras = {}
not_found = []
@@ -54,10 +83,11 @@ def find_nonscheduled_loras(consolidated_schedule):
return {k: v for (k, v) in candidate_loras.items() if k not in to_remove}
def smarter_split(separator, string):
def smarter_split(separator: str, string: str) -> list[str]:
"""Does not break () when splitting"""
splits = []
prev = 0
idx = 0
stack = 0
escape = False
for idx, x in enumerate(string):
@@ -74,7 +104,7 @@ def smarter_split(separator, string):
return splits
def find_closing_paren(text, start):
def find_closing_paren(text: str, start: int) -> int:
stack = 1
for i, char in enumerate(text[start:]):
if char == ")":
@@ -83,90 +113,124 @@ def find_closing_paren(text, start):
stack += 1
if stack == 0:
return start + i
# Implicit closing paren after end
return len(text)
return -1
def get_function(text, func, defaults, return_func_name=False, placeholder="", return_dict=False):
rex = re.compile(rf"\b{func}\b", re.MULTILINE)
instances = []
def find_function_spans(
text: str, func: str, require_args: bool, defaults: FunctionArgs | None
) -> Iterator[tuple[int, int, str, FunctionArgs]]:
if require_args:
rex = re.compile(rf"\b{func}\(", re.MULTILINE)
else:
rex = re.compile(rf"\b{func}\b", re.MULTILINE)
idx = 0
match = rex.search(text)
count = 0
while match:
# Match start, content start
start, at_paren = match.span()
if require_args:
at_paren = at_paren - 1
funcname = text[start:at_paren]
after_first_paren = at_paren + 1
if text[at_paren:after_first_paren] == "(":
end = find_closing_paren(text, after_first_paren)
if end < 0:
continue
args = parse_strings(text[after_first_paren:end], defaults)
end += 1
else:
end = at_paren
args = None
args = defaults or []
yield idx + start, idx + end, funcname, args
idx = idx + end
text = text[end:]
match = rex.search(text)
def get_function(
text: str, func: str, defaults: list[str] | None, placeholder: str = "", require_args: bool = True
) -> tuple[str, list[FunctionSpec]]:
spans = [x.span() for x in re.finditer(r'".+?"', text)]
instances = []
count = 0
chunks = []
current = 0
for start, end, funcname, args in find_function_spans(text, func, require_args, defaults):
ph = None
if spans_include(spans, start, end):
continue
if placeholder:
ph = f"\0{placeholder}{count}\0"
if return_dict:
instances.append(
{
"name": funcname,
"args": args,
"position": start,
"placeholder": ph,
}
)
elif return_func_name:
instances.append((funcname, args))
else:
instances.append(args)
if placeholder:
text = text[:start] + f"\0{placeholder}{count}\0" + text[end:]
else:
text = text[:start] + text[end:]
match = rex.search(text)
instances.append(FunctionSpec(funcname, args, start, ph))
chunks.append(text[current:start] + (ph or ""))
current = end
count += 1
chunks.append(text[current:])
text = "".join(chunks)
return text, instances
def split_by_function(text, func, defaults=None):
def spans_include(spans: list[tuple[int, int]], s: int, e: int) -> bool:
return any((s > a and e < b) for a, b in spans)
def split_quotable(text: str, regexp: str) -> Iterator[str]:
start_from = 0
spans = [x.span() for x in re.finditer(r'".+?"', text)]
for x in re.finditer(regexp, text):
s, e = x.span()
if not spans_include(spans, s, e):
yield text[start_from:s].strip()
start_from = e
yield text[start_from:].strip()
def split_by_function(
text: str, func: str, defaults: list[str] | None = None, require_args: bool = True
) -> tuple[str, list[tuple[str, FunctionSpec]]]:
"""
Splits a string by function calls, returning the text preceding the first call and a list of dictionaries with a "text" key with the prompt before the next split or until hthe end of the text.
Splits a string by function calls, returning the leftover text along with a list of functions with their associated text chunk.
"""
text, functions = get_function(text, func, defaults, return_dict=True)
text, functions = get_function(text, func, defaults, require_args=require_args)
chunks = []
prev = 0
for f in functions:
chunks.append(text[prev : f["position"]])
prev = f["position"]
chunks.append(text[prev : f.position])
prev = f.position
chunks.append(text[prev:])
r = []
for i, f in enumerate(functions):
f["text"] = chunks[i + 1]
return chunks[0], functions
r.append((chunks[i + 1], f))
return chunks[0], r
def parse_args(strings, arg_spec, strip=True):
T = TypeVar("T")
def parse_args(strings: list[str], arg_spec: list[tuple[Any, T]], strip: bool = True) -> list[T]:
args = [s[1] for s in arg_spec]
for i, spec in list(enumerate(arg_spec))[: len(strings)]:
try:
if strip:
strings[i] = strings[i].strip()
args[i] = spec[0](strings[i])
f = spec[0]
args[i] = f(strings[i])
except ValueError:
pass
return args
def parse_floats(string, defaults, split_re=","):
def parse_floats(string: str, defaults: list[float], split_re: str = ",") -> list[float]:
spec = [(float, d) for d in defaults]
return parse_args(re.split(split_re, string.strip()), spec)
def parse_strings(string, defaults, split_re=r"(?<!\\),", replace=(r"\,", ",")):
def parse_strings(
string: str, defaults: FunctionArgs | None, split_re: str = r"(?<!\\),", replace: tuple[str, str] = (r"\,", ",")
) -> FunctionArgs:
if defaults is None:
return string
spec = [(lambda x: x, d) for d in defaults]
return [string]
spec = [(str, d) for d in defaults]
splits = re.split(split_re, string)
if replace:
f, t = replace
@@ -174,7 +238,7 @@ def parse_strings(string, defaults, split_re=r"(?<!\\),", replace=(r"\,", ",")):
return parse_args(splits, spec, strip=False)
def safe_float(f, default):
def safe_float(f: Any, default: float) -> float:
if f is None:
return default
try:
@@ -183,7 +247,7 @@ def safe_float(f, default):
return default
def lora_name_to_file(name):
def lora_name_to_file(name: str) -> str | None:
filenames = get_filename_list("loras")
# Return exact matches as is
if name in filenames:
@@ -194,4 +258,45 @@ def lora_name_to_file(name):
p = Path(f).with_suffix("")
if p.name == n or str(p) == n:
return f
# Finally, try to find unique match from parts
parts = name.split()
search = [f for f in filenames if all(p in f for p in parts)]
if len(search) == 1:
return search[0]
return None
def map_inputs(input_map, inputs):
new_inputs = {}
for k in inputs:
key = inputs[k]
new_inputs[k] = key
if isinstance(key, list):
key = tuple(key)
x = input_map.get(key, inputs[k])
new_inputs[k] = x
return new_inputs
def expand_graph(node_mappings, graph):
input_map = {}
new_graph = copy.deepcopy(graph)
for k in graph:
data = graph[k]
if not isinstance(data, dict) or "class_type" not in data or data["class_type"] not in node_mappings:
continue
node = node_mappings[data["class_type"]]()
inputs = map_inputs(input_map, data["inputs"].copy())
inputs["unique_id"] = k
fn = getattr(node, getattr(node, "FUNCTION"))
expansion = fn(**inputs)
for i, v in enumerate(expansion["result"]):
input_map[(k, i)] = v
del new_graph[k]
new_graph.update(expansion["expand"])
for k in new_graph:
data = new_graph[k]
data["inputs"] = map_inputs(input_map, data["inputs"])
return new_graph
+6 -2
View File
@@ -1,7 +1,7 @@
[project]
name = "comfyui-prompt-control"
description = "Nodes for convenient prompt editing, making many common operations prompt-controllable"
version = "2.0.0-rc.8"
description = "Provides nodes for prompt editing and LoRA scheduling, advanced regional prompting (including attention masking) and more, all controlled through your text prompt"
version = "2.1.1"
license = { file = "LICENSE" }
# some lark versions older than 1.1.9 apparently have a bug that breaks things, see https://github.com/asagi4/comfyui-prompt-control/issues/35
dependencies = ["lark >= 1.1.9"]
@@ -13,3 +13,7 @@ Repository = "https://github.com/asagi4/comfyui-prompt-control"
PublisherId = "asagi4"
DisplayName = "ComfyUI Prompt Control"
Icon = ""
[tool.pyright]
extraPaths = ["../../"]
exclude = ["prompt_control/*test*"]
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env python3
from prompt_control.utils import expand_graph
from prompt_control.nodes_lazy import NODE_CLASS_MAPPINGS as LN
import json
import sys
# Needs ComfyUI in Python path
# Usage: PYTHONPATH=../..:. python tools/expand_graph < graph_in_api_format.json > out.json
if __name__ == "__main__":
graph = json.load(sys.stdin)
new = expand_graph(LN, graph)
print(json.dumps(new))
+3
View File
@@ -0,0 +1,3 @@
# PC: Attach Mask
Attaches custom masks to a CLIP object so that they can be referred to in prompts using `PCTextEncode` or `PC: Schedule prompt`.
+1
View File
@@ -0,0 +1 @@
PCAddMaskToCLIP.md
@@ -0,0 +1,7 @@
# PC: Attention Couple (batch negative)
This node applies an optimization that re-enables negative cond batching when Attention Couple is in use.
It improves performance when negative prompts are not scheduled, but slightly affects outputs and is not required for Attention Couple to work.
Simply add it to your workflow and pass in your positive and negative prompts. It is always safe to use, as it will not do anything when it detects that the optimization can't be applied (eg. when negative prompts contain schedules)
+7
View File
@@ -0,0 +1,7 @@
# PC: Schedule LoRAs
This node is the core of Prompt Control. It evaluates a prompt schedule and dynamically expands into a scheduled workflow consisting of necessary calls to `LoRALoader` and `Create Hook LoRA` (for scheduled LoRAs).
You can use it in place or in addition to your usual `LoRA Loader` nodes; just pass in a text prompt containing your LoRA schedule (it can be shared with `PC: Schedule Prompt`). Then connect your MODEL output as usual and the CLIP output to your `PC: Schedule Prompt` nodes.
For documentation on syntax, for now see the [documentation on GitHub](https://github.com/asagi4/comfyui-prompt-control/blob/master/doc/schedules.md)
+1
View File
@@ -0,0 +1 @@
PCLazyLoraLoader.md
+7
View File
@@ -0,0 +1,7 @@
# PC: Schedule Prompt
This node is the core of Prompt Control. It evaluates a prompt schedule and dynamically expands into a scheduled workflow consisting of calls to `PCTextEncode`, `SetConditioningTimesteps` and other necessary nodes.
To use it, simply replace your usual `CLIP Text Encode` nodes with `PC: Schedule Prompt` nodes. For LoRA Loading, you should use `PC: Schedule LoRAs` in place (or in addition to) of your usual LoRA Loader node.
For documentation on syntax, for now see the [documentation on GitHub](https://github.com/asagi4/comfyui-prompt-control/blob/master/doc/schedules.md)
+1
View File
@@ -0,0 +1 @@
PCLazyTextEncode.md
+5
View File
@@ -0,0 +1,5 @@
# PC: LoRA Hooks from Text (non-lazy)
Creates cond hooks from a LoRA schedule, if you want to apply them manually for some reason.
You should not need to use this. Use `PC: Schedule LoRAs`.
+5
View File
@@ -0,0 +1,5 @@
# PC: Expand Macros
Expands [prompt macros](https://github.com/asagi4/comfyui-prompt-control/blob/master/doc/macros.md)
You should not need to use this directly. Use `PC: Schedule Prompt` instead.
+7
View File
@@ -0,0 +1,7 @@
# PC: Configure PCTextEncode
Configures a CLIP object with new default values used by `PCTextEncode`. Apply it before everything else.
This is needed if you want to do scheduling with steps instead of denoising percentages, but otherwise it's completely optional.
Note that steps are simply syntactic sugar for percentages and may not correspond to actual steps depending on the scheduler used.
+5
View File
@@ -0,0 +1,5 @@
# PC: Text Encode (no scheduling)
This node encodes text using some special syntax for advanced features. You should rarely need to use this node directly, and instead use `PC: Schedule Prompt` which uses this node under the hood.
For documentation on syntax, see the [documentation on GitHub](https://github.com/asagi4/comfyui-prompt-control/blob/master/doc/basic.md)