* Fix process wildcards setting.

* Better debug log.
* Set commands support lazy evaluation and content addition.
* Unified in one tree and processing phase.
* Supports Flux.
* Fix evaluation of string variable as truthy.
* Show invalid wildcards.
* Better detection of installed UI.
* Improved cache of results.
* Works as a ComfyUI node.
* Debug setting changed to debug level.
* Fixes in variable use.
* Added timing reporting.
* Detection of more model types.
* Installation script for requirements.
* Extension metadata for A1111.
* Lark grammar in separate file and cached.
* Improved conditions for if command.
* Avoid repeats in the processing of prompts.
* Fix multiple loading of settings.

.markdownlint.jsonc

@@ -0,0 +1,5 @@
+{
+    "default": true,
+    "MD013": false,
+    "MD024": false
+}

.vscode/launch.json

@@ -6,7 +6,7 @@
     "configurations": [
         {
             "name": "Tests",
-            "type": "python",
+            "type": "debugpy",
             "request": "launch",
             "program": "tests/tests.py",
             "console": "integratedTerminal",

.vscode/settings.json

@@ -9,7 +9,7 @@
     ],
     "python.testing.pytestEnabled": false,
     "python.testing.unittestEnabled": true,
-    "python.analysis.typeCheckingMode": "basic",
+    "python.analysis.typeCheckingMode": "off",
     "black-formatter.args": [
         "--line-length=120"
     ]

README.md

@@ -1,111 +1,355 @@
-# Send to Negative for Stable Diffusion WebUI
+# Prompt Postprocessor for Stable Diffusion WebUI and ComfyUI
 
-Extension for the [AUTOMATIC1111 Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) or compatible UIs.
+The Prompt Postprocessor, formerly known as "sd-webui-sendtonegative", is an extension designed to process the prompt, possibly after other extensions have modified it. This extension is compatible with:
 
-## Purpose
+* [AUTOMATIC1111 Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
+* [SD.Next](https://github.com/vladmandic/automatic).
+* [Forge](https://github.com/lllyasviel/stable-diffusion-webui-forge)
+* [reForge](https://github.com/Panchovix/stable-diffusion-webui-reForge)
+* ...and probably other forks
+* [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
 
-This extension allows the tagging of parts of the prompt and moves them to the
+Currently this extension has these functions:
-negative prompt. This allows useful tricks when using a wildcard extension
-since you can add negative content from choices made in the positive prompt.
 
-Note: The extension must be loaded after the installed wildcards extension. Extensions
+* Sending parts of the prompt to the negative prompt. This allows for useful tricks when using wildcards since you can add negative content from choices made in the positive prompt.
-load by their folder in alphanumeric order.
+* Set and modify local variables.
+* Filter content based on the loaded SD model or a variable.
+* Process wildcards. Compatible with Dynamic Prompts formats. Can also detect invalid wildcards and act as you choose.
+* Clean up the prompt and negative prompt.
 
-With the ["Dynamic Prompts" extension](https://github.com/adieyal/sd-dynamic-prompts)
+Note: when used in an A1111 compatible webui, the extension must be loaded after any other extension that modifies the prompt (like another wildcards extension). Usually extensions load by their folder name in alphanumeric order, so if the extensions are not loading in the correct order just rename this extension's folder so the ordering works out. When in doubt, just rename this extension's folder with a "z" in front (for example) so that it is the last one to load, or manually set such folder name when installing it.
-this happens by default due to default folder names for both extensions. But if
-this is not the case, you can just rename the extension folder so the ordering
-works out.
 
-With the ["AUTOMATIC1111 Wildcards" extension](https://github.com/AUTOMATIC1111/stable-diffusion-webui-wildcards)
+Notes:
-you will have to rename one of the folders, so that it loads before than "Send to Negative".
 
-When in doubt, just rename this extension's folder with a "z" in front (for example) so that it is the last one to load, or manually set such folder name when installing it.
+1. Other than its own commands, it only recognizes regular A1111 prompt formats. So:
 
-Note: it only recognizes regular A1111 prompt formats. So:
+    * **Attention**: `\[prompt\] (prompt) (prompt:weight)`
+    * **Alternation**: `\[prompt1|prompt2|...\]`
+    * **Scheduling**: `\[prompt1:prompt2:step\]`
+    * **Extra networks**: `\<kind:model...\>`
+    * **BREAK**: `prompt1 BREAK prompt2`
+    * **Composable Diffusion**: `prompt1:weight1 AND prompt2:weight2`
 
-* Attention: \[prompt\] (prompt) (prompt:weight)
+    In SD.Next that means only the *A1111* or *Full* parsers. It will warn you if you use the *Compel* parser.
-* Alternation: \[prompt1|prompt2|...\]
+2. It recognizes wildcards in the *\_\_wildcard\_\_* and *{choice|choice}* formats (and anything that [Dynamic Prompts](https://github.com/adieyal/sd-dynamic-prompts) supports).
-* Scheduling: \[prompt1:prompt2:step\]
+3. It does not create *AND/BREAK* constructs when moving content to the negative prompt.
-
-In SD.Next that means only the A1111 or Full parsers.
-
-It does not build AND/BREAK separations into the negative prompt.
 
 ## Installation
 
+On A1111 compatible webuis:
+
 1. Go to Extensions > Install from URL
-2. Paste <https://github.com/acorderob/sd-webui-sendtonegative> in the URL for extension's git repository text field
+2. Paste <https://github.com/acorderob/sd-webui-prompt-postprocessor> in the URL for extension's git repository text field
 3. Click the Install button
 4. Restart the webui
 
+On ComfyUI:
+
+1. Go to Manager > Custom Nodes Manager
+2. Install through ComfyUI Manager
+3. Click Install via Git URL and enter <https://github.com/acorderob/sd-webui-prompt-postprocessor>
+4. Restart
+
 ## Usage
 
-The format of the tags is like this:
+### Commands
+
+The extension uses a format for its commands similar to an extranetwork, but it has a "ppp:" prefix followed by the command, and then a space and any parameters (if any).
 
 ```text
-<!content!>
+<ppp:command parameters>
 ```
 
-And an optional position in the negative prompt can be specified like this:
+When a command is associated with any content, it will be between an opening and a closing command:
 
 ```text
-<!!position!content!>
+<ppp:command parameters>content<ppp:/command>
 ```
 
-Where position can be:
+For wildcards and choices it uses the formats from the Dynamic Prompts extension, but sometimes with some additional options for more functionality.
 
-* s: at the start (the default)
+### Choices
-* e: at the end
+
-* pN: at the position of the insertion point "<!!iN!!>" with N being 0-9
+The generic format is:
+
+```text
+{parameters$$opt1::choice1|opt2::choice2|opt3::choice3}
+```
+
+Both the construct parameters (up to the '$$') and the individual choice options (up to the '::') are optional.
+
+There is also a format where instead of "parameters$$" you just put the sampler, for compatibility with Dynamic Prompts.
+
+The construct parameters can be written with the following options (all are optional):
+
+* "**~**" or "**@**": sampler (for compatibility with Dynamic Prompts), but only "**~**" (random) is allowed.
+* "**r**": means it allows repetition of the choices.
+* "**n**" or "**n-m**" or "**n-**" or "**-m**": number or range of choices to select. Allows zero as the start of a range. Default is 1.
+* "**$$sep**": separator when multiple choices are selected. Default is set in settings.
+* "**$$**": end of the parameters.
+
+The choice options are as follows:
+
+* "**n**": weight of the choice (default 1)
+* "**if condition**": filters out the choice if the condition is false (this is an extension to the Dynamic Prompts syntax). Same conditions as in the `if` command.
+* "**::**": end of choice options
+
+Whitespace is allowed between parameters.
+
+These are examples of formats you can use to insert a choice construct:
+
+```text
+{opt1|5::opt2|3::opt3}             # select 1 choice, two have weights
+{3$$opt1|5 if _is_sd1::opt2|opt3}  # select 3 choices, one has a weight and a condition
+{2-3$$opt1|opt2|opt3}              # select 2 to 3 choices
+{r2-3$$opt1|opt2|opt3}             # select 2 to 3 choices allowing repetition
+{2-3$$ / $$opt1|opt2|opt3}         # select 2 to 3 choices with separator " / "
+```
+
+Notes:
+
+* The Dynamic Prompts format `{2$$__flavours__}` does not work as expected. It will only output one value. You can write is as `{r2$$__flavours__}` to get two values, but they may repeat since the evaluation of the wildcard is independent of the choices selection.
+* Whitespace in the choices is not ignored like in Dynamic Prompts, but will be cleaned up if the appropiate settings are checked.
+
+### Wildcards
+
+The generic format is:
+
+```text
+__parameters$$path/to/wildcard(var=value)__
+```
+
+The parameters and the setting of a variable are optional. The parameters follow the same format as for the choices. The variable value only applies during the evaluation of the selected choices and is discarded afterward (the variable keeps its original value if there was one).
+
+In the wildcard definition (which supports the text, json and yaml formats), if the first choice follows the format of these parameters, it will be used as default parameters for that wildcard (see examples in the tests folder). The choices of the wildcard follow the same format as in the choices construct. If using the object format for a choice you can use a new "if" property for the condition in addition to the standard "weight" and "text"/"content".
+
+Wildcards can contain just one choice. In json and yaml formats this allows the use of a string value for the keys, rather than an array.
+
+These are examples of formats you can use to insert a wildcard:
+
+```text
+__path/wildcard__            # select 1 choice
+__3$$path/wildcard__         # select 3 choices
+__2-3$$path/wildcard__       # select 2 to 3 choices
+__r2-3$$path/wildcard__      # select 2 to 3 choices allowing repetition
+__2-3$$ / $$path/wildcard__  # select 2 to 3 choices with separator " / "
+__path/wildcard(var=value)__ # select 1 choice using the specified variable value in the evaluation.
+```
+
+#### Detection of remaining wildcards
+
+This extension should run after any other wildcard extensions, so if you don't use the internal wildcards processing, any remaining wildcards present in the prompt or negative_prompt at this point must be invalid. Usually you might not notice this problem until you check the image metadata, so this option gives you some ways to detect and treat the problem.
+
+### Set command
+
+This command sets the value of a variable that can be checked later.
+
+The format is:
+
+```text
+<ppp:set varname>value<ppp:/set>
+<ppp:set varname evaluate>value<ppp:/set>
+<ppp:set varname add>value<ppp:/set>
+<ppp:set varname evaluate add>value<ppp:/set>
+```
+
+The `evaluate` parameter makes it so the value of the variable is evaluated at this moment, instead of when it is used.
+
+With the `add` parameter the value is added to the current value of the variable. It does not force an immediate evaluation of the old nor the added value.
+
+The Dynamic Prompts format also works:
+
+```text
+${var=value}
+${var=!value}   # immmediate evaluation
+```
+
+If also supports the addition as an extension of the Dynamic Prompts format:
+
+```text
+${var+=value}
+${var+=!value}
+```
+
+### Echo command
+
+This command prints the value of a variable.
+
+The format is:
+
+```text
+<ppp:echo varname>
+<ppp:echo varname>default<ppp:/echo>
+```
+
+The Dynamic Prompts format is:
+
+```text
+${var}
+${var:default}
+```
+
+### If command
+
+This command allows you to filter content based on conditions.
+
+The full format is:
+
+```text
+<ppp:if condition1>content one<ppp:elif condition2>content two<ppp:else>other content<ppp:/if>
+```
+
+The *conditionN* compares a variable with a value or a list of values. The allowed formats are:
+
+```text
+[not] variable
+[not] variable operation value
+variable [not] operation value
+[not] variable operation (value1,value2...)
+variable [not] operation (value1,value2...)
+```
+
+When there is no value it will check if the variable is truthy.
+
+For a simple value the allowed operations are `eq`, `ne`, `gt`, `lt`, `ge`, `le`, `contains` and the value can be a quoted string or an integer.
+
+For a list of values the allowed operations are `contains`, `in` and the value of the variable is checked against all the elements of the list until one matches.
+
+The variable can be one set with the `set` or `add` commands or you can use internal variables like these (names starting with an underscore are reserved):
+
+* `_sd` : the loaded model version (`"sd1"`, `"sd2"`, `"sdxl"`)
+* `_sdname` : the loaded model filename (without path)
+* `_sdfullname`: the loaded model filename (with path)
+* `_is_sd`: true if the loaded model version is any version of SD
+* `_is_sd1`: true if the loaded model version is SD 1.x
+* `_is_sd2`: true if the loaded model version is SD 2.x
+* `_is_sdxl`: true if the loaded model version is SDXL (includes Pony models)
+* `_is_ssd`: true if the loaded model version is SSD (Segmind Stable Diffusion 1B). Note that for an SSD model `_is_sdxl` will also be true.
+* `_is_sdxl_no_ssd`: true if the loaded model version is SDXL and not an SSD model.
+* `_is_pony`: true if the loaded model version is SDXL and a Pony model (based on its filename). Note that for a pony model `_is_sdxl` will also be true.
+* `_is_sdxl_no_pony`: true if the loaded model version is SDXL and not a Pony model.
+* `_is_sd3`: true if the loaded model version is SD 3.x
+* `_is_flux`: true if the loaded model is Flux
+
+Any `elif`s (there can be multiple) and the `else` are optional.
+
+#### Example
+
+(multiline to be easier to read)
+
+```text
+<ppp:if _is_sd1><lora:test_sd1> test sd1x
+<ppp:elif _sd_pony><lora:test_pony> test pony
+<ppp:elif _sd_sdxl><lora:test_sdxl> test sdxl
+<ppp:else>unknown model
+<ppp:/if>
+```
+
+Only one of the options will end up in the prompt, depending on the loaded model.
+
+### Sending content to the negative prompt
+
+The new format for this command is like this:
+
+```text
+<ppp:stn position>content<ppp:/stn>
+```
+
+Where position is optional (defaults to the start) and can be:
+
+* **s**: at the start of the negative prompt
+* **e**: at the end of the negative prompt
+* **pN**: at the position of the insertion point in the negative prompt with N being 0-9
+
+The format of the insertion point to be used in the negative prompt is:
+
+```text
+<ppp:stn iN>
+```
 
 If the insertion point is not found it inserts at the start.
 
-## Example
+#### Example
 
-You have a wildcard for hair colors (\_\_haircolors\_\_) with one being
+You have a wildcard for hair colors (\_\_haircolors\_\_) with one being strawberry blonde, but you don't want strawberries. So in that option you add a command to add to the negative prompt, like so:
-strawberry blonde, but you don't want strawberries. So in that option you add a
-tag to add to the negative prompt, like so:
 
 ```text
 blonde
-strawberry blonde <!strawberry!>
+strawberry blonde <ppp:stn>strawberry<ppp:/stn>
 brunette
 ```
 
-Then, if that option is chosen this extension will process it later and move
+Then, if that option is chosen this extension will process it later and move that part to the negative prompt.
-that part to the negative prompt.
 
-## Configuration
+#### Old format
 
-Separator used when adding to the negative prompt: You can specify the separator used when adding to the negative prompt (by default it's ", ").
+The old format (`<!...!>`) is not supported anymore.
 
-Ignore tags with repeated content: by default it ignores repeated content to avoid repetitions in the negative prompt.
+### Notes on negative commands
 
-Join attention modifiers (weights) when possible: by default it joins attention modifiers when possible (joins into one, multipliying their values).
+Positional insertion commands have less priority that start/end commands, so even if they are at the start or end of the negative prompt, they will end up inside any start/end (and default position) commands.
 
-Try to clean-up the prompt after processing: by default cleans up the positive prompt after processing, removing extra spaces and separators.
+The content of the negative commands is not processed and is copied as-is to the negative prompt. Other modifiers around the commands are processed in the following way.
 
-## Notes
+#### Attention modifiers (weights)
-
-The content of the negative tags is not processed and is copied as is to the negative prompt. Other modifiers around the tags are processed in the following way.
-
-### Attention modifiers (weights)
 
 They will be translated to the negative prompt. For example:
 
-* `(red<!square!>:1.5)` will end up as `(square:1.5)` in the negative prompt
+* `(red<ppp:stn>square<ppp:/stn>:1.5)` will end up as `(square:1.5)` in the negative prompt
-* `(red[<!square!>]:1.5)` will end up as `(square:1.35)` in the negative prompt (weight=1.5*0.9)
+* `(red[<ppp:stn>square<ppp:/stn>]:1.5)` will end up as `(square:1.35)` in the negative prompt (weight=1.5*0.9)
-* However `(red<![square]!>:1.5)` will end up as `([square]:1.5)` in the negative prompt. The content of the negative tag is copied as is, and not joined with the surrounding modifier.
+* However `(red<ppp:stn>[square]<ppp:/stn>:1.5)` will end up as `([square]:1.5)` in the negative prompt. The content of the negative tag is copied as is, and not joined with the surrounding modifier.
 
-### Prompt editing constructs (alternation and scheduling)
+#### Prompt editing constructs (alternation and scheduling)
 
-Negative tags inside such constructs will copy the construct to the negative prompt, but separating its elements. For example:
+Negative commands inside such constructs will copy the construct to the negative prompt, but separating its elements. For example:
 
-* Alternation: `[red<!square!>|blue<!circle!>]` will end up as `[square|], [|circle]` in the negative prompt, instead of `[square|circle]`
+* **Alternation**: `[red<ppp:stn>square<ppp:/stn>|blue<ppp:stn>circle<ppp:/stn>]` will end up as `[square|], [|circle]` in the negative prompt, instead of `[square|circle]`
-* Scheduling: `[red<!square!>:blue<!circle!>:0.5]` will end up as `[square::0.5], [:circle:0.5]` instead of `[square:circle:0.5]`
+* **Scheduling**: `[red<ppp:stn>square<ppp:/stn>:blue<ppp:stn>circle<ppp:/stn>:0.5]` will end up as `[square::0.5], [:circle:0.5]` instead of `[square:circle:0.5]`
 
 This should still work as intended, and the only negative point i see is the unnecessary separators.
 
+## Configuration
+
+### General settings
+
+* **Debug level**: what to write to the console. Note: in SD.Next debug messages only show if you launch it with the --debug argument.
+* **Pony substrings**: list of substrings to detect a Pony model.
+* **Apply in img2img**: check if you want to do the processing in img2img processes (does not apply to ComfyUI node).
+
+### Wildcard settings
+
+* **Process wildcards**: you can choose to process them with this extension or use a different one.
+* **Wildcards folders**: you can enter multiple folders separated by commas. In ComfyUI you can leave it empty and add a "wildcards" entry in the extra_model_paths.yaml file.
+* **What to do with remaining wildcards?**: select what do you want to do with any found wildcards.
+  * **Ignore**: do not try to detect wildcards.
+  * **Remove**: detect wildcards and remove them.
+  * **Add visible warning**: detect wildcards and add a warning text to the prompt, that hopefully produces a noticeable generation.
+  * **Stop the generation**: detect wildcards and stop the generation.
+* **Default separator used when adding multiple choices**: what do you want to use by default to separate multiple choices when the options allow it (by default it's ", ").
+* **Keep the order of selected choices**: if checked, a multiple choice construct will return them in the order they are in the construct.
+
+### Send to negative prompt settings
+
+* **Separator used when adding to the negative prompt**: you can specify the separator used when adding to the negative prompt (by default it's ", ").
+* **Ignore repeated content**: it ignores repeated content to avoid repetitions in the negative prompt.
+* **Join attention modifiers (weights) when possible**: it joins attention modifiers when possible (joins into one, multipliying their values).
+
+### Clean up settings
+
+* **Remove empty constructs**: removes attention/scheduling/alternation constructs when they are invalid.
+* **Remove extra separators**: removes unnecessary separators. This applies to the configured separator and regular commas.
+* **Remove additional extra separators**: removes unnecessary separators at start or end of lines. This applies to the configured separator and regular commas.
+* **Clean up around BREAKs**: removes consecutive BREAKs and unnecessary commas and space around them.
+* **Use EOL instead of Space before BREAKs**: add a newline before BREAKs.
+* **Clean up around ANDs**: removes consecutive ANDs and unnecessary commas and space around them.
+* **Use EOL instead of Space before ANDs**: add a newline before ANDs.
+* **Clean up around extra network tags**: removes spaces around them.
+* **Remove extra spaces**: removes other unnecessary spaces.
+
+### Content removal settings
+
+* **Remove extra network tags**: removes all extra network tags.
+
 ## License
 
 MIT

__init__.py

@@ -0,0 +1,28 @@
+"""
+@author: ACB
+@title: Prompt Post Processor
+@nickname: ACB PPP
+@description: Node for processing prompts. Includes the following options: send to negative prompt, set variables, if/elif/else command for conditional content, wildcards and choices.
+"""
+
+import sys
+import os
+
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+
+from .ppp_comfyui import PromptPostProcessorComfyUINode
+
+NODE_CLASS_MAPPINGS = {"ACBPromptPostProcessor": PromptPostProcessorComfyUINode}
+
+NODE_DISPLAY_NAME_MAPPINGS = {"ACBPromptPostProcessor": "ACB Prompt Post Processor"}
+
+MANIFEST = {
+    "name": "ACB Prompt Post Processor",
+    "version": PromptPostProcessorComfyUINode.VERSION,
+    "author": "ACB",
+    "project": "https://github.com/acorderob/sd-webui-prompt-postprocessor",
+    "description": "Node for processing prompts",
+    "license": "MIT",
+}
+
+__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]

grammar.lark

@@ -0,0 +1,98 @@
+%import common (LETTER, DIGIT, INT, CNAME, SIGNED_NUMBER, NUMBER)
+
+_WHITESPACE: /\s+/
+STRING:      /("(?!"").*?(?<!\\)(\\\\)*?"|'(?!'').*?(?<!\\)(\\\\)*?')/i
+BOOLEAN:     /true|false/i
+
+// plain text and weights
+?plain:                  /((?!__|\bAND\b|\${)[^\\()\[\]:<>${]|\\.)+/s // exclude only the starting ones
+?plain_choice:           /((?!__|\bAND\b|\${|\$\$)[^\\()\[\]:<>${|}~@]|\\.)+/s // add the specific internal choice ones
+?plain_alternate:        /((?!__|\bAND\b|\${)[^\\()\[\]:<>${|]|\\.)+/s // add the specific internal alternate ones
+?plain_var:              /((?!__|\bAND\b|\${)[^\\()\[\]:<>${}]|\\.)+/s  // add the specific internal var ones
+?specialchars:           /[_{()\[\]:<>]|\$(?![{$])/ // include only the starting ones
+?specialchars_negtag:    /[_{()\[\]:<>!|}]|\$(?![{$])/ // add the internal negtag ones
+?specialchars_alternate: /[_{()\[\]:<>|]|\$(?![{$])/ // add the internal alternate ones
+?specialchars_choice:    /[_{()\[\]:<>|}]|\$(?![{$])/ // add the internal choice ones
+?specialchars_var:       /[_{()\[\]:<>}]|\$(?![{$])/ // add the internal var ones
+?numpar: _WHITESPACE? SIGNED_NUMBER _WHITESPACE?
+
+start: promptcomp | content
+
+// prompt composition with AND
+promptcomp.4: promptcomppart ([":" numpar] (/\bAND\b/ promptcomppart [":" numpar])+)+
+promptcomppart: content
+
+// simple prompts
+?content.2: (old_content | new_content | plain | specialchars)*
+?content_choice.2: (old_content | new_content | plain_choice | specialchars_choice)*
+?content_var.2: (old_content | new_content | plain_var | specialchars_var)*
+?content_negtag.2: (old_content | new_content_negtag | plain | specialchars_negtag)*
+?content_alternate.2: (old_content | new_content | plain_alternate | specialchars_alternate)*
+?old_content.2: (emphasized | deemphasized | scheduled | alternate | extranetworktag)+
+?new_content.3: (variableset | variableuse | commandstn | commandstni | commandset | commandecho | commandif | wildcard | choices)+
+?new_content_negtag.3: (variableset | variableuse | commandset | commandecho | commandif | wildcard | choices)+
+
+// attention modifiers
+emphasized: "(" content [":" numpar] ")"
+deemphasized: "[" content "]"
+
+// prompt scheduling and alternation
+alternate: "[" alternateoption ("|" alternateoption)+ "]"
+alternateoption: content_alternate
+scheduled: "[" [content ":"] content ":" numpar "]"
+
+// extra network tags
+extranetworktag: "<" /(?!ppp:)[^>]+/ ">"
+
+// command: stn (send to negative)
+commandstn: "<ppp:stn" [_WHITESPACE /s|e|p\d/] _WHITESPACE? ">" content_negtag "<ppp:/stn>"
+commandstni: "<ppp:stn" _WHITESPACE /i\d/ _WHITESPACE? ">"
+
+// command: if
+commandif.2: commandif_if commandif_elif* commandif_else? "<ppp:/if>"
+commandif_if: "<ppp:if" _WHITESPACE condition _WHITESPACE? ">" ifvalue
+commandif_elif: "<ppp:elif" _WHITESPACE condition _WHITESPACE? ">" ifvalue
+commandif_else: "<ppp:else" _WHITESPACE? ">" ifvalue
+ifvalue: content
+
+// conditions
+condition: conditionsimplevalue | conditionlistvalue | conditionnocomparison
+conditionnocomparison: (/not/ _WHITESPACE)? IDENTIFIER
+conditionsimplevalue: (/not/ _WHITESPACE)? IDENTIFIER _WHITESPACE (/not/ _WHITESPACE)? /eq|ne|gt|lt|ge|le|contains/ _WHITESPACE SIMPLEVALUE
+conditionlistvalue: (/not/ _WHITESPACE)? IDENTIFIER _WHITESPACE (/not/ _WHITESPACE)? /contains|in/ _WHITESPACE listvalue
+IDENTIFIER: CNAME
+SIMPLEVALUE: STRING | INT | BOOLEAN
+listvalue: "(" _WHITESPACE? SIMPLEVALUE (_WHITESPACE? "," _WHITESPACE? SIMPLEVALUE)* _WHITESPACE? ")"
+
+// command: set
+commandset: "<ppp:set" _WHITESPACE IDENTIFIER [ _WHITESPACE? /evaluate/ ] [ _WHITESPACE? /add/ ] _WHITESPACE? ">" content "<ppp:/set>"
+
+// command: echo
+commandecho: "<ppp:echo" _WHITESPACE IDENTIFIER _WHITESPACE? ">" [ content "<ppp:/echo>" ]
+
+// variable set
+variableset.2: "${" _WHITESPACE? IDENTIFIER _WHITESPACE? [/\+/] "=" [/!/] varvalue "}"
+
+// variable use
+variableuse.2: "${" _WHITESPACE? IDENTIFIER _WHITESPACE? [":" varvalue] "}"
+varvalue: content_var
+
+// wildcards
+wildcard.2: "__" [choicesoptions_sampler | (choicesoptions _WHITESPACE? "$$")] /(?:(?!__|\$\$|\()\S)+/ [ wildcard_var ] "__"
+wildcard_var: "(" _WHITESPACE? IDENTIFIER _WHITESPACE? "=" varvalue ")"
+
+// choices
+choices.2: "{" [choicesoptions_sampler | (choicesoptions _WHITESPACE? "$$")] choice ("|" choice)* "}"
+
+choicesoptions: [choicesoptions_sampler] [_WHITESPACE? choicesoptions_rep] (([_WHITESPACE? choicesoptions_from] "-" [_WHITESPACE? choicesoptions_to]) | [_WHITESPACE? choicesoptions_num] ) [_WHITESPACE? choicesoptions_sep]
+choicesoptions_sampler: /[~@]/ // ~ for random, @ for cyclical
+choicesoptions_rep: /r/
+choicesoptions_num: INT
+choicesoptions_from: INT
+choicesoptions_to: INT
+choicesoptions_sep: "$$" plain
+
+choice: [[_WHITESPACE? choiceweight] [_WHITESPACE? choiceif] _WHITESPACE? "::"] choicevalue
+choiceweight: NUMBER
+choiceif: "if" _WHITESPACE condition
+choicevalue: content_choice

install.py

@@ -0,0 +1,5 @@
+import os
+import launch
+
+requirements_filename = os.path.join(os.path.dirname(os.path.realpath(__file__)), "requirements.txt")
+launch.run_pip(f'install -r "{requirements_filename}"', "requirements for Prompt Post-Processor")

metadata.ini

@@ -0,0 +1,5 @@
+[Extension]
+Name = sd-webui-prompt-postprocessor
+
+[Scripts]
+After = sd-dynamic-prompts, stable-diffusion-webui-wildcards

ppp_cache.py

@@ -0,0 +1,24 @@
+from collections import OrderedDict
+from typing import Tuple
+
+
+class PPPLRUCache:
+
+    ProcessInput = Tuple[int, str, str]
+    ProcessResult = Tuple[str, str]
+
+    def __init__(self, capacity: int):
+        self.cache = OrderedDict()
+        self.capacity = capacity
+
+    def get(self, key: ProcessInput) -> ProcessResult:
+        if key not in self.cache:
+            return None
+        self.cache.move_to_end(key)
+        return self.cache[key]
+
+    def put(self, key: ProcessInput, value: ProcessResult) -> None:
+        self.cache[key] = value
+        self.cache.move_to_end(key)
+        if len(self.cache) > self.capacity:
+            self.cache.popitem(last=False)

ppp_comfyui.py

@@ -0,0 +1,406 @@
+# pylint: disable=missing-module-docstring, missing-class-docstring, missing-function-docstring, invalid-name
+
+import os
+
+# pylint: disable=import-error
+import folder_paths  # type: ignore
+import nodes  # type: ignore
+
+from .ppp import PromptPostProcessor
+from .ppp_logging import DEBUG_LEVEL, PromptPostProcessorLogFactory
+from .ppp_wildcards import PPPWildcards
+
+if __name__ == "__main__":
+    raise SystemExit("This script must be run from ComfyUI")
+
+
+class PromptPostProcessorComfyUINode:
+
+    VERSION = PromptPostProcessor.VERSION
+
+    logger = None
+
+    def __init__(self):
+        lf = PromptPostProcessorLogFactory()
+        self.logger = lf.log
+        grammar_filename = os.path.join(os.path.dirname(os.path.realpath(__file__)), "grammar.lark")
+        with open(grammar_filename, "r", encoding="utf-8") as file:
+            self.grammar_content = file.read()
+        self.wildcards_obj = PPPWildcards(lf.log)
+
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "model": (
+                    "MODEL",
+                    {
+                        "forceInput": True,
+                    },
+                ),
+                "modelname": (
+                    "STRING",
+                    {
+                        "default": "",
+                        "forceInput": True,
+                    },
+                ),
+                "pos_prompt": (
+                    "STRING",
+                    {
+                        "multiline": True,
+                        "default": "",
+                        "forceInput": True,
+                    },
+                ),
+                "neg_prompt": (
+                    "STRING",
+                    {
+                        "multiline": True,
+                        "default": "",
+                        "forceInput": True,
+                    },
+                ),
+            },
+            "optional": {
+                "seed": (
+                    "INT",
+                    {
+                        "default": -1,
+                        "forceInput": False,
+                    },
+                ),
+                "debug_level": (
+                    [e.value for e in DEBUG_LEVEL],
+                    {
+                        "default": DEBUG_LEVEL.minimal.value,
+                        "tooltip": "Debug level",
+                    },
+                ),
+                "pony_substrings": (
+                    "STRING",
+                    {
+                        "default": PromptPostProcessor.DEFAULT_PONY_SUBSTRINGS,
+                        "placeholder": "comma separated list",
+                        "tooltip": "Comma separated list of substrings to look for in the modelname to determine if the model is a pony model",
+                    },
+                ),
+                "wc_process_wildcards": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Process wildcards in the prompt",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "wc_wildcards_folders": (
+                    "STRING",
+                    {
+                        "default": "",
+                        "tooltip": "Comma separated list of wildcards folders",
+                    },
+                ),
+                "wc_if_wildcards": (
+                    [e.value for e in PromptPostProcessor.IFWILDCARDS_CHOICES],
+                    {
+                        "default": PromptPostProcessor.IFWILDCARDS_CHOICES.ignore.value,
+                        "tooltip": "How to handle invalid wildcards in the prompt",
+                    },
+                ),
+                "wc_choice_separator": (
+                    "STRING",
+                    {
+                        "default": PromptPostProcessor.DEFAULT_CHOICE_SEPARATOR,
+                        "tooltip": "Default separator for selected choices",
+                    },
+                ),
+                "wc_keep_choices_order": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Keep the order of the choices in the prompt",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "stn_separator": (
+                    "STRING",
+                    {
+                        "default": PromptPostProcessor.DEFAULT_STN_SEPARATOR,
+                        "tooltip": "Separator for the content added to the negative prompt",
+                    },
+                ),
+                "stn_ignore_repeats": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Ignore repeated content added to the negative prompt",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "stn_join_attention": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Merge attention in the content added to the negative prompt",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "cleanup_extra_spaces": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Remove extra spaces",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "cleanup_empty_constructs": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Remove empty constructs",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "cleanup_extra_separators": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Remove extra separators",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "cleanup_extra_separators2": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Remove extra separators (additional cases)",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "cleanup_breaks": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Cleanup around BREAKs",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "cleanup_breaks_eol": (
+                    "BOOLEAN",
+                    {
+                        "default": False,
+                        "tooltip": "Set BREAKs in their own line",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "cleanup_ands": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": "Cleanup around ANDs",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "cleanup_ands_eol": (
+                    "BOOLEAN",
+                    {
+                        "default": False,
+                        "tooltip": "Set ANDs in their own line",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "cleanup_extranetwork_tags": (
+                    "BOOLEAN",
+                    {
+                        "default": False,
+                        "tooltip": "Clean up around extra network tags",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+                "remove_extranetwork_tags": (
+                    "BOOLEAN",
+                    {
+                        "default": False,
+                        "tooltip": "Remove extra network tags",
+                        "label_on": "Yes",
+                        "label_off": "No",
+                    },
+                ),
+            },
+        }
+
+    RETURN_TYPES = (
+        "STRING",
+        "STRING",
+    )
+    RETURN_NAMES = (
+        "pos_prompt",
+        "neg_prompt",
+    )
+
+    FUNCTION = "process"
+
+    CATEGORY = "ACB"
+
+    @classmethod
+    def IS_CHANGED(
+        cls,
+        model,
+        modelname,
+        pos_prompt,
+        neg_prompt,
+        seed,
+        debug_level,  # pylint: disable=unused-argument
+        pony_substrings,
+        wc_process_wildcards,
+        wc_wildcards_folders,
+        wc_if_wildcards,
+        wc_choice_separator,
+        wc_keep_choices_order,
+        stn_separator,
+        stn_ignore_repeats,
+        stn_join_attention,
+        cleanup_extra_spaces,
+        cleanup_empty_constructs,
+        cleanup_extra_separators,
+        cleanup_extra_separators2,
+        cleanup_breaks,
+        cleanup_breaks_eol,
+        cleanup_ands,
+        cleanup_ands_eol,
+        cleanup_extranetwork_tags,
+        remove_extranetwork_tags,
+    ):
+        new_run = {
+            "model": model,
+            "modelname": modelname,
+            "pos_prompt": pos_prompt,
+            "neg_prompt": neg_prompt,
+            "seed": seed,
+            "pony_substrings": pony_substrings,
+            "process_wildcards": wc_process_wildcards,
+            "wildcards_folders": wc_wildcards_folders,
+            "if_wildcards": wc_if_wildcards,
+            "choice_separator": wc_choice_separator,
+            "keep_choices_order": wc_keep_choices_order,
+            "stn_separator": stn_separator,
+            "stn_ignore_repeats": stn_ignore_repeats,
+            "stn_join_attention": stn_join_attention,
+            "cleanup_extra_spaces": cleanup_extra_spaces,
+            "cleanup_empty_constructs": cleanup_empty_constructs,
+            "cleanup_extra_separators": cleanup_extra_separators,
+            "cleanup_extra_separators2": cleanup_extra_separators2,
+            "cleanup_breaks": cleanup_breaks,
+            "cleanup_breaks_eol": cleanup_breaks_eol,
+            "cleanup_ands": cleanup_ands,
+            "cleanup_ands_eol": cleanup_ands_eol,
+            "cleanup_extranetwork_tags": cleanup_extranetwork_tags,
+            "remove_extranetwork_tags": remove_extranetwork_tags,
+        }
+        return new_run.__hash__
+        # return float("NaN")
+
+    def process(
+        self,
+        model,
+        modelname,
+        pos_prompt,
+        neg_prompt,
+        seed,
+        debug_level,
+        pony_substrings,
+        wc_process_wildcards,
+        wc_wildcards_folders,
+        wc_if_wildcards,
+        wc_choice_separator,
+        wc_keep_choices_order,
+        stn_separator,
+        stn_ignore_repeats,
+        stn_join_attention,
+        cleanup_extra_spaces,
+        cleanup_empty_constructs,
+        cleanup_extra_separators,
+        cleanup_extra_separators2,
+        cleanup_breaks,
+        cleanup_breaks_eol,
+        cleanup_ands,
+        cleanup_ands_eol,
+        cleanup_extranetwork_tags,
+        remove_extranetwork_tags,
+    ):
+        model_info = {
+            "models_path": folder_paths.models_dir,
+            "model_filename": modelname, # path is relative to checkpoints folder
+            "is_sd1": model.model.model_config.__class__.__name__ in ("SD15", "SD15_instructpix2pix"),
+            "is_sd2": model.model.model_config.__class__.__name__ in ("SD20", "SD21UnclipL", "SD21UnclipH"),
+            "is_sdxl": model.model.model_config.__class__.__name__
+            in (
+                "SDXL",
+                "SDXLRefiner",
+                "SDXL_instructpix2pix",
+                "Segmind_Vega",
+                "KOALA_700M",
+                "KOALA_1B",
+            ),
+            "is_ssd": model.model.model_config.__class__.__name__ in ("SSD1B"),
+            "is_sd3": model.model.model_config.__class__.__name__ in ("SD3"),
+            "is_flux": model.model.model_config.__class__.__name__ in ("Flux"),
+        }
+        # SVD_img2vid, SVD3D_u, SVD3_p, Stable_Zero123, SD_X4Upscaler,
+        # Stable_Cascade_C, Stable_Cascade_B, StableAudio
+
+        if wc_wildcards_folders == "":
+            wc_wildcards_folders = ",".join(folder_paths.get_folder_paths("wildcards") or [])
+        if wc_wildcards_folders == "":
+            wc_wildcards_folders = os.getenv("WILDCARD_DIR", PPPWildcards.DEFAULT_WILDCARDS_FOLDER)
+        wildcards_folders = [
+            (f if os.path.isabs(f) else os.path.abspath(os.path.join(folder_paths.models_dir, f)))
+            for f in wc_wildcards_folders.split(",")
+            if f.strip() != ""
+        ]
+        options = {
+            "debug_level": debug_level,
+            "pony_substrings": pony_substrings,
+            "process_wildcards": wc_process_wildcards,
+            "if_wildcards": wc_if_wildcards,
+            "choice_separator": wc_choice_separator,
+            "keep_choices_order": wc_keep_choices_order,
+            "stn_separator": stn_separator,
+            "stn_ignore_repeats": stn_ignore_repeats,
+            "stn_join_attention": stn_join_attention,
+            "cleanup_extra_spaces": cleanup_extra_spaces,
+            "cleanup_empty_constructs": cleanup_empty_constructs,
+            "cleanup_extra_separators": cleanup_extra_separators,
+            "cleanup_extra_separators2": cleanup_extra_separators2,
+            "cleanup_breaks": cleanup_breaks,
+            "cleanup_breaks_eol": cleanup_breaks_eol,
+            "cleanup_ands": cleanup_ands,
+            "cleanup_ands_eol": cleanup_ands_eol,
+            "cleanup_extranetwork_tags": cleanup_extranetwork_tags,
+            "remove_extranetwork_tags": remove_extranetwork_tags,
+        }
+        self.wildcards_obj.refresh_wildcards(debug_level, wildcards_folders if options["process_wildcards"] else None)
+        ppp = PromptPostProcessor(
+            self.logger, self.interrupt, model_info, options, self.grammar_content, self.wildcards_obj
+        )
+        pos_prompt, neg_prompt = ppp.process_prompt(pos_prompt, neg_prompt, seed if seed is not None else 1)
+        return (
+            pos_prompt,
+            neg_prompt,
+        )
+
+    def interrupt(self):
+        nodes.interrupt_processing(True)

ppp_logging.py

@@ -0,0 +1,92 @@
+from enum import Enum
+import logging
+import sys
+import copy
+
+
+class DEBUG_LEVEL(Enum):
+    none = "none"
+    minimal = "minimal"
+    full = "full"
+
+
+class PromptPostProcessorLogFactory:  # pylint: disable=too-few-public-methods
+    """
+    Factory class for creating loggers for the PromptPostProcessor module.
+    """
+
+    class ColoredFormatter(logging.Formatter):
+        """
+        A custom logging formatter that adds color to log records based on their level.
+
+        Attributes:
+            COLORS (dict): A dictionary mapping log levels to ANSI escape codes for colors.
+
+        Methods:
+            format(record): Formats the log record with color based on its level.
+
+        """
+
+        COLORS = {
+            "DEBUG": "\033[0;36m",  # CYAN
+            "INFO": "\033[0;32m",  # GREEN
+            "WARNING": "\033[0;33m",  # YELLOW
+            "ERROR": "\033[0;31m",  # RED
+            "CRITICAL": "\033[0;37;41m",  # WHITE ON RED
+            "RESET": "\033[0m",  # RESET COLOR
+        }
+
+        def format(self, record):
+            """
+            Formats the log record with color based on the log level.
+
+            Args:
+                record (LogRecord): The log record to be formatted.
+
+            Returns:
+                str: The formatted log record.
+            """
+            colored_record = copy.copy(record)
+            levelname = colored_record.levelname
+            seq = self.COLORS.get(levelname, self.COLORS["RESET"])
+            colored_record.levelname = f"{seq}{levelname:8s}{self.COLORS['RESET']}"
+            return super().format(colored_record)
+
+    def __init__(self):
+        """
+        Initializes the PromptPostProcessor class.
+
+        This method sets up the logger for the PromptPostProcessor class and configures its log level and handlers.
+
+        Args:
+            None
+
+        Returns:
+            None
+        """
+        ppplog = logging.getLogger("PromptPostProcessor")
+        ppplog.propagate = False
+        if not ppplog.handlers:
+            handler = logging.StreamHandler(sys.stdout)
+            handler.setFormatter(self.ColoredFormatter("%(asctime)s %(levelname)s %(message)s")) # Used in A1111 / Forge / reForge / ComfyUI, but not in SD.Next
+            ppplog.addHandler(handler)
+        ppplog.setLevel(logging.DEBUG)
+        self.log = PromptPostProcessorLogCustomAdapter(ppplog)
+
+
+class PromptPostProcessorLogCustomAdapter(logging.LoggerAdapter):
+    """
+    Custom logger adapter for the PromptPostProcessor.
+    This adapter adds a prefix to log messages to indicate that they are related to the PromptPostProcessor.
+    """
+
+    def process(self, msg, kwargs):
+        """
+        Process the log message and keyword arguments.
+        Args:
+            msg (str): The log message.
+            kwargs (dict): The keyword arguments.
+        Returns:
+            tuple: A tuple containing the processed log message and keyword arguments.
+        """
+        return f"[PPP] {msg}", kwargs

ppp_wildcards.py

@@ -0,0 +1,209 @@
+import os
+import json
+from typing import Optional
+import yaml
+
+from ppp_logging import DEBUG_LEVEL
+
+
+class PPPWildcards:
+
+    DEFAULT_WILDCARDS_FOLDER = "wildcards"
+
+    def __init__(self, logger):
+        self.logger = logger
+        self.debug_level = DEBUG_LEVEL.none
+        self.wildcards_folders = []
+        self.wildcards = {}
+        self.wildcard_files = {}
+
+    def refresh_wildcards(self, debug_level: DEBUG_LEVEL, wildcards_folders: Optional[list[str]]):
+        """
+        Initialize the wildcards.
+        """
+        self.debug_level = debug_level
+        self.wildcards_folders = wildcards_folders
+        if wildcards_folders is not None:
+            # if self.debug_level != DEBUG_LEVEL.none:
+            #     self.logger.info("Initializing wildcards...")
+            # t1 = time.time()
+            for fullpath in self.wildcard_files.keys():
+                path = os.path.dirname(fullpath)
+                if not os.path.exists(fullpath) or not any(
+                    os.path.commonpath([path, folder]) == folder for folder in self.wildcards_folders
+                ):
+                    self.__remove_wildcards_from_file(fullpath)
+            for f in self.wildcards_folders:
+                self.__get_wildcards_in_directory(f, f)
+            # t2 = time.time()
+            # if self.debug_level != DEBUG_LEVEL.none:
+            #     self.logger.info(f"Wildcards init time: {t2 - t1:.3f} seconds")
+        else:
+            self.wildcards_folders = []
+            self.wildcards = {}
+            self.wildcard_files = {}
+
+    def __get_keys_in_dict(self, dictionary: dict, prefix="") -> list[str]:
+        """
+        Get all keys in a dictionary.
+
+        Args:
+            dictionary (dict): The dictionary to check.
+            prefix (str): The prefix for the current key.
+
+        Returns:
+            list: A list of all keys in the dictionary, including nested keys.
+        """
+        keys = []
+        for key in dictionary.keys():
+            if isinstance(dictionary[key], dict):
+                keys.extend(self.__get_keys_in_dict(dictionary[key], prefix + key + "/"))
+            else:
+                keys.append(prefix + str(key))
+        return keys
+
+    def __get_nested(self, dictionary: dict, keys: str) -> object:
+        """
+        Get a nested value from a dictionary.
+
+        Args:
+            dictionary (dict): The dictionary to check.
+            keys (str): The keys to get the value from.
+
+        Returns:
+            object: The value of the nested keys in the dictionary.
+        """
+        keys = keys.split("/")
+        current_dict = dictionary
+        for key in keys:
+            current_dict = current_dict.get(key)
+            if current_dict is None:
+                return None
+        return current_dict
+
+    def __remove_wildcards_from_file(self, full_path: str):
+        """
+        Clear all wildcards in a file.
+
+        Args:
+            full_path (str): The path to the file.
+        """
+        last_modified_cached = self.wildcard_files.get(full_path, None)
+        if last_modified_cached is not None and self.debug_level != DEBUG_LEVEL.none:
+            self.logger.debug(f"Removing wildcards from file: {full_path}")
+        if full_path in self.wildcard_files.keys():
+            del self.wildcard_files[full_path]
+        for key in list(self.wildcards.keys()):
+            if self.wildcards[key]["file"] == full_path:
+                del self.wildcards[key]
+
+    def __get_wildcards_in_file(self, base, full_path: str):
+        """
+        Get all wildcards in a file.
+
+        Args:
+            base (str): The base path for the wildcards.
+            full_path (str): The path to the file.
+        """
+        last_modified = os.path.getmtime(full_path)
+        last_modified_cached = self.wildcard_files.get(full_path, None)
+        if last_modified_cached is not None and last_modified == self.wildcard_files[full_path]:
+            return
+        filename = os.path.basename(full_path)
+        name, extension = os.path.splitext(filename)
+        if extension not in (".txt", ".json", ".yaml", ".yml"):
+            return
+        self.__remove_wildcards_from_file(full_path)
+        if last_modified_cached is not None and self.debug_level != DEBUG_LEVEL.none:
+            self.logger.debug(f"Updating wildcards from file: {full_path}")
+        relfolders = os.path.relpath(os.path.dirname(full_path), base)
+        if relfolders == ".":
+            relfolders = ""
+        elif relfolders != "":
+            relfolders += "/"
+        if extension == ".txt":
+            self.__get_wildcards_in_text_file(full_path, name, relfolders)
+        elif extension in (".json", ".yaml", ".yml"):
+            self.__get_wildcards_in_structured_file(full_path, extension, relfolders)
+        self.wildcard_files[full_path] = last_modified
+
+    def __get_wildcards_in_structured_file(self, full_path, extension, relfolders):
+        with open(full_path, "r", encoding="utf-8") as file:
+            if extension == ".json":
+                content = json.loads(file.read())
+            else:
+                content = yaml.safe_load(file)
+        keys = self.__get_keys_in_dict(content)
+        for key in keys:
+            fullkey = f"{relfolders}{key}"
+            if self.wildcards.get(fullkey) is not None:
+                self.logger.warning(
+                    f"Duplicate wildcard '{fullkey}' in file '{full_path}' and '{self.wildcards[fullkey]['file']}'!"
+                )
+            else:
+                obj = self.__get_nested(content, key)
+                if obj is not None:
+                    if isinstance(obj, str):
+                        choices = [obj]
+                    elif isinstance(obj, (int, float, bool)):
+                        choices = [str(obj)]
+                    elif isinstance(obj, list) and len(obj) > 0:
+                        choices = []
+                        for c in obj:
+                            if isinstance(c, str):
+                                choices.append(c)
+                            elif isinstance(c, dict):  # we convert the dict to a string
+                                d = ""
+                                if "weight" in c.keys():
+                                    d += str(c["weight"])
+                                if "if" in c.keys():
+                                    d += f" if {c['if']}"
+                                if d != "":
+                                    d += "::"
+                                if "text" in c.keys():
+                                    d += c["text"]
+                                elif "content" in c.keys():
+                                    d += c["content"]
+                                choices.append(d)
+                    else:
+                        obj = None
+                if obj is None:
+                    self.logger.warning(f"Invalid wildcard '{fullkey}' in file '{full_path}'!")
+                else:
+                    self.wildcards[fullkey] = {"file": full_path, "choices": choices}
+
+    def __get_wildcards_in_text_file(self, full_path, name, relfolders):
+        with open(full_path, "r", encoding="utf-8") as file:
+            text_content = map(lambda x: x.strip("\n\r"), file.readlines())
+        text_content = list(filter(lambda x: x.strip() != "" and not x.strip().startswith("#"), text_content))
+        text_content = [x.split("#")[0].rstrip() if len(x.split("#")) > 1 else x for x in text_content]
+        fullkey = f"{relfolders}{name}"
+        if self.wildcards.get(fullkey) is not None:
+            self.logger.warning(
+                f"Duplicate wildcard '{fullkey}' in file '{full_path}' and '{self.wildcards[fullkey]['file']}'!"
+            )
+        else:
+            if len(text_content) == 0:
+                self.logger.warning(f"Invalid wildcard in file '{full_path}'!")
+            else:
+                self.wildcards[fullkey] = {"file": full_path, "choices": text_content}
+
+    def __get_wildcards_in_directory(self, base: str, directory: str):
+        """
+        Get all wildcards in a directory.
+
+        Args:
+            base (str): The base path for the wildcards.
+            directory (str): The path to the directory.
+        """
+        if not os.path.exists(directory):
+            self.logger.warning(f"Wildcard directory '{directory}' does not exist!")
+            return
+        for filename in os.listdir(directory):
+            full_path = os.path.abspath(os.path.join(directory, filename))
+            if os.path.basename(full_path).startswith("."):
+                continue
+            if os.path.isdir(full_path):
+                self.__get_wildcards_in_directory(base, full_path)
+            elif os.path.isfile(full_path):
+                self.__get_wildcards_in_file(base, full_path)

requirements.txt

@@ -0,0 +1 @@
+lark

scripts/extension.py

@@ -1,73 +0,0 @@
-if __name__ == "__main__":
-    raise SystemExit("This script must be run from a Stable Diffusion WebUI")
-
-import sys
-import os
-
-sys.path.insert(1, os.path.join(sys.path[0], ".."))
-
-
-# pylint: disable=import-error
-
-from modules import scripts, shared, script_callbacks
-from modules.processing import StableDiffusionProcessing
-from modules.shared import opts
-from sendtonegative import SendToNegative
-from stnlogging import SendToNegativeLogFactory
-
-
-class SendToNegativeScript(scripts.Script):
-    def __init__(self):
-        if not hasattr(self, "callbacks_added"):
-            lf = SendToNegativeLogFactory()
-            self.__logstn = lf.log
-            script_callbacks.on_ui_settings(self.__on_ui_settings)
-            self.callbacks_added = True
-
-    def title(self):
-        return f"{SendToNegative.NAME} v{SendToNegative.VERSION}"
-
-    def show(self, is_img2img):
-        return scripts.AlwaysVisible
-
-    def process(self, p: StableDiffusionProcessing, *args, **kwargs):
-        stn = SendToNegative(self.__logstn, opts=opts)
-        for i in range(len(p.all_prompts)):  # pylint: disable=consider-using-enumerate
-            p.all_prompts[i], p.all_negative_prompts[i] = stn.process_prompt(
-                p.all_prompts[i], p.all_negative_prompts[i]
-            )
-
-    def __on_ui_settings(self):
-        section = ("send-to-negative", SendToNegative.NAME)
-        shared.opts.add_option(
-            key="stn_separator",
-            info=shared.OptionInfo(
-                SendToNegative.DEFAULT_SEPARATOR,
-                label="Separator used when adding to the negative prompt",
-                section=section,
-            ),
-        )
-        shared.opts.add_option(
-            key="stn_ignorerepeats",
-            info=shared.OptionInfo(
-                True,
-                label="Ignore tags with repeated content",
-                section=section,
-            ),
-        )
-        shared.opts.add_option(
-            key="stn_joinattention",
-            info=shared.OptionInfo(
-                True,
-                label="Join attention modifiers (weights) when possible",
-                section=section,
-            ),
-        )
-        shared.opts.add_option(
-            key="stn_cleanup",
-            info=shared.OptionInfo(
-                True,
-                label="Try to clean-up the prompt after processing (removes extra spaces or the configured separator)",
-                section=section,
-            ),
-        )

scripts/ppp_script.py

@@ -0,0 +1,513 @@
+if __name__ == "__main__":
+    raise SystemExit("This script must be run from a Stable Diffusion WebUI")
+
+import sys
+import os
+import time
+
+sys.path.append(os.path.join(sys.path[0], ".."))
+
+
+from modules import scripts, shared, script_callbacks
+from modules.processing import StableDiffusionProcessing
+from modules.shared import opts
+from modules.paths import models_path
+import gradio as gr
+from ppp import PromptPostProcessor
+from ppp_logging import DEBUG_LEVEL, PromptPostProcessorLogFactory
+from ppp_cache import PPPLRUCache
+from ppp_wildcards import PPPWildcards
+
+
+class PromptPostProcessorA1111Script(scripts.Script):
+    """
+    This class represents a script for prompt post-processing.
+    It is responsible for processing prompts and applying various settings and cleanup operations.
+
+    Attributes:
+        callbacks_added (bool): Flag indicating whether the script callbacks have been added.
+
+    Methods:
+        __init__(): Initializes the PromptPostProcessorScript object.
+        title(): Returns the title of the script.
+        show(is_img2img): Determines whether the script should be shown based on the input type.
+        process(p, *args, **kwargs): Processes the prompts and applies post-processing operations.
+        ppp_interrupt(): Interrupts the generation.
+        __on_ui_settings(): Callback function for UI settings.
+    """
+
+    def __init__(self):
+        """
+        Initializes the PromptPostProcessor object.
+
+        This method adds callbacks for UI settings and initializes the logger.
+
+        Parameters:
+            None
+
+        Returns:
+            None
+        """
+        lf = PromptPostProcessorLogFactory()
+        self.name = PromptPostProcessor.NAME
+        self.ppp_logger = lf.log
+        self.ppp_debug_level = DEBUG_LEVEL(getattr(opts, "ppp_gen_debug_level", DEBUG_LEVEL.none.value))
+        self.lru_cache = PPPLRUCache(1000)
+        grammar_filename = os.path.join(os.path.dirname(os.path.realpath(__file__)), "../grammar.lark")
+        with open(grammar_filename, "r", encoding="utf-8") as file:
+            self.grammar_content = file.read()
+        self.wildcards_obj = PPPWildcards(lf.log)
+
+    def title(self):
+        """
+        Returns the title of the script.
+
+        Returns:
+            str: The title of the script.
+        """
+        return PromptPostProcessor.NAME
+
+    def show(self, is_img2img):
+        """
+        Determines whether the script should be shown based on the kind of processing.
+
+        Args:
+            is_img2img (bool): Flag indicating whether the processing is image-to-image.
+
+        Returns:
+            scripts.Visibility: The visibility setting for the script.
+        """
+        return scripts.AlwaysVisible
+
+    def process(self, p: StableDiffusionProcessing, *args, **kwargs):  # pylint: disable=unused-argument
+        """
+        Processes the prompts and applies post-processing operations.
+
+        Args:
+            p (StableDiffusionProcessing): The StableDiffusionProcessing object containing the prompts.
+
+        Returns:
+            None
+        """
+        t1 = time.time()
+        if getattr(opts, "prompt_attention", "") == "Compel parser":
+            self.ppp_logger.warning("Compel parser is not supported!")
+        is_i2i = getattr(p, "init_images", [None])[0] is not None
+        self.ppp_debug_level = DEBUG_LEVEL(getattr(opts, "ppp_gen_debug_level", DEBUG_LEVEL.none.value))
+        do_i2i = getattr(opts, "ppp_gen_doi2i", False)
+        if is_i2i and not do_i2i:
+            if self.ppp_debug_level != DEBUG_LEVEL.none:
+                self.ppp_logger.info("Not processing the prompt for i2i")
+            return
+        if self.ppp_debug_level != DEBUG_LEVEL.none:
+            self.ppp_logger.info(f"Post-processing prompts ({'i2i' if is_i2i else 't2i'})")
+        model_info = {
+            "models_path": models_path,
+            "model_filename": getattr(p.sd_model.sd_checkpoint_info, "filename", ""),  # path is absolute
+            "is_sd1": False,  # Stable Diffusion 1
+            "is_sd2": False,  # Stable Diffusion 2
+            "is_sdxl": False,  # Stable Diffusion XL
+            "is_ssd": False,  # Segmind Stable Diffusion 1B
+            "is_sd3": False,  # Stable Diffusion 3
+            "is_flux": False,  # Flux
+        }
+        app = (
+            "forge"
+            if hasattr(p.sd_model, "model_config")
+            else "sdnext" if hasattr(p.sd_model, "is_sdxl") and not hasattr(p.sd_model, "is_ssd") else "a1111"
+        )
+        if app == "sdnext":
+            # cannot differenciate SD1 and SD2, we set True to both
+            # LatentDiffusion is for the original backend, StableDiffusionPipeline is for the diffusers backend
+            model_info["is_sd1"] = p.sd_model.__class__.__name__ in ("LatentDiffusion", "StableDiffusionPipeline")
+            model_info["is_sd2"] = p.sd_model.__class__.__name__ in ("LatentDiffusion", "StableDiffusionPipeline")
+            model_info["is_sdxl"] = p.sd_model.__class__.__name__ == "StableDiffusionXLPipeline"
+            model_info["is_ssd"] = False  # ?
+            model_info["is_sd3"] = p.sd_model.__class__.__name__ == "StableDiffusion3Pipeline"
+            model_info["is_flux"] = False
+        elif app == "forge":
+            model_info["is_sd1"] = getattr(p.sd_model, "is_sd1", False)
+            model_info["is_sd2"] = getattr(p.sd_model, "is_sd2", False)
+            model_info["is_sdxl"] = getattr(p.sd_model, "is_sdxl", False)
+            model_info["is_ssd"] = False  # ?
+            model_info["is_sd3"] = getattr(p.sd_model, "is_sd3", False)
+            model_info["is_flux"] = p.sd_model.model_config.__class__.__name__ == "Flux"
+        else:  # assume A1111 compatible (p.sd_model.__class__.__name__=="DiffusionEngine")
+            model_info["is_sd1"] = getattr(p.sd_model, "is_sd1", False)
+            model_info["is_sd2"] = getattr(p.sd_model, "is_sd2", False)
+            model_info["is_sdxl"] = getattr(p.sd_model, "is_sdxl", False)
+            model_info["is_ssd"] = getattr(p.sd_model, "is_ssd", False)
+            model_info["is_sd3"] = getattr(p.sd_model, "is_sd3", False)
+            model_info["is_flux"] = False
+        wc_wildcards_folders = getattr(opts, "ppp_wil_wildcardsfolders", "")
+        if wc_wildcards_folders == "":
+            wc_wildcards_folders = os.getenv("WILDCARD_DIR", PPPWildcards.DEFAULT_WILDCARDS_FOLDER)
+        wildcards_folders = [
+            (f if os.path.isabs(f) else os.path.abspath(os.path.join(models_path, f)))
+            for f in wc_wildcards_folders.split(",")
+            if f.strip() != ""
+        ]
+        options = {
+            "debug_level": getattr(opts, "ppp_gen_debug_level", DEBUG_LEVEL.none.value),
+            "pony_substrings": getattr(opts, "ppp_gen_ponysubstrings", PromptPostProcessor.DEFAULT_PONY_SUBSTRINGS),
+            "process_wildcards": getattr(opts, "ppp_wil_processwildcards", True),
+            "if_wildcards": getattr(opts, "ppp_wil_ifwildcards", PromptPostProcessor.IFWILDCARDS_CHOICES.ignore.value),
+            "choice_separator": getattr(opts, "ppp_wil_choice_separator", PromptPostProcessor.DEFAULT_CHOICE_SEPARATOR),
+            "keep_choices_order": getattr(opts, "ppp_wil_keep_choices_order", False),
+            "stn_separator": getattr(opts, "ppp_stn_separator", PromptPostProcessor.DEFAULT_STN_SEPARATOR),
+            "stn_ignore_repeats": getattr(opts, "ppp_stn_ignorerepeats", True),
+            "stn_join_attention": getattr(opts, "ppp_stn_joinattention", True),
+            "cleanup_extra_spaces": getattr(opts, "ppp_cup_extraspaces", True),
+            "cleanup_empty_constructs": getattr(opts, "ppp_cup_emptyconstructs", True),
+            "cleanup_extra_separators": getattr(opts, "ppp_cup_extraseparators", True),
+            "cleanup_extra_separators2": getattr(opts, "ppp_cup_extraseparators2", True),
+            "cleanup_breaks": getattr(opts, "ppp_cup_breaks", True),
+            "cleanup_breaks_eol": getattr(opts, "ppp_cup_breaks_eol", False),
+            "cleanup_ands": getattr(opts, "ppp_cup_ands", True),
+            "cleanup_ands_eol": getattr(opts, "ppp_cup_ands_eol", False),
+            "cleanup_extranetwork_tags": getattr(opts, "ppp_cup_extranetworktags", False),
+            "remove_extranetwork_tags": getattr(opts, "ppp_rem_removeextranetworktags", False),
+        }
+        self.wildcards_obj.refresh_wildcards(
+            self.ppp_debug_level, wildcards_folders if options["process_wildcards"] else None
+        )
+        ppp = PromptPostProcessor(
+            self.ppp_logger, self.ppp_interrupt, model_info, options, self.grammar_content, self.wildcards_obj
+        )
+        prompts_list = []
+
+        seeds = getattr(p, "all_seeds", [])
+        subseeds = getattr(p, "all_subseeds", [])
+        subseed_strength = getattr(p, "subseed_strength", 0.0)
+        if subseed_strength > 0:
+            calculated_seeds = [
+                int(subseed * subseed_strength + seed * (1 - subseed_strength))
+                for seed, subseed in zip(seeds, subseeds)
+            ]
+        else:
+            calculated_seeds = seeds
+        if len(set(calculated_seeds)) < len(calculated_seeds):
+            self.ppp_logger.info("Adjusting seeds because some are equal.")
+            calculated_seeds = [seed + i for i, seed in enumerate(calculated_seeds)]
+
+        # adds regular prompts
+        rpr = getattr(p, "all_prompts", None)
+        rnr = getattr(p, "all_negative_prompts", None)
+        if rpr is not None and rnr is not None:
+            prompts_list += [
+                ("regular", seed, prompt, negative_prompt)
+                for seed, prompt, negative_prompt in zip(calculated_seeds, rpr, rnr)
+                if (seed, prompt, negative_prompt) not in prompts_list
+            ]
+        # make it compatible with A1111 hires fix
+        rph = getattr(p, "all_hr_prompts", None)
+        rnh = getattr(p, "all_hr_negative_prompts", None)
+        if rph is not None and rnh is not None:
+            prompts_list += [
+                ("hiresfix", seed, prompt, negative_prompt)
+                for seed, prompt, negative_prompt in zip(calculated_seeds, rph, rnh)
+                if (seed, prompt, negative_prompt) not in prompts_list
+            ]
+
+        # processes prompts
+        for i, (prompttype, seed, prompt, negative_prompt) in enumerate(prompts_list):
+            if self.ppp_debug_level != DEBUG_LEVEL.none:
+                self.ppp_logger.info(f"processing prompts[{i+1}] ({prompttype})")
+            if self.lru_cache.get((seed, prompt, negative_prompt)) is None:
+                pp, np = ppp.process_prompt(prompt, negative_prompt, seed)
+                self.lru_cache.put((seed, prompt, negative_prompt), (pp, np))
+                # adds also the result so i2i doesn't process it unnecessarily
+                self.lru_cache.put((seed, pp, np), (pp, np))
+            elif self.ppp_debug_level != DEBUG_LEVEL.none:
+                self.ppp_logger.info("result already in cache")
+
+        # updates the prompts
+        if rpr is not None and rnr is not None:
+            for i, (seed, prompt, negative_prompt) in enumerate(zip(calculated_seeds, rpr, rnr)):
+                found = self.lru_cache.get((seed, prompt, negative_prompt))
+                if found is not None:
+                    rpr[i] = found[0]
+                    rnr[i] = found[1]
+        if rph is not None and rnh is not None:
+            for i, (seed, prompt, negative_prompt) in enumerate(zip(calculated_seeds, rph, rnh)):
+                found = self.lru_cache.get((seed, prompt, negative_prompt))
+                if found is not None:
+                    rph[i] = found[0]
+                    rnh[i] = found[1]
+
+        t2 = time.time()
+        if self.ppp_debug_level != DEBUG_LEVEL.none:
+            self.ppp_logger.info(f"process time: {t2 - t1:.3f} seconds")
+
+    def ppp_interrupt(self):
+        """
+        Interrupts the generation.
+
+        Returns:
+            None
+        """
+        shared.state.interrupted = True
+
+
+def on_ui_settings():
+    """
+    Callback function for UI settings.
+
+    Returns:
+        None
+    """
+
+    section = ("prompt-post-processor", PromptPostProcessor.NAME)
+
+    def import_old_settings(names, default):
+        for name in names:
+            if hasattr(opts, name):
+                return getattr(opts, name)
+        return default
+
+    def import_bool_to_any(name, value_false, value_true, default):
+        if hasattr(opts, name):
+            return value_true if getattr(opts, name) else value_false
+        return default
+
+    def new_html_title(title):
+        info = shared.OptionInfo(
+            title,
+            "",
+            gr.HTML,
+            section=section,
+        )
+        info.do_not_save = True
+        return info
+
+    # general settings
+    shared.opts.add_option(
+        key="ppp_gen_sep",
+        info=new_html_title("<h2>General settings</h2>"),
+    )
+    shared.opts.add_option(
+        key="ppp_gen_debug_level",
+        info=shared.OptionInfo(
+            default=import_bool_to_any(
+                "ppp_gen_debug",
+                DEBUG_LEVEL.minimal.value,
+                DEBUG_LEVEL.full.value,
+                DEBUG_LEVEL.minimal.value,
+            ),
+            label="Debug level",
+            component=gr.Radio,
+            component_args={
+                "choices": (
+                    ("None", DEBUG_LEVEL.none.value),
+                    ("Minimal", DEBUG_LEVEL.minimal.value),
+                    ("Full", DEBUG_LEVEL.full.value),
+                ),
+            },
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_gen_ponysubstrings",
+        info=shared.OptionInfo(
+            PromptPostProcessor.DEFAULT_PONY_SUBSTRINGS,
+            label="Comma separated list of substrings to look for in the model full filename to flag it as Pony (case insensitive)",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_gen_doi2i",
+        info=shared.OptionInfo(
+            False,
+            label="Apply in img2img",
+            comment_after='<span class="info">(this includes any pass that contains an initial image, like adetailer)</span>',
+            section=section,
+        ),
+    )
+
+    # wildcard settings
+    shared.opts.add_option(
+        key="ppp_wil_sep",
+        info=new_html_title('<br><h2>Wildcard settings</h2>'),
+    )
+    shared.opts.add_option(
+        key="ppp_wil_processwildcards",
+        info=shared.OptionInfo(
+            True,
+            label="Process wildcards",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_wil_wildcardsfolders",
+        info=shared.OptionInfo(
+            PPPWildcards.DEFAULT_WILDCARDS_FOLDER,
+            label="Wildcards folders",
+            comment_after='<span class="info">(absolute or relative to the models folder)</span>',
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_wil_ifwildcards",
+        info=shared.OptionInfo(
+            default=import_old_settings(
+                ["ppp_gen_ifwildcards", "ppp_ifwildcards"],
+                PromptPostProcessor.IFWILDCARDS_CHOICES.ignore.value,
+            ),
+            label="What to do with remaining/invalid wildcards?",
+            component=gr.Radio,
+            component_args={
+                "choices": (
+                    ("Ignore", PromptPostProcessor.IFWILDCARDS_CHOICES.ignore.value),
+                    ("Remove", PromptPostProcessor.IFWILDCARDS_CHOICES.remove.value),
+                    ("Add visible warning", PromptPostProcessor.IFWILDCARDS_CHOICES.warn.value),
+                    ("Stop the generation", PromptPostProcessor.IFWILDCARDS_CHOICES.stop.value),
+                )
+            },
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_wil_choice_separator",
+        info=shared.OptionInfo(
+            PromptPostProcessor.DEFAULT_CHOICE_SEPARATOR,
+            label="Default separator used when adding multiple choices",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_wil_keep_choices_order",
+        info=shared.OptionInfo(
+            False,
+            label="Keep the order of selected choices",
+            section=section,
+        ),
+    )
+
+    # content removal settings
+    shared.opts.add_option(
+        key="ppp_rem_sep",
+        info=new_html_title('<br><h2>Content removal settings</h2>'),
+    )
+    shared.opts.add_option(
+        key="ppp_rem_removeextranetworktags",
+        info=shared.OptionInfo(
+            False,
+            label="Remove extra network tags",
+            section=section,
+        ),
+    )
+
+    # send to negative settings
+    shared.opts.add_option(
+        key="ppp_stn_sep",
+        info=new_html_title('<br><h2>Send to Negative settings</h2>'),
+    )
+    shared.opts.add_option(
+        key="ppp_stn_separator",
+        info=shared.OptionInfo(
+            PromptPostProcessor.DEFAULT_STN_SEPARATOR,
+            label="Separator used when adding to the negative prompt",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_stn_ignorerepeats",
+        info=shared.OptionInfo(
+            True,
+            label="Ignore repeated content",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_stn_joinattention",
+        info=shared.OptionInfo(
+            True,
+            label="Join attention modifiers (weights) when possible",
+            section=section,
+        ),
+    )
+    # clean-up settings
+    shared.opts.add_option(
+        key="ppp_cup_sep",
+        info=new_html_title('<br><h2>Clean-up settings</h2>'),
+    )
+    shared.opts.add_option(
+        key="ppp_cup_emptyconstructs",
+        info=shared.OptionInfo(
+            True,
+            label="Remove empty constructs (attention, alternation, scheduling)",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_cup_extraseparators",
+        info=shared.OptionInfo(
+            True,
+            label="Remove extra separators",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_cup_extraseparators2",
+        info=shared.OptionInfo(
+            True,
+            label="Remove additional extra separators",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_cup_breaks",
+        info=shared.OptionInfo(
+            True,
+            label="Clean up around BREAKs",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_cup_breaks_eol",
+        info=shared.OptionInfo(
+            False,
+            label="Use EOL instead of Space before BREAKs",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_cup_ands",
+        info=shared.OptionInfo(
+            True,
+            label="Clean up around ANDs",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_cup_ands_eol",
+        info=shared.OptionInfo(
+            False,
+            label="Use EOL instead of Space before ANDs",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_cup_extranetworktags",
+        info=shared.OptionInfo(
+            False,
+            label="Clean up around extra network tags",
+            section=section,
+        ),
+    )
+    shared.opts.add_option(
+        key="ppp_cup_extraspaces",
+        info=shared.OptionInfo(
+            True,
+            label="Remove extra spaces",
+            section=section,
+        ),
+    )
+
+    # Remove old settings
+    # for name in ["ppp_gen_ifwildcards", "ppp_ifwildcards", "ppp_gen_debug", "ppp_stn_doi2i", "ppp_cup_doi2i"]:
+    #     if hasattr(opts, name):
+    #         delattr(opts, name)
+
+
+script_callbacks.on_ui_settings(on_ui_settings)

sendtonegative.py

@@ -1,324 +0,0 @@
-from collections import namedtuple
-import logging
-import re
-import lark
-
-
-class SendToNegative:  # pylint: disable=too-few-public-methods
-    NAME = "Send to Negative"
-    VERSION = "2.1"
-
-    DEFAULT_SEPARATOR = ", "
-
-    def __init__(
-        self,
-        log,
-        separator=None,
-        ignore_repeats=None,
-        join_attention=None,
-        cleanup=None,
-        opts=None,
-    ):
-        """
-        Format for the tag:
-            <!content!>
-
-            <!!x!content!>
-
-        with x being:
-            s - content is added at the start of the negative prompt. This is the default if no parameter exists.
-
-            e - content is added at the end of the negative prompt.
-
-            pN - content is added where the insertion point N is in the negative prompt or at the start if it does not exist. N can be 0 to 9.
-
-            iN - tags the position of insertion point N. Used only in the negative prompt and does not accept content. N can be 0 to 9.
-        """
-        self.__logger = log
-        if opts is not None and opts.prompt_attention == "Compel parser":
-            self.__logger.warning("Compel parser is not supported!")
-        self.__ignore_repeats = (
-            ignore_repeats if ignore_repeats is not None else getattr(opts, "stn_ignorerepeats", True)
-        )
-        self.__join_attention = (
-            join_attention
-            if join_attention is not None
-            else getattr(opts, "stn_joinattention", True)
-            if opts is not None
-            else True
-        )
-        self.__cleanup = (
-            cleanup if cleanup is not None else getattr(opts, "stn_cleanup", True) if opts is not None else True
-        )
-        self.__separator = (
-            separator
-            if separator is not None
-            else getattr(opts, "stn_separator", self.DEFAULT_SEPARATOR)
-            if opts is not None
-            else self.DEFAULT_SEPARATOR
-        )
-        self.__insertion_point_tags = [f"<!!i{x}!!>" for x in range(10)]
-        # Process with lark (debug with https://www.lark-parser.org/ide/)
-        self.__schedule_parser = lark.Lark(
-            r"""
-                start: (prompt | /[\][():|<>!]/+)*
-                ?prompt: (emphasized | deemphasized | scheduled | alternate | modeltag | negtag | plain)*
-                ?nonegprompt: (emphasized | deemphasized | scheduled | alternate | modeltag | plain)*
-                emphasized: "(" prompt [":" numpar] ")"
-                deemphasized: "[" prompt "]"
-                scheduled: "[" [prompt ":"] prompt ":" numpar "]"
-                alternate: "[" alternateoption ("|" alternateoption)+ "]"
-                alternateoption: prompt
-                negtag: "<!" [negtagparameters] nonegprompt "!>"
-                negtagparameters: "!" /s|e|[ip]\d/ "!"
-                modeltag: "<" /(?!!)[^>]+/ ">"
-                numpar: WHITESPACE* NUMBER WHITESPACE*
-                WHITESPACE: /\s+/
-                ?plain: /([^\\[\]():|<>!]|\\.)+/s
-                %import common.SIGNED_NUMBER -> NUMBER
-            """,
-            propagate_positions=True,
-        )
-
-    class ReadTree(lark.visitors.Interpreter):
-        def __init__(self, logger, ignorerepeats, joinattention, prompt, add_at):
-            super().__init__()
-            self.__logger = logger
-            self.__ignore_repeats = ignorerepeats
-            self.__join_attention = joinattention
-            self.__prompt = prompt
-            self.AccumulatedShell = namedtuple("AccumulatedShell", ["type", "info1", "info2"])
-            AccumulatedShell = self.AccumulatedShell
-            self.__shell: list[AccumulatedShell] = []
-            self.NegTag = namedtuple("NegTag", ["start", "end", "content", "parameters", "shell"])
-            NegTag = self.NegTag
-            self.__negtags: list[NegTag] = []
-            self.__already_processed = []
-            self.add_at = add_at
-            self.remove = []
-
-        def scheduled(self, tree):
-            if len(tree.children) > 2:  # before & after
-                before = tree.children[0]
-            else:
-                before = None
-            after = tree.children[-2]
-            numpar = tree.children[-1]
-            pos = float(numpar.children[0].value)
-            if pos >= 1:
-                pos = int(pos)
-            # self.__shell.append(self.AccumulatedShell("sc", tree.meta.start_pos, pos))
-            if before is not None and hasattr(before, "data"):
-                self.__logger.debug(
-                    f"Shell scheduled before at {[before.meta.start_pos,before.meta.end_pos] if hasattr(before,'meta') else '?'} : {pos}"
-                )
-                self.__shell.append(self.AccumulatedShell("scb", pos, None))
-                self.visit(before)
-                self.__shell.pop()
-            if hasattr(after, "data"):
-                self.__logger.debug(
-                    f"Shell scheduled after at {[after.meta.start_pos,after.meta.end_pos] if hasattr(after,'meta') else '?'} : {pos}"
-                )
-                self.__shell.append(self.AccumulatedShell("sca", pos, None))
-                self.visit(after)
-                self.__shell.pop()
-            # self.__shell.pop()
-
-        def alternate(self, tree):
-            # self.__shell.append(self.AccumulatedShell("al", tree.meta.start_pos, len(tree.children)))
-            for i, opt in enumerate(tree.children):
-                self.__logger.debug(
-                    f"Shell alternate at {[opt.meta.start_pos,opt.meta.end_pos] if hasattr(opt,'meta') else '?'} : {i+1}"
-                )
-                if hasattr(opt, "data"):
-                    self.__shell.append(self.AccumulatedShell("alo", i + 1, len(tree.children)))
-                    self.visit(opt)
-                    self.__shell.pop()
-            # self.__shell.pop()
-
-        def emphasized(self, tree):
-            numpar = tree.children[-1]
-            weight = float(numpar.children[0].value) if numpar is not None else 1.1
-            self.__logger.debug(
-                f"Shell attention at {[tree.meta.start_pos,tree.meta.end_pos] if hasattr(tree,'meta') else '?'}: {weight}"
-            )
-            self.__shell.append(self.AccumulatedShell("at", weight, None))
-            self.visit_children(tree)
-            self.__shell.pop()
-
-        def deemphasized(self, tree):
-            weight = 0.9
-            self.__logger.debug(
-                f"Shell attention at {[tree.meta.start_pos,tree.meta.end_pos] if hasattr(tree,'meta') else '?'}: {weight}"
-            )
-            self.__shell.append(self.AccumulatedShell("at", weight, None))
-            self.visit_children(tree)
-            self.__shell.pop()
-
-        def negtag(self, tree):
-            negtagparameters = tree.children[0]
-            parameters = negtagparameters.children[0].value if negtagparameters is not None else ""
-            rest = []
-            for x in tree.children[1::]:
-                rest.append(self.__prompt[x.meta.start_pos : x.meta.end_pos] if hasattr(x, "meta") else x.value)
-            content = "".join(rest)
-            self.__negtags.append(
-                self.NegTag(tree.meta.start_pos, tree.meta.end_pos, content, parameters, self.__shell.copy())
-            )
-            self.__logger.debug(
-                f"Negative tag at {[tree.meta.start_pos,tree.meta.end_pos] if hasattr(tree,'meta') else '?'}: {parameters}: {content.encode('unicode_escape').decode('utf-8')}"
-            )
-
-        def start(self, tree):
-            self.visit_children(tree)
-            # process the found negtags
-            for nt in self.__negtags:
-                if self.__join_attention:
-                    # join consecutive attention elements
-                    for i in range(len(nt.shell) - 1, 0, -1):
-                        if nt.shell[i].type == "at" and nt.shell[i - 1].type == "at":
-                            nt.shell[i - 1] = self.AccumulatedShell(
-                                "at",
-                                (100 * nt.shell[i - 1].info1 * nt.shell[i].info1) / 100,  # we limit to two decimals
-                                None,
-                            )
-                            nt.shell.pop(i)
-                start = ""
-                end = ""
-                for s in nt.shell:
-                    match s.type:
-                        case "at":
-                            if s.info1 == 0.9:
-                                start += "["
-                                end = "]" + end
-                            elif s.info1 == 1.1:
-                                start += "("
-                                end = ")" + end
-                            else:
-                                start += "("
-                                end = f":{s.info1})" + end
-                        # case "sc":
-                        case "scb":
-                            start += "["
-                            end = f"::{s.info1}]" + end
-                        case "sca":
-                            start += "["
-                            end = f":{s.info1}]" + end
-                        # case "al":
-                        case "alo":
-                            start += "[" + ("|" * int(s.info1 - 1))
-                            end = ("|" * int(s.info2 - s.info1)) + "]" + end
-                content = start + nt.content + end
-                position = nt.parameters or "s"
-                if len(content) > 0:
-                    if content not in self.__already_processed:
-                        if self.__ignore_repeats:
-                            self.__already_processed.append(content)
-                        self.__logger.debug(
-                            f"Adding content at position {position}: {content.encode('unicode_escape').decode('utf-8')}"
-                        )
-                        if position == "e":
-                            self.add_at["end"].append(content)
-                        elif position.startswith("p"):
-                            n = int(position[1])
-                            self.add_at["insertion_point"][n].append(content)
-                        else:  # position == "s" or invalid
-                            self.add_at["start"].append(content)
-                    else:
-                        self.__logger.warning(
-                            f"Ignoring repeated content: {content.encode('unicode_escape').decode('utf-8')}"
-                        )
-                # remove from prompt
-                self.remove.append([nt.start, nt.end])
-
-    def process_prompt(self, original_prompt, original_negative_prompt):
-        """
-        Extract from the prompt the tagged parts and add them to the negative prompt
-        """
-        try:
-            prompt = original_prompt
-            negative_prompt = original_negative_prompt
-            self.__logger.debug(f"Input prompt: {prompt.encode('unicode_escape').decode('utf-8')}")
-            self.__logger.debug(f"Input negative_prompt: {negative_prompt.encode('unicode_escape').decode('utf-8')}")
-            prompt, add_at = self.__find_tags(prompt)
-            negative_prompt = self.__add_to_insertion_points(negative_prompt, add_at["insertion_point"])
-            if len(add_at["start"]) > 0:
-                negative_prompt = self.__add_to_start(negative_prompt, add_at["start"])
-            if len(add_at["end"]) > 0:
-                negative_prompt = self.__add_to_end(negative_prompt, add_at["end"])
-            self.__logger.debug(f"Output prompt: {prompt.encode('unicode_escape').decode('utf-8')}")
-            self.__logger.debug(f"Output negative_prompt: {negative_prompt.encode('unicode_escape').decode('utf-8')}")
-            return prompt, negative_prompt
-        except Exception as e:  # pylint: disable=broad-exception-caught
-            self.__logger.exception(e)
-            return original_prompt, original_negative_prompt
-
-    def __find_tags(self, prompt):
-        add_at = {"start": [], "insertion_point": [[] for x in range(10)], "end": []}
-        tree = self.__schedule_parser.parse(prompt)
-        self.__logger.debug(f"Initial tree:\n{tree.pretty()}")
-
-        readtree = self.ReadTree(self.__logger, self.__ignore_repeats, self.__join_attention, prompt, add_at)
-        readtree.visit(tree)
-
-        for r in readtree.remove[::-1]:
-            prompt = prompt[: r[0]] + prompt[r[1] :]
-        if self.__cleanup:
-            prompt = re.sub(r"\((?::[+-]?[\d\.]+)?\)", "", prompt)  # clean up empty attention
-            prompt = re.sub(r"\[\]", "", prompt)  # clean up empty attention
-            prompt = re.sub(r"\[:?:[+-]?[\d\.]+\]", "", prompt)  # clean up empty scheduling
-            prompt = re.sub(r"\[\|+\]", "", prompt)  # clean up empty alternation
-            # clean up whitespace and extra separators
-            prompt = (
-                prompt.replace("  ", " ")
-                .replace(self.__separator + self.__separator, self.__separator)
-                .replace(" " + self.__separator, self.__separator)
-                .removeprefix(self.__separator)
-                .removesuffix(self.__separator)
-                .strip()
-            )
-        add_at = readtree.add_at
-        self.__logger.debug(f"New negative additions: {add_at}")
-
-        return prompt, add_at
-
-    def __add_to_insertion_points(self, negative_prompt, add_at_insertion_point):
-        for n in range(10):
-            ipp = negative_prompt.find(self.__insertion_point_tags[n])
-            if ipp >= 0:
-                ipl = len(self.__insertion_point_tags[n])
-                if negative_prompt[ipp - len(self.__separator) : ipp] == self.__separator:
-                    ipp -= len(self.__separator)  # adjust for existing start separator
-                    ipl += len(self.__separator)
-                add_at_insertion_point[n].insert(0, negative_prompt[:ipp])
-                if negative_prompt[ipp + ipl : ipp + ipl + len(self.__separator)] == self.__separator:
-                    ipl += len(self.__separator)  # adjust for existing end separator
-                endPart = negative_prompt[ipp + ipl :]
-                if len(endPart) > 0:
-                    add_at_insertion_point[n].append(endPart)
-                negative_prompt = self.__separator.join(add_at_insertion_point[n])
-            else:
-                ipp = 0
-                if negative_prompt.startswith(self.__separator):
-                    ipp = len(self.__separator)
-                add_at_insertion_point[n].append(negative_prompt[ipp:])
-                negative_prompt = self.__separator.join(add_at_insertion_point[n])
-        return negative_prompt
-
-    def __add_to_start(self, negative_prompt, add_at_start):
-        if len(negative_prompt) > 0:
-            ipp = 0
-            if negative_prompt.startswith(self.__separator):
-                ipp = len(self.__separator)  # adjust for existing end separator
-            add_at_start.append(negative_prompt[ipp:])
-        negative_prompt = self.__separator.join(add_at_start)
-        return negative_prompt
-
-    def __add_to_end(self, negative_prompt, add_at_end):
-        if len(negative_prompt) > 0:
-            ipl = len(negative_prompt)
-            if negative_prompt.endswith(self.__separator):
-                ipl -= len(self.__separator)  # adjust for existing start separator
-            add_at_end.insert(0, negative_prompt[:ipl])
-        negative_prompt = self.__separator.join(add_at_end)
-        return negative_prompt

stnlogging.py

@@ -1,40 +0,0 @@
-import sys
-import copy
-import logging
-
-
-class SendToNegativeLogFactory:
-    class ColoredFormatter(logging.Formatter):
-        COLORS = {
-            "DEBUG": "\033[0;36m",  # CYAN
-            "INFO": "\033[0;32m",  # GREEN
-            "WARNING": "\033[0;33m",  # YELLOW
-            "ERROR": "\033[0;31m",  # RED
-            "CRITICAL": "\033[0;37;41m",  # WHITE ON RED
-            "RESET": "\033[0m",  # RESET COLOR
-        }
-
-        def format(self, record):
-            colored_record = copy.copy(record)
-            levelname = colored_record.levelname
-            seq = self.COLORS.get(levelname, self.COLORS["RESET"])
-            colored_record.levelname = f"{seq}{levelname}{self.COLORS['RESET']}"
-            return super().format(colored_record)
-
-    def __init__(self):
-        logsd = logging.getLogger("sd")
-        stnlog = logging.getLogger("SendToNegative")
-        stnlog.setLevel(logging.INFO)
-        stnlog.handlers = logsd.handlers
-        if not stnlog.handlers:
-            handler = logging.StreamHandler(sys.stdout)
-            handler.setFormatter(self.ColoredFormatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
-            stnlog.addHandler(handler)
-            self.log = stnlog
-        else:
-            self.log = SendToNegativeLogCustomAdapter(stnlog)
-
-
-class SendToNegativeLogCustomAdapter(logging.LoggerAdapter):
-    def process(self, msg, kwargs):
-        return f"[SendToNegative] {msg}", kwargs

tests/tests.py

@@ -1,183 +1,691 @@
+from collections import namedtuple
 import logging
 import unittest
 import sys
 import os
 
-sys.path.insert(1, os.path.join(sys.path[0], ".."))
+from ppp_wildcards import PPPWildcards
 
-from sendtonegative import SendToNegative  # pylint: disable=import-error
+sys.path.append(os.path.join(sys.path[0], ".."))
-from stnlogging import SendToNegativeLogFactory
+
+from ppp import PromptPostProcessor
+from ppp_logging import DEBUG_LEVEL, PromptPostProcessorLogFactory
 
 
-class TestSendToNegative(unittest.TestCase):
+PromptPair = namedtuple("PromptPair", ["prompt", "negative_prompt"], defaults=["", ""])
+
+
+class TestPromptPostProcessor(unittest.TestCase):
+    """
+    A test case class for testing the PromptPostProcessor class.
+    """
+
     def setUp(self):
-        lf = SendToNegativeLogFactory()
+        """
-        self.__log = lf.log
+        Set up the test case by initializing the necessary objects and configurations.
-        self.__log.setLevel(logging.DEBUG)
+        """
-        self.defstn = SendToNegative(self.__log, separator=", ", ignore_repeats=True, join_attention=True, cleanup=True)
+        lf = PromptPostProcessorLogFactory()
+        self.__ppp_logger = lf.log
+        self.__ppp_logger.setLevel(logging.DEBUG)
+        self.__defopts = {
+            "debug_level": DEBUG_LEVEL.full.value,
+            "pony_substrings": PromptPostProcessor.DEFAULT_PONY_SUBSTRINGS,
+            "process_wildcards": True,
+            "if_wildcards": PromptPostProcessor.IFWILDCARDS_CHOICES.ignore.value,
+            "choice_separator": ", ",
+            "keep_choices_order": False,
+            "stn_separator": ", ",
+            "stn_ignore_repeats": True,
+            "stn_join_attention": True,
+            "cleanup_empty_constructs": True,
+            "cleanup_extra_separators": True,
+            "cleanup_extra_separators2": True,
+            "cleanup_extra_spaces": True,
+            "cleanup_breaks": True,
+            "cleanup_breaks_eol": False,
+            "cleanup_ands": True,
+            "cleanup_ands_eol": False,
+            "cleanup_extranetwork_tags": True,
+            "remove_extranetwork_tags": False,
+        }
+        self.__def_model_info = {
+            "is_sd1": False,
+            "is_sd2": False,
+            "is_sdxl": True,
+            "is_ssd": False,
+            "is_sd3": False,
+            "is_flux": False,
+            "models_path": "./webui/models",
+            "model_filename": "./webui/models/Stable-diffusion/testmodel.safetensors",
+        }
+        self.__interrupted = False
+        self.__wildcards_obj = PPPWildcards(lf.log)
+        self.__wildcards_obj.refresh_wildcards(
+            DEBUG_LEVEL.full,
+            [
+                os.path.abspath(os.path.join(os.path.dirname(__file__), "wildcards")),
+                os.path.abspath(os.path.join(os.path.dirname(__file__), "wildcards2")),
+            ],
+        )
+        grammar_filename = os.path.join(os.path.dirname(os.path.realpath(__file__)), "../grammar.lark")
+        with open(grammar_filename, "r", encoding="utf-8") as file:
+            self.__grammar_content = file.read()
+        self.__defppp = PromptPostProcessor(
+            self.__ppp_logger,
+            self.__interrupt,
+            self.__def_model_info,
+            self.__defopts,
+            self.__grammar_content,
+            self.__wildcards_obj,
+        )
+        self.__nocupppp = PromptPostProcessor(
+            self.__ppp_logger,
+            self.__interrupt,
+            self.__def_model_info,
+            {
+                **self.__defopts,
+                "cleanup_empty_constructs": False,
+                "cleanup_extra_separators": False,
+                "cleanup_extra_separators2": False,
+                "cleanup_extra_spaces": False,
+                "cleanup_breaks": False,
+                "cleanup_breaks_eol": False,
+                "cleanup_ands": False,
+                "cleanup_ands_eol": False,
+                "cleanup_extranetwork_tags": False,
+            },
+            self.__grammar_content,
+            self.__wildcards_obj,
+        )
 
-    def process(
+    def __interrupt(self):
+        self.__interrupted = True
+
+    def __process(
         self,
-        prompt,
+        input_prompts: PromptPair,
-        negative_prompt,
+        expected_output_prompts: PromptPair | list[PromptPair],
-        expected_prompt,
+        seed: int = 1,
-        expected_negative_prompt,
+        ppp=None,
-        stn=None,
+        interrupted=False,
     ):
-        the_obj = self.defstn if stn is None else stn
+        """
-        result_prompt, result_negative_prompt = the_obj.process_prompt(prompt, negative_prompt)
+        Process the prompt and compare the results with the expected prompts.
-        self.assertEqual(result_prompt, expected_prompt, f"Prompt should be '{expected_prompt}'")
+
-        self.assertEqual(
+        Args:
-            result_negative_prompt,
+            input_prompts (PromptPair): The input prompts.
-            expected_negative_prompt,
+            expected_output_prompts (PromptPair | list[PromptPair]): The expected prompts.
-            f"Negative Prompt should be '{expected_negative_prompt}'",
+            seed (int, optional): The seed value. Defaults to 1.
+            ppp (object, optional): The post-processor object. Defaults to None.
+            interrupted (bool, optional): The interrupted flag. Defaults to False.
+
+        Returns:
+            None
+        """
+        the_obj = ppp or self.__defppp
+        out = expected_output_prompts if isinstance(expected_output_prompts, list) else [expected_output_prompts]
+        for eo in out:
+            result_prompt, result_negative_prompt = the_obj.process_prompt(
+                input_prompts.prompt,
+                input_prompts.negative_prompt,
+                seed,
+            )
+            self.assertEqual(self.__interrupted, interrupted, "Interrupted flag is incorrect")
+            if not self.__interrupted:
+                self.assertEqual(result_prompt, eo.prompt, "Incorrect prompt")
+                self.assertEqual(result_negative_prompt, eo.negative_prompt, "Incorrect negative prompt")
+            seed += 1
+
+    # Send To Negative tests
+
+    def test_stn_simple(self):  # negtags with different parameters and separations
+        self.__process(
+            PromptPair(
+                "flowers<ppp:stn>red<ppp:/stn>, <ppp:stn s>green<ppp:/stn>, <ppp:stn e>blue<ppp:/stn><ppp:stn p0>yellow<ppp:/stn>, <ppp:stn p1>purple<ppp:/stn><ppp:stn p2>black<ppp:/stn>",
+                "<ppp:stn i0>normal quality<ppp:stn i1>, worse quality<ppp:stn i2>",
+            ),
+            PromptPair("flowers", "red, green, yellow, normal quality, purple, worse quality, black, blue"),
         )
 
-    def test_tag_default(self):
+    def test_stn_complex(self):  # complex negtags
-        self.process(
+        self.__process(
-            "flowers<!red!>",
+            PromptPair(
-            "normal quality, worse quality",
+                "<ppp:stn>red<ppp:/stn> ((<ppp:stn s>pink<ppp:/stn>)), flowers <ppp:stn e>purple<ppp:/stn>, <ppp:stn p0>mauve<ppp:/stn><ppp:stn e>blue<ppp:/stn>, <ppp:stn p0>yellow<ppp:/stn> <ppp:stn p1>green<ppp:/stn>",
-            "flowers",
+                "normal quality, <ppp:stn i0>, bad quality<ppp:stn i1>, worse quality",
-            "red, normal quality, worse quality",
+            ),
+            PromptPair(
+                "flowers",
+                "red, (pink:1.21), normal quality, mauve, yellow, bad quality, green, worse quality, purple, blue",
+            ),
         )
 
-    def test_tag_start(self):
+    def test_stn_complex_nocleanup(self):  # complex negtags with no cleanup
-        self.process(
+        self.__process(
-            "flowers<!!s!red!>",
+            PromptPair(
-            "normal quality, worse quality",
+                "<ppp:stn>red<ppp:/stn> ((<ppp:stn s>pink<ppp:/stn>)), flowers <ppp:stn e>purple<ppp:/stn>, <ppp:stn p0>mauve<ppp:/stn><ppp:stn e>blue<ppp:/stn>, <ppp:stn p0>yellow<ppp:/stn> <ppp:stn p1>green<ppp:/stn>",
-            "flowers",
+                "normal quality, <ppp:stn i0>, bad quality<ppp:stn i1>, worse quality",
-            "red, normal quality, worse quality",
+            ),
+            PromptPair(
+                " (()), flowers , ,  ",
+                "red, (pink:1.21), normal quality, mauve, yellow, bad quality, green, worse quality, purple, blue",
+            ),
+            ppp=self.__nocupppp,
         )
 
-    def test_tag_end(self):
+    def test_stn_inside_attention(self):  # negtag inside attention
-        self.process(
+        self.__process(
-            "flowers<!!e!red!>",
+            PromptPair(
-            "normal quality, worse quality",
+                "[<ppp:stn>neg1<ppp:/stn>] this is a ((test<ppp:stn e>neg2<ppp:/stn>) (test:2.0): 1.5 ) (red<ppp:stn>[square]<ppp:/stn>:1.5)",
-            "flowers",
+                "normal quality",
-            "normal quality, worse quality, red",
+            ),
+            PromptPair(
+                "this is a ((test) (test:2.0):1.5) (red:1.5)", "[neg1], ([square]:1.5), normal quality, (neg2:1.65)"
+            ),
         )
 
-    def test_tag_insertion_mid_sep(self):
+    def test_stn_inside_alternation(self):  # negtag inside alternation
-        self.process(
+        self.__process(
-            "flowers<!!p0!red!>",
+            PromptPair(
-            "normal quality, <!!i0!!>, worse quality",
+                "this is a (([complex<ppp:stn>neg1<ppp:/stn>|simple<ppp:stn>neg2<ppp:/stn>|regular<ppp:stn>neg3<ppp:/stn>] test)(test:2.0):1.5)",
-            "flowers",
+                "normal quality",
-            "normal quality, red, worse quality",
+            ),
+            PromptPair(
+                "this is a (([complex|simple|regular] test)(test:2.0):1.5)",
+                "([neg1||]:1.65), ([|neg2|]:1.65), ([||neg3]:1.65), normal quality",
+            ),
         )
 
-    def test_tag_insertion_mid_no_sep(self):
+    def test_stn_inside_alternation_recursive(self):  # negtag inside alternation (recursive alternation)
-        self.process(
+        self.__process(
-            "flowers<!!p0!red!>",
+            PromptPair(
-            "normal quality<!!i0!!>worse quality",
+                "this is a (([complex<ppp:stn>neg1<ppp:/stn>[one|two<ppp:stn>neg12<ppp:/stn>||three|four(<ppp:stn>neg14<ppp:/stn>)]|simple<ppp:stn>neg2<ppp:/stn>|regular<ppp:stn>neg3<ppp:/stn>] test)(test:2.0):1.5)",
-            "flowers",
+                "normal quality",
-            "normal quality, red, worse quality",
+            ),
+            PromptPair(
+                "this is a (([complex[one|two||three|four]|simple|regular] test)(test:2.0):1.5)",
+                "([neg1||]:1.65), ([[|neg12|||]||]:1.65), ([[||||(neg14)]||]:1.65), ([|neg2|]:1.65), ([||neg3]:1.65), normal quality",
+            ),
         )
 
-    def test_tag_insertion_start_sep(self):
+    def test_stn_inside_scheduling(self):  # negtag inside scheduling
-        self.process(
+        self.__process(
-            "flowers<!!p0!red!>",
+            PromptPair("this is [abc<ppp:stn>neg1<ppp:/stn>:def<ppp:stn e>neg2<ppp:/stn>: 5 ]", "normal quality"),
-            "<!!i0!!>, normal quality, worse quality",
+            [PromptPair("this is [abc:def:5]", "[neg1::5], normal quality, [neg2:5]")],
-            "flowers",
-            "red, normal quality, worse quality",
         )
 
-    def test_tag_insertion_start_no_sep(self):
+    def test_stn_complex_features(self):  # complex negtags with AND, BREAK and other features
-        self.process(
+        self.__process(
-            "flowers<!!p0!red!>",
+            PromptPair(
-            "<!!i0!!>normal quality, worse quality",
+                "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loratrigger <lora:xxx:1> AND AND hypernettrigger <hypernet:yyy>:0.3",
-            "flowers",
+                "normal quality, <ppp:stn i0>",
-            "red, normal quality, worse quality",
+            ),
+            PromptPair(
+                "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5)\nBREAK with [abc:def:5]:0.5 AND loratrigger <lora:xxx:1> AND hypernettrigger <hypernet:yyy>:0.3",
+                "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+            ),
         )
 
-    def test_tag_insertion_end_sep(self):
+    def test_stn_complex_features_newformat(self):  # complex negtags with AND, BREAK and other features (new format)
-        self.process(
+        self.__process(
-            "flowers<!!p0!red!>",
+            PromptPair(
-            "normal quality, worse quality, <!!i0!!>",
+                "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loratrigger <lora:xxx:1> AND AND hypernettrigger <hypernet:yyy>:0.3",
-            "flowers",
+                "normal quality, <ppp:stn i0>",
-            "normal quality, worse quality, red",
+            ),
+            PromptPair(
+                "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5)\nBREAK with [abc:def:5]:0.5 AND loratrigger <lora:xxx:1> AND hypernettrigger <hypernet:yyy>:0.3",
+                "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+            ),
         )
 
-    def test_tag_insertion_end_no_sep(self):
+    # Cleanup tests
-        self.process(
+
-            "flowers<!!p0!red!>",
+    def test_cl_simple(self):  # simple cleanup
-            "normal quality, worse quality<!!i0!!>",
+        self.__process(
-            "flowers",
+            PromptPair("  this is a ((test ), , ,  (), ,   [] ( , test ,:2.0):1.5) (red:1.5)  ", "  normal quality  "),
-            "normal quality, worse quality, red",
+            PromptPair("this is a ((test), (test,:2.0):1.5) (red:1.5)", "normal quality"),
         )
 
-    def test_complex(self):
+    def test_cl_complex(self):  # complex cleanup
-        self.process(
+        self.__process(
-            "<!red!> (<!!s!pink!>), flowers <!!e!purple!>, <!!e!blue!>, <!!p0!yellow!> <!!p1!green!>",
+            PromptPair(
-            "normal quality, <!!i0!!>, bad quality<!!i1!!>, worse quality",
+                "  this is BREAKABLE a ((test)), ,AND AND(() [] <lora:test> ANDERSON (test:2.0):1.5) :o BREAK \n BREAK (red:1.5)  ",
-            "flowers",
+                "  [:hands, feet, :0.15]normal quality  ",
-            "red, (pink), normal quality, yellow, bad quality, green, worse quality, purple, blue",
+            ),
+            PromptPair(
+                "this is BREAKABLE a ((test)) AND(<lora:test> ANDERSON (test:2.0):1.5) :o BREAK (red:1.5)",
+                "[:hands, feet, :0.15]normal quality",
+            ),
         )
 
-    def test_complex_no_cleanup(self):
+    def test_cl_removenetworktags(self):  # remove network tags
-        self.process(
+        self.__process(
-            "<!red!> (<!!s!pink!>), flowers <!!e!purple!>, <!!e!blue!>, <!!p0!yellow!> <!!p1!green!>",
+            PromptPair("this is a <lora:test> test", ""),
-            "normal quality, <!!i0!!>, bad quality<!!i1!!>, worse quality",
+            PromptPair("this is a test", ""),
-            " (), flowers , ,  ",
+            ppp=PromptPostProcessor(
-            "red, (pink), normal quality, yellow, bad quality, green, worse quality, purple, blue",
+                self.__ppp_logger,
-            SendToNegative(self.__log, separator=", ", ignore_repeats=True, join_attention=True, cleanup=False),
+                self.__interrupt,
+                self.__def_model_info,
+                {**self.__defopts, "remove_extranetwork_tags": True},
+            ),
         )
 
-    def test_inside_attention1(self):
+    def test_cl_dontremoveseparatorsoneol(self):  # dont remove separators on eol
-        self.process(
+        self.__process(
-            "[<!neg1!>] this is a ((test<!!e!neg2!>) (test:2.0):1.5)",
+            PromptPair("this is a test,\nsecond line", ""),
-            "normal quality",
+            PromptPair("this is a test,\nsecond line", ""),
-            "this is a ((test) (test:2.0):1.5)",
+            ppp=PromptPostProcessor(
-            "[neg1], normal quality, (neg2:1.65)",
+                self.__ppp_logger,
+                self.__interrupt,
+                self.__def_model_info,
+                {**self.__defopts, "cleanup_extra_separators2": False},
+            ),
         )
 
-    def test_inside_attention2(self):
+    # Command tests
-        self.process(
+
-            "(red<![square]!>:1.5)",
+    def test_cmd_stn_complex_features(self):  # complex stn command with AND, BREAK and other features
-            "",
+        self.__process(
-            "(red:1.5)",
+            PromptPair(
-            "([square]:1.5)",
+                "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loratrigger <lora:xxx:1> AND AND hypernettrigger <hypernet:yyy>:0.3",
+                "normal quality, <ppp:stn i0>",
+            ),
+            PromptPair(
+                "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5)\nBREAK with [abc:def:5]:0.5 AND loratrigger <lora:xxx:1> AND hypernettrigger <hypernet:yyy>:0.3",
+                "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+            ),
         )
 
-    def test_inside_alternation1(self):
+    def test_cmd_if_complex_features(self):  # complex if command
-        self.process(
+        self.__process(
-            "this is a (([complex|simple<!neg1!>|regular] test)(test:2.0):1.5)",
+            PromptPair(
-            "normal quality",
+                "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK, BREAK <ppp:if _is_sd1>with [abc<ppp:stn>neg4<ppp:/stn>:def:5]<ppp:/if>:0.5 AND <ppp:if _is_sd1>loratrigger <lora:xxx:1><ppp:elif _is_sdxl>hypernettrigger <hypernet:yyy><ppp:else>nothing<ppp:/if>:0.3",
-            "this is a (([complex|simple|regular] test)(test:2.0):1.5)",
+                "normal quality",
-            "([|neg1|]:1.65), normal quality",
+            ),
+            PromptPair(
+                "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5)\nBREAK :0.5 AND hypernettrigger <hypernet:yyy>:0.3",
+                "normal quality",
+            ),
         )
 
-    def test_inside_alternation2(self):
+    def test_cmd_if_nested(self):  # nested if command
-        self.process(
+        self.__process(
-            "this is a (([complex<!neg1!>|simple<!neg2!>|regular<!neg3!>] test)(test:2.0):1.5)",
+            PromptPair(
-            "normal quality",
+                "this is <ppp:if _sd eq 'sd1'>SD1<ppp:else><ppp:if _is_pony>PONY<ppp:else>SD2<ppp:/if><ppp:/if>", ""
-            "this is a (([complex|simple|regular] test)(test:2.0):1.5)",
+            ),
-            "([neg1||]:1.65), ([|neg2|]:1.65), ([||neg3]:1.65), normal quality",
+            PromptPair("this is PONY", ""),
+            ppp=PromptPostProcessor(
+                self.__ppp_logger,
+                self.__interrupt,
+                {
+                    **self.__def_model_info,
+                    "model_filename": "./webui/models/Stable-diffusion/ponymodel.safetensors",
+                },
+                self.__defopts,
+            ),
         )
 
-    def test_inside_alternation3(self):
+    def test_cmd_set_if(self):  # set and if commands
-        self.process(
+        self.__process(
-            "this is a (([complex<!neg1!>[one|two<!neg12!>|three|four(<!neg14!>)]|simple<!neg2!>|regular<!neg3!>] test)(test:2.0):1.5)",
+            PromptPair("<ppp:set v>value<ppp:/set>this test is <ppp:if v>OK<ppp:else>not OK<ppp:/if>", ""),
-            "normal quality",
+            PromptPair("this test is OK", ""),
-            "this is a (([complex[one|two|three|four]|simple|regular] test)(test:2.0):1.5)",
-            "([neg1||]:1.65), ([[|neg12||]||]:1.65), ([[|||(neg14)]||]:1.65), ([|neg2|]:1.65), ([||neg3]:1.65), normal quality",
         )
 
-    def test_inside_scheduling(self):
+    def test_cmd_set_eval_if(self):  # set and if commands
-        self.process(
+        self.__process(
-            "this is [abc<!neg1!>:def<!!e!neg2!>:5]",
+            PromptPair("<ppp:set v evaluate>value<ppp:/set>this test is <ppp:if v>OK<ppp:else>not OK<ppp:/if>", ""),
-            "normal quality",
+            PromptPair("this test is OK", ""),
-            "this is [abc:def:5]",
-            "[neg1::5], normal quality, [neg2:5]",
         )
 
-    def test_complex_features(self):
+    def test_cmd_set_if_echo_nested(self):  # nested set, if and echo commands
-        self.process(
+        self.__process(
-            "[<!neg5!>] this is: a (([complex|simple<!neg6!>|regular] test<!neg1!>)(test:2.0):1.5) \nBREAK with [abc<!neg4!>:def<!!p0!neg2(neg3:1.6)!>:5] <lora:xxx:1>",
+            PromptPair(
-            "normal quality, <!!i0!!>",
+                "<ppp:set v1>1<ppp:/set><ppp:if v1 gt 0><ppp:set v2>OK<ppp:/set><ppp:/if><ppp:if v2 eq 'OK'><ppp:echo v2><ppp:else>not OK<ppp:/if> <ppp:echo v2>NOK<ppp:/echo> <ppp:echo v3>OK<ppp:/echo>",
-            "this is: a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5] <lora:xxx:1>",
+                "",
-            "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+            ),
+            PromptPair("OK OK OK", ""),
         )
 
+    def test_cmd_set_if2(self):  # set and more complex if commands
+        self.__process(
+            PromptPair(
+                "First: <ppp:set v>value1<ppp:/set>this test is <ppp:if v in ('value1','value2')>OK<ppp:else>not OK<ppp:/if>\nSecond: <ppp:set v2>value3<ppp:/set>this test is <ppp:if not v2 in ('value1','value2')>OK<ppp:else>not OK<ppp:/if>",
+                "",
+            ),
+            PromptPair("First: this test is OK\nSecond: this test is OK", ""),
+        )
+
+    def test_cmd_set_add_if(self):  # set, add and if commands
+        self.__process(
+            PromptPair(
+                "<ppp:set v>value<ppp:/set><ppp:set v add>2<ppp:/set>this test is <ppp:if v eq 'value2'>OK<ppp:else>not OK<ppp:/if>",
+                "",
+            ),
+            PromptPair("this test is OK", ""),
+        )
+
+    def test_cmd_set_add_DP_if(self):  # set, add (DP format) and if commands
+        self.__process(
+            PromptPair(
+                "${v=value}${v+=2}this test is <ppp:if v eq 'value2'>OK<ppp:else>not OK<ppp:/if>",
+                "",
+            ),
+            PromptPair("this test is OK", ""),
+        )
+
+    def test_cmd_set_immediateeval(self):  # set (DP format) with mixed evaluation
+        self.__process(
+            PromptPair(
+                "${var=!__yaml/wildcard1__}the choices are: ${var}, ${var}, ${var2:default}, ${var3=__yaml/wildcard1__}${var3}, ${var3}",
+                "",
+            ),
+            PromptPair("the choices are: choice2, choice2, default, choice3, choice1", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_cmd_set_mixeval(self):  # set and add (DP format) with mixed evaluation
+        self.__process(
+            PromptPair(
+                "${var=__yaml/wildcard1__}the choices are: ${var}, ${var}, ${var+=, __yaml/wildcard2__}${var}, ${var}, ${var+=!, __yaml/wildcard3__}${var}, ${var}",
+                "",
+            ),
+            PromptPair(
+                "the choices are: choice2, choice3, choice1, choice1- choice2 -choice3, choice2,  choice2 -choice1-choice3, choice2, choice3-choice1- choice2 , choice1, choice2 , choice2, choice3-choice1- choice2 , choice1, choice2 ",
+                "",
+            ),
+            ppp=self.__nocupppp,
+        )
+
+    # Choices tests
+
+    def test_ch_choices(self):  # simple choices with weights
+        self.__process(
+            PromptPair("the choices are: {3::choice1|2::choice2|choice3}", ""),
+            PromptPair("the choices are: choice2", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_ch_unsupportedsampler(self):  # unsupported sampler
+        self.__process(
+            PromptPair("the choices are: {@choice1|choice2|choice3}", ""),
+            PromptPair("", ""),
+            ppp=self.__nocupppp,
+            interrupted=True,
+        )
+
+    def test_ch_choices_withcomments(self):  # choices with comments and multiline
+        self.__process(
+            PromptPair(
+                "the choices are: {\n3::choice1 # this is option 1\n|2::choice2\n# this was option 2\n|choice3 # this is option 3\n}",
+                "",
+            ),
+            PromptPair("the choices are: choice2", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_ch_choices_multiple(self):  # choices with multiple selection
+        self.__process(
+            PromptPair("the choices are: {~2$$, $$3::choice1|2:: choice2 |choice3}", ""),
+            PromptPair("the choices are:  choice2 , choice3", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_ch_choices_if_multiple(self):  # choices with if and multiple selection
+        self.__process(
+            PromptPair("the choices are: {2$$, $$3::choice1|2 if _is_sd1::choice2|choice3}", ""),
+            PromptPair("the choices are: choice1, choice3", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_ch_choices_set_if_multiple(self):  # choices with if user variable and multiple selection
+        self.__process(
+            PromptPair("${var=test}the choices are: {2$$, $$3::choice1|2 if not var eq 'test'::choice2|choice3}", ""),
+            PromptPair("the choices are: choice1, choice3", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_ch_choices_set_if_nested(self):  # nested choices with if user variable and multiple selection
+        self.__process(
+            PromptPair(
+                "${var=test}the choices are: {2$$, $$3::choice1${var2=test2} {if var2 eq 'test2'::choice11|choice12}|2 if not var eq 'test'::choice2|choice3}",
+                "",
+            ),
+            PromptPair("the choices are: choice1 choice11, choice3", ""),
+            ppp=self.__nocupppp,
+        )
+
+    # Wildcards tests
+
+    def test_wc_ignore(self):  # wildcards with ignore option
+        self.__process(
+            PromptPair("__bad_wildcard__", "{option1|option2}"),
+            PromptPair("__bad_wildcard__", "{option1|option2}"),
+            ppp=PromptPostProcessor(
+                self.__ppp_logger,
+                self.__interrupt,
+                self.__def_model_info,
+                {
+                    **self.__defopts,
+                    "process_wildcards": False,
+                    "if_wildcards": PromptPostProcessor.IFWILDCARDS_CHOICES.ignore.value,
+                },
+            ),
+        )
+
+    def test_wc_remove(self):  # wildcards with remove option
+        self.__process(
+            PromptPair(
+                "[<ppp:stn>neg5<ppp:/stn>] this is: __bad_wildcard__ a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5] <lora:xxx:1>",
+                "normal quality, <ppp:stn i0> {option1|option2}",
+            ),
+            PromptPair(
+                "this is: a (([complex|simple|regular] test)(test:2.0):1.5)\nBREAK with [abc:def:5]<lora:xxx:1>",
+                "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+            ),
+            ppp=PromptPostProcessor(
+                self.__ppp_logger,
+                self.__interrupt,
+                self.__def_model_info,
+                {
+                    **self.__defopts,
+                    "process_wildcards": False,
+                    "if_wildcards": PromptPostProcessor.IFWILDCARDS_CHOICES.remove.value,
+                },
+            ),
+        )
+
+    def test_wc_warn(self):  # wildcards with warn option
+        self.__process(
+            PromptPair("__bad_wildcard__", "{option1|option2}"),
+            PromptPair(PromptPostProcessor.WILDCARD_WARNING + "__bad_wildcard__", "{option1|option2}"),
+            ppp=PromptPostProcessor(
+                self.__ppp_logger,
+                self.__interrupt,
+                self.__def_model_info,
+                {
+                    **self.__defopts,
+                    "process_wildcards": False,
+                    "if_wildcards": PromptPostProcessor.IFWILDCARDS_CHOICES.warn.value,
+                },
+            ),
+        )
+
+    def test_wc_stop(self):  # wildcards with stop option
+        self.__process(
+            PromptPair("__bad_wildcard__", "{option1|option2}"),
+            PromptPair(
+                PromptPostProcessor.WILDCARD_STOP + "__bad_wildcard__",
+                PromptPostProcessor.WILDCARD_STOP + "{option1|option2}",
+            ),
+            ppp=PromptPostProcessor(
+                self.__ppp_logger,
+                self.__interrupt,
+                self.__def_model_info,
+                {
+                    **self.__defopts,
+                    "process_wildcards": False,
+                    "if_wildcards": PromptPostProcessor.IFWILDCARDS_CHOICES.stop.value,
+                },
+            ),
+            interrupted=True,
+        )
+
+    def test_wc_wildcard1a_text(self):  # simple text wildcard
+        self.__process(
+            PromptPair("the choices are: __text/wildcard1__", ""),
+            PromptPair("the choices are: choice2", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard1a_json(self):  # simple json wildcard
+        self.__process(
+            PromptPair("the choices are: __json/wildcard1__", ""),
+            PromptPair("the choices are: choice2", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard1a_yaml(self):  # simple yaml wildcard
+        self.__process(
+            PromptPair("the choices are: __yaml/wildcard1__", ""),
+            PromptPair("the choices are: choice2", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard1b_text(self):  # simple text wildcard with multiple choices
+        self.__process(
+            PromptPair("the choices are: __2-$$text/wildcard1__", ""),
+            PromptPair("the choices are: choice3, choice1", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard1b_json(self):  # simple json wildcard with multiple choices
+        self.__process(
+            PromptPair("the choices are: __2-$$json/wildcard1__", ""),
+            PromptPair("the choices are: choice3, choice1", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard1b_yaml(self):  # simple yaml wildcard with multiple choices
+        self.__process(
+            PromptPair("the choices are: __2-$$yaml/wildcard1__", ""),
+            PromptPair("the choices are: choice3, choice1", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard2_text(self):  # simple text wildcard with default options
+        self.__process(
+            PromptPair("the choices are: __text/wildcard2__", ""),
+            PromptPair("the choices are: choice3-choice1", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard2_json(self):  # simple json wildcard with default options
+        self.__process(
+            PromptPair("the choices are: __json/wildcard2__", ""),
+            PromptPair("the choices are: choice3-choice1", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard2_yaml(self):  # simple yaml wildcard with default options
+        self.__process(
+            PromptPair("the choices are: __yaml/wildcard2__", ""),
+            PromptPair("the choices are: choice3-choice1", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_nested_wildcard_text(self):  # nested text wildcard with repeating multiple choices
+        self.__process(
+            PromptPair("the choices are: __r3$$-$$text/wildcard3__", ""),
+            PromptPair("the choices are: choice3,choice1- choice2 ,choice3", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_nested_wildcard_json(self):  # nested json wildcard with repeating multiple choices
+        self.__process(
+            PromptPair("the choices are: __r3$$-$$json/wildcard3__", ""),
+            PromptPair("the choices are: choice3,choice1- choice2 ,choice3", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_nested_wildcard_yaml(self):  # nested yaml wildcard with repeating multiple choices
+        self.__process(
+            PromptPair("the choices are: __r3$$-$$yaml/wildcard3__", ""),
+            PromptPair("the choices are: choice3,choice1- choice2 ,choice3", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard4_yaml(self):  # simple yaml wildcard with one option
+        self.__process(
+            PromptPair("the choices are: __yaml/wildcard4__", ""),
+            PromptPair("the choices are: inline text", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcard6_yaml(self):  # simple yaml wildcard with object formatted choices
+        self.__process(
+            PromptPair("the choices are: __yaml/wildcard6__", ""),
+            PromptPair("the choices are: choice2", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_choice_wildcard_mix(self):  # choices with wildcard mix
+        self.__process(
+            PromptPair("the choices are: {__~2$$yaml/wildcard2__|choice0}", ""),
+            [
+                PromptPair("the choices are: choice0", ""),
+                PromptPair("the choices are: choice1, choice3", ""),
+                PromptPair("the choices are: choice1, choice3", ""),
+            ],
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_unsupportedsampler(self):  # unsupported sampler
+        self.__process(
+            PromptPair("the choices are: __@yaml/wildcard2__", ""),
+            PromptPair("", ""),
+            ppp=self.__nocupppp,
+            interrupted=True,
+        )
+
+    def test_wc_wildcard_globbing(self):  # wildcard with globbing
+        self.__process(
+            PromptPair("the choices are: __yaml/wildcard[12]__, __yaml/wildcard*__", ""),
+            PromptPair("the choices are: choice3-choice2, choice3-choice1- choice2 ", ""),
+            ppp=self.__nocupppp,
+        )
+
+    def test_wc_wildcardwithvar(self):  # wildcard with inline variable
+        self.__process(
+            PromptPair("the choices are: __yaml/wildcard5(var=test)__, __yaml/wildcard5__", ""),
+            PromptPair("the choices are: inline test, inline default", ""),
+            ppp=self.__nocupppp,
+        )
+
+    # def test_mix(self):
+    #     self.__process(
+    #         PromptPair(
+    #             "__text/wildcard1__ (__text/wildcard2__) (__text/wildcard3__:1.5) [__text/wildcard1__] [__text/wildcard2__:__text/wildcard3__:0.5] [__text/wildcard1__|__text/wildcard2__] # <lora:__text/wildcard3__:1> {opt1_1|opt1_2} ({opt2_1|opt2_2}) ({opt3_1|opt3_2}:1.5) [{opt4_1|opt4_2}] [{opt5_1|opt5_2}:{opt6_1|opt6_2}:0.5] [{opt7_1|opt7_2}|{opt8_1|opt8_2}] # <lora:{opt9_1|opt9_2}:1> {opt1_1|__text/wildcard1__} ({opt2_1|__text/wildcard2__}) ({opt3_1|__text/wildcard3__}:1.5) [{opt4_1|__text/wildcard1__}] [{opt5_1|__text/wildcard2__}# :{opt6_1|__text/wildcard3__}:0.5] [{opt7_1|__text/wildcard1__}|{opt8_1|__text/wildcard2__}] {<lora:opt9_1:1>|<lora:__text/wildcard3__:1>}",
+    #             "",
+    #         ),
+    #         PromptPair(
+    #             "choice2 ( choice2 -choice1) (choice1, choice2 :1.5) [choice1] [choice1-choice1:choice3, choice2 :0.5] [choice2| choice2 - choice2 ] <lora: choice2 ,choice1:1> opt1_1 # (opt2_2) (opt3_1:1.5) [opt4_2] [opt5_1:opt6_2:0.5] [opt7_1|opt8_1] <lora:opt9_2:1> choice3 (opt2_1) (choice1,choice3:1.5) [choice1] [choice3-choice3:opt6_1:0.5] [opt7_1|# opt8_1] <lora:choice1, choice2 :1>",
+    #             "",
+    #         ),
+    #         ppp=self.__nocupppp,
+    #     )
+
+    # def test_real(self):
+    #     self.__wildcards_obj.refresh_wildcards(
+    #         DEBUG_LEVEL.full,
+    #         ["D:\\AI\\SD\\_configuraciones\\acb-wildcards\\wildcards"],
+    #     )
+    #     self.__process(
+    #         PromptPair(
+    #             "${separator=()}, __quality/high__ __misc/sep__, photograph of a __character__",
+    #             "__negatives/ng_generic__",
+    #         ),
+    #         PromptPair("", ""),
+    #     )
+
 
 if __name__ == "__main__":
     unittest.main()

tests/wildcards/test.json

@@ -0,0 +1,19 @@
+{
+  "json": {
+    "wildcard1": [
+      "choice1",
+      "choice2",
+      "choice3"
+    ],
+    "wildcard2": [
+      "r2-3$$-",
+      "4::choice1",
+      "3:: choice2 ",
+      "2::choice3",
+      "5 if _is_sd1::choice4"
+    ],
+    "wildcard3": [
+      "__2$$,$$json/wildcard2__"
+    ]
+  }
+}

tests/wildcards/test.yaml

@@ -0,0 +1,25 @@
+yaml:
+  wildcard1:
+    - choice1
+    - choice2
+    - choice3
+
+  wildcard2:
+    - ~r2-3$$-
+    - 4::choice1
+    - "3:: choice2 "
+    - 2::choice3
+    - 5 if _is_sd1::choice4
+
+  wildcard3:
+    - __2$$,$$yaml/wildcard2__
+
+  wildcard4: inline text
+
+  wildcard5: inline ${var:default}
+
+  wildcard6:
+    - { weight: 2, text: choice1 }
+    - { weight: 3, content: choice2 }
+    - { text: choice3 }
+    - { weight: 4, if: "_is_ssd", text: choice4 }

tests/wildcards2/text/wildcard1.txt

@@ -0,0 +1,4 @@
+# wildcard1
+choice1
+choice2
+choice3

tests/wildcards2/text/wildcard2.txt

@@ -0,0 +1,6 @@
+# wildcard2
+r2-3$$-
+4::choice1
+3:: choice2 
+2::choice3
+5 if _is_sd1::choice4

tests/wildcards2/text/wildcard3.txt

@@ -0,0 +1,2 @@
+# wildcard3
+__2$$,$$text/wildcard2__