* New cleanup options: Remove additional extra separators, Use EOL instead of Space before BREAKs, Use EOL instead of Space before ANDs

* New format for commands
* New commands: "set", "echo" and "if/elif/else"
* Option to Remove extra network tags.

.markdownlint.jsonc

@@ -0,0 +1,5 @@
+{
+    "default": true,
+    "MD013": false,
+    "MD024": false
+}

README.md

@@ -1,111 +1,232 @@
-# Send to Negative for Stable Diffusion WebUI
+# Prompt Postprocessor for Stable Diffusion WebUI
 
-Extension for the [AUTOMATIC1111 Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) or compatible UIs.
+The Prompt Postprocessor for Stable Diffusion WebUI, formerly known as "sd-webui-sendtonegative", is an extension designed to process the prompt after other extensions have potentially modified it. This extension is compatible with the [AUTOMATIC1111 Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) and [SD.Next](https://github.com/vladmandic/automatic).
 
-## Purpose
+Currently this extension has these functions:
 
-This extension allows the tagging of parts of the prompt and moves them to the
+* Allows marking parts of the prompt and moves them to the negative prompt. This allows for useful tricks when using a wildcard extension since you can add negative content from choices made in the positive prompt.
-negative prompt. This allows useful tricks when using a wildcard extension
+* Set values to local variables.
-since you can add negative content from choices made in the positive prompt.
+* Filter content based on the loaded SD model version or a set variable.
+* Detect invalid wildcards and act on them.
+* Clean up the prompt and negative prompt.
 
-Note: The extension must be loaded after the installed wildcards extension. Extensions
+Note: The extension must be loaded after the installed wildcards extension (or any other that modifies the prompt or has it's own syntax expressions). Extensions load by their folder name in alphanumeric order.
-load by their folder in alphanumeric order.
 
-With the ["Dynamic Prompts" extension](https://github.com/adieyal/sd-dynamic-prompts)
+With the ["Dynamic Prompts" extension](https://github.com/adieyal/sd-dynamic-prompts) this happens by default due to default folder names for both extensions. But if this is not the case, you can just rename this extension's folder so the ordering works out.
-this happens by default due to default folder names for both extensions. But if
-this is not the case, you can just rename the extension folder so the ordering
-works out.
 
-With the ["AUTOMATIC1111 Wildcards" extension](https://github.com/AUTOMATIC1111/stable-diffusion-webui-wildcards)
+With the ["AUTOMATIC1111 Wildcards" extension](https://github.com/AUTOMATIC1111/stable-diffusion-webui-wildcards) you will have to rename one of the folders, so that it loads before than this extension.
-you will have to rename one of the folders, so that it loads before than "Send to Negative".
 
 When in doubt, just rename this extension's folder with a "z" in front (for example) so that it is the last one to load, or manually set such folder name when installing it.
 
-Note: it only recognizes regular A1111 prompt formats. So:
+Notes:
 
-* Attention: \[prompt\] (prompt) (prompt:weight)
+1. It only recognizes regular A1111 prompt formats. So:
-* Alternation: \[prompt1|prompt2|...\]
-* Scheduling: \[prompt1:prompt2:step\]
 
-In SD.Next that means only the A1111 or Full parsers.
+    * **Attention**: `\[prompt\] (prompt) (prompt:weight)`
+    * **Alternation**: `\[prompt1|prompt2|...\]`
+    * **Scheduling**: `\[prompt1:prompt2:step\]`
+    * **Extra networks**: `\<kind:model...\>`
+    * **BREAK**: `prompt1 BREAK prompt2`
+    * **Composable Diffusion**: `prompt1 AND prompt2`
 
-It does not build AND/BREAK separations into the negative prompt.
+    In SD.Next that means only the *A1111* or *Full* parsers. It will warn you if you use the *Compel* parser.
+2. It only recognizes wildcards in the *\_\_wildcard\_\_* and *{choice|choice}* formats.
+3. Since it should run after other extensions that apply to the prompt, the content should have already been processed by them and there should't be any non recognized syntax anymore.
+4. It does not create *AND/BREAK* constructs when moving content to the negative prompt.
 
 ## Installation
 
 1. Go to Extensions > Install from URL
-2. Paste <https://github.com/acorderob/sd-webui-sendtonegative> in the URL for extension's git repository text field
+2. Paste <https://github.com/acorderob/sd-webui-prompt-postprocessor> in the URL for extension's git repository text field
 3. Click the Install button
 4. Restart the webui
 
 ## Usage
 
-The format of the tags is like this:
+### Detection of remaining wildcards
+
+This extension should run after any wildcard extensions, so any remaining wildcards present in the prompt or negative_prompt at this point of processing must be invalid. Usually you might not notice this problem until you check the image metadata, so this option gives you some ways to detect and treat the problem.
+
+If you choose to not ignore wildcards, the extension will look for any *\_\_wildcard\_\_* or *{choice|choice}* constructs and act as configured.
+
+### Commands
+
+The extension uses now a new format for its commands. The format is similar to an extranetwork, but it has a "ppp:" prefix followed by the command, and then a space and any parameters (if any).
+
+```text
+<ppp:command parameters>
+```
+
+When a command is associated with any content, it will be between an opening and a closing command:
+
+```text
+<ppp:command parameters>content<ppp:/command>
+```
+
+The `set` and `if` commands are the first to be processed.
+
+### Set command
+
+This command sets the value of a variable that can be checked later.
+
+The format is:
+
+```text
+<ppp:set varname>value<ppp:/set>
+```
+
+### Echo command
+
+This command prints the value of a variable.
+
+The format is:
+
+```text
+<ppp:echo varname>
+```
+
+### If command
+
+This command allows you to filter content based on conditions.
+
+The format is:
+
+```text
+<ppp:if condition1>content one<ppp:elif condition2>content two<ppp:else>other content<ppp:/if>
+```
+
+The *conditionN* compares a variable with a value. The operation can be `eq`, `ne`, `gt`, `lt`, `ge`, `le` and the value can be a quoted string or an integer.
+
+The variable can be one set with the `set` command or special variables like:
+
+* `_sd` : the loaded model version (`"sd1"`, `"sd2"`, `"sdxl"`)
+
+Any `elif`s (there can be multiple) and the `else` are optional.
+
+#### Example
+
+(multiline to be easier to read)
+
+```text
+<ppp:if _sd eq "sd1"><lora:test_sd1> test sd1
+<ppp:elif _sd eq "sd2"><lora:test_sd2> test sd2
+<ppp:elif _sd eq "sdxl"><lora:test_sdxl> test sdxl
+<ppp:else>unknown model
+<ppp:/if>
+```
+
+Only one of the options will end up in the prompt, depending on the loaded model.
+
+### Sending content to the negative prompt
+
+The new format for this command is like this:
+
+```text
+<ppp:stn position>content<ppp:/stn>
+```
+
+Where position is optional (defaults to the start) and can be:
+
+* **s**: at the start of the negative prompt
+* **e**: at the end of the negative prompt
+* **pN**: at the position of the insertion point in the negative prompt with N being 0-9
+
+The format of the insertion point to be used in the negative prompt is:
+
+```text
+<ppp:stn iN>
+```
+
+If the insertion point is not found it inserts at the start.
+
+#### Example
+
+You have a wildcard for hair colors (\_\_haircolors\_\_) with one being strawberry blonde, but you don't want strawberries. So in that option you add a command to add to the negative prompt, like so:
+
+```text
+blonde
+strawberry blonde <ppp:stn>strawberry<ppp:/stn>
+brunette
+```
+
+Then, if that option is chosen this extension will process it later and move that part to the negative prompt.
+
+#### Old format
+
+The old format is still supported (for now) and is like this:
 
 ```text
 <!content!>
 ```
 
-And an optional position in the negative prompt can be specified like this:
+And an optional position can be specified like this:
 
 ```text
 <!!position!content!>
 ```
 
-Where position can be:
+With the insertion point like this:
-
-* s: at the start (the default)
-* e: at the end
-* pN: at the position of the insertion point "<!!iN!!>" with N being 0-9
-
-If the insertion point is not found it inserts at the start.
-
-## Example
-
-You have a wildcard for hair colors (\_\_haircolors\_\_) with one being
-strawberry blonde, but you don't want strawberries. So in that option you add a
-tag to add to the negative prompt, like so:
 
 ```text
-blonde
+<!!iN!!>
-strawberry blonde <!strawberry!>
-brunette
 ```
 
-Then, if that option is chosen this extension will process it later and move
+### Notes on negative commands
-that part to the negative prompt.
 
-## Configuration
+Positional insertion commands have less priority that start/end commands, so even if they are at the start or end of the negative prompt, they will end up inside any start/end (and default position) commands.
 
-Separator used when adding to the negative prompt: You can specify the separator used when adding to the negative prompt (by default it's ", ").
+The content of the negative commands is not processed and is copied as-is to the negative prompt. Other modifiers around the commands are processed in the following way.
 
-Ignore tags with repeated content: by default it ignores repeated content to avoid repetitions in the negative prompt.
+#### Attention modifiers (weights)
-
-Join attention modifiers (weights) when possible: by default it joins attention modifiers when possible (joins into one, multipliying their values).
-
-Try to clean-up the prompt after processing: by default cleans up the positive prompt after processing, removing extra spaces and separators.
-
-## Notes
-
-The content of the negative tags is not processed and is copied as is to the negative prompt. Other modifiers around the tags are processed in the following way.
-
-### Attention modifiers (weights)
 
 They will be translated to the negative prompt. For example:
 
-* `(red<!square!>:1.5)` will end up as `(square:1.5)` in the negative prompt
+* `(red<ppp:stn>square<ppp:/stn>:1.5)` will end up as `(square:1.5)` in the negative prompt
-* `(red[<!square!>]:1.5)` will end up as `(square:1.35)` in the negative prompt (weight=1.5*0.9)
+* `(red[<ppp:stn>square<ppp:/stn>]:1.5)` will end up as `(square:1.35)` in the negative prompt (weight=1.5*0.9)
-* However `(red<![square]!>:1.5)` will end up as `([square]:1.5)` in the negative prompt. The content of the negative tag is copied as is, and not joined with the surrounding modifier.
+* However `(red<ppp:stn>[square]<ppp:/stn>:1.5)` will end up as `([square]:1.5)` in the negative prompt. The content of the negative tag is copied as is, and not joined with the surrounding modifier.
 
-### Prompt editing constructs (alternation and scheduling)
+#### Prompt editing constructs (alternation and scheduling)
 
-Negative tags inside such constructs will copy the construct to the negative prompt, but separating its elements. For example:
+Negative commands inside such constructs will copy the construct to the negative prompt, but separating its elements. For example:
 
-* Alternation: `[red<!square!>|blue<!circle!>]` will end up as `[square|], [|circle]` in the negative prompt, instead of `[square|circle]`
+* **Alternation**: `[red<ppp:stn>square<ppp:/stn>|blue<ppp:stn>circle<ppp:/stn>]` will end up as `[square|], [|circle]` in the negative prompt, instead of `[square|circle]`
-* Scheduling: `[red<!square!>:blue<!circle!>:0.5]` will end up as `[square::0.5], [:circle:0.5]` instead of `[square:circle:0.5]`
+* **Scheduling**: `[red<ppp:stn>square<ppp:/stn>:blue<ppp:stn>circle<ppp:/stn>:0.5]` will end up as `[square::0.5], [:circle:0.5]` instead of `[square:circle:0.5]`
 
 This should still work as intended, and the only negative point i see is the unnecessary separators.
 
+## Configuration
+
+### General settings
+
+* **Debug**: writes debugging information to the console.
+* **What to do with remaining wildcards?**: select what do you want to do with any found wildcards.
+  * **Ignore**: do not try to detect wildcards.
+  * **Remove**: detect wildcards and remove them.
+  * **Add visible warning**: detect wildcards and add a warning text to the prompt, that hopefully produces a noticeable generation.
+  * **Stop the generation**: detect wildcards and stop the generation.
+
+### Content removal settings
+
+* **Remove extra network tags**: removes all extra network tags.
+
+### Send to negative prompt settings
+
+* **Apply in img2img**: check if you want to do this processing in img2img processes.
+* **Separator used when adding to the negative prompt**: you can specify the separator used when adding to the negative prompt (by default it's ", ").
+* **Ignore repeated content**: it ignores repeated content to avoid repetitions in the negative prompt.
+* **Join attention modifiers (weights) when possible**: it joins attention modifiers when possible (joins into one, multipliying their values).
+
+### Clean up settings
+
+* **Apply in img2img**: check if you want to do this processing in img2img processes.
+* **Remove empty constructs**: removes attention/scheduling/alternation constructs when they are invalid.
+* **Remove extra separators**: removes unnecessary separators. This applies to the configured separator and regular commas.
+* **Remove additional extra separators**: removes unnecessary separators at start or end of lines. This applies to the configured separator and regular commas.
+* **Clean up around BREAKs**: removes consecutive BREAKs and unnecessary commas and space around them.
+* **Clean up around ANDs**: removes consecutive ANDs and unnecessary commas and space around them.
+* **Clean up around extra network tags**: removes spaces around them.
+* **Remove extra spaces**: removes other unnecessary spaces.
+
 ## License
 
 MIT

ppp_logging.py

@@ -0,0 +1,87 @@
+import logging
+import sys
+import copy
+
+
+class PromptPostProcessorLogFactory:  # pylint: disable=too-few-public-methods
+    """
+    Factory class for creating loggers for the PromptPostProcessor module.
+    """
+
+    class ColoredFormatter(logging.Formatter):
+        """
+        A custom logging formatter that adds color to log records based on their level.
+
+        Attributes:
+            COLORS (dict): A dictionary mapping log levels to ANSI escape codes for colors.
+
+        Methods:
+            format(record): Formats the log record with color based on its level.
+
+        """
+
+        COLORS = {
+            "DEBUG": "\033[0;36m",  # CYAN
+            "INFO": "\033[0;32m",  # GREEN
+            "WARNING": "\033[0;33m",  # YELLOW
+            "ERROR": "\033[0;31m",  # RED
+            "CRITICAL": "\033[0;37;41m",  # WHITE ON RED
+            "RESET": "\033[0m",  # RESET COLOR
+        }
+
+        def format(self, record):
+            """
+            Formats the log record with color based on the log level.
+
+            Args:
+                record (LogRecord): The log record to be formatted.
+
+            Returns:
+                str: The formatted log record.
+            """
+            colored_record = copy.copy(record)
+            levelname = colored_record.levelname
+            seq = self.COLORS.get(levelname, self.COLORS["RESET"])
+            colored_record.levelname = f"{seq}{levelname}{self.COLORS['RESET']}"
+            return super().format(colored_record)
+
+    def __init__(self):
+        """
+        Initializes the PromptPostProcessor class.
+
+        This method sets up the logger for the PromptPostProcessor class and configures its log level and handlers.
+
+        Args:
+            None
+
+        Returns:
+            None
+        """
+        ppplog = logging.getLogger("PromptPostProcessor")
+        ppplog.propagate = False
+        if not ppplog.handlers:
+            handler = logging.StreamHandler(sys.stdout)
+            handler.setFormatter(self.ColoredFormatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
+            ppplog.addHandler(handler)
+        ppplog.setLevel(logging.INFO)
+        self.log = PromptPostProcessorLogCustomAdapter(ppplog)
+
+
+class PromptPostProcessorLogCustomAdapter(logging.LoggerAdapter):
+    """
+    Custom logger adapter for the PromptPostProcessor.
+    This adapter adds a prefix to log messages to indicate that they are related to the PromptPostProcessor.
+    """
+
+    def process(self, msg, kwargs):
+        """
+        Process the log message and keyword arguments.
+
+        Args:
+            msg (str): The log message.
+            kwargs (dict): The keyword arguments.
+
+        Returns:
+            tuple: A tuple containing the processed log message and keyword arguments.
+        """
+        return f"[PromptPostProcessor] {msg}", kwargs

scripts/extension.py

@@ -1,73 +0,0 @@
-if __name__ == "__main__":
-    raise SystemExit("This script must be run from a Stable Diffusion WebUI")
-
-import sys
-import os
-
-sys.path.insert(1, os.path.join(sys.path[0], ".."))
-
-
-# pylint: disable=import-error
-
-from modules import scripts, shared, script_callbacks
-from modules.processing import StableDiffusionProcessing
-from modules.shared import opts
-from sendtonegative import SendToNegative
-from stnlogging import SendToNegativeLogFactory
-
-
-class SendToNegativeScript(scripts.Script):
-    def __init__(self):
-        if not hasattr(self, "callbacks_added"):
-            lf = SendToNegativeLogFactory()
-            self.__logstn = lf.log
-            script_callbacks.on_ui_settings(self.__on_ui_settings)
-            self.callbacks_added = True
-
-    def title(self):
-        return f"{SendToNegative.NAME} v{SendToNegative.VERSION}"
-
-    def show(self, is_img2img):
-        return scripts.AlwaysVisible
-
-    def process(self, p: StableDiffusionProcessing, *args, **kwargs):
-        stn = SendToNegative(self.__logstn, opts=opts)
-        for i in range(len(p.all_prompts)):  # pylint: disable=consider-using-enumerate
-            p.all_prompts[i], p.all_negative_prompts[i] = stn.process_prompt(
-                p.all_prompts[i], p.all_negative_prompts[i]
-            )
-
-    def __on_ui_settings(self):
-        section = ("send-to-negative", SendToNegative.NAME)
-        shared.opts.add_option(
-            key="stn_separator",
-            info=shared.OptionInfo(
-                SendToNegative.DEFAULT_SEPARATOR,
-                label="Separator used when adding to the negative prompt",
-                section=section,
-            ),
-        )
-        shared.opts.add_option(
-            key="stn_ignorerepeats",
-            info=shared.OptionInfo(
-                True,
-                label="Ignore tags with repeated content",
-                section=section,
-            ),
-        )
-        shared.opts.add_option(
-            key="stn_joinattention",
-            info=shared.OptionInfo(
-                True,
-                label="Join attention modifiers (weights) when possible",
-                section=section,
-            ),
-        )
-        shared.opts.add_option(
-            key="stn_cleanup",
-            info=shared.OptionInfo(
-                True,
-                label="Try to clean-up the prompt after processing (removes extra spaces or the configured separator)",
-                section=section,
-            ),
-        )

scripts/ppp_script.py

@@ -0,0 +1,284 @@
+if __name__ == "__main__":
+    raise SystemExit("This script must be run from a Stable Diffusion WebUI")
+
+import sys
+import os
+
+sys.path.insert(1, os.path.join(sys.path[0], ".."))
+
+
+from modules import scripts, shared, script_callbacks
+from modules.processing import StableDiffusionProcessing
+from modules.shared import opts
+import gradio as gr
+from ppp import PromptPostProcessor
+from ppp_logging import PromptPostProcessorLogFactory
+
+
+class PromptPostProcessorScript(scripts.Script):
+    """
+    This class represents a script for prompt post-processing.
+    It is responsible for processing prompts and applying various settings and cleanup operations.
+
+    Attributes:
+        callbacks_added (bool): Flag indicating whether the script callbacks have been added.
+
+    Methods:
+        __init__(): Initializes the PromptPostProcessorScript object.
+        title(): Returns the title of the script.
+        show(is_img2img): Determines whether the script should be shown based on the input type.
+        process(p, *args, **kwargs): Processes the prompts and applies post-processing operations.
+        __on_ui_settings(): Callback function for UI settings.
+    """
+
+    def __init__(self):
+        """
+        Initializes the PromptPostProcessor object.
+
+        This method adds callbacks for UI settings and initializes the logger.
+
+        Parameters:
+            None
+
+        Returns:
+            None
+        """
+        if not hasattr(self, "ppp_callbacks_added"):
+            lf = PromptPostProcessorLogFactory()
+            self.ppp_logger = lf.log
+            self.ppp_debug = getattr(opts, "ppp_gen_debug", False) if opts is not None else False
+            script_callbacks.on_ui_settings(self.__on_ui_settings)
+            self.ppp_callbacks_added = True
+
+    def title(self):
+        """
+        Returns the title of the script.
+
+        Returns:
+            str: The title of the script.
+        """
+        return PromptPostProcessor.NAME
+
+    def show(self, is_img2img):
+        """
+        Determines whether the script should be shown based on the kind of processing.
+
+        Args:
+            is_img2img (bool): Flag indicating whether the processing is image-to-image.
+
+        Returns:
+            scripts.Visibility: The visibility setting for the script.
+        """
+        return scripts.AlwaysVisible
+
+    def process(self, p: StableDiffusionProcessing, *args, **kwargs):  # pylint: disable=unused-argument
+        """
+        Processes the prompts and applies post-processing operations.
+
+        Args:
+            p (StableDiffusionProcessing): The StableDiffusionProcessing object containing the prompts.
+
+        Returns:
+            None
+        """
+        is_i2i = getattr(p, "init_images", [None])[0] is not None
+        self.ppp_debug = getattr(opts, "ppp_gen_debug", False) if opts is not None else False
+        if self.ppp_debug:
+            self.ppp_logger.info(f"Post-processing prompts ({'i2i' if is_i2i else 't2i'} mode)")
+        ppp = PromptPostProcessor(self, p, shared.state, opts, is_i2i)
+        # processes regular prompts
+        if (
+            hasattr(p, "all_prompts")
+            and p.all_prompts is not None
+            and hasattr(p, "all_negative_prompts")
+            and p.all_negative_prompts is not None
+        ):
+            for i, (prompt, negative_prompt) in enumerate(zip(p.all_prompts, p.all_negative_prompts)):
+                p.all_prompts[i], p.all_negative_prompts[i] = ppp.process_prompt(prompt, negative_prompt)
+        # make it compatible with A1111 hires fix
+        if (
+            hasattr(p, "all_hr_prompts")
+            and p.all_hr_prompts is not None
+            and hasattr(p, "all_hr_negative_prompts")
+            and p.all_hr_negative_prompts is not None
+        ):
+            for i, (hr_prompt, hr_negative_prompt) in enumerate(zip(p.all_hr_prompts, p.all_hr_negative_prompts)):
+                p.all_hr_prompts[i], p.all_hr_negative_prompts[i] = ppp.process_prompt(hr_prompt, hr_negative_prompt)
+
+    def ppp_interrupt(self):
+        """
+        Interrupts the generation.
+
+        Returns:
+            None
+        """
+        shared.state.interrupted = True
+
+    def __on_ui_settings(self):
+        """
+        Callback function for UI settings.
+
+        Returns:
+            None
+        """
+        # general settings
+        section = ("prompt-post-processor", PromptPostProcessor.NAME)
+        shared.opts.add_option(
+            key="ppp_gen_sep", info=shared.OptionInfo("<h2>General settings</h2>", "", gr.HTML, section=section)
+        )
+        shared.opts.add_option(
+            key="ppp_gen_debug",
+            info=shared.OptionInfo(
+                False,
+                label="Debug",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_gen_ifwildcards",
+            info=shared.OptionInfo(
+                default=PromptPostProcessor.IFWILDCARDS_CHOICES["ignore"],
+                label="What to do with remaining wildcards?",
+                component=gr.Radio,
+                component_args={"choices": PromptPostProcessor.IFWILDCARDS_CHOICES.values()},
+                section=section,
+            ),
+        )
+
+        # content removal settings
+        shared.opts.add_option(
+            key="ppp_rem_sep", info=shared.OptionInfo("<br/><h2>Content removal settings</h2>", "", gr.HTML, section=section)
+        )
+        shared.opts.add_option(
+            key="ppp_rem_removeextranetworktags",
+            info=shared.OptionInfo(
+                False,
+                label="Remove extra network tags",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_rem_if", info=shared.OptionInfo("<p style=\"font-style:italic\">* Parsing of the 'if' commands cannot be disabled</p>", "", gr.HTML, section=section)
+        )
+
+        # send to negative settings
+        shared.opts.add_option(
+            key="ppp_stn_sep",
+            info=shared.OptionInfo("<br/><h2>Send to Negative settings</h2>", "", gr.HTML, section=section),
+        )
+        shared.opts.add_option(
+            key="ppp_stn_doi2i",
+            info=shared.OptionInfo(
+                False,
+                label="Apply in img2img (this includes any pass that contains an initial image, like refiner, hires fix, adetailer)",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_stn_separator",
+            info=shared.OptionInfo(
+                PromptPostProcessor.DEFAULT_STN_SEPARATOR,
+                label="Separator used when adding to the negative prompt",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_stn_ignorerepeats",
+            info=shared.OptionInfo(
+                True,
+                label="Ignore repeated content",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_stn_joinattention",
+            info=shared.OptionInfo(
+                True,
+                label="Join attention modifiers (weights) when possible",
+                section=section,
+            ),
+        )
+        # clean-up settings
+        shared.opts.add_option(
+            key="ppp_cup_sep", info=shared.OptionInfo("<br/><h2>Clean-up settings</h2>", "", gr.HTML, section=section)
+        )
+        shared.opts.add_option(
+            key="ppp_cup_doi2i",
+            info=shared.OptionInfo(
+                False,
+                label="Apply in img2img (this includes any pass that contains an initial image, like refiner, hires fix, adetailer)",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_emptyconstructs",
+            info=shared.OptionInfo(
+                True,
+                label="Remove empty constructs (attention, alternation, scheduling)",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_extraseparators",
+            info=shared.OptionInfo(
+                True,
+                label="Remove extra separators",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_extraseparators2",
+            info=shared.OptionInfo(
+                True,
+                label="Remove additional extra separators",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_breaks",
+            info=shared.OptionInfo(
+                True,
+                label="Clean up around BREAKs",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_breaks_eol",
+            info=shared.OptionInfo(
+                False,
+                label="Use EOL instead of Space before BREAKs",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_ands",
+            info=shared.OptionInfo(
+                True,
+                label="Clean up around ANDs",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_ands_eol",
+            info=shared.OptionInfo(
+                False,
+                label="Use EOL instead of Space before ANDs",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_extranetworktags",
+            info=shared.OptionInfo(
+                False,
+                label="Clean up around extra network tags",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_extraspaces",
+            info=shared.OptionInfo(
+                True,
+                label="Remove extra spaces",
+                section=section,
+            ),
+        )

sendtonegative.py

@@ -1,323 +0,0 @@
-from collections import namedtuple
-import re
-import lark
-
-
-class SendToNegative:  # pylint: disable=too-few-public-methods
-    NAME = "Send to Negative"
-    VERSION = "2.1.1"
-
-    DEFAULT_SEPARATOR = ", "
-
-    def __init__(
-        self,
-        log,
-        separator=None,
-        ignore_repeats=None,
-        join_attention=None,
-        cleanup=None,
-        opts=None,
-    ):
-        """
-        Format for the tag:
-            <!content!>
-
-            <!!x!content!>
-
-        with x being:
-            s - content is added at the start of the negative prompt. This is the default if no parameter exists.
-
-            e - content is added at the end of the negative prompt.
-
-            pN - content is added where the insertion point N is in the negative prompt or at the start if it does not exist. N can be 0 to 9.
-
-            iN - tags the position of insertion point N. Used only in the negative prompt and does not accept content. N can be 0 to 9.
-        """
-        self.__logger = log
-        if opts is not None and getattr(opts, "prompt_attention", "") == "Compel parser":
-            self.__logger.warning("Compel parser is not supported!")
-        self.__ignore_repeats = (
-            ignore_repeats if ignore_repeats is not None else getattr(opts, "stn_ignorerepeats", True)
-        )
-        self.__join_attention = (
-            join_attention
-            if join_attention is not None
-            else getattr(opts, "stn_joinattention", True)
-            if opts is not None
-            else True
-        )
-        self.__cleanup = (
-            cleanup if cleanup is not None else getattr(opts, "stn_cleanup", True) if opts is not None else True
-        )
-        self.__separator = (
-            separator
-            if separator is not None
-            else getattr(opts, "stn_separator", self.DEFAULT_SEPARATOR)
-            if opts is not None
-            else self.DEFAULT_SEPARATOR
-        )
-        self.__insertion_point_tags = [f"<!!i{x}!!>" for x in range(10)]
-        # Process with lark (debug with https://www.lark-parser.org/ide/)
-        self.__schedule_parser = lark.Lark(
-            r"""
-                start: (prompt | /[\][():|<>!]/+)*
-                ?prompt: (emphasized | deemphasized | scheduled | alternate | modeltag | negtag | plain)*
-                ?nonegprompt: (emphasized | deemphasized | scheduled | alternate | modeltag | plain)*
-                emphasized: "(" prompt [":" numpar] ")"
-                deemphasized: "[" prompt "]"
-                scheduled: "[" [prompt ":"] prompt ":" numpar "]"
-                alternate: "[" alternateoption ("|" alternateoption)+ "]"
-                alternateoption: prompt
-                negtag: "<!" [negtagparameters] nonegprompt "!>"
-                negtagparameters: "!" /s|e|[ip]\d/ "!"
-                modeltag: "<" /(?!!)[^>]+/ ">"
-                numpar: WHITESPACE* NUMBER WHITESPACE*
-                WHITESPACE: /\s+/
-                ?plain: /([^\\[\]():|<>!]|\\.)+/s
-                %import common.SIGNED_NUMBER -> NUMBER
-            """,
-            propagate_positions=True,
-        )
-
-    class ReadTree(lark.visitors.Interpreter):
-        def __init__(self, logger, ignorerepeats, joinattention, prompt, add_at):
-            super().__init__()
-            self.__logger = logger
-            self.__ignore_repeats = ignorerepeats
-            self.__join_attention = joinattention
-            self.__prompt = prompt
-            self.AccumulatedShell = namedtuple("AccumulatedShell", ["type", "info1", "info2"])
-            AccumulatedShell = self.AccumulatedShell
-            self.__shell: list[AccumulatedShell] = []
-            self.NegTag = namedtuple("NegTag", ["start", "end", "content", "parameters", "shell"])
-            NegTag = self.NegTag
-            self.__negtags: list[NegTag] = []
-            self.__already_processed = []
-            self.add_at = add_at
-            self.remove = []
-
-        def scheduled(self, tree):
-            if len(tree.children) > 2:  # before & after
-                before = tree.children[0]
-            else:
-                before = None
-            after = tree.children[-2]
-            numpar = tree.children[-1]
-            pos = float(numpar.children[0].value)
-            if pos >= 1:
-                pos = int(pos)
-            # self.__shell.append(self.AccumulatedShell("sc", tree.meta.start_pos, pos))
-            if before is not None and hasattr(before, "data"):
-                self.__logger.debug(
-                    f"Shell scheduled before at {[before.meta.start_pos,before.meta.end_pos] if hasattr(before,'meta') else '?'} : {pos}"
-                )
-                self.__shell.append(self.AccumulatedShell("scb", pos, None))
-                self.visit(before)
-                self.__shell.pop()
-            if hasattr(after, "data"):
-                self.__logger.debug(
-                    f"Shell scheduled after at {[after.meta.start_pos,after.meta.end_pos] if hasattr(after,'meta') else '?'} : {pos}"
-                )
-                self.__shell.append(self.AccumulatedShell("sca", pos, None))
-                self.visit(after)
-                self.__shell.pop()
-            # self.__shell.pop()
-
-        def alternate(self, tree):
-            # self.__shell.append(self.AccumulatedShell("al", tree.meta.start_pos, len(tree.children)))
-            for i, opt in enumerate(tree.children):
-                self.__logger.debug(
-                    f"Shell alternate at {[opt.meta.start_pos,opt.meta.end_pos] if hasattr(opt,'meta') else '?'} : {i+1}"
-                )
-                if hasattr(opt, "data"):
-                    self.__shell.append(self.AccumulatedShell("alo", i + 1, len(tree.children)))
-                    self.visit(opt)
-                    self.__shell.pop()
-            # self.__shell.pop()
-
-        def emphasized(self, tree):
-            numpar = tree.children[-1]
-            weight = float(numpar.children[0].value) if numpar is not None else 1.1
-            self.__logger.debug(
-                f"Shell attention at {[tree.meta.start_pos,tree.meta.end_pos] if hasattr(tree,'meta') else '?'}: {weight}"
-            )
-            self.__shell.append(self.AccumulatedShell("at", weight, None))
-            self.visit_children(tree)
-            self.__shell.pop()
-
-        def deemphasized(self, tree):
-            weight = 0.9
-            self.__logger.debug(
-                f"Shell attention at {[tree.meta.start_pos,tree.meta.end_pos] if hasattr(tree,'meta') else '?'}: {weight}"
-            )
-            self.__shell.append(self.AccumulatedShell("at", weight, None))
-            self.visit_children(tree)
-            self.__shell.pop()
-
-        def negtag(self, tree):
-            negtagparameters = tree.children[0]
-            parameters = negtagparameters.children[0].value if negtagparameters is not None else ""
-            rest = []
-            for x in tree.children[1::]:
-                rest.append(self.__prompt[x.meta.start_pos : x.meta.end_pos] if hasattr(x, "meta") else x.value)
-            content = "".join(rest)
-            self.__negtags.append(
-                self.NegTag(tree.meta.start_pos, tree.meta.end_pos, content, parameters, self.__shell.copy())
-            )
-            self.__logger.debug(
-                f"Negative tag at {[tree.meta.start_pos,tree.meta.end_pos] if hasattr(tree,'meta') else '?'}: {parameters}: {content.encode('unicode_escape').decode('utf-8')}"
-            )
-
-        def start(self, tree):
-            self.visit_children(tree)
-            # process the found negtags
-            for nt in self.__negtags:
-                if self.__join_attention:
-                    # join consecutive attention elements
-                    for i in range(len(nt.shell) - 1, 0, -1):
-                        if nt.shell[i].type == "at" and nt.shell[i - 1].type == "at":
-                            nt.shell[i - 1] = self.AccumulatedShell(
-                                "at",
-                                (100 * nt.shell[i - 1].info1 * nt.shell[i].info1) / 100,  # we limit to two decimals
-                                None,
-                            )
-                            nt.shell.pop(i)
-                start = ""
-                end = ""
-                for s in nt.shell:
-                    match s.type:
-                        case "at":
-                            if s.info1 == 0.9:
-                                start += "["
-                                end = "]" + end
-                            elif s.info1 == 1.1:
-                                start += "("
-                                end = ")" + end
-                            else:
-                                start += "("
-                                end = f":{s.info1})" + end
-                        # case "sc":
-                        case "scb":
-                            start += "["
-                            end = f"::{s.info1}]" + end
-                        case "sca":
-                            start += "["
-                            end = f":{s.info1}]" + end
-                        # case "al":
-                        case "alo":
-                            start += "[" + ("|" * int(s.info1 - 1))
-                            end = ("|" * int(s.info2 - s.info1)) + "]" + end
-                content = start + nt.content + end
-                position = nt.parameters or "s"
-                if len(content) > 0:
-                    if content not in self.__already_processed:
-                        if self.__ignore_repeats:
-                            self.__already_processed.append(content)
-                        self.__logger.debug(
-                            f"Adding content at position {position}: {content.encode('unicode_escape').decode('utf-8')}"
-                        )
-                        if position == "e":
-                            self.add_at["end"].append(content)
-                        elif position.startswith("p"):
-                            n = int(position[1])
-                            self.add_at["insertion_point"][n].append(content)
-                        else:  # position == "s" or invalid
-                            self.add_at["start"].append(content)
-                    else:
-                        self.__logger.warning(
-                            f"Ignoring repeated content: {content.encode('unicode_escape').decode('utf-8')}"
-                        )
-                # remove from prompt
-                self.remove.append([nt.start, nt.end])
-
-    def process_prompt(self, original_prompt, original_negative_prompt):
-        """
-        Extract from the prompt the tagged parts and add them to the negative prompt
-        """
-        try:
-            prompt = original_prompt
-            negative_prompt = original_negative_prompt
-            self.__logger.debug(f"Input prompt: {prompt.encode('unicode_escape').decode('utf-8')}")
-            self.__logger.debug(f"Input negative_prompt: {negative_prompt.encode('unicode_escape').decode('utf-8')}")
-            prompt, add_at = self.__find_tags(prompt)
-            negative_prompt = self.__add_to_insertion_points(negative_prompt, add_at["insertion_point"])
-            if len(add_at["start"]) > 0:
-                negative_prompt = self.__add_to_start(negative_prompt, add_at["start"])
-            if len(add_at["end"]) > 0:
-                negative_prompt = self.__add_to_end(negative_prompt, add_at["end"])
-            self.__logger.debug(f"Output prompt: {prompt.encode('unicode_escape').decode('utf-8')}")
-            self.__logger.debug(f"Output negative_prompt: {negative_prompt.encode('unicode_escape').decode('utf-8')}")
-            return prompt, negative_prompt
-        except Exception as e:  # pylint: disable=broad-exception-caught
-            self.__logger.exception(e)
-            return original_prompt, original_negative_prompt
-
-    def __find_tags(self, prompt):
-        add_at = {"start": [], "insertion_point": [[] for x in range(10)], "end": []}
-        tree = self.__schedule_parser.parse(prompt)
-        self.__logger.debug(f"Initial tree:\n{tree.pretty()}")
-
-        readtree = self.ReadTree(self.__logger, self.__ignore_repeats, self.__join_attention, prompt, add_at)
-        readtree.visit(tree)
-
-        for r in readtree.remove[::-1]:
-            prompt = prompt[: r[0]] + prompt[r[1] :]
-        if self.__cleanup:
-            prompt = re.sub(r"\((?::[+-]?[\d\.]+)?\)", "", prompt)  # clean up empty attention
-            prompt = re.sub(r"\[\]", "", prompt)  # clean up empty attention
-            prompt = re.sub(r"\[:?:[+-]?[\d\.]+\]", "", prompt)  # clean up empty scheduling
-            prompt = re.sub(r"\[\|+\]", "", prompt)  # clean up empty alternation
-            # clean up whitespace and extra separators
-            prompt = (
-                prompt.replace("  ", " ")
-                .replace(self.__separator + self.__separator, self.__separator)
-                .replace(" " + self.__separator, self.__separator)
-                .removeprefix(self.__separator)
-                .removesuffix(self.__separator)
-                .strip()
-            )
-        add_at = readtree.add_at
-        self.__logger.debug(f"New negative additions: {add_at}")
-
-        return prompt, add_at
-
-    def __add_to_insertion_points(self, negative_prompt, add_at_insertion_point):
-        for n in range(10):
-            ipp = negative_prompt.find(self.__insertion_point_tags[n])
-            if ipp >= 0:
-                ipl = len(self.__insertion_point_tags[n])
-                if negative_prompt[ipp - len(self.__separator) : ipp] == self.__separator:
-                    ipp -= len(self.__separator)  # adjust for existing start separator
-                    ipl += len(self.__separator)
-                add_at_insertion_point[n].insert(0, negative_prompt[:ipp])
-                if negative_prompt[ipp + ipl : ipp + ipl + len(self.__separator)] == self.__separator:
-                    ipl += len(self.__separator)  # adjust for existing end separator
-                endPart = negative_prompt[ipp + ipl :]
-                if len(endPart) > 0:
-                    add_at_insertion_point[n].append(endPart)
-                negative_prompt = self.__separator.join(add_at_insertion_point[n])
-            else:
-                ipp = 0
-                if negative_prompt.startswith(self.__separator):
-                    ipp = len(self.__separator)
-                add_at_insertion_point[n].append(negative_prompt[ipp:])
-                negative_prompt = self.__separator.join(add_at_insertion_point[n])
-        return negative_prompt
-
-    def __add_to_start(self, negative_prompt, add_at_start):
-        if len(negative_prompt) > 0:
-            ipp = 0
-            if negative_prompt.startswith(self.__separator):
-                ipp = len(self.__separator)  # adjust for existing end separator
-            add_at_start.append(negative_prompt[ipp:])
-        negative_prompt = self.__separator.join(add_at_start)
-        return negative_prompt
-
-    def __add_to_end(self, negative_prompt, add_at_end):
-        if len(negative_prompt) > 0:
-            ipl = len(negative_prompt)
-            if negative_prompt.endswith(self.__separator):
-                ipl -= len(self.__separator)  # adjust for existing start separator
-            add_at_end.insert(0, negative_prompt[:ipl])
-        negative_prompt = self.__separator.join(add_at_end)
-        return negative_prompt

stnlogging.py

@@ -1,40 +0,0 @@
-import sys
-import copy
-import logging
-
-
-class SendToNegativeLogFactory:
-    class ColoredFormatter(logging.Formatter):
-        COLORS = {
-            "DEBUG": "\033[0;36m",  # CYAN
-            "INFO": "\033[0;32m",  # GREEN
-            "WARNING": "\033[0;33m",  # YELLOW
-            "ERROR": "\033[0;31m",  # RED
-            "CRITICAL": "\033[0;37;41m",  # WHITE ON RED
-            "RESET": "\033[0m",  # RESET COLOR
-        }
-
-        def format(self, record):
-            colored_record = copy.copy(record)
-            levelname = colored_record.levelname
-            seq = self.COLORS.get(levelname, self.COLORS["RESET"])
-            colored_record.levelname = f"{seq}{levelname}{self.COLORS['RESET']}"
-            return super().format(colored_record)
-
-    def __init__(self):
-        logsd = logging.getLogger("sd")
-        stnlog = logging.getLogger("SendToNegative")
-        stnlog.setLevel(logging.INFO)
-        stnlog.handlers = logsd.handlers
-        if not stnlog.handlers:
-            handler = logging.StreamHandler(sys.stdout)
-            handler.setFormatter(self.ColoredFormatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
-            stnlog.addHandler(handler)
-            self.log = stnlog
-        else:
-            self.log = SendToNegativeLogCustomAdapter(stnlog)
-
-
-class SendToNegativeLogCustomAdapter(logging.LoggerAdapter):
-    def process(self, msg, kwargs):
-        return f"[SendToNegative] {msg}", kwargs

tests/tests.py

@@ -5,16 +5,83 @@ import os
 
 sys.path.insert(1, os.path.join(sys.path[0], ".."))
 
-from sendtonegative import SendToNegative  # pylint: disable=import-error
+from ppp import PromptPostProcessor
-from stnlogging import SendToNegativeLogFactory
+from ppp_logging import PromptPostProcessorLogFactory
 
 
-class TestSendToNegative(unittest.TestCase):
+class DictToObj:  # pylint: disable=too-few-public-methods
+    """
+    Converts a dictionary to an object with attribute access.
+    from https://joelmccune.com/python-dictionary-as-object/
+    """
+
+    def __init__(self, in_dict: dict):
+        assert isinstance(in_dict, dict)
+        for key, val in in_dict.items():
+            if isinstance(val, (list, tuple)):
+                setattr(self, key, [DictToObj(x) if isinstance(x, dict) else x for x in val])
+            else:
+                setattr(self, key, DictToObj(val) if isinstance(val, dict) else val)
+
+
+class TestPromptPostProcessor(unittest.TestCase):
+    """
+    A test case class for testing the PromptPostProcessor class.
+    """
+
     def setUp(self):
-        lf = SendToNegativeLogFactory()
+        """
-        self.__log = lf.log
+        Set up the test case by initializing the necessary objects and configurations.
-        self.__log.setLevel(logging.DEBUG)
+        """
-        self.defstn = SendToNegative(self.__log, separator=", ", ignore_repeats=True, join_attention=True, cleanup=True)
+        lf = PromptPostProcessorLogFactory()
+        self.ppp_logger = lf.log
+        self.ppp_logger.setLevel(logging.DEBUG)
+        self.__defopts = DictToObj(
+            {
+                "ppp_gen_debug": True,
+                "ppp_gen_ifwildcards": PromptPostProcessor.IFWILDCARDS_CHOICES["ignore"],
+                "ppp_stn_doi2i": False,
+                "ppp_stn_separator": ", ",
+                "ppp_stn_ignore_repeats": True,
+                "ppp_stn_join_attention": True,
+                "ppp_cup_doi2i": False,
+                "ppp_cup_emptyconstructs": True,
+                "ppp_cup_extraseparators": True,
+                "ppp_cup_extraseparators2": True,
+                "ppp_cup_extraspaces": True,
+                "ppp_cup_breaks": True,
+                "ppp_cup_breaks_eol": False,
+                "ppp_cup_ands": True,
+                "ppp_cup_ands_eol": False,
+                "ppp_cup_extranetworktags": True,
+                "ppp_rem_removeextranetworktags": False,
+            }
+        )
+        self.__nocupopts = DictToObj(
+            {
+                "ppp_gen_debug": True,
+                "ppp_gen_ifwildcards": PromptPostProcessor.IFWILDCARDS_CHOICES["ignore"],
+                "ppp_stn_doi2i": False,
+                "ppp_stn_separator": ", ",
+                "ppp_stn_ignore_repeats": True,
+                "ppp_stn_join_attention": True,
+                "ppp_cup_doi2i": False,
+                "ppp_cup_emptyconstructs": False,
+                "ppp_cup_extraseparators": False,
+                "ppp_cup_extraseparators2": False,
+                "ppp_cup_extraspaces": False,
+                "ppp_cup_breaks": False,
+                "ppp_cup_breaks_eol": False,
+                "ppp_cup_ands": False,
+                "ppp_cup_ands_eol": False,
+                "ppp_cup_extranetworktags": False,
+                "ppp_rem_removeextranetworktags": False,
+            }
+        )
+        self.__defprocessing = DictToObj({"sd_model": DictToObj({"is_sd1": False, "is_sd2": False, "is_sdxl": True})})
+        self.__defstate = None
+        self.defppp = PromptPostProcessor(self, self.__defprocessing, self.__defstate, self.__defopts)
+        self.nocupppp = PromptPostProcessor(self, self.__defprocessing, self.__defstate, self.__nocupopts)
 
     def process(
         self,
@@ -22,9 +89,22 @@ class TestSendToNegative(unittest.TestCase):
         negative_prompt,
         expected_prompt,
         expected_negative_prompt,
-        stn=None,
+        ppp=None,
     ):
-        the_obj = self.defstn if stn is None else stn
+        """
+        Process the prompt and compare the results with the expected prompts.
+
+        Args:
+            prompt (str): The input prompt.
+            negative_prompt (str): The input negative prompt.
+            expected_prompt (str): The expected output prompt.
+            expected_negative_prompt (str): The expected output negative prompt.
+            ppp (object, optional): The post-processor object. Defaults to None.
+
+        Returns:
+            None
+        """
+        the_obj = self.defppp if ppp is None else ppp
         result_prompt, result_negative_prompt = the_obj.process_prompt(prompt, negative_prompt)
         self.assertEqual(result_prompt, expected_prompt, f"Prompt should be '{expected_prompt}'")
         self.assertEqual(
@@ -33,151 +113,255 @@ class TestSendToNegative(unittest.TestCase):
             f"Negative Prompt should be '{expected_negative_prompt}'",
         )
 
-    def test_tag_default(self):
+    # Send To Negative tests
+
+    def test_nt_simple_oldformat(self):  # negtags with different parameters and separations
         self.process(
-            "flowers<!red!>",
+            "flowers<!red!>, <!!s!green!>, <!!e!blue!><!!p0!yellow!>, <!!p1!purple!><!!p2!black!>",
-            "normal quality, worse quality",
+            "<!!i0!!>normal quality<!!i1!!>, worse quality<!!i2!!>",
             "flowers",
-            "red, normal quality, worse quality",
+            "red, green, yellow, normal quality, purple, worse quality, black, blue",
         )
 
-    def test_tag_start(self):
+    def test_nt_simple(self):  # negtags with different parameters and separations
         self.process(
-            "flowers<!!s!red!>",
+            "flowers<ppp:stn>red<ppp:/stn>, <ppp:stn s>green<ppp:/stn>, <ppp:stn e>blue<ppp:/stn><ppp:stn p0>yellow<ppp:/stn>, <ppp:stn p1>purple<ppp:/stn><ppp:stn p2>black<ppp:/stn>",
-            "normal quality, worse quality",
+            "<ppp:stn i0>normal quality<ppp:stn i1>, worse quality<ppp:stn i2>",
             "flowers",
-            "red, normal quality, worse quality",
+            "red, green, yellow, normal quality, purple, worse quality, black, blue",
         )
 
-    def test_tag_end(self):
+    def test_nt_complex(self):  # complex negtags
         self.process(
-            "flowers<!!e!red!>",
+            "<ppp:stn>red<ppp:/stn> ((<ppp:stn s>pink<ppp:/stn>)), flowers <ppp:stn e>purple<ppp:/stn>, <ppp:stn p0>mauve<ppp:/stn><ppp:stn e>blue<ppp:/stn>, <ppp:stn p0>yellow<ppp:/stn> <ppp:stn p1>green<ppp:/stn>",
-            "normal quality, worse quality",
+            "normal quality, <ppp:stn i0>, bad quality<ppp:stn i1>, worse quality",
             "flowers",
-            "normal quality, worse quality, red",
+            "red, (pink:1.21), normal quality, mauve, yellow, bad quality, green, worse quality, purple, blue",
         )
 
-    def test_tag_insertion_mid_sep(self):
+    def test_nt_complex_nocleanup(self):  # complex negtags with no cleanup
         self.process(
-            "flowers<!!p0!red!>",
+            "<ppp:stn>red<ppp:/stn> ((<ppp:stn s>pink<ppp:/stn>)), flowers <ppp:stn e>purple<ppp:/stn>, <ppp:stn p0>mauve<ppp:/stn><ppp:stn e>blue<ppp:/stn>, <ppp:stn p0>yellow<ppp:/stn> <ppp:stn p1>green<ppp:/stn>",
-            "normal quality, <!!i0!!>, worse quality",
+            "normal quality, <ppp:stn i0>, bad quality<ppp:stn i1>, worse quality",
-            "flowers",
+            " (()), flowers , ,  ",
-            "normal quality, red, worse quality",
+            "red, (pink:1.21), normal quality, mauve, yellow, bad quality, green, worse quality, purple, blue",
+            self.nocupppp,
         )
 
-    def test_tag_insertion_mid_no_sep(self):
+    def test_nt_inside_attention(self):  # negtag inside attention
         self.process(
-            "flowers<!!p0!red!>",
+            "[<ppp:stn>neg1<ppp:/stn>] this is a ((test<ppp:stn e>neg2<ppp:/stn>) (test:2.0): 1.5 ) (red<ppp:stn>[square]<ppp:/stn>:1.5)",
-            "normal quality<!!i0!!>worse quality",
-            "flowers",
-            "normal quality, red, worse quality",
-        )
-
-    def test_tag_insertion_start_sep(self):
-        self.process(
-            "flowers<!!p0!red!>",
-            "<!!i0!!>, normal quality, worse quality",
-            "flowers",
-            "red, normal quality, worse quality",
-        )
-
-    def test_tag_insertion_start_no_sep(self):
-        self.process(
-            "flowers<!!p0!red!>",
-            "<!!i0!!>normal quality, worse quality",
-            "flowers",
-            "red, normal quality, worse quality",
-        )
-
-    def test_tag_insertion_end_sep(self):
-        self.process(
-            "flowers<!!p0!red!>",
-            "normal quality, worse quality, <!!i0!!>",
-            "flowers",
-            "normal quality, worse quality, red",
-        )
-
-    def test_tag_insertion_end_no_sep(self):
-        self.process(
-            "flowers<!!p0!red!>",
-            "normal quality, worse quality<!!i0!!>",
-            "flowers",
-            "normal quality, worse quality, red",
-        )
-
-    def test_complex(self):
-        self.process(
-            "<!red!> (<!!s!pink!>), flowers <!!e!purple!>, <!!e!blue!>, <!!p0!yellow!> <!!p1!green!>",
-            "normal quality, <!!i0!!>, bad quality<!!i1!!>, worse quality",
-            "flowers",
-            "red, (pink), normal quality, yellow, bad quality, green, worse quality, purple, blue",
-        )
-
-    def test_complex_no_cleanup(self):
-        self.process(
-            "<!red!> (<!!s!pink!>), flowers <!!e!purple!>, <!!e!blue!>, <!!p0!yellow!> <!!p1!green!>",
-            "normal quality, <!!i0!!>, bad quality<!!i1!!>, worse quality",
-            " (), flowers , ,  ",
-            "red, (pink), normal quality, yellow, bad quality, green, worse quality, purple, blue",
-            SendToNegative(self.__log, separator=", ", ignore_repeats=True, join_attention=True, cleanup=False),
-        )
-
-    def test_inside_attention1(self):
-        self.process(
-            "[<!neg1!>] this is a ((test<!!e!neg2!>) (test:2.0):1.5)",
             "normal quality",
-            "this is a ((test) (test:2.0):1.5)",
+            "this is a ((test) (test:2.0):1.5) (red:1.5)",
-            "[neg1], normal quality, (neg2:1.65)",
+            "[neg1], ([square]:1.5), normal quality, (neg2:1.65)",
         )
 
-    def test_inside_attention2(self):
+    def test_nt_inside_alternation(self):  # negtag inside alternation
         self.process(
-            "(red<![square]!>:1.5)",
+            "this is a (([complex<ppp:stn>neg1<ppp:/stn>|simple<ppp:stn>neg2<ppp:/stn>|regular<ppp:stn>neg3<ppp:/stn>] test)(test:2.0):1.5)",
-            "",
-            "(red:1.5)",
-            "([square]:1.5)",
-        )
-
-    def test_inside_alternation1(self):
-        self.process(
-            "this is a (([complex|simple<!neg1!>|regular] test)(test:2.0):1.5)",
-            "normal quality",
-            "this is a (([complex|simple|regular] test)(test:2.0):1.5)",
-            "([|neg1|]:1.65), normal quality",
-        )
-
-    def test_inside_alternation2(self):
-        self.process(
-            "this is a (([complex<!neg1!>|simple<!neg2!>|regular<!neg3!>] test)(test:2.0):1.5)",
             "normal quality",
             "this is a (([complex|simple|regular] test)(test:2.0):1.5)",
             "([neg1||]:1.65), ([|neg2|]:1.65), ([||neg3]:1.65), normal quality",
         )
 
-    def test_inside_alternation3(self):
+    def test_nt_inside_alternation_recursive(self):  # negtag inside alternation (recursive alternation)
         self.process(
-            "this is a (([complex<!neg1!>[one|two<!neg12!>|three|four(<!neg14!>)]|simple<!neg2!>|regular<!neg3!>] test)(test:2.0):1.5)",
+            "this is a (([complex<ppp:stn>neg1<ppp:/stn>[one|two<ppp:stn>neg12<ppp:/stn>||three|four(<ppp:stn>neg14<ppp:/stn>)]|simple<ppp:stn>neg2<ppp:/stn>|regular<ppp:stn>neg3<ppp:/stn>] test)(test:2.0):1.5)",
             "normal quality",
-            "this is a (([complex[one|two|three|four]|simple|regular] test)(test:2.0):1.5)",
+            "this is a (([complex[one|two||three|four]|simple|regular] test)(test:2.0):1.5)",
-            "([neg1||]:1.65), ([[|neg12||]||]:1.65), ([[|||(neg14)]||]:1.65), ([|neg2|]:1.65), ([||neg3]:1.65), normal quality",
+            "([neg1||]:1.65), ([[|neg12|||]||]:1.65), ([[||||(neg14)]||]:1.65), ([|neg2|]:1.65), ([||neg3]:1.65), normal quality",
         )
 
-    def test_inside_scheduling(self):
+    def test_nt_inside_scheduling(self):  # negtag inside scheduling
         self.process(
-            "this is [abc<!neg1!>:def<!!e!neg2!>:5]",
+            "this is [abc<ppp:stn>neg1<ppp:/stn>:def<ppp:stn e>neg2<ppp:/stn>: 5 ]",
             "normal quality",
             "this is [abc:def:5]",
             "[neg1::5], normal quality, [neg2:5]",
         )
 
-    def test_complex_features(self):
+    def test_nt_complex_features(self):  # complex negtags with AND, BREAK and other features
         self.process(
-            "[<!neg5!>] this is: a (([complex|simple<!neg6!>|regular] test<!neg1!>)(test:2.0):1.5) \nBREAK with [abc<!neg4!>:def<!!p0!neg2(neg3:1.6)!>:5] <lora:xxx:1>",
+            "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loraword <lora:xxx:1> AND AND hypernetword <hypernet:yyy>:0.3",
-            "normal quality, <!!i0!!>",
+            "normal quality, <ppp:stn i0>",
-            "this is: a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5] <lora:xxx:1>",
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5]:0.5 AND loraword <lora:xxx:1> AND hypernetword <hypernet:yyy>:0.3",
             "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
         )
 
+    def test_nt_complex_features_newformat(self):  # complex negtags with AND, BREAK and other features (new format)
+        self.process(
+            "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loraword <lora:xxx:1> AND AND hypernetword <hypernet:yyy>:0.3",
+            "normal quality, <ppp:stn i0>",
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5]:0.5 AND loraword <lora:xxx:1> AND hypernetword <hypernet:yyy>:0.3",
+            "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+        )
+
+    # Wildcard tests
+
+    def test_wc_ignore(self):  # wildcards with ignore option
+        self.process(
+            "__bad_wildcard__",
+            "{option1|option2}",
+            "__bad_wildcard__",
+            "{option1|option2}",
+            PromptPostProcessor(
+                self,
+                self.__defprocessing,
+                self.__defstate,
+                DictToObj(
+                    {
+                        **self.__defopts.__dict__,
+                        "ppp_gen_ifwildcards": PromptPostProcessor.IFWILDCARDS_CHOICES["ignore"],
+                    }
+                ),
+            ),
+        )
+
+    def test_wc_remove(self):  # wildcards with remove option
+        self.process(
+            "[<ppp:stn>neg5<ppp:/stn>] this is: __bad_wildcard__ a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5] <lora:xxx:1>",
+            "normal quality, <ppp:stn i0> {option1|option2}",
+            "this is: a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5]<lora:xxx:1>",
+            "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+            PromptPostProcessor(
+                self,
+                self.__defprocessing,
+                self.__defstate,
+                DictToObj(
+                    {
+                        **self.__defopts.__dict__,
+                        "ppp_gen_ifwildcards": PromptPostProcessor.IFWILDCARDS_CHOICES["remove"],
+                    }
+                ),
+            ),
+        )
+
+    def test_wc_warn(self):  # wildcards with warn option
+        self.process(
+            "__bad_wildcard__",
+            "{option1|option2}",
+            PromptPostProcessor.WILDCARD_WARNING + "__bad_wildcard__",
+            "{option1|option2}",
+            PromptPostProcessor(
+                self,
+                self.__defprocessing,
+                self.__defstate,
+                DictToObj(
+                    {
+                        **self.__defopts.__dict__,
+                        "ppp_gen_ifwildcards": PromptPostProcessor.IFWILDCARDS_CHOICES["warn"],
+                    }
+                ),
+            ),
+        )
+
+    def test_wc_stop(self):  # wildcards with stop option
+        self.process(
+            "__bad_wildcard__",
+            "{option1|option2}",
+            PromptPostProcessor.WILDCARD_STOP + "__bad_wildcard__",
+            PromptPostProcessor.WILDCARD_STOP + "{option1|option2}",
+            PromptPostProcessor(
+                self,
+                self.__defprocessing,
+                self.__defstate,
+                DictToObj(
+                    {
+                        **self.__defopts.__dict__,
+                        "ppp_gen_ifwildcards": PromptPostProcessor.IFWILDCARDS_CHOICES["stop"],
+                    }
+                ),
+            ),
+        )
+
+    # Cleanup tests
+
+    def test_cl_simple(self):  # simple cleanup
+        self.process(
+            "  this is a ((test ), ,  () [] ( , test ,:2.0):1.5) (red:1.5)  ",
+            "  normal quality  ",
+            "this is a ((test), (test,:2.0):1.5) (red:1.5)",
+            "normal quality",
+        )
+
+    def test_cl_complex(self):  # complex cleanup
+        self.process(
+            "  this is BREAKABLE a ((test), ,AND AND() [] <lora:test> ANDERSON (test:2.0):1.5) :o BREAK \n BREAK (red:1.5)  ",
+            "  [:hands, feet, :0.15]normal quality  ",
+            "this is BREAKABLE a ((test) AND <lora:test> ANDERSON (test:2.0):1.5) :o BREAK (red:1.5)",
+            "[:hands, feet, :0.15]normal quality",
+        )
+
+    def test_cl_removenetworktags(self):  # remove network tags
+        self.process(
+            "this is a <lora:test> test",
+            "",
+            "this is a test",
+            "",
+            PromptPostProcessor(
+                self,
+                self.__defprocessing,
+                self.__defstate,
+                DictToObj({**self.__defopts.__dict__, "ppp_rem_removeextranetworktags": True}),
+            ),
+        )
+
+    def test_cl_dontremoveseparatorsoneol(self):  # dont remove separators on eol
+        self.process(
+            "this is a test,\nsecond line",
+            "",
+            "this is a test,\nsecond line",
+            "",
+            PromptPostProcessor(
+                self,
+                self.__defprocessing,
+                self.__defstate,
+                DictToObj({**self.__defopts.__dict__, "ppp_cup_extraseparators2": False}),
+            ),
+        )
+
+    # Command tests
+
+    def test_cmd_stn_complex_features(self):  # complex stn command with AND, BREAK and other features
+        self.process(
+            "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loraword <lora:xxx:1> AND AND hypernetword <hypernet:yyy>:0.3",
+            "normal quality, <ppp:stn i0>",
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5]:0.5 AND loraword <lora:xxx:1> AND hypernetword <hypernet:yyy>:0.3",
+            "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+        )
+
+    def test_cmd_if_complex_features(self):  # complex if command
+        self.process(
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK, BREAK <ppp:if _sd eq 'sd1'>with [abc<ppp:stn>neg4<ppp:/stn>:def:5]<ppp:/if>:0.5 AND <ppp:if _sd eq 'sd1'>loraword <lora:xxx:1><ppp:elif _sd eq 'sdxl'>hypernetword <hypernet:yyy><ppp:else>nothing<ppp:/if>:0.3",
+            "normal quality",
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK :0.5 AND hypernetword <hypernet:yyy>:0.3",
+            "normal quality",
+        )
+
+    def test_cmd_if_nested(self):  # nested if command
+        self.process(
+            "this is <ppp:if _sd eq 'sd1'>SD1<ppp:else><ppp:if _sd eq 'sdxl'>SDXL<ppp:else>SD2<ppp:/if><ppp:/if>",
+            "",
+            "this is SDXL",
+            "",
+        )
+
+    def test_cmd_set_if(self):  # set and if commands
+        self.process(
+            "<ppp:set v>value<ppp:/set>this test is <ppp:if v eq 'value'>OK<ppp:else>not OK<ppp:/if>",
+            "",
+            "this test is OK",
+            "",
+        )
+
+    def test_cmd_set_if_echo_nested(self):  # nested set, if and echo commands
+        self.process(
+            "<ppp:set v1>1<ppp:/set><ppp:if v1 gt 0><ppp:set v2>OK<ppp:/set><ppp:/if><ppp:if v2 eq 'OK'><ppp:echo v2><ppp:else>not OK<ppp:/if>",
+            "",
+            "OK",
+            "",
+        )
+
 
 if __name__ == "__main__":
     unittest.main()