* New cleanup options: Remove additional extra separators, Use EOL instead of Space before BREAKs, Use EOL instead of Space before ANDs

* New format for commands
* New commands: "set", "echo" and "if/elif/else"
* Option to Remove extra network tags.

.markdownlint.jsonc

@@ -0,0 +1,5 @@
+{
+    "default": true,
+    "MD013": false,
+    "MD024": false
+}

README.md

@@ -4,11 +4,13 @@ The Prompt Postprocessor for Stable Diffusion WebUI, formerly known as "sd-webui
 
 Currently this extension has these functions:
 
-* Allows the tagging of parts of the prompt and moves them to the negative prompt. This allows for useful tricks when using a wildcard extension since you can add negative content from choices made in the positive prompt.
+* Allows marking parts of the prompt and moves them to the negative prompt. This allows for useful tricks when using a wildcard extension since you can add negative content from choices made in the positive prompt.
+* Set values to local variables.
+* Filter content based on the loaded SD model version or a set variable.
 * Detect invalid wildcards and act on them.
 * Clean up the prompt and negative prompt.
 
-Note: The extension must be loaded after the installed wildcards extension (or any other that modifies the prompt). Extensions load by their folder name in alphanumeric order.
+Note: The extension must be loaded after the installed wildcards extension (or any other that modifies the prompt or has it's own syntax expressions). Extensions load by their folder name in alphanumeric order.
 
 With the ["Dynamic Prompts" extension](https://github.com/adieyal/sd-dynamic-prompts) this happens by default due to default folder names for both extensions. But if this is not the case, you can just rename this extension's folder so the ordering works out.
 
@@ -20,14 +22,17 @@ Notes:
 
 1. It only recognizes regular A1111 prompt formats. So:
 
-    * **Attention**: \[prompt\] (prompt) (prompt:weight)
-    * **Alternation**: \[prompt1|prompt2|...\]
-    * **Scheduling**: \[prompt1:prompt2:step\]
-    * **Models**: \<model\>
+    * **Attention**: `\[prompt\] (prompt) (prompt:weight)`
+    * **Alternation**: `\[prompt1|prompt2|...\]`
+    * **Scheduling**: `\[prompt1:prompt2:step\]`
+    * **Extra networks**: `\<kind:model...\>`
+    * **BREAK**: `prompt1 BREAK prompt2`
+    * **Composable Diffusion**: `prompt1 AND prompt2`
 
     In SD.Next that means only the *A1111* or *Full* parsers. It will warn you if you use the *Compel* parser.
 2. It only recognizes wildcards in the *\_\_wildcard\_\_* and *{choice|choice}* formats.
-3. It does not translate equivalent *AND/BREAK* separations into the negative prompt.
+3. Since it should run after other extensions that apply to the prompt, the content should have already been processed by them and there should't be any non recognized syntax anymore.
+4. It does not create *AND/BREAK* constructs when moving content to the negative prompt.
 
 ## Installation
 
@@ -44,39 +49,150 @@ This extension should run after any wildcard extensions, so any remaining wildca
 
 If you choose to not ignore wildcards, the extension will look for any *\_\_wildcard\_\_* or *{choice|choice}* constructs and act as configured.
 
+### Commands
+
+The extension uses now a new format for its commands. The format is similar to an extranetwork, but it has a "ppp:" prefix followed by the command, and then a space and any parameters (if any).
+
+```text
+<ppp:command parameters>
+```
+
+When a command is associated with any content, it will be between an opening and a closing command:
+
+```text
+<ppp:command parameters>content<ppp:/command>
+```
+
+The `set` and `if` commands are the first to be processed.
+
+### Set command
+
+This command sets the value of a variable that can be checked later.
+
+The format is:
+
+```text
+<ppp:set varname>value<ppp:/set>
+```
+
+### Echo command
+
+This command prints the value of a variable.
+
+The format is:
+
+```text
+<ppp:echo varname>
+```
+
+### If command
+
+This command allows you to filter content based on conditions.
+
+The format is:
+
+```text
+<ppp:if condition1>content one<ppp:elif condition2>content two<ppp:else>other content<ppp:/if>
+```
+
+The *conditionN* compares a variable with a value. The operation can be `eq`, `ne`, `gt`, `lt`, `ge`, `le` and the value can be a quoted string or an integer.
+
+The variable can be one set with the `set` command or special variables like:
+
+* `_sd` : the loaded model version (`"sd1"`, `"sd2"`, `"sdxl"`)
+
+Any `elif`s (there can be multiple) and the `else` are optional.
+
+#### Example
+
+(multiline to be easier to read)
+
+```text
+<ppp:if _sd eq "sd1"><lora:test_sd1> test sd1
+<ppp:elif _sd eq "sd2"><lora:test_sd2> test sd2
+<ppp:elif _sd eq "sdxl"><lora:test_sdxl> test sdxl
+<ppp:else>unknown model
+<ppp:/if>
+```
+
+Only one of the options will end up in the prompt, depending on the loaded model.
+
 ### Sending content to the negative prompt
 
-The format of the tags is like this:
+The new format for this command is like this:
+
+```text
+<ppp:stn position>content<ppp:/stn>
+```
+
+Where position is optional (defaults to the start) and can be:
+
+* **s**: at the start of the negative prompt
+* **e**: at the end of the negative prompt
+* **pN**: at the position of the insertion point in the negative prompt with N being 0-9
+
+The format of the insertion point to be used in the negative prompt is:
+
+```text
+<ppp:stn iN>
+```
+
+If the insertion point is not found it inserts at the start.
+
+#### Example
+
+You have a wildcard for hair colors (\_\_haircolors\_\_) with one being strawberry blonde, but you don't want strawberries. So in that option you add a command to add to the negative prompt, like so:
+
+```text
+blonde
+strawberry blonde <ppp:stn>strawberry<ppp:/stn>
+brunette
+```
+
+Then, if that option is chosen this extension will process it later and move that part to the negative prompt.
+
+#### Old format
+
+The old format is still supported (for now) and is like this:
 
 ```text
 <!content!>
 ```
 
-And an optional position in the negative prompt can be specified like this:
+And an optional position can be specified like this:
 
 ```text
 <!!position!content!>
 ```
 
-Where position can be:
-
-* **s**: at the start (the default)
-* **e**: at the end
-* **pN**: at the position of the insertion point "**<!!iN!!>**" with N being 0-9
-
-The insertion point of course must be in the negative prompt. If the insertion point is not found it inserts at the start.
-
-#### Example
-
-You have a wildcard for hair colors (\_\_haircolors\_\_) with one being strawberry blonde, but you don't want strawberries. So in that option you add a tag to add to the negative prompt, like so:
+With the insertion point like this:
 
 ```text
-blonde
-strawberry blonde <!strawberry!>
-brunette
+<!!iN!!>
 ```
 
-Then, if that option is chosen this extension will process it later and move that part to the negative prompt.
+### Notes on negative commands
+
+Positional insertion commands have less priority that start/end commands, so even if they are at the start or end of the negative prompt, they will end up inside any start/end (and default position) commands.
+
+The content of the negative commands is not processed and is copied as-is to the negative prompt. Other modifiers around the commands are processed in the following way.
+
+#### Attention modifiers (weights)
+
+They will be translated to the negative prompt. For example:
+
+* `(red<ppp:stn>square<ppp:/stn>:1.5)` will end up as `(square:1.5)` in the negative prompt
+* `(red[<ppp:stn>square<ppp:/stn>]:1.5)` will end up as `(square:1.35)` in the negative prompt (weight=1.5*0.9)
+* However `(red<ppp:stn>[square]<ppp:/stn>:1.5)` will end up as `([square]:1.5)` in the negative prompt. The content of the negative tag is copied as is, and not joined with the surrounding modifier.
+
+#### Prompt editing constructs (alternation and scheduling)
+
+Negative commands inside such constructs will copy the construct to the negative prompt, but separating its elements. For example:
+
+* **Alternation**: `[red<ppp:stn>square<ppp:/stn>|blue<ppp:stn>circle<ppp:/stn>]` will end up as `[square|], [|circle]` in the negative prompt, instead of `[square|circle]`
+* **Scheduling**: `[red<ppp:stn>square<ppp:/stn>:blue<ppp:stn>circle<ppp:/stn>:0.5]` will end up as `[square::0.5], [:circle:0.5]` instead of `[square:circle:0.5]`
+
+This should still work as intended, and the only negative point i see is the unnecessary separators.
 
 ## Configuration
 
@@ -89,41 +205,27 @@ Then, if that option is chosen this extension will process it later and move tha
   * **Add visible warning**: detect wildcards and add a warning text to the prompt, that hopefully produces a noticeable generation.
   * **Stop the generation**: detect wildcards and stop the generation.
 
+### Content removal settings
+
+* **Remove extra network tags**: removes all extra network tags.
+
 ### Send to negative prompt settings
 
 * **Apply in img2img**: check if you want to do this processing in img2img processes.
 * **Separator used when adding to the negative prompt**: you can specify the separator used when adding to the negative prompt (by default it's ", ").
-* **Ignore tags with repeated content**: it ignores repeated content to avoid repetitions in the negative prompt.
+* **Ignore repeated content**: it ignores repeated content to avoid repetitions in the negative prompt.
 * **Join attention modifiers (weights) when possible**: it joins attention modifiers when possible (joins into one, multipliying their values).
 
 ### Clean up settings
 
 * **Apply in img2img**: check if you want to do this processing in img2img processes.
 * **Remove empty constructs**: removes attention/scheduling/alternation constructs when they are invalid.
-* **Remove extra separators**: removes unnecesary separators. This applies to the configured separator and regular commas.
-* **Clean up around BREAKs**: removes consecutive BREAKs and unnecesary commas and space around them.
-* **Remove extra spaces**: removes unnecesary spaces.
-
-## Notes
-
-The content of the negative tags is not processed and is copied as-is to the negative prompt. Other modifiers around the tags are processed in the following way.
-
-### Attention modifiers (weights)
-
-They will be translated to the negative prompt. For example:
-
-* `(red<!square!>:1.5)` will end up as `(square:1.5)` in the negative prompt
-* `(red[<!square!>]:1.5)` will end up as `(square:1.35)` in the negative prompt (weight=1.5*0.9)
-* However `(red<![square]!>:1.5)` will end up as `([square]:1.5)` in the negative prompt. The content of the negative tag is copied as is, and not joined with the surrounding modifier.
-
-### Prompt editing constructs (alternation and scheduling)
-
-Negative tags inside such constructs will copy the construct to the negative prompt, but separating its elements. For example:
-
-* **Alternation**: `[red<!square!>|blue<!circle!>]` will end up as `[square|], [|circle]` in the negative prompt, instead of `[square|circle]`
-* **Scheduling**: `[red<!square!>:blue<!circle!>:0.5]` will end up as `[square::0.5], [:circle:0.5]` instead of `[square:circle:0.5]`
-
-This should still work as intended, and the only negative point i see is the unnecessary separators.
+* **Remove extra separators**: removes unnecessary separators. This applies to the configured separator and regular commas.
+* **Remove additional extra separators**: removes unnecessary separators at start or end of lines. This applies to the configured separator and regular commas.
+* **Clean up around BREAKs**: removes consecutive BREAKs and unnecessary commas and space around them.
+* **Clean up around ANDs**: removes consecutive ANDs and unnecessary commas and space around them.
+* **Clean up around extra network tags**: removes spaces around them.
+* **Remove extra spaces**: removes other unnecessary spaces.
 
 ## License
 

scripts/ppp_script.py

@@ -85,7 +85,7 @@ class PromptPostProcessorScript(scripts.Script):
         self.ppp_debug = getattr(opts, "ppp_gen_debug", False) if opts is not None else False
         if self.ppp_debug:
             self.ppp_logger.info(f"Post-processing prompts ({'i2i' if is_i2i else 't2i'} mode)")
-        ppp = PromptPostProcessor(self, opts, is_i2i)
+        ppp = PromptPostProcessor(self, p, shared.state, opts, is_i2i)
         # processes regular prompts
         if (
             hasattr(p, "all_prompts")
@@ -145,10 +145,26 @@ class PromptPostProcessorScript(scripts.Script):
             ),
         )
 
+        # content removal settings
+        shared.opts.add_option(
+            key="ppp_rem_sep", info=shared.OptionInfo("<br/><h2>Content removal settings</h2>", "", gr.HTML, section=section)
+        )
+        shared.opts.add_option(
+            key="ppp_rem_removeextranetworktags",
+            info=shared.OptionInfo(
+                False,
+                label="Remove extra network tags",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_rem_if", info=shared.OptionInfo("<p style=\"font-style:italic\">* Parsing of the 'if' commands cannot be disabled</p>", "", gr.HTML, section=section)
+        )
+
         # send to negative settings
         shared.opts.add_option(
             key="ppp_stn_sep",
-            info=shared.OptionInfo("<h2>Send to Negative settings</h2>", "", gr.HTML, section=section),
+            info=shared.OptionInfo("<br/><h2>Send to Negative settings</h2>", "", gr.HTML, section=section),
         )
         shared.opts.add_option(
             key="ppp_stn_doi2i",
@@ -170,7 +186,7 @@ class PromptPostProcessorScript(scripts.Script):
             key="ppp_stn_ignorerepeats",
             info=shared.OptionInfo(
                 True,
-                label="Ignore tags with repeated content",
+                label="Ignore repeated content",
                 section=section,
             ),
         )
@@ -184,7 +200,7 @@ class PromptPostProcessorScript(scripts.Script):
         )
         # clean-up settings
         shared.opts.add_option(
-            key="ppp_cup_sep", info=shared.OptionInfo("<h2>Clean-up settings</h2>", "", gr.HTML, section=section)
+            key="ppp_cup_sep", info=shared.OptionInfo("<br/><h2>Clean-up settings</h2>", "", gr.HTML, section=section)
         )
         shared.opts.add_option(
             key="ppp_cup_doi2i",
@@ -194,14 +210,6 @@ class PromptPostProcessorScript(scripts.Script):
                 section=section,
             ),
         )
-        shared.opts.add_option(
-            key="ppp_cup_extraspaces",
-            info=shared.OptionInfo(
-                True,
-                label="Remove extra spaces",
-                section=section,
-            ),
-        )
         shared.opts.add_option(
             key="ppp_cup_emptyconstructs",
             info=shared.OptionInfo(
@@ -218,6 +226,14 @@ class PromptPostProcessorScript(scripts.Script):
                 section=section,
             ),
         )
+        shared.opts.add_option(
+            key="ppp_cup_extraseparators2",
+            info=shared.OptionInfo(
+                True,
+                label="Remove additional extra separators",
+                section=section,
+            ),
+        )
         shared.opts.add_option(
             key="ppp_cup_breaks",
             info=shared.OptionInfo(
@@ -226,3 +242,43 @@ class PromptPostProcessorScript(scripts.Script):
                 section=section,
             ),
         )
+        shared.opts.add_option(
+            key="ppp_cup_breaks_eol",
+            info=shared.OptionInfo(
+                False,
+                label="Use EOL instead of Space before BREAKs",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_ands",
+            info=shared.OptionInfo(
+                True,
+                label="Clean up around ANDs",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_ands_eol",
+            info=shared.OptionInfo(
+                False,
+                label="Use EOL instead of Space before ANDs",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_extranetworktags",
+            info=shared.OptionInfo(
+                False,
+                label="Clean up around extra network tags",
+                section=section,
+            ),
+        )
+        shared.opts.add_option(
+            key="ppp_cup_extraspaces",
+            info=shared.OptionInfo(
+                True,
+                label="Remove extra spaces",
+                section=section,
+            ),
+        )

tests/tests.py

@@ -47,14 +47,41 @@ class TestPromptPostProcessor(unittest.TestCase):
                 "ppp_cup_doi2i": False,
                 "ppp_cup_emptyconstructs": True,
                 "ppp_cup_extraseparators": True,
+                "ppp_cup_extraseparators2": True,
                 "ppp_cup_extraspaces": True,
                 "ppp_cup_breaks": True,
+                "ppp_cup_breaks_eol": False,
+                "ppp_cup_ands": True,
+                "ppp_cup_ands_eol": False,
+                "ppp_cup_extranetworktags": True,
+                "ppp_rem_removeextranetworktags": False,
             }
         )
-        self.defppp = PromptPostProcessor(self, self.__defopts)
-
-    def ppp_interrupt(self):
-        pass # fake interrupt
+        self.__nocupopts = DictToObj(
+            {
+                "ppp_gen_debug": True,
+                "ppp_gen_ifwildcards": PromptPostProcessor.IFWILDCARDS_CHOICES["ignore"],
+                "ppp_stn_doi2i": False,
+                "ppp_stn_separator": ", ",
+                "ppp_stn_ignore_repeats": True,
+                "ppp_stn_join_attention": True,
+                "ppp_cup_doi2i": False,
+                "ppp_cup_emptyconstructs": False,
+                "ppp_cup_extraseparators": False,
+                "ppp_cup_extraseparators2": False,
+                "ppp_cup_extraspaces": False,
+                "ppp_cup_breaks": False,
+                "ppp_cup_breaks_eol": False,
+                "ppp_cup_ands": False,
+                "ppp_cup_ands_eol": False,
+                "ppp_cup_extranetworktags": False,
+                "ppp_rem_removeextranetworktags": False,
+            }
+        )
+        self.__defprocessing = DictToObj({"sd_model": DictToObj({"is_sd1": False, "is_sd2": False, "is_sdxl": True})})
+        self.__defstate = None
+        self.defppp = PromptPostProcessor(self, self.__defprocessing, self.__defstate, self.__defopts)
+        self.nocupppp = PromptPostProcessor(self, self.__defprocessing, self.__defstate, self.__nocupopts)
 
     def process(
         self,
@@ -88,165 +115,90 @@ class TestPromptPostProcessor(unittest.TestCase):
 
     # Send To Negative tests
 
-    def test_tag_default(self):  # negtag with no parameters
+    def test_nt_simple_oldformat(self):  # negtags with different parameters and separations
         self.process(
-            "flowers<!red!>",
-            "normal quality, worse quality",
+            "flowers<!red!>, <!!s!green!>, <!!e!blue!><!!p0!yellow!>, <!!p1!purple!><!!p2!black!>",
+            "<!!i0!!>normal quality<!!i1!!>, worse quality<!!i2!!>",
             "flowers",
-            "red, normal quality, worse quality",
+            "red, green, yellow, normal quality, purple, worse quality, black, blue",
         )
 
-    def test_tag_start(self):  # negtag with s parameter
+    def test_nt_simple(self):  # negtags with different parameters and separations
         self.process(
-            "flowers<!!s!red!>",
-            "normal quality, worse quality",
+            "flowers<ppp:stn>red<ppp:/stn>, <ppp:stn s>green<ppp:/stn>, <ppp:stn e>blue<ppp:/stn><ppp:stn p0>yellow<ppp:/stn>, <ppp:stn p1>purple<ppp:/stn><ppp:stn p2>black<ppp:/stn>",
+            "<ppp:stn i0>normal quality<ppp:stn i1>, worse quality<ppp:stn i2>",
             "flowers",
-            "red, normal quality, worse quality",
+            "red, green, yellow, normal quality, purple, worse quality, black, blue",
         )
 
-    def test_tag_end(self):  # negtag with e parameter
+    def test_nt_complex(self):  # complex negtags
         self.process(
-            "flowers<!!e!red!>",
-            "normal quality, worse quality",
+            "<ppp:stn>red<ppp:/stn> ((<ppp:stn s>pink<ppp:/stn>)), flowers <ppp:stn e>purple<ppp:/stn>, <ppp:stn p0>mauve<ppp:/stn><ppp:stn e>blue<ppp:/stn>, <ppp:stn p0>yellow<ppp:/stn> <ppp:stn p1>green<ppp:/stn>",
+            "normal quality, <ppp:stn i0>, bad quality<ppp:stn i1>, worse quality",
             "flowers",
-            "normal quality, worse quality, red",
+            "red, (pink:1.21), normal quality, mauve, yellow, bad quality, green, worse quality, purple, blue",
         )
 
-    def test_tag_insertion_mid_sep(self):  # negtag with p parameter and insertion in the middle
+    def test_nt_complex_nocleanup(self):  # complex negtags with no cleanup
         self.process(
-            "flowers<!!p0!red!>",
-            "normal quality, <!!i0!!>, worse quality",
-            "flowers",
-            "normal quality, red, worse quality",
+            "<ppp:stn>red<ppp:/stn> ((<ppp:stn s>pink<ppp:/stn>)), flowers <ppp:stn e>purple<ppp:/stn>, <ppp:stn p0>mauve<ppp:/stn><ppp:stn e>blue<ppp:/stn>, <ppp:stn p0>yellow<ppp:/stn> <ppp:stn p1>green<ppp:/stn>",
+            "normal quality, <ppp:stn i0>, bad quality<ppp:stn i1>, worse quality",
+            " (()), flowers , ,  ",
+            "red, (pink:1.21), normal quality, mauve, yellow, bad quality, green, worse quality, purple, blue",
+            self.nocupppp,
         )
 
-    def test_tag_insertion_mid_no_sep(self):  # negtag with p parameter and insertion in the middle without separator
+    def test_nt_inside_attention(self):  # negtag inside attention
         self.process(
-            "flowers<!!p0!red!>",
-            "normal quality<!!i0!!>worse quality",
-            "flowers",
-            "normal quality, red, worse quality",
-        )
-
-    def test_tag_insertion_start_sep(self):  # negtag with p parameter and insertion at the start
-        self.process(
-            "flowers<!!p0!red!>",
-            "<!!i0!!>, normal quality, worse quality",
-            "flowers",
-            "red, normal quality, worse quality",
-        )
-
-    def test_tag_insertion_start_no_sep(self):  # negtag with p parameter and insertion at the start without separator
-        self.process(
-            "flowers<!!p0!red!>",
-            "<!!i0!!>normal quality, worse quality",
-            "flowers",
-            "red, normal quality, worse quality",
-        )
-
-    def test_tag_insertion_end_sep(self):  # negtag with p parameter and insertion at the end
-        self.process(
-            "flowers<!!p0!red!>",
-            "normal quality, worse quality, <!!i0!!>",
-            "flowers",
-            "normal quality, worse quality, red",
-        )
-
-    def test_tag_insertion_end_no_sep(self):  # negtag with p parameter and insertion at the end without separator
-        self.process(
-            "flowers<!!p0!red!>",
-            "normal quality, worse quality<!!i0!!>",
-            "flowers",
-            "normal quality, worse quality, red",
-        )
-
-    def test_complex(self):  # complex negtags
-        self.process(
-            "<!red!> (<!!s!pink!>), flowers <!!e!purple!>, <!!e!blue!>, <!!p0!yellow!> <!!p1!green!>",
-            "normal quality, <!!i0!!>, bad quality<!!i1!!>, worse quality",
-            "flowers",
-            "red, (pink), normal quality, yellow, bad quality, green, worse quality, purple, blue",
-        )
-
-    def test_complex_no_cleanup(self):  # complex negtags with no cleanup
-        self.process(
-            "<!red!> (<!!s!pink!>), flowers <!!e!purple!>, <!!e!blue!>, <!!p0!yellow!> <!!p1!green!>",
-            "normal quality, <!!i0!!>, bad quality<!!i1!!>, worse quality",
-            " (), flowers , ,  ",
-            "red, (pink), normal quality, yellow, bad quality, green, worse quality, purple, blue",
-            PromptPostProcessor(
-                self,
-                DictToObj(
-                    {
-                        **self.__defopts.__dict__,
-                        "ppp_cup_emptyconstructs": False,
-                        "ppp_cup_extraseparators": False,
-                        "ppp_cup_extraspaces": False,
-                        "ppp_cup_breaks": False,
-                    }
-                ),
-            ),
-        )
-
-    def test_inside_attention1(self):  # negtag inside attention
-        self.process(
-            "[<!neg1!>] this is a ((test<!!e!neg2!>) (test:2.0): 1.5 )",
+            "[<ppp:stn>neg1<ppp:/stn>] this is a ((test<ppp:stn e>neg2<ppp:/stn>) (test:2.0): 1.5 ) (red<ppp:stn>[square]<ppp:/stn>:1.5)",
             "normal quality",
-            "this is a ((test) (test:2.0):1.5)",
-            "[neg1], normal quality, (neg2:1.65)",
+            "this is a ((test) (test:2.0):1.5) (red:1.5)",
+            "[neg1], ([square]:1.5), normal quality, (neg2:1.65)",
         )
 
-    def test_inside_attention2(self):  # negtag inside attention
+    def test_nt_inside_alternation(self):  # negtag inside alternation
         self.process(
-            "(red<![square]!>:1.5)",
-            "",
-            "(red:1.5)",
-            "([square]:1.5)",
-        )
-
-    def test_inside_alternation1(self):  # negtag inside alternation
-        self.process(
-            "this is a (([complex|simple<!neg1!>|regular] test)(test:2.0):1.5)",
-            "normal quality",
-            "this is a (([complex|simple|regular] test)(test:2.0):1.5)",
-            "([|neg1|]:1.65), normal quality",
-        )
-
-    def test_inside_alternation2(self):  # negtag inside alternation
-        self.process(
-            "this is a (([complex<!neg1!>|simple<!neg2!>|regular<!neg3!>] test)(test:2.0):1.5)",
+            "this is a (([complex<ppp:stn>neg1<ppp:/stn>|simple<ppp:stn>neg2<ppp:/stn>|regular<ppp:stn>neg3<ppp:/stn>] test)(test:2.0):1.5)",
             "normal quality",
             "this is a (([complex|simple|regular] test)(test:2.0):1.5)",
             "([neg1||]:1.65), ([|neg2|]:1.65), ([||neg3]:1.65), normal quality",
         )
 
-    def test_inside_alternation3(self):  # negtag inside alternation (recursive alternation)
+    def test_nt_inside_alternation_recursive(self):  # negtag inside alternation (recursive alternation)
         self.process(
-            "this is a (([complex<!neg1!>[one|two<!neg12!>||three|four(<!neg14!>)]|simple<!neg2!>|regular<!neg3!>] test)(test:2.0):1.5)",
+            "this is a (([complex<ppp:stn>neg1<ppp:/stn>[one|two<ppp:stn>neg12<ppp:/stn>||three|four(<ppp:stn>neg14<ppp:/stn>)]|simple<ppp:stn>neg2<ppp:/stn>|regular<ppp:stn>neg3<ppp:/stn>] test)(test:2.0):1.5)",
             "normal quality",
             "this is a (([complex[one|two||three|four]|simple|regular] test)(test:2.0):1.5)",
             "([neg1||]:1.65), ([[|neg12|||]||]:1.65), ([[||||(neg14)]||]:1.65), ([|neg2|]:1.65), ([||neg3]:1.65), normal quality",
         )
 
-    def test_inside_scheduling(self):  # negtag inside scheduling
+    def test_nt_inside_scheduling(self):  # negtag inside scheduling
         self.process(
-            "this is [abc<!neg1!>:def<!!e!neg2!>: 5 ]",
+            "this is [abc<ppp:stn>neg1<ppp:/stn>:def<ppp:stn e>neg2<ppp:/stn>: 5 ]",
             "normal quality",
             "this is [abc:def:5]",
             "[neg1::5], normal quality, [neg2:5]",
         )
 
-    def test_complex_features(self):  # complex negtags with features
+    def test_nt_complex_features(self):  # complex negtags with AND, BREAK and other features
         self.process(
-            "[<!neg5!>] this is: a (([complex|simple<!neg6!>|regular] test<!neg1!>)(test:2.0):1.5) \nBREAK, BREAK with [abc<!neg4!>:def<!!p0!neg2(neg3:1.6)!>:5] <lora:xxx:1>",
-            "normal quality, <!!i0!!>",
-            "this is: a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5] <lora:xxx:1>",
+            "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loraword <lora:xxx:1> AND AND hypernetword <hypernet:yyy>:0.3",
+            "normal quality, <ppp:stn i0>",
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5]:0.5 AND loraword <lora:xxx:1> AND hypernetword <hypernet:yyy>:0.3",
+            "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+        )
+
+    def test_nt_complex_features_newformat(self):  # complex negtags with AND, BREAK and other features (new format)
+        self.process(
+            "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loraword <lora:xxx:1> AND AND hypernetword <hypernet:yyy>:0.3",
+            "normal quality, <ppp:stn i0>",
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5]:0.5 AND loraword <lora:xxx:1> AND hypernetword <hypernet:yyy>:0.3",
             "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
         )
 
     # Wildcard tests
 
-    def test_wildcards_ignore(self):  # wildcards with ignore option
+    def test_wc_ignore(self):  # wildcards with ignore option
         self.process(
             "__bad_wildcard__",
             "{option1|option2}",
@@ -254,6 +206,8 @@ class TestPromptPostProcessor(unittest.TestCase):
             "{option1|option2}",
             PromptPostProcessor(
                 self,
+                self.__defprocessing,
+                self.__defstate,
                 DictToObj(
                     {
                         **self.__defopts.__dict__,
@@ -263,14 +217,16 @@ class TestPromptPostProcessor(unittest.TestCase):
             ),
         )
 
-    def test_wildcards_remove(self):  # wildcards with remove option
+    def test_wc_remove(self):  # wildcards with remove option
         self.process(
-            "[<!neg5!>] this is: __bad_wildcard__ a (([complex|simple<!neg6!>|regular] test<!neg1!>)(test:2.0):1.5) \nBREAK, BREAK with [abc<!neg4!>:def<!!p0!neg2(neg3:1.6)!>:5] <lora:xxx:1>",
-            "normal quality, <!!i0!!> {option1|option2}",
-            "this is: a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5] <lora:xxx:1>",
+            "[<ppp:stn>neg5<ppp:/stn>] this is: __bad_wildcard__ a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5] <lora:xxx:1>",
+            "normal quality, <ppp:stn i0> {option1|option2}",
+            "this is: a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5]<lora:xxx:1>",
             "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
             PromptPostProcessor(
                 self,
+                self.__defprocessing,
+                self.__defstate,
                 DictToObj(
                     {
                         **self.__defopts.__dict__,
@@ -280,7 +236,7 @@ class TestPromptPostProcessor(unittest.TestCase):
             ),
         )
 
-    def test_wildcards_warn(self):  # wildcards with warn option
+    def test_wc_warn(self):  # wildcards with warn option
         self.process(
             "__bad_wildcard__",
             "{option1|option2}",
@@ -288,6 +244,8 @@ class TestPromptPostProcessor(unittest.TestCase):
             "{option1|option2}",
             PromptPostProcessor(
                 self,
+                self.__defprocessing,
+                self.__defstate,
                 DictToObj(
                     {
                         **self.__defopts.__dict__,
@@ -297,7 +255,7 @@ class TestPromptPostProcessor(unittest.TestCase):
             ),
         )
 
-    def test_wildcards_stop(self):  # wildcards with stop option
+    def test_wc_stop(self):  # wildcards with stop option
         self.process(
             "__bad_wildcard__",
             "{option1|option2}",
@@ -305,6 +263,8 @@ class TestPromptPostProcessor(unittest.TestCase):
             PromptPostProcessor.WILDCARD_STOP + "{option1|option2}",
             PromptPostProcessor(
                 self,
+                self.__defprocessing,
+                self.__defstate,
                 DictToObj(
                     {
                         **self.__defopts.__dict__,
@@ -314,6 +274,94 @@ class TestPromptPostProcessor(unittest.TestCase):
             ),
         )
 
+    # Cleanup tests
+
+    def test_cl_simple(self):  # simple cleanup
+        self.process(
+            "  this is a ((test ), ,  () [] ( , test ,:2.0):1.5) (red:1.5)  ",
+            "  normal quality  ",
+            "this is a ((test), (test,:2.0):1.5) (red:1.5)",
+            "normal quality",
+        )
+
+    def test_cl_complex(self):  # complex cleanup
+        self.process(
+            "  this is BREAKABLE a ((test), ,AND AND() [] <lora:test> ANDERSON (test:2.0):1.5) :o BREAK \n BREAK (red:1.5)  ",
+            "  [:hands, feet, :0.15]normal quality  ",
+            "this is BREAKABLE a ((test) AND <lora:test> ANDERSON (test:2.0):1.5) :o BREAK (red:1.5)",
+            "[:hands, feet, :0.15]normal quality",
+        )
+
+    def test_cl_removenetworktags(self):  # remove network tags
+        self.process(
+            "this is a <lora:test> test",
+            "",
+            "this is a test",
+            "",
+            PromptPostProcessor(
+                self,
+                self.__defprocessing,
+                self.__defstate,
+                DictToObj({**self.__defopts.__dict__, "ppp_rem_removeextranetworktags": True}),
+            ),
+        )
+
+    def test_cl_dontremoveseparatorsoneol(self):  # dont remove separators on eol
+        self.process(
+            "this is a test,\nsecond line",
+            "",
+            "this is a test,\nsecond line",
+            "",
+            PromptPostProcessor(
+                self,
+                self.__defprocessing,
+                self.__defstate,
+                DictToObj({**self.__defopts.__dict__, "ppp_cup_extraseparators2": False}),
+            ),
+        )
+
+    # Command tests
+
+    def test_cmd_stn_complex_features(self):  # complex stn command with AND, BREAK and other features
+        self.process(
+            "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loraword <lora:xxx:1> AND AND hypernetword <hypernet:yyy>:0.3",
+            "normal quality, <ppp:stn i0>",
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK with [abc:def:5]:0.5 AND loraword <lora:xxx:1> AND hypernetword <hypernet:yyy>:0.3",
+            "[neg5], ([|neg6|]:1.65), (neg1:1.65), [neg4::5], normal quality, [neg2(neg3:1.6):5]",
+        )
+
+    def test_cmd_if_complex_features(self):  # complex if command
+        self.process(
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK, BREAK <ppp:if _sd eq 'sd1'>with [abc<ppp:stn>neg4<ppp:/stn>:def:5]<ppp:/if>:0.5 AND <ppp:if _sd eq 'sd1'>loraword <lora:xxx:1><ppp:elif _sd eq 'sdxl'>hypernetword <hypernet:yyy><ppp:else>nothing<ppp:/if>:0.3",
+            "normal quality",
+            "this \\(is\\): a (([complex|simple|regular] test)(test:2.0):1.5) \nBREAK :0.5 AND hypernetword <hypernet:yyy>:0.3",
+            "normal quality",
+        )
+
+    def test_cmd_if_nested(self):  # nested if command
+        self.process(
+            "this is <ppp:if _sd eq 'sd1'>SD1<ppp:else><ppp:if _sd eq 'sdxl'>SDXL<ppp:else>SD2<ppp:/if><ppp:/if>",
+            "",
+            "this is SDXL",
+            "",
+        )
+
+    def test_cmd_set_if(self):  # set and if commands
+        self.process(
+            "<ppp:set v>value<ppp:/set>this test is <ppp:if v eq 'value'>OK<ppp:else>not OK<ppp:/if>",
+            "",
+            "this test is OK",
+            "",
+        )
+
+    def test_cmd_set_if_echo_nested(self):  # nested set, if and echo commands
+        self.process(
+            "<ppp:set v1>1<ppp:/set><ppp:if v1 gt 0><ppp:set v2>OK<ppp:/set><ppp:/if><ppp:if v2 eq 'OK'><ppp:echo v2><ppp:else>not OK<ppp:/if>",
+            "",
+            "OK",
+            "",
+        )
+
 
 if __name__ == "__main__":
     unittest.main()