* Wildcard identifiers cannot start with underscore.

* Fix cleaning constructs with only whitespace.
* Logging supports saving to file (used optionally in tests).
* Fix en refresh wildcards.

.gitignore

@@ -5,3 +5,4 @@
 !.vscode/launch.json
 
 tests/tests_local.py
+tests/logs

README.md

@@ -6,7 +6,6 @@ The Prompt PostProcessor (PPP), formerly known as "sd-webui-sendtonegative", is
 * [AUTOMATIC1111 Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
 * [Forge](https://github.com/lllyasviel/stable-diffusion-webui-forge)
 * [reForge](https://github.com/Panchovix/stable-diffusion-webui-reForge)
-* [SD.Next](https://github.com/vladmandic/automatic)
 * ...and probably other forks
 
 Currently this extension has these functions:
@@ -50,7 +49,7 @@ On *A1111* compatible webuis:
 3. Click the Install button
 4. Restart the webui
 
-On *SD.Next* I recommend you disable the native wildcard processing.
+On *SD.Next* I recommend you to disable the native wildcard processing and use the old UI.
 
 On *ComfyUI*:
 

docs/SYNTAX.md

@@ -4,7 +4,7 @@
 
 The extension uses a format for its commands similar to an extranetwork, but it has a "ppp:" prefix followed by the command, and then a space and any parameters (if any).
 
-`<ppp:command parameters>`
+`<ppp:command parameters/>`
 
 When a command is associated with any content, it will be between an opening and a closing command:
 
@@ -33,6 +33,7 @@ Regarding the "optional" flag, consider this scenario: due to their conditions n
 
 The choice options are as follows:
 
+* "**%**": indicates that the content of the choice is a command
 * "**'identifiers'**": comma separated labels for the choice (optional, quotes can be single or double). Only makes sense inside a wildcard definition. Can be used when specifying the wildcard to select this specific choice. It's case insensitive.
 * "**n**": weight of the choice (optional, default 1).
 * "**if condition**": filters out the choice if the condition is false (optional; this is an extension to the *Dynamic Prompts* syntax). Same conditions as in the `if` command.
@@ -40,6 +41,8 @@ The choice options are as follows:
 
 Whitespace is allowed between parameters/options.
 
+The only command available is `include wildcard`, which will include the choices of the specified wildcard in place of this choice. This allows composing choices from multiple wildcards. It also works in the choices of a wildcard, but note that in yaml you cannot start an array element with "%" and you will have to put the full choice in quotes, or use the object format.
+
 These are examples of formats you can use to insert a choice construct:
 
 | Construct                                      | Result |
@@ -50,10 +53,11 @@ These are examples of formats you can use to insert a choice construct:
 | `{r2-3$$choice1\|choice2\|choice3}`            | select 2 to 3 choices allowing repetition |
 | `{2-3$$ / $$choice1\|choice2\|choice3}`        | select 2 to 3 choices with separator " / " |
 | `{o$$if _is_sd1::choice1\|if _is_sd2::choice2}`| select 1 choice, both have conditions, if none matches it is allowed because we indicate that it is optional |
+| `{choice1\|choice2\|%0.5::path/wildcard}`      | select 1 choice from the two specified and the ones inside the path/wildcard wildcard, which will be weighted with half their weights |
 
 Notes:
 
-* The *Dynamic Prompts* format `{2$$__flavours__}` does not work as expected. It will only output one value. You can write is as `{r2$$__flavours__}` to get two values, but they may repeat since the evaluation of the wildcard is independent of the choices selection.
+* The *Dynamic Prompts* format `{2$$__flavours__}` does not work as expected because the wildcard is considered only one possible choice (it will only output one value). You can write it instead as a wildcard with parameters `__2$$flavours__`.
 * Whitespace around the choices is not ignored like in *Dynamic Prompts*, but will be cleaned up if the appropriate cleaning settings are selected.
 
 ## Wildcards
@@ -62,26 +66,40 @@ The generic format is: `__parameters$$wildcard'filter'(var=value)__`
 
 The parameters, the filter, and the setting of a variable are optional. The parameters follow the same format as for the choices.
 
-The wildcard identifier can have a relative path and contain globbing formatting, to read multiple wildcards and merge their choices. Note that if there are no parameters specified, the globbing will use the ones from the first wildcard that matches and have parameters (sorted by keys), so if you don't want that you might want to specify them. Also note that, unlike with *Dynamic Prompts*, the wildcard name has to be specified with its full path (unless you use globbing).
+Wildcards cannot be used inside an extranetwork tag (because some lora names contain double underscores). If you need to choose from multiple loras put the whole extranetwork tag inside a wildcard, or use choices.
 
-The filter can be used to filter specific choices from the wildcard. The filtering works before applying the choice conditions (if any). The surrounding quotes can be single or double. The filter is a comma separated list of an integer (positional choice index; zero-based) or choice label. You can also compound them with `+`. That is, the comma separated items act as an OR and the `+` inside them as an AND. Using labels can simplify the definitions of complex wildcards where you want to have direct access to specific choices on occasion (you don't need to create wildcards for each individual choice). There are some additional formats when using filters. You can specify `^wildcard` as a filter to use the filter of a previous wildcard in the chain. You can start the filter (regular or inherited) with `#` and it will not be applied to the current wildcard choices, but the filter will remain in memory to use by other descendant wildcards. You use `#` and `^` when you want to pass a filter to inner wildcards (see the test files).
+### Identifier
+
+* Allowed characters are letters, numbers, underscore (`_`), dash (`-`), dot (`.`), and the path separators (`/` and `\`). It cannot start with an underscore because it would be ambiguous whether it's part of the name or just precedes the wildcard.
+* Can have a relative path and contain globbing formatting, to read multiple wildcards and merge their choices. Note that if there are no parameters specified, the globbing will use the ones from the first wildcard that matches and have parameters (sorted by keys), so if you don't want that you might want to specify them. Also note that, unlike with *Dynamic Prompts*, the wildcard name has to be specified with its full path (unless you use globbing).
+* You can use variables, with the `${name}`, `${name:default}`, `<ppp:echo name/>` or `<ppp:echo name>default<ppp:/>echo>` formats, to build a dynamic identifier.
+
+### Filter
+
+The filter can be used to filter specific choices from the wildcard. The filtering works before applying the choice conditions (if any). The surrounding quotes can be single or double.
+
+The filter is a comma separated list of an integer (positional choice index, zero-based) or choice label. You can also compound them with `+`. That is, the comma separated items act as an OR and the `+` inside them as an AND. Using labels can simplify the definitions of complex wildcards where you want to have direct access to specific choices on occasion (you don't need to create wildcards for each individual choice). There are some additional formats when using filters. You can specify `^wildcard` as a filter to use the filter of a previous wildcard in the chain. You can start the filter (regular or inherited) with `#` and it will not be applied to the current wildcard choices, but the filter will remain in memory to use by other descendant wildcards. You use `#` and `^` when you want to pass a filter to inner wildcards (see the test files).
+
+### Variable
 
 The variable value only applies during the evaluation of the selected choices and is discarded afterward (the variable keeps its original value if there was one).
 
+### Examples
+
 These are examples of formats you can use to insert a wildcard:
 
-| Construct                            | Result |
-| ---------                            | ------ |
-| `__wildcard__`                       | select 1 choice |
-| `__path/wildcard'0'__`               | select the first choice |
-| `__path/wildcard'label'__`           | select the choices with label "label" |
-| `__path/wildcard'0,label1,label2'__` | select the first choice and those with labels "label1" or "label2" |
+| Construct                            | Result                                                                   |
+| ---------                            | ------                                                                   |
+| `__wildcard__`                       | select 1 choice                                                          |
+| `__path/wildcard'0'__`               | select the first choice                                                  |
+| `__path/wildcard'label'__`           | select the choices with label "label"                                    |
+| `__path/wildcard'0,label1,label2'__` | select the first choice and those with labels "label1" or "label2"       |
 | `__path/wildcard'0,label1+label2'__` | select the first choice and those with both labels "label1" and "label2" |
-| `__3$$path/wildcard__`               | select 3 choices |
-| `__2-3$$path/wildcard__`             | select 2 to 3 choices |
-| `__r2-3$$path/wildcard__`            | select 2 to 3 choices allowing repetition |
-| `__2-3$$ / $$path/wildcard__`        | select 2 to 3 choices with separator " / " |
-| `__path/wildcard(var=value)__`       | select 1 choice using the specified variable value in the evaluation. |
+| `__3$$path/wildcard__`               | select 3 choices                                                         |
+| `__2-3$$path/wildcard__`             | select 2 to 3 choices                                                    |
+| `__r2-3$$path/wildcard__`            | select 2 to 3 choices allowing repetition                                |
+| `__2-3$$ / $$path/wildcard__`        | select 2 to 3 choices with separator " / "                               |
+| `__path/wildcard(var=value)__`       | select 1 choice using the specified variable value in the evaluation.    |
 
 ### Wildcard definitions
 
@@ -94,10 +112,10 @@ The best format is a yaml file with a dictionary of wildcards inside. An editor
 
 In a choice, the content after a `#`  is ignored.
 
-If the first choice follows the format of wildcard parameters (*including the final `$$`*), it will be used as default parameters for that wildcard (see examples in the tests folder). The choices of the wildcard follow the same format as in the choices construct, or the object format of *Dynamic Prompts* (only in structured files). If using the object format for a choice you can use a new `if` property for the condition, and the `labels` property (an array of strings) in addition to the standard `weight` and `text`/`content`.
+If the first choice follows the format of wildcard parameters (*including the final `$$`*), it will be used as default parameters for that wildcard (see examples in the tests folder). The choices of the wildcard follow the same format as in the choices construct, or the object format of *Dynamic Prompts* (only in structured files). If using the object format for a choice you can use a new `if` property for the condition, and the `labels` property (an array of strings) and `command` property (a boolean) in addition to the standard `weight` and `text`/`content`.
 
 ```yaml
-{ labels: ["some_label"], weight: 2, if: "_is_pony", content: "the text" } # "text" property can be used instead of "content"
+{ command: false, labels: ["some_label"], weight: 2, if: "_is_pony", content: "the text" } # "text" property can be used instead of "content"
 ```
 
 Wildcard parameters in a json/yaml file can also be in object format, and support two additional properties, prefix and suffix:
@@ -115,6 +133,12 @@ Wildcards can contain just one choice. In json and yaml formats this allows the
 
 A choice inside a wildcard can also be a list or a dictionary of one element containing a list. These are considered anonymous wildcards. With a list it will be an anonymous wildcard with no choice options, and with a dictionary the key will be the options for the choice containing the anonymous wildcard and the value the choices of the anonymous wildcard. Anonymous wildcards can help formatting complex choice values that are used in only one place and thus creating a regular wildcard is not necessary. See test.yaml for examples.
 
+Remember you can use the include command on choices to compose a wildcard from other wildcards' choices.
+
+Note: the files should have UTF-8 encoding. The extension will also try with windows-1252 if that fails.
+
+Wildcard definitions are reloaded automatically on each generation if they change.
+
 ### Detection of remaining wildcards
 
 This extension should run after any other wildcard extensions, so if you don't use the internal wildcards processing, any remaining wildcards present in the prompt or negative_prompt at this point must be invalid. Usually you might not notice this problem until you check the image metadata, so this option gives you some ways to detect and treat the problem.
@@ -135,18 +159,18 @@ The `add` and `ifundefined` modifiers are mutually exclusive and cannot be used
 
 The *Dynamic Prompts* format also works:
 
-| Construct       | Meaning |
-| ---------       | ------- |
-| `${var=value}`  | regular evaluation |
+| Construct       | Meaning              |
+| ---------       | -------              |
+| `${var=value}`  | regular evaluation   |
 | `${var=!value}` | immediate evaluation |
 
 If also supports the addition and undefined check as an extension of the *Dynamic Prompts* format:
 
-| Construct        | Meaning |
-| ---------        | ------- |
-| `${var+=value}`  | equivalent to "add" |
-| `${var+=!value}` | equivalent to "evaluate add" |
-| `${var?=value}`  | equivalent to "ifundefined" |
+| Construct        | Meaning                              |
+| ---------        | -------                              |
+| `${var+=value}`  | equivalent to "add"                  |
+| `${var+=!value}` | equivalent to "evaluate add"         |
+| `${var?=value}`  | equivalent to "ifundefined"          |
 | `${var?=!value}` | equivalent to "evaluate ifundefined" |
 
 ## Echo command
@@ -157,7 +181,7 @@ The format is:
 
 | Construct                              |
 | ---------                              |
-| `<ppp:echo varname>`                   |
+| `<ppp:echo varname/>`                  |
 | `<ppp:echo varname>default<ppp:/echo>` |
 
 The *Dynamic Prompts* format is:
@@ -179,10 +203,10 @@ Any `elif`s (there can be multiple) and the `else` are optional.
 
 The `conditionN` can be:
 
-| Construct                                      | Meaning |
-| ---------                                      | ------- |
-| `variable`                                     | check truthyness of the variable |
-| `variable [not] operation value`               | check the variable against a value |
+| Construct                                      | Meaning                                     |
+| ---------                                      | -------                                     |
+| `variable`                                     | check truthyness of the variable            |
+| `variable [not] operation value`               | check the variable against a value          |
 | `variable [not] operation (value1,value2,...)` | check the variable against a list of values |
 
 For a simple value the allowed operations are `eq`, `ne`, `gt`, `lt`, `ge`, `le`, `contains` and the value can be a quoted string or an integer. For a list of values the allowed operations are `contains`, `in` and the value of the variable is checked against all the elements of the list until one matches. The operation can be preceded by `not` for readability, instead of using it in the front.
@@ -232,7 +256,7 @@ This command is a shortcut to add an extranetwork (usually a lora), and its trig
 The full format is:
 
 `<ppp:ext type name [parameters] [if condition]>[triggers]<ppp:/ext>`
-`<ppp:ext type name [parameters] [if condition]>`
+`<ppp:ext type name [parameters] [if condition]/>`
 
 The `type` is the kind of extranetwork, like `lora` or `hypernet`.
 
@@ -253,7 +277,7 @@ If the condition passes (or if there is no condition) the extranetwork tag will
 ```text
 <ppp:ext lora test_sd1 if _is_sd1>test sd1x<ppp:/ext>
 <ppp:ext lora test_pony 0.5 if _is_pony>test pony<ppp:/ext>
-<ppp:ext lora test_ilxl if _is_illustrious>
+<ppp:ext lora test_ilxl if _is_illustrious/>
 <ppp:ext lora 'test sdxl' '1:0.8' if _is_pure_sdxl>test sdxl<ppp:/ext>
 ```
 
@@ -286,7 +310,7 @@ extnettype:
 Used like this:
 
 ```text
-<ppp:ext $lora mappingname>
+<ppp:ext $lora mappingname/>
 <ppp:ext $lora mappingname>inline triggers<ppp:/ext>
 ```
 
@@ -306,10 +330,10 @@ See the file in the tests folder as an example.
 
 The new format for this command is like this:
 
-| Construct                             | Meaning |
-| ---------                             | ------- |
-| `<ppp:stn position>content<ppp:/stn>` | send to negative prompt |
-| `<ppp:stn iN>`                        | insertion point to be used in the negative prompt as destination for the pN position |
+| Construct                             | Meaning                                                                              |
+| ---------                             | -------                                                                              |
+| `<ppp:stn position>content<ppp:/stn>` | send to negative prompt                                                              |
+| `<ppp:stn iN/>`                       | insertion point to be used in the negative prompt as destination for the pN position |
 
 Where position is optional (defaults to the start) and can be:
 

grammar.lark

@@ -1,12 +1,13 @@
 %import common (LETTER, DIGIT, INT, CNAME, SIGNED_NUMBER, NUMBER)
 
-_WHITESPACE:   /\s+/
-STRING:        /("(?!"").*?(?<!\\)(\\\\)*?"|'(?!'').*?(?<!\\)(\\\\)*?')/i
-BOOLEAN:       /true|false/i
-WILDCARD_NAME: /(?:(?!__|\$\$|[('"])\S)+/
-INDEX:         INT | IDENTIFIER
-IDENTIFIER:    CNAME
-SIMPLEVALUE:   STRING | SIGNED_NUMBER | BOOLEAN
+_WHITESPACE:         /\s+/
+STRING:              /("(?!"").*?(?<!\\)(\\\\)*?"|'(?!'').*?(?<!\\)(\\\\)*?')/i // string with single or double quotes
+BOOLEAN:             /true|false/i
+WC_NAME_PLAIN_START: /(?:(?!__|\$\$|\$\{|<ppp:|[('"])[a-zA-Z0-9\-.\\\/])+/u // starting characters, excluding end and other inside constructs
+WC_NAME_PLAIN:       /(?:(?!__|\$\$|\$\{|<ppp:|[('"])[a-zA-Z0-9\-.\\\/_#\*\?\[!\]])+/u // continuing characters, including globbing characters and anonymous wildcard mark, excluding end and other inside constructs
+INDEX:               INT | IDENTIFIER
+IDENTIFIER:          CNAME
+SIMPLEVALUE:         STRING | SIGNED_NUMBER | BOOLEAN
 
 // plain text and weights
 ?plain:                  /((?!__|\bAND\b|\${)[^\\()\[\]:<>${]|\\.)+/s // exclude only the starting ones
@@ -31,43 +32,51 @@ promptcomppart: content
 ?old_content.2: ( attention | scheduled | alternate | extranetworktag )+
 //#if ALLOW_NEW_CONTENT
     ?content.2: ( old_content | new_content | plain | specialchars )*
-    ?inside_content.2: ( new_content | plain | specialchars )*
     ?content_choice.2: ( old_content | new_content | plain_choice | specialchars_choice )*
     ?content_var.2: ( old_content | new_content | plain_var | specialchars_var )*
     ?content_negtag.2: ( old_content | new_content_negtag | plain | specialchars_negtag )*
     ?content_alternate.2: ( old_content | new_content | plain_alternate | specialchars_alternate )*
+    ?content_en.2: (new_content_en | plain | specialchars )*
     //#if ALLOW_WILDCARDS ALLOW_CHOICES ALLOW_COMMVARS
         ?new_content.3: ( variableset | variableuse | commandstn | commandstni | commandset | commandecho | commandif | commandext | wildcard | choices )+
         ?new_content_negtag.3: ( variableset | variableuse | commandset | commandecho | commandif | commandext | wildcard | choices )+
+        ?new_content_en.3: ( variableset | variableuse | commandset | commandecho | commandif | choices )+
     //#elif ALLOW_WILDCARDS !ALLOW_CHOICES ALLOW_COMMVARS
         ?new_content.3: ( variableset | variableuse | commandstn | commandstni | commandset | commandecho | commandif | commandext | wildcard )+
         ?new_content_negtag.3: ( variableset | variableuse | commandset | commandecho | commandif | commandext | wildcard )+
+        ?new_content_en.3: ( variableset | variableuse | commandset | commandecho | commandif )+
     //#elif !ALLOW_WILDCARDS ALLOW_CHOICES ALLOW_COMMVARS
         ?new_content.3: ( variableset | variableuse | commandstn | commandstni | commandset | commandecho | commandif | commandext | choices )+
         ?new_content_negtag.3: ( variableset | variableuse | commandset | commandecho | commandif | commandext | choices )+
+        ?new_content_en.3: ( variableset | variableuse | commandset | commandecho | commandif | choices )+
     //#elif ALLOW_WILDCARDS ALLOW_CHOICES !ALLOW_COMMVARS
         ?new_content.3: ( wildcard | choices )+
         ?new_content_negtag.3: ( wildcard | choices )+
+        ?new_content_en.3: ( choices )+
     //#elif ALLOW_WILDCARDS !ALLOW_CHOICES !ALLOW_COMMVARS
         ?new_content.3: ( wildcard )+
         ?new_content_negtag.3: ( wildcard )+
+        ?new_content_en.3: /(?!)./ // never matches
     //#elif !ALLOW_WILDCARDS ALLOW_CHOICES !ALLOW_COMMVARS
         ?new_content.3: ( choices )+
         ?new_content_negtag.3: ( choices )+
+        ?new_content_en.3: ( choices )+
     //#elif !ALLOW_WILDCARDS !ALLOW_CHOICES ALLOW_COMMVARS
         ?new_content.3: ( variableset | variableuse | commandstn | commandstni | commandset | commandecho | commandif | commandext )+
         ?new_content_negtag.3: ( variableset | variableuse | commandset | commandecho | commandif | commandext )+
+        ?new_content_en.3: ( variableset | variableuse | commandset | commandecho | commandif )+
     //#else
         ?new_content.3: ( variableset | variableuse | commandstn | commandstni | commandset | commandecho | commandif | commandext | wildcard | choices )+
         ?new_content_negtag.3: ( variableset | variableuse | commandset | commandecho | commandif | wildcard | commandext | choices )+
+        ?new_content_en.3: ( variableset | variableuse | commandset | commandecho | commandif | choices )+
     //#endif
 //#else
     ?content.2: ( old_content | plain | specialchars )*
-    ?inside_content.2: ( plain | specialchars )*
     ?content_choice.2: ( old_content | plain_choice | specialchars_choice )*
     ?content_var.2: ( old_content | plain_var | specialchars_var )*
     ?content_negtag.2: ( old_content | plain | specialchars_negtag )*
     ?content_alternate.2: ( old_content | plain_alternate | specialchars_alternate )*
+    ?content_en.2: (plain | specialchars )*
 //#endif
 
 // attention modifiers
@@ -79,11 +88,12 @@ alternateoption: content_alternate
 scheduled: "[" [ content ":" ] content ":" numpar "]"
 
 // extra network tags
-extranetworktag: "<" /(?!ppp:)\w+:/ inside_content ">"
+extranetworktag: "<" /(?!ppp:)\w+:/ encontent ">"
+?encontent.3: content_en
 
 // command: stn (send to negative)
 commandstn: "<ppp:stn" [ _WHITESPACE /s|e|p\d/ ] _WHITESPACE? ">" content_negtag "<ppp:/stn>"
-commandstni: "<ppp:stn" _WHITESPACE /i\d/ _WHITESPACE? ">"
+commandstni: "<ppp:stn" _WHITESPACE /i\d/ _WHITESPACE? ( "/>" | ">" )
 
 // command: if
 commandif.2: commandif_if commandif_elif* commandif_else? "<ppp:/if>"
@@ -112,11 +122,11 @@ commandsetmodifiers: (_WHITESPACE /evaluate|ifundefined|add/ )+
 ?commandsetcontent.3: content
 
 // command: echo
-commandecho: "<ppp:echo" _WHITESPACE IDENTIFIER _WHITESPACE? ">" [ commandechodefault "<ppp:/echo>" ]
+commandecho: "<ppp:echo" _WHITESPACE IDENTIFIER _WHITESPACE? ( ">" [ commandechodefault "<ppp:/echo>" ] | "/>" )
 ?commandechodefault.3: content
 
 // command: ext
-commandext: "<ppp:ext" _WHITESPACE commandexttype _WHITESPACE commandextid [ _WHITESPACE? commandextparams ] [ _WHITESPACE? commandextif ] _WHITESPACE? ">" [ commandexttriggers "<ppp:/ext>" ]
+commandext: "<ppp:ext" _WHITESPACE commandexttype _WHITESPACE commandextid [ _WHITESPACE? commandextparams ] [ _WHITESPACE? commandextif ] _WHITESPACE? ( ">" [ commandexttriggers "<ppp:/ext>" ] | "/>" )
 commandexttype: [/\$/] IDENTIFIER
 ?commandextid: STRING | CNAME
 ?commandextparams: STRING | SIGNED_NUMBER
@@ -132,8 +142,9 @@ variableuse.2: "${" _WHITESPACE? IDENTIFIER _WHITESPACE? [ ":" varvalue ] "}"
 varvalue: content_var
 
 // wildcards
-wildcard.2: "__" [ choicesoptions_sampler | ( choicesoptions _WHITESPACE? "$$" ) ] WILDCARD_NAME [ wc_filter ] [ wildcardvar ] "__"
-wc_filter: /["']/ ( [ /#/ ] wc_filter_or | ( /#?\^/ WILDCARD_NAME ) ) /["']/
+wildcard.2: "__" [ choicesoptions_sampler | ( choicesoptions _WHITESPACE? "$$" ) ] wildcard_name [ wc_filter ] [ wildcardvar ] "__"
+wildcard_name.2: ( WC_NAME_PLAIN_START | variableuse | commandecho ) ( WC_NAME_PLAIN | variableuse | commandecho )*
+wc_filter: /["']/ ( [ /#/ ] wc_filter_or | ( /#?\^/ wildcard_name ) ) /["']/
 wc_filter_or: wc_filter_and ( _WHITESPACE? "," _WHITESPACE? wc_filter_and )*
 wc_filter_and: INDEX ( _WHITESPACE? "+" _WHITESPACE? INDEX )*
 wildcardvar.7: "(" _WHITESPACE? IDENTIFIER _WHITESPACE? "=" varvalue ")"
@@ -149,7 +160,8 @@ choicesoptions_from: INT
 choicesoptions_to: INT
 choicesoptions_sep: "$$" plain
 
-choice: [ [ _WHITESPACE? choicelabels ] [ _WHITESPACE? choiceweight ] [ _WHITESPACE? choiceif ] _WHITESPACE? "::" ] choicevalue
+choice: [ [ _WHITESPACE? choiceiscmd ] [ _WHITESPACE? choicelabels ] [ _WHITESPACE? choiceweight ] [ _WHITESPACE? choiceif ] _WHITESPACE? "::" ] choicevalue
+choiceiscmd: /%/ // the option text is a special command
 choicelabels: /["']/ IDENTIFIER ( _WHITESPACE? "," _WHITESPACE? IDENTIFIER )* /["']/
 choiceweight: NUMBER
 choiceif: "if" _WHITESPACE condition

ppp.py

@@ -716,9 +716,17 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
         negative_prompt = n_processor.start_visit("negative prompt", n_parsed, True)
 
         var_keys = set(self.user_variables.keys()).union(set(self.echoed_variables.keys()))
-        all_variables.update(
-            {k: self.echoed_variables.get(k, p_processor.get_final_user_variable(k)) for k in var_keys}
-        )
+        for k in var_keys:
+            ev = self.echoed_variables.get(k)
+            if ev is None:
+                ev = self.user_variables.get(k)
+            if ev is None or not isinstance(ev, str):
+                if self.debug_level == DEBUG_LEVEL.full:
+                    self.logger.debug(self.format_output(f"Completing variable: {k}"))
+                ev = p_processor.get_final_user_variable(k)
+            all_variables[k] = ev
+        if self.debug_level == DEBUG_LEVEL.full:
+            self.logger.debug(self.format_output(f"All variables: {all_variables}"))
 
         # Insertions in the negative prompt
         if self.debug_level == DEBUG_LEVEL.full:
@@ -902,6 +910,7 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
             self.__already_processed: list[str] = []
             self.__is_negative = False
             self.__wildcard_filters = {}
+            self.__seen_wildcards: list[str] = []
             self.add_at: dict = {"start": [], "insertion_point": [[] for x in range(10)], "end": []}
             self.insertion_at: list[tuple[int, int]] = [None for x in range(10)]
             self.detectedWildcards: list[str] = []
@@ -1258,7 +1267,7 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
             self.result += ":"
             self.__visit(after)
             self.__shell.pop()
-            if self.__ppp.cup_emptyconstructs and self.result == start_result + "[:":
+            if self.__ppp.cup_emptyconstructs and re.fullmatch(re.escape(start_result) + r"\[:\s*", self.result):
                 self.result = start_result
             else:
                 self.result += f":{pos_str}]"
@@ -1285,7 +1294,7 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                 self.__visit(opt)
                 self.__shell.pop()
             self.result += "]"
-            if self.__ppp.cup_emptyconstructs and self.result == start_result + "[]":
+            if self.__ppp.cup_emptyconstructs and re.fullmatch(re.escape(start_result) + r"\[\s*\]", self.result):
                 self.result = start_result
             # self.__shell.pop()
             t2 = time.monotonic_ns()
@@ -1353,7 +1362,7 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                 self.result += starttag
                 self.__visit(current_tree)
                 endtag = f":{weight_str})"
-            if self.__ppp.cup_emptyconstructs and self.result == start_result + starttag:
+            if self.__ppp.cup_emptyconstructs and re.fullmatch(re.escape(start_result + starttag) + r"\s*", self.result):
                 self.result = start_result
             else:
                 self.result += endtag
@@ -1512,13 +1521,13 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
             """
             Process a DP use variable command in the tree.
             """
-            self.__varecho("variableuse", str(tree.children[0]), tree.children[1])
+            self.__varecho("variableuse", str(tree.children[0]), tree.children[1] if len(tree.children) > 1 else None)
 
         def commandecho(self, tree: lark.Tree):
             """
             Process an echo command in the tree.
             """
-            self.__varecho("commandecho", str(tree.children[0]), tree.children[1])
+            self.__varecho("commandecho", str(tree.children[0]), tree.children[1] if len(tree.children) > 1 else None)
 
         def commandif(self, tree: lark.Tree):
             """
@@ -1576,42 +1585,49 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                     extra_triggers = None
                     compiled_extra_triggers = None
                     if is_mapping:
-                        found_mappings: list[PPPENMappingVariant] = []
-                        else_mapping = None
-                        if self.__ppp.extranetwork_mappings_obj:
-                            enmapping = self.__ppp.extranetwork_mappings_obj.extranetwork_mappings.get(extnet_id, None)
-                            if enmapping:
-                                for v in enmapping.variants:
-                                    if v.condition:
-                                        try:
-                                            cnd = self.__ppp.parse_prompt(
-                                                "condition", v.condition, self.__ppp.parser_condition, True
-                                            )
-                                        except lark.exceptions.UnexpectedInput as e:
-                                            self.warn_or_stop(
-                                                f"Error parsing condition '{v.condition}' in extranetwork mapping '{extnet_id}'! : {e.__class__.__name__}",
-                                                e,
-                                            )
-                                            cnd = None
-                                    else:
-                                        cnd = "True"
-                                    if cnd is not None and (cnd == "True" or self.__eval_condition(cnd)):
-                                        if v.condition:
-                                            found_mappings.append(v)
-                                        else:
-                                            else_mapping = v
-                        if found_mappings:
-                            found = found_mappings[
-                                self.__ppp.rng.choice(
-                                    len(found_mappings),
-                                    p=[v.weight or 1 for v in found_mappings],
+                        found = self.__ppp.extranetwork_mappings_obj.cached_mappings.get(extnet_id, None)
+                        # we assume the conditions do not change inside the prompt
+                        found_in_cache = found is not None
+                        if found is None:
+                            found_mappings: list[PPPENMappingVariant] = []
+                            else_mapping = None
+                            if self.__ppp.extranetwork_mappings_obj:
+                                enmapping = self.__ppp.extranetwork_mappings_obj.extranetwork_mappings.get(
+                                    extnet_id, None
                                 )
-                            ]
-                        else:
-                            found = else_mapping
+                                if enmapping:
+                                    for v in enmapping.variants:
+                                        if v.condition:
+                                            try:
+                                                cnd = self.__ppp.parse_prompt(
+                                                    "condition", v.condition, self.__ppp.parser_condition, True
+                                                )
+                                            except lark.exceptions.UnexpectedInput as e:
+                                                self.warn_or_stop(
+                                                    f"Error parsing condition '{v.condition}' in extranetwork mapping '{extnet_id}'! : {e.__class__.__name__}",
+                                                    e,
+                                                )
+                                                cnd = None
+                                        else:
+                                            cnd = "True"
+                                        if cnd is not None and (cnd == "True" or self.__eval_condition(cnd)):
+                                            if v.condition:
+                                                found_mappings.append(v)
+                                            else:
+                                                else_mapping = v
+                            if found_mappings:
+                                found = found_mappings[
+                                    self.__ppp.rng.choice(
+                                        len(found_mappings),
+                                        p=[v.weight or 1 for v in found_mappings],
+                                    )
+                                ]
+                            else:
+                                found = else_mapping
+                            self.__ppp.extranetwork_mappings_obj.cached_mappings[extnet_id] = found
                         if found:
                             if found.name:
-                                if self.__ppp.debug_level != DEBUG_LEVEL.none:
+                                if not found_in_cache and self.__ppp.debug_level != DEBUG_LEVEL.none:
                                     self.__ppp.logger.info(
                                         f"Mapping extranetwork '{extnet_id}' to '{extnet_type}:{found.name}'"
                                     )
@@ -1631,10 +1647,12 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                                 elif f_parameters is not None and parameters_defaulted:
                                     parameters = f_parameters
                             elif found.triggers:
-                                self.__ppp.logger.info(f"Mapping extranetwork '{extnet_id}' to just triggers")
+                                if not found_in_cache and self.__ppp.debug_level != DEBUG_LEVEL.none:
+                                    self.__ppp.logger.info(f"Mapping extranetwork '{extnet_id}' to just triggers")
                                 extnet_id = None
                             else:
-                                self.__ppp.logger.info(f"Mapping extranetwork '{extnet_id}' to nothing")
+                                if not found_in_cache and self.__ppp.debug_level != DEBUG_LEVEL.none:
+                                    self.__ppp.logger.info(f"Mapping extranetwork '{extnet_id}' to nothing")
                                 extnet_id = None
                             if found.triggers:
                                 extra_triggers = ", ".join(found.triggers)
@@ -1685,41 +1703,13 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
             t2 = time.monotonic_ns()
             self.__debug_end("extranetworktag", start_result, t2 - t1)
 
-        def __get_choices_internal(
+        def __get_choices_internal_get(
             self,
-            options: dict | None,
             choice_values: list[dict],
             filter_specifier: Optional[list[list[str]]] = None,
             wildcard_key: str = None,
-        ) -> tuple[str, list[str], str, str]:
-            """
-            Select choices based on the options.
-
-            Args:
-                options (dict): The object representing the options construct.
-                choice_values (list[dict]): A list of choice objects.
-                filter_specifier (list[list[str]]): The filter specifier.
-                wildcard_key (str): The wildcard key if it is a wildcard.
-
-            Returns:
-                tuple: A tuple containing the prefix, selected choices, separator and suffix
-            """
-            if options is None:
-                options = {}
-            sampler: str = options.get("sampler", "~")
-            repeating: bool = options.get("repeating", False)
-            optional: bool = options.get("optional", False)
-            if "count" in options:
-                from_value = options["count"]
-                to_value = from_value
-            else:
-                from_value: int = options.get("from", 1)
-                to_value: int = options.get("to", 1)
-            separator: str = options.get("separator", self.__ppp.wil_choice_separator)
+        ) -> list[dict]:
             msg_where = f"wildcard '{wildcard_key}'" if wildcard_key else "choices"
-            if sampler != "~":
-                self.warn_or_stop(f"Unsupported sampler '{sampler}' in {msg_where} options!")
-                sampler = "~"
             if filter_specifier is not None:
                 filtered_choice_values = []
                 for i, c in enumerate(choice_values):
@@ -1745,12 +1735,85 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                     )
             else:
                 filtered_choice_values = choice_values.copy()
+            expanded_choice_values = []
+            for i, c in enumerate(filtered_choice_values):
+                if c.get("command", False):
+                    content_text = self.__visit(c.get("content", ""), False, True).strip()
+                    (cmd, cmd_args) = content_text.split()
+                    if cmd == "include":
+                        wcs = self.__ppp.wildcard_obj.get_wildcards(cmd_args)
+                        if not wcs:
+                            self.warn_or_stop(f"Not found included wildcard '{cmd_args}' at {msg_where}!")
+                        c_weight = float(c.get("weight", 1.0))
+                        for wc in wcs:
+                            if wc.key in self.__seen_wildcards:
+                                self.warn_or_stop(
+                                    f"Circular reference detected including wildcard '{wc.key}' at {msg_where} (chain starts at '{self.__seen_wildcards[0]}')!"
+                                )
+                                continue
+                            self.__seen_wildcards.append(wc.key)
+                            if self.__ppp.debug_level == DEBUG_LEVEL.full:
+                                self.__ppp.logger.debug(f"Seen wildcard '{wc.key}'")
+                                self.__ppp.logger.debug(f"Including choices from wildcard '{wc.key}'")
+                            (_, choice_values) = self.__check_wildcard_initialization(wc)
+                            if choice_values is not None:
+                                ch_values = self.__get_choices_internal_get(choice_values, None, wc.key)
+                                for cv in ch_values:
+                                    expanded_choice_values.append(
+                                        {
+                                            **cv,
+                                            "weight": float(cv.get("weight", 1.0) * c_weight),  # we adjust the weight
+                                        }
+                                    )
+                    else:
+                        self.warn_or_stop(f"Unsupported choice command '{cmd}' at {msg_where}!")
+                else:
+                    expanded_choice_values.append(c)
+            return expanded_choice_values
+
+        def __get_choices_internal_select(
+            self,
+            options: dict | None,
+            choice_values: list[dict],
+            filter_specifier: Optional[list[list[str]]] = None,
+            wildcard_key: str = None,
+        ) -> tuple[str, list[str], str, str]:
+            """
+            Select choices based on the options.
+
+            Args:
+                options (dict): The object representing the options construct.
+                choice_values (list[dict]): A list of choice objects.
+                filter_specifier (list[list[str]]): The filter specifier.
+                wildcard_key (str): The wildcard key if it is a wildcard.
+
+            Returns:
+                tuple: A tuple containing the prefix, selected choices, separator and suffix
+            """
+            seen_wildcards_len = len(self.__seen_wildcards)
+            if options is None:
+                options = {}
+            sampler: str = options.get("sampler", "~")
+            repeating: bool = options.get("repeating", False)
+            optional: bool = options.get("optional", False)
+            if "count" in options:
+                from_value = options["count"]
+                to_value = from_value
+            else:
+                from_value: int = options.get("from", 1)
+                to_value: int = options.get("to", 1)
+            separator: str = options.get("separator", self.__ppp.wil_choice_separator)
+            msg_where = f"wildcard '{wildcard_key}'" if wildcard_key else "choices"
+            if sampler != "~":
+                self.warn_or_stop(f"Unsupported sampler '{sampler}' at {msg_where} options!")
+                sampler = "~"
+            expanded_choice_values = self.__get_choices_internal_get(choice_values, filter_specifier, wildcard_key)
             available_choices: list[dict] = []
             weights = []
             included_choices = 0
             excluded_choices = 0
             excluded_weights_sum = 0
-            for i, c in enumerate(filtered_choice_values):
+            for i, c in enumerate(expanded_choice_values):
                 c["choice_index"] = i  # we index them to later sort the results
                 weight = float(c.get("weight", 1.0))
                 condition = c.get("if", None)
@@ -1833,8 +1896,15 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                     suffix = " " + suffix
                 # remove comments
                 results = [re.sub(r"\s*#[^\n]*(?:\n|$)", "", r, flags=re.DOTALL) for r in selected_choices_text]
-                return (prefix, results, separator, suffix)
-            return ("", [], separator, "")
+            else:
+                prefix = ""
+                suffix = ""
+                results = []
+            if self.__ppp.debug_level == DEBUG_LEVEL.full:
+                list_unseen = [f"'{x}'" for x in self.__seen_wildcards[seen_wildcards_len:]]
+                self.__ppp.logger.debug(f"Unseen wildcards: {', '.join(list_unseen)}")
+            self.__seen_wildcards = self.__seen_wildcards[:seen_wildcards_len]
+            return (prefix, results, separator, suffix)
 
         def __get_choices(
             self,
@@ -1843,7 +1913,7 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
             filter_specifier: Optional[list[list[str]]] = None,
             wildcard_key: str = None,
         ) -> str:
-            r = self.__get_choices_internal(options, choice_values, filter_specifier, wildcard_key)
+            r = self.__get_choices_internal_select(options, choice_values, filter_specifier, wildcard_key)
             if r[1]:
                 return r[0] + r[2].join(r[1]) + r[3]
             return ""
@@ -1901,23 +1971,26 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                 dict: The converted choice.
             """
             choice_dict = {}
-            c_label_obj = choice.children[0]
+            choice_dict["command"] = choice.children[0] is not None
+            c_label_obj = choice.children[1]
             choice_dict["labels"] = (
                 [x.value.lower() for x in c_label_obj.children[1:-1]]  # should be a token
                 if c_label_obj is not None
                 else []
             )
-            choice_dict["weight"] = float(choice.children[1].children[0]) if choice.children[1] is not None else 1.0
-            choice_dict["if"] = choice.children[2].children[0] if choice.children[2] is not None else None
-            choice_dict["content"] = choice.children[3]
+            choice_dict["weight"] = float(choice.children[2].children[0]) if choice.children[2] is not None else 1.0
+            choice_dict["if"] = choice.children[3].children[0] if choice.children[3] is not None else None
+            choice_dict["content"] = choice.children[-1]
             return choice_dict
 
-        def __check_wildcard_initialization(self, wildcard: PPPWildcard):
+        def __check_wildcard_initialization(self, wildcard: PPPWildcard) -> tuple[dict | None, list[dict] | None]:
             """
             Initializes a wildcard if it hasn't been yet.
 
             Args:
                 wildcard (PPPWildcard): The wildcard to check.
+            Returns:
+                tuple: A tuple containing the options and choice values of the wildcard.
             """
             choice_values = wildcard.choices
             options = wildcard.options
@@ -2032,9 +2105,10 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
             Process a wildcard construct in the tree.
             """
             t1 = time.monotonic_ns()
+            seen_wildcards_len = len(self.__seen_wildcards)
             start_result = self.result
             applied_options = self.__convert_choices_options(tree.children[0])
-            wildcard_key: str = tree.children[1].value  # should be a token
+            wildcard_key: str = self.__visit(tree.children[1], False, True)
             wc = self.__get_original_node_content(tree, f"?__{wildcard_key}__")
             if self.__ppp.wil_process_wildcards:
                 if self.__ppp.debug_level == DEBUG_LEVEL.full:
@@ -2054,7 +2128,8 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                         and filter_object.children[1] is not None
                         and "^" in filter_object.children[1]
                     ):
-                        filter_specifier = self.__wildcard_filters.get(filter_object.children[2].value, None)
+                        filter_wildcard_key = self.__visit(filter_object.children[2], False, True)
+                        filter_specifier = self.__wildcard_filters.get(filter_wildcard_key, None)
                         if self.__ppp.debug_level == DEBUG_LEVEL.full:
                             self.__ppp.logger.debug("Filtering choices with inherited filter")
                     else:
@@ -2093,6 +2168,14 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                         t2 = time.monotonic_ns()
                         self.__debug_end("wildcard", start_result, t2 - t1, wc)
                         return
+                    if wildcard.key in self.__seen_wildcards:
+                        self.warn_or_stop(
+                            f"Circular reference detected with wildcard '{self.__seen_wildcards[-1]}' (chain starts at '{self.__seen_wildcards[0]}')!"
+                        )
+                        continue
+                    self.__seen_wildcards.append(wildcard.key)
+                    if self.__ppp.debug_level == DEBUG_LEVEL.full:
+                        self.__ppp.logger.debug(f"Seen wildcard '{wildcard.key}'")
                     (options, choice_values) = self.__check_wildcard_initialization(wildcard)
                     if options is not None:
                         if applied_options is None:
@@ -2111,6 +2194,10 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
             elif self.__ppp.wil_ifwildcards != self.__ppp.IFWILDCARDS_CHOICES.remove:
                 self.detectedWildcards.append(wc)
                 self.result += wc
+            if self.__ppp.debug_level == DEBUG_LEVEL.full:
+                list_unseen = [f"'{x}'" for x in self.__seen_wildcards[seen_wildcards_len:]]
+                self.__ppp.logger.debug(f"Unseen wildcards: {', '.join(list_unseen)}")
+            self.__seen_wildcards = self.__seen_wildcards[:seen_wildcards_len]
             t2 = time.monotonic_ns()
             self.__debug_end("wildcard", start_result, t2 - t1, f"'{wc}'")
 

ppp_comfyui.py

@@ -2,7 +2,7 @@ import os
 
 # pylint: disable=import-error
 import folder_paths  # type: ignore
-import nodes
+import nodes  # type: ignore
 
 from .ppp import PromptPostProcessor
 from .ppp_hosts import SUPPORTED_APPS

ppp_enmappings.py

@@ -43,7 +43,9 @@ class PPPENMapping:
         self.kind: str = kind
         self.name: str = name
         self.variants: list[PPPENMappingVariant] = [
-            PPPENMappingVariant(**{**{"condition": None, "name": None, "parameters": None, "triggers": None, "weight": 1.0}, **v})
+            PPPENMappingVariant(
+                **{**{"condition": None, "name": None, "parameters": None, "triggers": None, "weight": 1.0}, **v}
+            )
             for v in variants
         ]
 
@@ -72,13 +74,17 @@ class PPPExtraNetworkMappings:
         self.__enmappings_folders = []
         self.__enmappings_files = {}
         self.extranetwork_mappings: dict[str, PPPENMapping] = {}
+        self.cached_mappings = {}
 
     def __hash__(self) -> int:
         return hash(deep_freeze(self.extranetwork_mappings))
 
     def __sizeof__(self):
         return (
-            self.extranetwork_mappings.__sizeof__() + self.__enmappings_folders.__sizeof__() + self.__enmappings_files.__sizeof__()
+            self.extranetwork_mappings.__sizeof__()
+            + self.__enmappings_folders.__sizeof__()
+            + self.__enmappings_files.__sizeof__()
+            + self.cached_mappings.__sizeof__()
         )
 
     def refresh_extranetwork_mappings(
@@ -92,6 +98,7 @@ class PPPExtraNetworkMappings:
         # if self.__debug_level != DEBUG_LEVEL.none:
         #     self.__logger.info("Refreshing extra network mappings...")
         # t1 = time.monotonic_ns()
+        self.cached_mappings = {}
         for fullpath in list(self.__enmappings_files.keys()):
             if fullpath != self.LOCALINPUT_FILENAME:
                 path = os.path.dirname(fullpath)
@@ -233,9 +240,17 @@ class PPPExtraNetworkMappings:
             full_path (str): The path to the file.
             base (str): The base path for the extra network mappings.
         """
-        with open(full_path, "r", encoding="utf-8") as file:
-            content = yaml.safe_load(file)
-        self.__add_extranetwork_mapping(content, full_path)
+        try:
+            try:
+                with open(full_path, "r", encoding="utf-8") as file:
+                    content = yaml.safe_load(file)
+            except:  # pylint: disable=bare-except
+                self.__logger.warning(f"Could not read file '{full_path}' with utf-8 encoding, trying windows-1252...")
+                with open(full_path, "r", encoding="windows-1252") as file:
+                    content = yaml.safe_load(file)
+            self.__add_extranetwork_mapping(content, full_path)
+        except Exception as e:  # pylint: disable=broad-except
+            self.__logger.error(f"Error reading extra network mappings from file '{full_path}': {e}")
 
     def __get_extranetwork_mappings_in_directory(self, directory: str):
         """

ppp_logging.py

@@ -53,13 +53,14 @@ class PromptPostProcessorLogFactory:  # pylint: disable=too-few-public-methods
             colored_record.levelname = f"{seq}{levelname:8s}{self.COLORS['RESET']}"
             return super().format(colored_record)
 
-    def __init__(self, app: SUPPORTED_APPS = None):  # pylint: disable=unused-argument
+    def __init__(self, filename = None, app: SUPPORTED_APPS = None):  # pylint: disable=unused-argument
         """
         Initializes the PromptPostProcessor class.
 
         This method sets up the logger for the PromptPostProcessor class and configures its log level and handlers.
 
         Args:
+            filename (str, optional): The name of the file to log to. Defaults to None.
             app (SUPPORTED_APPS): The application for which the logger is being created.
 
         Returns:
@@ -71,6 +72,10 @@ class PromptPostProcessorLogFactory:  # pylint: disable=too-few-public-methods
             handler = logging.StreamHandler(sys.stdout)
             handler.setFormatter(self.ColoredFormatter("%(asctime)s %(levelname)s %(message)s"))
             ppplog.addHandler(handler)
+            if filename is not None:
+                file_handler = logging.FileHandler(filename, encoding="utf-8")
+                file_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s"))
+                ppplog.addHandler(file_handler)
         ppplog.setLevel(logging.DEBUG)
         self.log = PromptPostProcessorLogCustomAdapter(ppplog)
 

ppp_wildcards.py

@@ -66,7 +66,10 @@ class PPPWildcards:
         return self.wildcards.__sizeof__() + self.__wildcards_folders.__sizeof__() + self.__wildcard_files.__sizeof__()
 
     def refresh_wildcards(
-        self, debug_level: DEBUG_LEVEL, wildcards_folders: Optional[list[str]], wildcards_input: str = None
+        self,
+        debug_level: DEBUG_LEVEL,
+        wildcards_folders: Optional[list[str]],
+        wildcards_input: str = None,
     ):
         """
         Initialize the wildcards.
@@ -79,10 +82,19 @@ class PPPWildcards:
         for fullpath in list(self.__wildcard_files.keys()):
             if fullpath != self.LOCALINPUT_FILENAME:
                 path = os.path.dirname(fullpath)
-                if not os.path.exists(fullpath) or not any(
-                    os.path.commonpath([path, folder]) == folder for folder in self.__wildcards_folders
-                ):
+                if not os.path.exists(fullpath):
                     self.__remove_wildcards_from_path(fullpath)
+                else:
+                    a = False
+                    for folder in self.__wildcards_folders:
+                        try:
+                            if os.path.commonpath([folder, path]) == folder:
+                                a = True
+                                break
+                        except ValueError:
+                            pass
+                    if not a:
+                        self.__remove_wildcards_from_path(fullpath)
             elif wildcards_input is None:
                 self.__remove_wildcards_from_path(fullpath)
         if wildcards_folders is not None or wildcards_input is not None:
@@ -160,9 +172,9 @@ class PPPWildcards:
         last_modified_cached = self.__wildcard_files.get(full_path, None)  # a time or a hash
         if debug and last_modified_cached is not None and self.__debug_level != DEBUG_LEVEL.none:
             if full_path == self.LOCALINPUT_FILENAME:
-                self.__logger.debug("Removing wildcards from input")
+                self.__logger.debug("Removing from memory wildcards from input")
             else:
-                self.__logger.debug(f"Removing wildcards from file: {full_path}")
+                self.__logger.debug(f"Removing from memory wildcards from file: {full_path}")
         if full_path in self.__wildcard_files.keys():
             del self.__wildcard_files[full_path]
         for key in list(self.wildcards.keys()):
@@ -177,22 +189,25 @@ class PPPWildcards:
             base (str): The base path for the wildcards.
             full_path (str): The path to the file.
         """
-        last_modified = os.path.getmtime(full_path)
-        last_modified_cached = self.__wildcard_files.get(full_path, None)
-        if last_modified_cached is not None and last_modified == self.__wildcard_files[full_path]:
-            return
-        filename = os.path.basename(full_path)
-        _, extension = os.path.splitext(filename)
-        if extension not in (".txt", ".json", ".yaml", ".yml"):
-            return
-        self.__remove_wildcards_from_path(full_path, False)
-        if last_modified_cached is not None and self.__debug_level != DEBUG_LEVEL.none:
-            self.__logger.debug(f"Updating wildcards from file: {full_path}")
-        if extension == ".txt":
-            self.__get_wildcards_in_text_file(full_path, base)
-        elif extension in (".json", ".yaml", ".yml"):
-            self.__get_wildcards_in_structured_file(full_path, base)
-        self.__wildcard_files[full_path] = last_modified
+        try:
+            last_modified = os.path.getmtime(full_path)
+            last_modified_cached = self.__wildcard_files.get(full_path, None)
+            if last_modified_cached is not None and last_modified == self.__wildcard_files[full_path]:
+                return
+            filename = os.path.basename(full_path)
+            _, extension = os.path.splitext(filename)
+            if extension not in (".txt", ".json", ".yaml", ".yml"):
+                return
+            self.__remove_wildcards_from_path(full_path, False)
+            if last_modified_cached is not None and self.__debug_level != DEBUG_LEVEL.none:
+                self.__logger.debug(f"Updating wildcards from file: {full_path}")
+            if extension == ".txt":
+                self.__get_wildcards_in_text_file(full_path, base)
+            elif extension in (".json", ".yaml", ".yml"):
+                self.__get_wildcards_in_structured_file(full_path, base)
+            self.__wildcard_files[full_path] = last_modified
+        except Exception as e:  # pylint: disable=broad-except
+            self.__logger.error(f"Error reading wildcard file '{full_path}': {e}")
 
     def __get_wildcards_in_input(self, wildcards_input: str):
         """
@@ -201,23 +216,26 @@ class PPPWildcards:
         Args:
             wildcards_input (str): The input string containing wildcards in json or yaml format.
         """
-        new_h = hash(wildcards_input)
-        h = self.__wildcard_files.get(self.LOCALINPUT_FILENAME, None)
-        if h == new_h:
-            return
-        self.__remove_wildcards_from_path(self.LOCALINPUT_FILENAME, False)
-        if h is not None and self.__debug_level != DEBUG_LEVEL.none:
-            self.__logger.debug("Updating wildcards from input")
-        wildcards_input = wildcards_input.strip()
-        if wildcards_input != "":
-            try:
-                content = yaml.safe_load(wildcards_input)
-            except yaml.YAMLError as e:
-                self.__logger.warning(f"Invalid format for input wildcards: {e}")
+        try:
+            new_h = hash(wildcards_input)
+            h = self.__wildcard_files.get(self.LOCALINPUT_FILENAME, None)
+            if h == new_h:
                 return
-            if content is not None:
-                self.__add_wildcard(content, self.LOCALINPUT_FILENAME, [self.LOCALINPUT_FILENAME])
-        self.__wildcard_files[self.LOCALINPUT_FILENAME] = new_h
+            self.__remove_wildcards_from_path(self.LOCALINPUT_FILENAME, False)
+            if h is not None and self.__debug_level != DEBUG_LEVEL.none:
+                self.__logger.debug("Updating wildcards from input")
+            wildcards_input = wildcards_input.strip()
+            if wildcards_input != "":
+                try:
+                    content = yaml.safe_load(wildcards_input)
+                except yaml.YAMLError as e:
+                    self.__logger.warning(f"Invalid format for input wildcards: {e}")
+                    return
+                if content is not None:
+                    self.__add_wildcard(content, self.LOCALINPUT_FILENAME, [self.LOCALINPUT_FILENAME])
+            self.__wildcard_files[self.LOCALINPUT_FILENAME] = new_h
+        except Exception as e:  # pylint: disable=broad-except
+            self.__logger.error(f"Error reading wildcards input: {e}")
 
     def is_dict_choices_options(self, d: dict) -> bool:
         """
@@ -230,7 +248,8 @@ class PPPWildcards:
             bool: Whether the dictionary is a valid choices options dictionary or not.
         """
         return all(
-            k in ["sampler", "repeating", "optional", "count", "from", "to", "prefix", "suffix", "separator"] for k in d.keys()
+            k in ["sampler", "repeating", "optional", "count", "from", "to", "prefix", "suffix", "separator"]
+            for k in d.keys()
         )
 
     def is_dict_choice_options(self, d: dict) -> bool:
@@ -243,7 +262,7 @@ class PPPWildcards:
         Returns:
             bool: Whether the dictionary is a valid choice options dictionary or not.
         """
-        return all(k in ["labels", "weight", "if", "content", "text"] for k in d.keys())
+        return all(k in ["command", "labels", "weight", "if", "content", "text"] for k in d.keys())
 
     def __get_choices(self, obj: object, full_path: str, key_parts: list[str]) -> list:
         """
@@ -358,6 +377,8 @@ class PPPWildcards:
                     choices = self.__get_choices(obj, full_path, tmp_key_parts)
                     if choices is None:
                         self.__logger.warning(f"Invalid wildcard '{fullkey}' in file '{full_path}'!")
+                    elif fullkey.startswith("_"):
+                        self.__logger.warning(f"Invalid wildcard name '{fullkey}' in file '{full_path}'! (cannot start with underscore)")
                     else:
                         self.wildcards[fullkey] = PPPWildcard(full_path, fullkey, choices)
             return
@@ -377,6 +398,8 @@ class PPPWildcards:
             choices = self.__get_choices(content, full_path, key_parts)
             if choices is None:
                 self.__logger.warning(f"Invalid wildcard '{fullkey}' in file '{full_path}'!")
+            elif fullkey.startswith("_"):
+                self.__logger.warning(f"Invalid wildcard name '{fullkey}' in file '{full_path}'! (cannot start with underscore)")
             else:
                 self.wildcards[fullkey] = PPPWildcard(full_path, fullkey, choices)
 
@@ -390,8 +413,13 @@ class PPPWildcards:
         """
         external_key: str = os.path.relpath(os.path.splitext(full_path)[0], base)
         external_key_parts = external_key.split(os.sep)
-        with open(full_path, "r", encoding="utf-8") as file:
-            content = yaml.safe_load(file)
+        try:
+            with open(full_path, "r", encoding="utf-8") as file:
+                content = yaml.safe_load(file)
+        except:  # pylint: disable=bare-except
+            self.__logger.warning(f"Could not read file '{full_path}' with utf-8 encoding, trying windows-1252...")
+            with open(full_path, "r", encoding="windows-1252") as file:
+                content = yaml.safe_load(file)
         self.__add_wildcard(content, full_path, external_key_parts)
 
     def __get_wildcards_in_text_file(self, full_path, base):
@@ -404,8 +432,13 @@ class PPPWildcards:
         """
         external_key: str = os.path.relpath(os.path.splitext(full_path)[0], base)
         external_key_parts = external_key.split(os.sep)
-        with open(full_path, "r", encoding="utf-8") as file:
-            text_content = map(lambda x: x.strip("\n\r"), file.readlines())
+        try:
+            with open(full_path, "r", encoding="utf-8") as file:
+                text_content = map(lambda x: x.strip("\n\r"), file.readlines())
+        except:  # pylint: disable=bare-except
+            self.__logger.warning(f"Could not read file '{full_path}' with utf-8 encoding, trying windows-1252...")
+            with open(full_path, "r", encoding="windows-1252") as file:
+                text_content = map(lambda x: x.strip("\n\r"), file.readlines())
         text_content = list(filter(lambda x: x.strip() != "" and not x.strip().startswith("#"), text_content))
         text_content = [x.split("#")[0].rstrip() if len(x.split("#")) > 1 else x for x in text_content]
         self.__add_wildcard(text_content, full_path, external_key_parts)

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "sd-webui-prompt-postprocessor"
 description = "Stable Diffusion WebUI & ComfyUI extension to post-process the prompt, including sending content from the prompt to the negative prompt and wildcards."
-version = "2.12.0"
+version = "2.13.0"
 license = { file = "LICENSE.txt" }
 dependencies = ["lark", "numpy", "pyyaml"]
 

scripts/ppp_script.py

@@ -297,7 +297,10 @@ class PromptPostProcessorA1111Script(scripts.Script):
         ]
         en_mappings_folders = getattr(opts, "ppp_en_mappingsfolders", "")
         if en_mappings_folders == "":
-            en_mappings_folders = os.getenv("EXTRANETWORKMAPPINGS_DIR", PPPExtraNetworkMappings.DEFAULT_ENMAPPINGS_FOLDER)
+            en_mappings_folders = os.getenv(
+                "EXTRANETWORKMAPPINGS_DIR",
+                PPPExtraNetworkMappings.DEFAULT_ENMAPPINGS_FOLDER,
+            )
         enmappings_folders = [
             (f if os.path.isabs(f) else os.path.abspath(os.path.join(models_path, f)))
             for f in en_mappings_folders.split(",")

tests/tests.py

@@ -2,6 +2,7 @@ import os
 import logging
 from typing import NamedTuple, Optional
 import unittest
+import datetime
 
 from ppp_enmappings import PPPExtraNetworkMappings  # pylint: disable=import-error
 from ppp_wildcards import PPPWildcards  # pylint: disable=import-error
@@ -19,11 +20,23 @@ class TestPromptPostProcessorBase(unittest.TestCase):
     A test case class for testing the PromptPostProcessor class.
     """
 
-    def setUp(self):
+    def setUp(self, enable_file_logging=False):
         """
         Set up the test case by initializing the necessary objects and configurations.
+
+        Args:
+            enable_file_logging (bool): Whether to enable logging to a file. Defaults to True.
         """
-        self.lf = PromptPostProcessorLogFactory()
+        self.enable_file_logging = enable_file_logging
+        test_name = self.id().split(".")[-1]  # Extract the test method name
+        timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
+
+        if self.enable_file_logging:
+            log_filename = f"tests/logs/{test_name}_{timestamp}.log"
+        else:
+            log_filename = None  # Disable file logging
+
+        self.lf = PromptPostProcessorLogFactory(log_filename, app=None)
         self.ppp_logger = self.lf.log
         self.ppp_logger.setLevel(logging.DEBUG)
         self.grammar_content = None
@@ -183,13 +196,16 @@ class TestPromptPostProcessorBase(unittest.TestCase):
 
 class TestPromptPostProcessor(TestPromptPostProcessorBase):
 
+    def setUp(self):  # pylint: disable=arguments-differ
+        super().setUp(enable_file_logging=False)
+
     # Send To Negative tests
 
     def test_stn_simple(self):  # negtags with different parameters and separations
         self.process(
             PromptPair(
                 "flowers<ppp:stn>red<ppp:/stn>, <ppp:stn s>green<ppp:/stn>, <ppp:stn e>blue<ppp:/stn><ppp:stn p0>yellow<ppp:/stn>, <ppp:stn p1>purple<ppp:/stn><ppp:stn p2>black<ppp:/stn>",
-                "<ppp:stn i0>normal quality<ppp:stn i1>, worse quality<ppp:stn i2>",
+                "<ppp:stn i0/>normal quality<ppp:stn i1>, worse quality<ppp:stn i2/>",
             ),
             PromptPair("flowers", "red, green, yellow, normal quality, purple, worse quality, black, blue"),
         )
@@ -198,7 +214,7 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
         self.process(
             PromptPair(
                 "<ppp:stn>red<ppp:/stn> ((<ppp:stn s>pink<ppp:/stn>)), flowers <ppp:stn e>purple<ppp:/stn>, <ppp:stn p0>mauve<ppp:/stn><ppp:stn e>blue<ppp:/stn>, <ppp:stn p0>yellow<ppp:/stn> <ppp:stn p1>green<ppp:/stn>",
-                "normal quality, <ppp:stn i0>, bad quality<ppp:stn i1>, worse quality",
+                "normal quality, <ppp:stn i0/>, bad quality<ppp:stn i1/>, worse quality",
             ),
             PromptPair(
                 "flowers",
@@ -210,7 +226,7 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
         self.process(
             PromptPair(
                 "<ppp:stn>red<ppp:/stn> ((<ppp:stn s>pink<ppp:/stn>)), flowers <ppp:stn e>purple<ppp:/stn>, <ppp:stn p0>mauve<ppp:/stn><ppp:stn e>blue<ppp:/stn>, <ppp:stn p0>yellow<ppp:/stn> <ppp:stn p1>green<ppp:/stn>",
-                "normal quality, <ppp:stn i0>, bad quality<ppp:stn i1>, worse quality",
+                "normal quality, <ppp:stn i0/>, bad quality<ppp:stn i1/>, worse quality",
             ),
             PromptPair(
                 " (()), flowers , ,  ",
@@ -264,7 +280,7 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
         self.process(
             PromptPair(
                 "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loratrigger <lora:xxx:1> AND AND hypernettrigger <hypernet:yyy>:0.3",
-                "normal quality, <ppp:stn i0>",
+                "normal quality, <ppp:stn i0/>",
             ),
             PromptPair(
                 "this \\(is\\): a (([complex|simple|regular] test)(test:2):1.5)\nBREAK with [abc:def:5]:0.5 AND loratrigger <lora:xxx:1> AND hypernettrigger <hypernet:yyy>:0.3",
@@ -276,7 +292,7 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
         self.process(
             PromptPair(
                 "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loratrigger <lora:xxx:1> AND AND hypernettrigger <hypernet:yyy>:0.3",
-                "normal quality, <ppp:stn i0>",
+                "normal quality, <ppp:stn i0/>",
             ),
             PromptPair(
                 "this \\(is\\): a (([complex|simple|regular] test)(test:2):1.5)\nBREAK with [abc:def:5]:0.5 AND loratrigger <lora:xxx:1> AND hypernettrigger <hypernet:yyy>:0.3",
@@ -420,7 +436,7 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
         self.process(
             PromptPair(
                 "[<ppp:stn>neg5<ppp:/stn>] this \\(is\\): a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5]:0.5 AND loratrigger <lora:xxx:1> AND AND hypernettrigger <hypernet:yyy>:0.3",
-                "normal quality, <ppp:stn i0>",
+                "normal quality, <ppp:stn i0/>",
             ),
             PromptPair(
                 "this \\(is\\): a (([complex|simple|regular] test)(test:2):1.5)\nBREAK with [abc:def:5]:0.5 AND loratrigger <lora:xxx:1> AND hypernettrigger <hypernet:yyy>:0.3",
@@ -482,7 +498,7 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
     def test_cmd_set_if_echo_nested(self):  # nested set, if and echo commands
         self.process(
             PromptPair(
-                "<ppp:set v1>1<ppp:/set><ppp:if v1 gt 0><ppp:set v2>OK<ppp:/set><ppp:/if><ppp:if v2 eq 'OK'><ppp:echo v2><ppp:else>not OK<ppp:/if> <ppp:echo v2>NOK<ppp:/echo> <ppp:echo v3>OK<ppp:/echo>",
+                "<ppp:set v1>1<ppp:/set><ppp:if v1 gt 0><ppp:set v2>OK<ppp:/set><ppp:/if><ppp:if v2 eq 'OK'><ppp:echo v2/><ppp:else>not OK<ppp:/if> <ppp:echo v2>NOK<ppp:/echo> <ppp:echo v3>OK<ppp:/echo>",
                 "",
             ),
             PromptPair("OK OK OK", ""),
@@ -646,6 +662,15 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
             ),
         )
 
+    def test_cmd_ext_map_notrigger(self):  # ext mapping, no trigger
+        self.process(
+            PromptPair(
+                "<ppp:ext $lora lora1/><ppp:ext $lora lora1>",
+                "",
+            ),
+            PromptPair("triggergeneric1, triggergeneric2, two, triggergeneric1, triggergeneric2, two", ""),
+        )
+
     def test_cmd_ext_map1(self):  # ext mapping, no lora
         self.process(
             PromptPair(
@@ -799,12 +824,12 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
 
     def test_ch_choicesinsidelora(self):  # simple choices inside a lora
         self.process(
-            PromptPair("<lora:test1:1><lora:test2:{0.2|0.5|0.7|1}>", ""),
-            PromptPair("<lora:test1:1><lora:test2:0.7>", ""),
+            PromptPair("<lora:test1:1><lora:test__other__name:1><lora:test2:{0.2|0.5|0.7|1}>", ""),
+            PromptPair("<lora:test1:1><lora:test__other__name:1><lora:test2:0.7>", ""),
             ppp=self.nocupppp,
         )
 
-    def test_ch_removelorawithchoices(self):  # remove lora with choices inside
+    def test_ch_removelorawithchoices(self):
         self.process(
             PromptPair("<lora:test1:1><lora:test2:{0.2|0.5|0.7|1}>", ""),
             PromptPair("", ""),
@@ -819,6 +844,13 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
             ),
         )
 
+    def test_ch_cmd_includewildcard(self):
+        self.process(
+            PromptPair("{ch_one|ch_two|%0.5::include yaml/wildcard1}", ""),
+            PromptPair("ch_two", ""),
+            ppp=self.nocupppp,
+        )
+
     # Wildcards tests
 
     def test_wc_ignore(self):  # wildcards with ignore option
@@ -844,7 +876,7 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
         self.process(
             PromptPair(
                 "[<ppp:stn>neg5<ppp:/stn>] this is: __bad_wildcard__ a (([complex|simple<ppp:stn>neg6<ppp:/stn>|regular] test<ppp:stn>neg1<ppp:/stn>)(test:2.0):1.5) \nBREAK, BREAK with [abc<ppp:stn>neg4<ppp:/stn>:def<ppp:stn p0>neg2(neg3:1.6)<ppp:/stn>:5] <lora:xxx:1>",
-                "normal quality, <ppp:stn i0> {option1|option2}",
+                "normal quality, <ppp:stn i0/> {option1|option2}",
             ),
             PromptPair(
                 "this is: a (([complex|simple|regular] test)(test:2):1.5)\nBREAK with [abc:def:5]<lora:xxx:1>",
@@ -926,6 +958,13 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
             ),
         )
 
+    def test_wc_invalid_name(self):
+        self.process(
+            PromptPair("the choices are: ___invalid__", ""),
+            PromptPair("the choices are: ___invalid__", ""),
+            ppp=self.nocupppp,
+        )
+
     def test_wc_wildcard1a_text(self):  # simple text wildcard
         self.process(
             PromptPair("the choices are: __text/wildcard1__", ""),
@@ -1155,6 +1194,39 @@ class TestPromptPostProcessor(TestPromptPostProcessorBase):
             ppp=self.nocupppp,
         )
 
+    def test_wc_circular(self):  # wildcard circular reference
+        self.process(
+            PromptPair("the choices are: __yaml/circular1__", ""),
+            PromptPair("", ""),
+            ppp=self.nocupppp,
+            interrupted=True,
+        )
+
+    def test_wc_including(self):  # wildcard including another wildcard
+        self.process(
+            PromptPair("the choices are: __yaml/including__", ""),
+            PromptPair("the choices are: choice4", ""),
+            ppp=self.nocupppp,
+        )
+
+    def test_wc_circular_including(self):  # wildcard including another wildcard in a circular reference
+        self.process(
+            PromptPair("the choices are: __yaml/including1__", ""),
+            PromptPair("", ""),
+            ppp=self.nocupppp,
+            interrupted=True,
+        )
+
+    def test_wc_dynamicwildcard(self):  # wildcard built from variables
+        self.process(
+            PromptPair(
+                "the choices are: ${x={1|2|3}}${w=yaml/wildcard${x}}__yaml/wildcard${x}__ __${w}__ __<ppp:echo w/>__",
+                "",
+            ),
+            PromptPair("the choices are: choice1-choice3-choice1 choice3- choice2 - choice2  choice3", ""),
+            ppp=self.nocupppp,
+        )
+
     # Model variants tests
 
     def test_variants(self):

tests/wildcards/_invalid.txt

@@ -0,0 +1,3 @@
+# invalid wildcard name
+choice1
+choice2

tests/wildcards/test.yaml

@@ -76,3 +76,29 @@ yaml:
     - if false::3
     - if _sd in ("test1", "test2")::4
     - if (false or false)::5
+
+  circular1:
+    - 5::__yaml/circular2__
+    - choice1
+    - choice2
+
+  circular2:
+    - choice3
+    - choice4
+    - 5::__yaml/circular1__
+
+  including:
+    - "%0.5::include yaml/wildcard1"
+    - choice4
+    - choice5
+    - "%0.2::include yaml/wildcard7"
+
+  including1:
+    - choice1
+    - choice2
+    - "%5::include yaml/including2"
+
+  including2:
+    - choice3
+    - choice4
+    - "%5::include yaml/including1"

tests/wildcards2/text/wildcard3.txt

@@ -1,2 +1,3 @@
-# wildcard3
-__2$$,$$text/wildcard2__
+# wildcard3 (with windows-1252 encoding)
+__2$$,$$text/wildcard2__
+0::áüç