* Support for attention merge at the wildcard/choice boundary.

Co-authored-by: Copilot <copilot@github.com>

docs/CONFIG.md

@@ -118,7 +118,7 @@ Options for cleanup processing, in case you want to change them from the default
 * **around_ands**: Removes consecutive ANDs and unnecessary commas and space around them.
 * **ands_with_eol**: Add a newline before ANDs.
 * **around_extranetwork_tags**: Removes spaces around extra network tags.
-* **merge_attention**: It merges attention modifiers when possible (merges into one, multiplying their values). Only merges individually nested modifiers.
+* **merge_attention**: It merges attention modifiers when possible (merges into one, multiplying their values). Only merges individually nested attention (even through choice/wildcard boundaries).
 * **remove_extranetwork_tags**: Removes all extra network tags.
 
 Please note that *ComfyUI* does not natively support the `BREAK` and `AND` constructs, but the related settings are kept in that UI in case you use a node that supports them and the extension is configured to allow them (see the configuration file below).
@@ -179,6 +179,6 @@ Options for extranetworks mapping, in case you want to change them from the defa
 * **Clean up around ANDs**: Removes consecutive ANDs and unnecessary commas and space around them.
 * **Use EOL instead of Space before ANDs**: Add a newline before ANDs.
 * **Clean up around extra network tags**: Removes spaces around extra network tags.
-* **Merge attention modifiers (weights) when possible**: It merges attention modifiers when possible (merges into one, multiplying their values). Only merges individually nested modifiers.
+* **Merge attention modifiers (weights) when possible**: It merges attention modifiers when possible (merges into one, multiplying their values). Only merges individually nested attention (even through choice/wildcard boundaries).
 * **Remove extra spaces**: Removes other unnecessary spaces.
 * **Remove extra network tags**: Removes all extra network tags.

docs/COOKBOOK.md

@@ -411,11 +411,11 @@ Only choices labelled `fantasy` across all matched files are eligible.
 
 Note: if no parameters are specified in the glob call, the parameters from the first matching file that defines them (sorted by key) are used. To avoid that, specify parameters explicitly in the call.
 
-## Prefix/suffix on wildcard parameters
+## Prefix/suffix or container on wildcard parameters
 
-Using the object format for wildcard parameters you can add a prefix and/or suffix that wrap every result. This is cleaner than repeating the wrapper in each choice.
+Using the object format for wildcard parameters you can set a prefix and/or suffix, or a container, that wraps the result. This is cleaner than repeating the wrapper in each choice.
 
-Without prefix/suffix, every choice needs to repeat the attention modifier:
+In this example, without prefix/suffix, every choice needs to repeat the attention modifier:
 
 ```yaml
 qualities:
@@ -434,7 +434,17 @@ qualities:
     - "intricate details"
 ```
 
-The prefix and suffix are added around the joined result (including the separator when multiple choices are selected). They can themselves contain constructs.
+The prefix and suffix are added around the wildcard's result. They can themselves contain constructs.
+
+Another way is with the container property, which does the same in a slightly more flexible way:
+
+```yaml
+qualities:
+    - { container: "(${__choices[]}:1.3)" }  # parameters line
+    - "ultra detailed"
+    - "highly detailed"
+    - "intricate details"
+```
 
 ## `ifundefined` / `?=` for safe defaults
 

docs/SYNTAX.md

@@ -94,7 +94,7 @@ Wildcards cannot be used inside an extranetwork tag (because some LoRA names con
 
 The filter can be used to filter specific choices from the wildcard. The filtering works before applying the choice conditions (if any). The surrounding quotes can be single or double.
 
-The filter is a comma separated list of an integer/range (positional choice index, zero-based) or choice label. You can also compound them with `+`. That is, the comma separated items act as an OR and the `+` inside them as an AND. Using labels can simplify the definitions of complex wildcards where you want to have direct access to specific choices on occasion (you don't need to create wildcards for each individual choice).
+The filter is a comma separated list of an integer/range (positional choice index, zero-based) or choice label. You can also compound them with `+`. That is, the comma separated items act as an OR and the `+` inside them as an AND. Using labels can simplify the definitions of complex wildcards where you want to have direct access to specific choices on occasion (you don't need to create wildcards for each individual choice). You can use variables for individual labels (`${v}`) or a single variable for the whole filter (`${v}` or `${a[&',']}` or `${a[&'+']}`).
 
 There are some additional formats when using filters.
 
@@ -134,23 +134,36 @@ The best format is a yaml file with a dictionary of wildcards inside. An editor
 
 In a choice, the content after a `#`  is ignored.
 
-If the first choice follows the format of wildcard parameters (*including the final `$$`*), it will be used as default parameters for that wildcard (see examples in the tests folder).
+If the first choice follows the format of wildcard parameters (*including the final `$$`*), it will be used as default parameters for that wildcard (see examples in the tests folder). Unless the only property used is the wildcard description.
 
-The choices of the wildcard follow the same format as in the choices construct, or the object format of *Dynamic Prompts* (only in structured files). If using the object format for a choice you can use a new `if` property for the condition, and the `labels` property (an array of strings) and `command` property (a boolean) in addition to the standard `weight` and `text`/`content`.
+The choices of the wildcard follow the same format as in the choices construct, or the object format of *Dynamic Prompts* (only in structured files).
+
+If using the object format for a choice you can use the following in addition to the standard `weight` and `text`/`content`:
+
+* `if`: the condition (a string)
+* `labels`: list of labels (an array of strings)
+* `command`: indicates the content is a command (a boolean)
 
 ```yaml
 { command: false, labels: ["some_label"], weight: 2, if: "_is_pony", content: "the text" } # "text" property can be used instead of "content"
 ```
 
-Wildcard parameters in a json/yaml file can also be in object format, and support two additional properties, prefix and suffix:
+Wildcard parameters in a json/yaml file can also be in object format, and support some additional properties, that are not included in the string format:
+
+* `prefix`: content to prefix the list of choices
+* `suffix`: content to suffix the list of choices
+* `container`: includes the prefix, choices array variable, and suffix
 
 ```yaml
+{ sampler: "~", repeating: false, optional: false, from: 2, to: 3, description: "test wildcard", container: "prefix-${__choices[&'/']}-suffix" }
 { sampler: "~", repeating: false, optional: false, count: 2, description: "test wildcard", prefix: "prefix-", suffix: "-suffix", separator: "/" }
 { sampler: "~", repeating: false, optional: false, from: 2, to: 3, description: "test wildcard", prefix: "prefix-", suffix: "-suffix", separator: "/" }
 ```
 
 The prefix and suffix are added to the result along with the selected choices and separators. They can contain other constructs, but the separator can't.
 
+The container is a new option that replaces prefix/suffix/separator, and makes use of the recent support for array variables. Its value would be the concatenation of any prefix and/or suffix with the echoing (with the chosen separator) of a temporary `__choices[]` variable that holds the chosen values. This property is preferred over prefix/suffix/separator unless you only need the separator.
+
 It is recommended to use the object format for the wildcard parameters and for choices with complex options.
 
 If your first choice is interpreted as parameters, and you don't need parameters, you can avoid the problem by adding an empty parameters object `{}` as first choice.

ppp.py

@@ -1,4 +1,5 @@
 import dataclasses
+from enum import Enum
 import logging
 import os
 import re
@@ -269,6 +270,11 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
                     propagate_positions=True,
                     start="choicevalue",
                 ),
+                "wc_filter_or": lark.Lark(
+                    grammar_content_full,
+                    propagate_positions=True,
+                    start="wc_filter_or",
+                ),
             },
         )
         self.__init_sysvars()
@@ -578,6 +584,17 @@ class PromptPostProcessor:  # pylint: disable=too-few-public-methods,too-many-in
         """
         vs = self.state.variables
         vs.clear_system()
+
+        # Option related variables
+        for opt_name in self.defopt.keys():
+            opt_value = getattr(self.state.options, opt_name)
+            var_name = "_opt_" + opt_name
+            if isinstance(opt_value, (bool, str, int)):
+                vs.set_system(var_name, opt_value)
+            elif isinstance(opt_value, Enum):
+                vs.set_system(var_name, str(opt_value).split(".", 1)[-1])
+
+        # Model related variables
         sdchecks = {x: self.env_info.get("is_" + x, False) for x in self.known_models}
         sdchecks.update({"": True})
         model_name_val = next((k for k, v in sdchecks.items() if v), "")

ppp_enmappings.py

@@ -111,7 +111,7 @@ class PPPExtraNetworkMappings:
         if enmappings_folders is not None or enmappings_input is not None:
             if enmappings_folders is not None:
                 for f in self.__enmappings_folders:
-                    self.__get_extranetwork_mappings_in_directory(f)
+                    self.__get_extranetwork_mappings_in_path(f)
             if enmappings_input is not None:
                 self.__get_extranetwork_mappings_in_input(enmappings_input)
         else:
@@ -295,26 +295,26 @@ class PPPExtraNetworkMappings:
                 f"Error reading extra network mappings from file '{escape_single_quotes(full_path)}': {e}",
             )
 
-    def __get_extranetwork_mappings_in_directory(self, directory: str):
+    def __get_extranetwork_mappings_in_path(self, path: str):
         """
-        Get all extra network mappings in a directory.
+        Get all extra network mappings in a path.
 
         Args:
-            directory (str): The path to the directory.
+            path (str): The path (folder or file).
         """
-        if not os.path.exists(directory):
+        if not os.path.exists(path):
             log(
                 self.__logger,
                 self.__debug_level,
                 logging.WARNING,
-                f"Extra network mappings directory '{escape_single_quotes(directory)}' does not exist!",
+                f"Extra network mappings path '{escape_single_quotes(path)}' does not exist!",
             )
             return
-        for filename in os.listdir(directory):
-            full_path = os.path.abspath(os.path.join(directory, filename))
+        if os.path.isfile(path):
+            self.__get_extranetwork_mappings_in_file(path)
+            return
+        for filename in os.listdir(path):
+            full_path = os.path.abspath(os.path.join(path, filename))
             if os.path.basename(full_path).startswith("."):
                 continue
-            if os.path.isdir(full_path):
-                self.__get_extranetwork_mappings_in_directory(full_path)
-            elif os.path.isfile(full_path):
-                self.__get_extranetwork_mappings_in_file(full_path)
+            self.__get_extranetwork_mappings_in_path(full_path)

ppp_tree.py

@@ -933,6 +933,85 @@ class TreeProcessor(lark.visitors.Interpreter):
         t2 = time.monotonic_ns()
         self.__debug_end("alternate", start_result, t2 - t1)
 
+    @staticmethod
+    def _try_extract_attention(s: str) -> tuple[str, float] | None:
+        """
+        If s is entirely a single attention wrapper - (inner), (inner:W), or [inner] -
+        return (inner_content, weight). Otherwise return None.
+
+        Used for post-visit merging when a wildcard or choices construct expands to a
+        single attention that can be merged with an enclosing outer attention.
+        """
+        if not s:
+            return None
+        open_char = s[0]
+        if open_char == "(":
+            close_char = ")"
+        elif open_char == "[":
+            close_char = "]"
+        else:
+            return None
+        depth = 0
+        for i, c in enumerate(s):
+            if c == open_char:
+                depth += 1
+            elif c == close_char:
+                depth -= 1
+                if depth == 0:
+                    if i != len(s) - 1:
+                        return None  # wrapper closes before end of string -> multiple items
+                    break
+        else:
+            return None  # never fully closed
+        inner = s[1:-1]
+        if open_char == "[":
+            # Disambiguate from alternation [a|b] and scheduling [before:after:N].
+            # Both use [...] but are not attention constructs.
+            paren_depth = 0
+            bracket_depth = 0
+            top_level_pipes = 0
+            top_level_colons = 0
+            last_colon_pos = -1
+            for i, c in enumerate(inner):
+                if c == "(":
+                    paren_depth += 1
+                elif c == ")":
+                    paren_depth -= 1
+                elif c == "[":
+                    bracket_depth += 1
+                elif c == "]":
+                    bracket_depth -= 1
+                elif paren_depth == 0 and bracket_depth == 0:
+                    if c == "|":
+                        top_level_pipes += 1
+                    elif c == ":":
+                        top_level_colons += 1
+                        last_colon_pos = i
+            if top_level_pipes > 0:
+                return None  # alternation construct
+            if top_level_colons >= 2 and last_colon_pos >= 0:
+                try:
+                    float(inner[last_colon_pos + 1 :])
+                    return None  # scheduling construct: [before:after:N]
+                except ValueError:
+                    pass
+            return (inner, 0.9)
+        # Parenthesis form - scan backwards for a top-level :weight suffix
+        depth = 0
+        for i in range(len(inner) - 1, -1, -1):
+            c = inner[i]
+            if c in ")]":
+                depth += 1
+            elif c in "([":
+                depth -= 1
+            elif c == ":" and depth == 0:
+                try:
+                    w = float(inner[i + 1 :])
+                    return (inner[:i], w)
+                except ValueError:
+                    break
+        return (inner, 1.1)
+
     def attention(self, tree: lark.Tree):
         """
         Process a attention change construct in the tree and add it to the accumulated shell.
@@ -956,6 +1035,7 @@ class TreeProcessor(lark.visitors.Interpreter):
         self.log(logging.DEBUG, f"Shell attention with weight {weight}")
         current_tree = tree.children[0]
         if self.state.options.cup_merge_attention:
+            # we check while the children are attentions, in which case we merge the weights
             while isinstance(current_tree, lark.Tree) and current_tree.data == "attention":
                 # we merge the weights
                 if len(current_tree.children) == 2:
@@ -967,6 +1047,10 @@ class TreeProcessor(lark.visitors.Interpreter):
                 else:
                     inner_weight = 0.9
                 weight *= inner_weight
+                self.log(
+                    logging.DEBUG,
+                    f"Merging nested attention with weight {inner_weight}, cumulative weight now {weight}",
+                )
                 current_tree = current_tree.children[0]
             weight = math.floor(weight * 100) / 100  # we round to 2 decimals
             weight_str = f"{weight:.2f}".rstrip("0").rstrip(".")
@@ -1014,6 +1098,34 @@ class TreeProcessor(lark.visitors.Interpreter):
                 self.__result += starttag
                 self.__visit(current_tree)
                 endtag = f":{weight_str})"
+            # Post-visit merge: if the entire visited content is a single attention wrapper
+            # (e.g. from a wildcard or choices expansion), merge weights here.
+            # The static tree-walk above only covers direct attention children; this
+            # handles the case where the inner attention came from an expanded wildcard.
+            if self.state.options.cup_merge_attention:
+                visited_content = self.__result[len(start_result) + len(starttag) :]
+                merge = TreeProcessor._try_extract_attention(visited_content)
+                if merge is not None:
+                    inner_content, inner_weight = merge
+                    weight = math.floor(weight * inner_weight * 100) / 100
+                    self.log(
+                        logging.DEBUG,
+                        f"Merging nested attention with weight {inner_weight}, cumulative weight now {weight}",
+                    )
+                    weight_str = f"{weight:.2f}".rstrip("0").rstrip(".")
+                    if weight_str == "1.1":
+                        weight_kind = 2
+                        starttag = "("
+                        endtag = ")"
+                    elif weight_str == "0.9" and attention_processing != "parentheses":
+                        weight_kind = 1
+                        starttag = "["
+                        endtag = "]"
+                    else:
+                        weight_kind = 3
+                        starttag = "("
+                        endtag = f":{weight_str})"
+                    self.__result = start_result + starttag + inner_content
             if self.state.options.cup_empty_constructs and re.fullmatch(
                 re.escape(start_result + starttag) + r"\s*", self.__result
             ):
@@ -1231,6 +1343,11 @@ class TreeProcessor(lark.visitors.Interpreter):
                         f"Invalid use of starred value for '{escape_single_quotes(variable_name)}'! Starred values can only be assigned or added to unindexed array variables."
                     )
                     newvalue = ""
+                elif not isinstance(newvalue, list):
+                    self.warn_or_stop(
+                        f"Invalid starred value for '{escape_single_quotes(variable_name)}'! Starred values should be a list."
+                    )
+                    newvalue = ""
             if is_array:
                 if variable_specifier is not None:
                     # Accessing an existing index, we need to update the array
@@ -1644,7 +1761,7 @@ class TreeProcessor(lark.visitors.Interpreter):
         choice_values: list[dict],
         filter_specifier: Optional[list[list[str]]] = None,
         wildcard_key: str = None,
-    ) -> tuple[str, list[str], str, str]:
+    ) -> tuple[lark.Tree, list[str]]:
         """
         Select choices based on the options.
 
@@ -1655,7 +1772,7 @@ class TreeProcessor(lark.visitors.Interpreter):
             wildcard_key (str): The wildcard key if it is a wildcard.
 
         Returns:
-            tuple: A tuple containing the prefix, selected choices, separator and suffix
+            tuple[lark.Tree,list[str]]: The resulting container and list of chosen choices.
         """
         seen_wildcards_len = len(self.__seen_wildcards)
         if options is None:
@@ -1772,13 +1889,6 @@ class TreeProcessor(lark.visitors.Interpreter):
             if self.state.options.keep_choices_order:
                 selected_choices = sorted(selected_choices, key=lambda x: x["choice_index"])
             selected_choices_text = []
-            prefix: str = (
-                self.__visit(options.get("prefix", None), False, True)
-                if options.get("prefix", None) is not None
-                else ""
-            )
-            if prefix != "" and re.match(r"\w", prefix[-1]):
-                prefix += " "
             for i, c in enumerate(selected_choices):
                 t1 = time.monotonic_ns()
                 choice_content_obj = c.get("content", c.get("text", None))
@@ -1793,26 +1903,55 @@ class TreeProcessor(lark.visitors.Interpreter):
                     + textwrap.indent(re.sub(r"\n$", "", choice_content), "    "),
                 )
                 selected_choices_text.append(choice_content)
-            suffix: str = (
-                self.__visit(options.get("suffix", None), False, True)
-                if options.get("suffix", None) is not None
-                else ""
-            )
-            if suffix != "" and re.match(r"\w", suffix[0]):
-                suffix = " " + suffix
             # remove comments
             results = [re.sub(r"\s*#[^\n]*(?:\n|$)", "", r, flags=re.DOTALL) for r in selected_choices_text]
         else:
-            prefix = ""
-            suffix = ""
             results = []
+        container = options.get("container", None)
+        if container is None:
+            separator = options.get("separator", self.state.options.choice_separator)
+            container = lark.Tree(
+                lark.Token("RULE", "content"),
+                [
+                    lark.Tree(
+                        lark.Token("RULE", "variableuse"),
+                        [
+                            lark.Tree(
+                                lark.Token("RULE", "vardescriptor_get"),
+                                [
+                                    lark.Token("__identifier", "_choices"),
+                                    lark.Token("__openbracket", "["),
+                                    lark.Tree(
+                                        lark.Token("RULE", "separator_descriptor"),
+                                        [
+                                            lark.Token("__separatorflag", "&"),
+                                            lark.Token("STRING", "'" + separator + "'"),
+                                        ],
+                                    ),
+                                    lark.Token("__closebracket", "]"),
+                                ],
+                            ),
+                            None,
+                        ],
+                    ),
+                ],
+            )
         self.log(
             logging.DEBUG,
             "Unseen wildcards: "
             + ", ".join([f"'{escape_single_quotes(x)}'" for x in self.__seen_wildcards[seen_wildcards_len:]]),
         )
         self.__seen_wildcards = self.__seen_wildcards[:seen_wildcards_len]
-        return (prefix, results, separator, suffix)
+        return container, results
+
+    def __apply_container(self, container: lark.Tree, choices: list[str]) -> str:
+        # we save the choices variable in case there are nested choices
+        old_choices = self.state.variables.get_system("_choices[]", None)
+        self.state.variables.set_system("_choices[]", choices)
+        joined_results = self.__visit(container, False, True)
+        # we restore the old choices variable
+        self.state.variables.set_system("_choices[]", old_choices)
+        return joined_results
 
     def __convert_choices_options(self, options: Optional[lark.Tree], is_wcdef: bool = False) -> dict:
         """
@@ -1967,26 +2106,38 @@ class TreeProcessor(lark.visitors.Interpreter):
         if isinstance(wildcard.unprocessed_choices[0], dict):
             if self.state.wildcards_obj.is_dict_wcdef_options(wildcard.unprocessed_choices[0]):
                 options = wildcard.unprocessed_choices[0]
-                prefix = options.get("prefix", None)
-                if prefix is not None and isinstance(prefix, str):
+                container = options.get("container", None)
+                container_kind = "specified"
+                if container is None or not isinstance(container, str):
+                    container_kind = "assembled"
+                    has_separator = "separator" in options
+                    has_prefix = "prefix" in options
+                    has_suffix = "suffix" in options
+                    if has_separator or has_prefix or has_suffix:
+                        separator = options.get("separator", self.state.options.choice_separator)
+                        container = "${_choices[&'" + separator + "']}"
+                        prefix = options.get("prefix", None)
+                        if prefix is not None and isinstance(prefix, str):
+                            if prefix != "" and re.match(r"\w", prefix[-1]):
+                                prefix += " "
+                            container = prefix + container
+                        suffix = options.get("suffix", None)
+                        if suffix is not None and isinstance(suffix, str):
+                            if suffix != "" and re.match(r"\w", suffix[0]):
+                                suffix = " " + suffix
+                            container += suffix
+                    options.pop("separator", None)
+                    options.pop("prefix", None)
+                    options.pop("suffix", None)
+                    options.pop("container", None)
+                if container is not None:
                     try:
-                        options["prefix"] = parse_prompt(
-                            self.state, "choicevalue", prefix, self.state.parsers["choicevalue"], True
+                        options["container"] = parse_prompt(
+                            self.state, "choicevalue", container, self.state.parsers["choicevalue"], True
                         )
                     except lark.exceptions.UnexpectedInput as e:
                         self.warn_or_stop(
-                            f"Error parsing choice prefix '{escape_single_quotes(prefix)}' in wildcard '{escape_single_quotes(wildcard.key)}'! : {e.__class__.__name__}",
-                            e,
-                        )
-                suffix = options.get("suffix", None)
-                if suffix is not None and isinstance(suffix, str):
-                    try:
-                        options["suffix"] = parse_prompt(
-                            self.state, "choicevalue", suffix, self.state.parsers["choicevalue"], True
-                        )
-                    except lark.exceptions.UnexpectedInput as e:
-                        self.warn_or_stop(
-                            f"Error parsing choice suffix '{escape_single_quotes(suffix)}' in wildcard '{escape_single_quotes(wildcard.key)}'! : {e.__class__.__name__}",
+                            f"Error parsing choice {container_kind} container '{escape_single_quotes(container)}' in wildcard '{escape_single_quotes(wildcard.key)}'! : {e.__class__.__name__}",
                             e,
                         )
                 n = 1
@@ -2110,11 +2261,11 @@ class TreeProcessor(lark.visitors.Interpreter):
                             logging.DEBUG, f"Options for wildcard '{escape_single_quotes(wildcard.key)}' are ignored!"
                         )
                 choice_values_all += choice_values
-            prefix, chosen_choices, separator, suffix = self.__get_choices_select(
+            container, chosen_choices = self.__get_choices_select(
                 applied_options, choice_values_all, filter_specifier, wildcard_key
             )
             if chosen_choices:
-                self.__result += prefix + separator.join(chosen_choices) + suffix
+                self.__result += self.__apply_container(container, chosen_choices)
             if wildcard_key in self.__wildcard_filters:
                 del self.__wildcard_filters[wildcard_key]
             if variablename is not None:
@@ -2139,17 +2290,35 @@ class TreeProcessor(lark.visitors.Interpreter):
         self.__process_wildcard(tree)
 
     def __extract_filter_specifiers(self, filters: lark.Tree) -> list[list[str]]:
+        if (
+            len(filters.children) == 1
+            and len(filters.children[0].children) == 1
+            and isinstance(filters.children[0].children[0], lark.Tree)
+        ):
+            # special case when the whole filter is in a variable
+            # note that an individual label in a variable will also come through here
+            f = self.__visit(filters.children[0].children[0], False, True)
+            filters = parse_prompt(
+                self.state,
+                "filter specifier",
+                str(f),
+                self.state.parsers["wc_filter_or"],
+                True,
+            )
         filter_specifier = []
         for or_ in filters.children:
+            and_group = []
             for and_ in or_.children:
                 label = and_.children[0]
                 if isinstance(label, lark.Token):
                     # it's a literal, we can use it directly
-                    filter_specifier.append([str(label)])
+                    and_group.append(str(label))
                 else:
                     # it's a variable, we need to evaluate it
                     v = self.__visit(label, False, True)
-                    filter_specifier.append([v])
+                    # we remove commas and pluses to avoid confusion with the filter specifier syntax
+                    and_group.append(v.replace(",", "").replace("+", ""))
+            filter_specifier.append(and_group)
         return filter_specifier
 
     def choices(self, tree: lark.Tree):
@@ -2163,9 +2332,8 @@ class TreeProcessor(lark.visitors.Interpreter):
         ch = self.__get_original_node_content(tree, "?{...}")
         if self.state.options.process_wildcards:
             self.log(logging.DEBUG, "Processing choices:")
-            prefix, chosen_choices, separator, suffix = self.__get_choices_select(options, choice_values)
-            if chosen_choices:
-                self.__result += prefix + separator.join(chosen_choices) + suffix
+            container, chosen_choices = self.__get_choices_select(options, choice_values)
+            self.__result += self.__apply_container(container, chosen_choices)
         elif self.state.options.if_wildcards != IFWILDCARDS_CHOICES.remove:
             self.__detectedWildcards.append((ch, self.__is_negative))
             self.__result += ch

ppp_wildcards.py

@@ -92,7 +92,7 @@ class PPPWildcards:
         if wildcards_folders is not None or wildcards_input is not None:
             if wildcards_folders is not None:
                 for f in self.__wildcards_folders:
-                    self.__get_wildcards_in_directory(f, f)
+                    self.__get_wildcards_in_path(f if os.path.isdir(f) else os.path.dirname(f), f)
             if wildcards_input is not None:
                 self.__get_wildcards_in_input(wildcards_input)
         else:
@@ -261,6 +261,7 @@ class PPPWildcards:
                 "to",
                 "prefix",
                 "suffix",
+                "container",
                 "description",
                 "separator",
             ]
@@ -514,30 +515,30 @@ class PPPWildcards:
         text_content = [x.split("#")[0].rstrip() if len(x.split("#")) > 1 else x for x in text_content]
         self.__add_wildcard(text_content, full_path, external_key_parts)
 
-    def __get_wildcards_in_directory(self, base: str, directory: str):
+    def __get_wildcards_in_path(self, base: str, path: str):
         """
-        Get all wildcards in a directory.
+        Get all wildcards in a path.
 
         Args:
             base (str): The base path for the wildcards.
-            directory (str): The path to the directory.
+            path (str): The path (folder or file).
         """
-        if not os.path.exists(directory):
+        if not os.path.exists(path):
             log(
                 self.__logger,
                 self.__debug_level,
                 logging.WARNING,
-                f"Wildcard directory '{escape_single_quotes(directory)}' does not exist!",
+                f"Wildcard path '{escape_single_quotes(path)}' does not exist!",
             )
             return
-        for filename in os.listdir(directory):
-            full_path = os.path.abspath(os.path.join(directory, filename))
+        if os.path.isfile(path):
+            self.__get_wildcards_in_file(base, path)
+            return
+        for filename in os.listdir(path):
+            full_path = os.path.abspath(os.path.join(path, filename))
             if os.path.basename(full_path).startswith("."):
                 continue
-            if os.path.isdir(full_path):
-                self.__get_wildcards_in_directory(base, full_path)
-            elif os.path.isfile(full_path):
-                self.__get_wildcards_in_file(base, full_path)
+            self.__get_wildcards_in_path(base, full_path)
 
     def set_wildcard_default_filter(self, wildcard_key: str, filter_options: Optional[list[list[str]]]):
         """

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "sd-webui-prompt-postprocessor"
 description = "Stable Diffusion WebUI & ComfyUI extension to post-process the prompt, including sending content from the prompt to the negative prompt and wildcards."
-version = "3.0.0"
+version = "3.1.0"
 license = { file = "LICENSE.txt" }
 dependencies = ["lark", "numpy", "pyyaml", "pydantic"]
 

tests/base_tests.py

@@ -1,4 +1,5 @@
 from dataclasses import replace
+import difflib
 import os
 import logging
 from typing import Any, NamedTuple, Optional
@@ -114,7 +115,7 @@ class TestPromptPostProcessorBase(unittest.TestCase):
     def interrupt(self):
         self.interrupted = True
 
-    def init_obj(
+    def init_ppp(
         self,
         ppp: Optional[str | PromptPostProcessor] = None,
         combinatorial: bool = False,
@@ -181,6 +182,16 @@ class TestPromptPostProcessorBase(unittest.TestCase):
             )
         return the_obj
 
+    def _comp_diff(self, result: str, expected: str) -> list[str]:
+        return list(
+            difflib.ndiff(
+                result.splitlines(True),
+                expected.splitlines(True),
+                linejunk=None,
+                charjunk=None,
+            )
+        )
+
     def process(
         self,
         input_prompts: InputTuple,
@@ -220,7 +231,7 @@ class TestPromptPostProcessorBase(unittest.TestCase):
                 DEBUG_LEVEL.full,
                 specific_em_folders,
             )
-        the_obj: PromptPostProcessor = self.init_obj(ppp, combinatorial, combinatorial_limit)
+        the_obj: PromptPostProcessor = self.init_ppp(ppp, combinatorial, combinatorial_limit)
         out = (
             [OutputTuple("", "", None)]
             if expected_output is None
@@ -236,11 +247,13 @@ class TestPromptPostProcessorBase(unittest.TestCase):
                 seed,
             )
             the_obj.process_prompts_group_end()
-            if self.interrupted != interrupted:
-                errors.append(f"Interrupted flag is incorrect: expected {interrupted}, got {self.interrupted}")
-            elif not self.interrupted and expected_output is not None:
+            self.assertTrue(
+                self.interrupted == interrupted,
+                f"Interrupted flag is incorrect: got {self.interrupted} but expected {interrupted}",
+            )
+            if not self.interrupted and expected_output is not None:
                 if len(result) != len(out):
-                    errors.append(f"Incorrect number of combinations (expected {len(out)}, got {len(result)})")
+                    errors.append(f"Incorrect number of combinations: got {len(result)} but expected {len(out)}")
                 for out_prompt, out_negative_prompt, out_variables in out:
                     found = None
                     for r_prompt, r_negative_prompt, r_variables in result:
@@ -248,25 +261,47 @@ class TestPromptPostProcessorBase(unittest.TestCase):
                             found = OutputTuple(r_prompt, r_negative_prompt, r_variables)
                             break
                     if not found:
-                        errors.append(f"Combination '{out_prompt}' / '{out_negative_prompt}' not found in output")
+                        errors.extend(
+                            [
+                                "Combination not found in output",
+                                "Prompt:",
+                                out_prompt,
+                                "Negative Prompt:",
+                                out_negative_prompt,
+                            ]
+                        )
                     elif out_variables:
-                        unmatched_vars = {}
+                        missing_vars = {}
+                        incorrect_vars = {}
                         expected_values = {}
-                        for var_name, var_value in out_variables.items():
-                            if var_name not in found.variables or found.variables[var_name] != var_value:
-                                unmatched_vars[var_name] = (
-                                    found.variables[var_name] if var_name in found.variables else None
-                                )
+                        sorted_var_keys = sorted(out_variables.keys())
+                        for var_name in sorted_var_keys:
+                            var_value = out_variables[var_name]
+                            if var_name not in found.variables:
+                                missing_vars[var_name] = var_value
+                            elif found.variables[var_name] != var_value:
+                                incorrect_vars[var_name] = found.variables[var_name]
                                 expected_values[var_name] = var_value
-                        if unmatched_vars:
-                            errors.append(
-                                f"Combination '{out_prompt}' / '{out_negative_prompt}' found, but variables do not match: expected {expected_values}, got {unmatched_vars}"
+                        if missing_vars or incorrect_vars:
+                            errors.extend(
+                                [
+                                    "Combination found, but variables do not match",
+                                    "Prompt:",
+                                    out_prompt,
+                                    "Negative Prompt:",
+                                    out_negative_prompt,
+                                ]
                             )
-            self.assertFalse(
-                bool(errors),
-                "\n" + "\n".join(errors),
-            )
+                            if missing_vars:
+                                errors.append("Missing variables:")
+                                errors.append(str(missing_vars))
+                            if incorrect_vars:
+                                errors.append("Incorrect variables:")
+                                errors.extend(self._comp_diff(str(incorrect_vars), str(expected_values)))
+            if errors:
+                raise AssertionError("\n".join(errors))
             return
+
         # non-combinatorial
         errors = []
         the_obj.process_prompts_group_start()
@@ -276,30 +311,41 @@ class TestPromptPostProcessorBase(unittest.TestCase):
                 input_prompts.negative_prompt,
                 seed,
             )
-            if self.interrupted != interrupted:
-                errors.append(f"Interrupted flag is incorrect: expected {interrupted}, got {self.interrupted}")
-            elif not self.interrupted and expected_output is not None:
+            self.assertTrue(
+                self.interrupted == interrupted,
+                f"Interrupted flag is incorrect: got {self.interrupted} but expected {interrupted}",
+            )
+            if not self.interrupted and expected_output is not None:
                 result_prompt, result_negative_prompt, output_variables = result[0] if result else (None, None, None)
                 if result_prompt != eo.prompt or result_negative_prompt != eo.negative_prompt:
-                    errors.append(
-                        f"Incorrect result '{eo.prompt}' / '{eo.negative_prompt}', got '{result_prompt}' / '{result_negative_prompt}'"
-                    )
+                    errors.append("Incorrect result")
+                    if result_prompt != eo.prompt:
+                        errors.append("Prompt:")
+                        errors.extend(self._comp_diff(result_prompt, eo.prompt))
+                    if result_negative_prompt != eo.negative_prompt:
+                        errors.append("Negative Prompt:")
+                        errors.extend(self._comp_diff(result_negative_prompt, eo.negative_prompt))
                 if eo.variables:
-                    unmatched_vars = {}
+                    missing_vars = {}
+                    incorrect_vars = {}
                     expected_values = {}
-                    for var_name, var_value in eo.variables.items():
-                        if var_name not in output_variables or output_variables[var_name] != var_value:
-                            unmatched_vars[var_name] = (
-                                output_variables[var_name] if var_name in output_variables else None
-                            )
+                    sorted_var_keys = sorted(eo.variables.keys())
+                    for var_name in sorted_var_keys:
+                        var_value = eo.variables[var_name]
+                        if var_name not in output_variables:
+                            missing_vars[var_name] = var_value
+                        elif output_variables[var_name] != var_value:
+                            incorrect_vars[var_name] = output_variables[var_name]
                             expected_values[var_name] = var_value
-                    if unmatched_vars:
-                        errors.append(
-                            f"Result '{eo.prompt}' / '{eo.negative_prompt}' found, but variables do not match: expected {expected_values}, got {unmatched_vars}"
-                        )
+                    if missing_vars or incorrect_vars:
+                        errors.append("Result correct, but variables do not match")
+                        if missing_vars:
+                            errors.append("Missing variables:")
+                            errors.append(str(missing_vars))
+                        if incorrect_vars:
+                            errors.append("Incorrect variables:")
+                            errors.extend(self._comp_diff(str(incorrect_vars), str(expected_values)))
             seed += 1
         the_obj.process_prompts_group_end()
-        self.assertFalse(
-            bool(errors),
-            "\n" + "\n".join(errors),
-        )
+        if errors:
+            raise AssertionError("\n".join(errors))

tests/tests_choices.py

@@ -22,7 +22,7 @@ class TestChoices(TestPromptPostProcessorBase):
         )
 
     def test_ch_cyclical(self):  # cyclical sampler cycles through all choices
-        ppp_instance = self.init_obj("nocup")
+        ppp_instance = self.init_ppp("nocup")
         self.process(
             InputTuple("the choices are: {@choice1|choice2|choice3}", ""),
             [
@@ -35,7 +35,7 @@ class TestChoices(TestPromptPostProcessorBase):
         )
 
     def test_ch_cyclical_multiple_constructs(self):  # two independent @ constructs cycle together
-        ppp_instance = self.init_obj("nocup")
+        ppp_instance = self.init_ppp("nocup")
         self.process(
             InputTuple("{@a|b} {@c|d}", ""),
             [
@@ -49,7 +49,7 @@ class TestChoices(TestPromptPostProcessorBase):
         )
 
     def test_ch_cyclical_resets_on_prompt_change(self):  # state resets when the prompt pair changes
-        ppp_instance = self.init_obj("nocup")
+        ppp_instance = self.init_ppp("nocup")
         # Advance the cycle to position 1 (choice2).
         self.process(
             InputTuple("the choices are: {@choice1|choice2|choice3}", ""),
@@ -67,7 +67,7 @@ class TestChoices(TestPromptPostProcessorBase):
         )
 
     def test_ch_cyclical_mixed_samplers(self):  # @ construct cycles while a ~ construct alongside is unaffected
-        ppp_instance = self.init_obj("nocup")
+        ppp_instance = self.init_ppp("nocup")
         self.process(
             InputTuple("{@a|b|c} {x|y}", ""),
             [

tests/tests_varcomms.py

@@ -1064,3 +1064,9 @@ class TestVarCommands(TestPromptPostProcessorBase):
                 self.extranetwork_maps_obj,
             ),
         )
+
+    def test_var_attention_merge(self):  # attention merge at variable boundary
+        self.process(
+            InputTuple("${v!=[content]}(${v}:1.5)", ""),
+            OutputTuple("(content:1.35)", ""),
+        )

tests/tests_wildcards.py

@@ -205,6 +205,13 @@ class TestWildcards(TestPromptPostProcessorBase):
             ppp="nocup",
         )
 
+    def test_wc_test3_yaml_2(self):  # simple yaml wildcard build from variable
+        self.process(
+            InputTuple("${v=3}the choice is: __testwc/test${v}__", ""),
+            OutputTuple("the choice is: one choice", ""),
+            ppp="nocup",
+        )
+
     def test_wc_wildcard_filter_index(self):  # wildcard with positional index filter
         self.process(
             InputTuple("the choice is: __yaml/wildcard2'2'__", ""),
@@ -235,8 +242,8 @@ class TestWildcards(TestPromptPostProcessorBase):
 
     def test_wc_wildcard_filter_label3(self):  # wildcard with multiple label filter
         self.process(
-            InputTuple("the choice is: __yaml/wildcard2'label1,label2'__", ""),
-            OutputTuple("the choice is: choice3-choice1", ""),
+            InputTuple("the choice is: __4$$-$$yaml/wildcard2'label1,label2'__", ""),
+            OutputTuple("the choice is: choice1-choice3", ""),
             ppp="nocup",
         )
 
@@ -249,8 +256,15 @@ class TestWildcards(TestPromptPostProcessorBase):
 
     def test_wc_wildcard_filter_compound(self):  # wildcard with compound filter
         self.process(
-            InputTuple("the choice is: __yaml/wildcard2'label1+label3'__", ""),
-            OutputTuple("the choice is: choice3-choice1", ""),
+            InputTuple("the choice is: __4$$-$$yaml/wildcard2'label1+label3'__", ""),
+            OutputTuple("the choice is: choice3", ""),
+            ppp="nocup",
+        )
+
+    def test_wc_wildcard_filter_compound_var(self):  # wildcard with compound filter in a variable
+        self.process(
+            InputTuple("${v[]=*('label1','label3')}the choice is: __4$$-$$yaml/wildcard2'${v[&'+']}'__", ""),
+            OutputTuple("the choice is: choice3", ""),
             ppp="nocup",
         )
 
@@ -264,14 +278,14 @@ class TestWildcards(TestPromptPostProcessorBase):
     def test_wc_wildcard_filter_compound3(self):  # wildcard with doubly inherited compound filter
         self.process(
             InputTuple("the choice is: __yaml/wildcard2bisbis'#label1+label3'__", ""),
-            OutputTuple("the choice is: choice1bisbis", ""),
+            OutputTuple("the choice is: choice3bisbis", ""),
             ppp="nocup",
         )
 
     def test_wc_wildcard_filter_compound4(self):  # wildcard with doubly inherited compound filter with variable
         self.process(
             InputTuple("${v=label1}the choice is: __yaml/wildcard2bisbis'#${v}+label3'__", ""),
-            OutputTuple("the choice is: choice1bisbis", ""),
+            OutputTuple("the choice is: choice3bisbis", ""),
             ppp="nocup",
         )
 
@@ -281,7 +295,7 @@ class TestWildcards(TestPromptPostProcessorBase):
                 "<ppp:setwcdeffilter 'yaml/wildcard2' 'label1+label3' />the choice is: __yaml/wildcard2__, <ppp:setwcdeffilter 'yaml/wildcard2' />__yaml/wildcard2__",
                 "",
             ),
-            OutputTuple("the choice is: choice3-choice1, choice3-choice1- choice2 ", ""),
+            OutputTuple("the choice is: choice3-choice3, choice3-choice1- choice2 ", ""),
             ppp="nocup",
         )
 
@@ -291,7 +305,7 @@ class TestWildcards(TestPromptPostProcessorBase):
                 "${v=label1}<ppp:setwcdeffilter 'yaml/wildcard2' '${v}+label3' />the choice is: __yaml/wildcard2__, <ppp:setwcdeffilter 'yaml/wildcard2' />__yaml/wildcard2__",
                 "",
             ),
-            OutputTuple("the choice is: choice3-choice1, choice3-choice1- choice2 ", ""),
+            OutputTuple("the choice is: choice3-choice3, choice3-choice1- choice2 ", ""),
             ppp="nocup",
         )
 
@@ -369,6 +383,42 @@ class TestWildcards(TestPromptPostProcessorBase):
             ppp="nocup",
         )
 
+    def test_wc_wildcardPS2_yaml(self):  # yaml wildcard with object formatted choices and options and prefix and suffix
+        self.process(
+            InputTuple("the choices are: [__yaml/wildcardPS2__]", ""),
+            OutputTuple("the choices are: (prefix2-choice2-suffix:1.35)", ""),
+        )
+
+    def test_wc_wildcardContainer_yaml(self):  # yaml wildcard with object formatted choices and options and container
+        self.process(
+            InputTuple("the choices are: [__yaml/wildcardContainer__]", ""),
+            OutputTuple("the choices are: (prefix1-choice2/choice3-suffix:1.35)", ""),
+        )
+
+    def test_wc_wildcardAt_yaml(self):  # yaml wildcard with attention in choices
+        self.process(
+            InputTuple("the choices are: [__yaml/wildcardAt__]", ""),
+            OutputTuple("the choices are: (choice2:1.35)", ""),
+        )
+
+    def test_wc_merge_attention_bracket(self):  # bracket attention from wildcard merges with outer attention
+        self.process(
+            InputTuple("(__yaml/wildcardAtBracket__:1.5)", ""),
+            OutputTuple("(the content:1.35)", ""),
+        )
+
+    def test_wc_no_merge_attention_alternation(self):  # alternation from wildcard is not merged as attention
+        self.process(
+            InputTuple("(__yaml/wildcardAlt__:1.5)", ""),
+            OutputTuple("([cat|dog]:1.5)", ""),
+        )
+
+    def test_wc_no_merge_attention_scheduling(self):  # scheduling from wildcard is not merged as attention
+        self.process(
+            InputTuple("(__yaml/wildcardSched__:1.5)", ""),
+            OutputTuple("([cat:dog:0.5]:1.5)", ""),
+        )
+
     def test_wc_anonymouswildcard_yaml(self):  # yaml anonymous wildcard
         self.process(
             InputTuple("the choices are: __yaml/anonwildcards__", ""),

tests/wildcards/test.yaml

@@ -37,6 +37,20 @@ yaml:
     - <lora:test2:1>
     - <lora:test3:1>
 
+  wildcardAt:
+    - (choice1:0.5)
+    - (choice2:1.5)
+    - (choice3:1)
+
+  wildcardAtBracket:
+    - "[the content]"
+
+  wildcardAlt:
+    - "[cat|dog]"
+
+  wildcardSched:
+    - "[cat:dog:0.5]"
+
   wildcardPS:
     - {
         sampler: "~",
@@ -51,6 +65,24 @@ yaml:
     - { weight: 2, text: choice2 }
     - { weight: 1, text: choice3 }
 
+  wildcardPS2:
+    - {
+        prefix: "(prefix{1|2}-",
+        suffix: "-suffix:1.5)",
+      }
+    - { text: choice1 }
+    - { text: choice2 }
+    - { text: choice3 }
+  
+  wildcardContainer:
+    - {
+        count: 2,
+        container: "(prefix{1|2}-${_choices[&'/']}-suffix:1.5)",
+      }
+    - { weight: 3, text: choice1 }
+    - { weight: 2, text: choice2 }
+    - { weight: 1, text: choice3 }
+
   more_nested:
     even_more_nested: # this would be __yaml/more_nested/even_more_nested__
       - one