Test linked wildcard execution, cache reuse and Encode output parity

docs/E2E_TEST_STRATEGY.md

@@ -12,7 +12,7 @@ that setup does not extend the supported import scope.
 | Script | Setup through the UI | Assertions |
 | --- | --- | --- |
 | `tests/e2e_wildcards.py` | Create a blank workflow, search for and place ImpactWildcardProcessor and Preview as Text, drag a connection, enter text, select modes, click Run | UTF-8 file expansion, populate/fixed/reproduce behavior, editable/disabled text fields, websocket feedback, preview text |
-| `tests/test_e2e_wildcard_cache.py` | Create both wildcard node graphs, edit seeds/text, Run, native Export and reload | Seeded outputs, actual downstream cache hits, invalidation on text changes, and widget names/order/values under both restoration modes |
+| `tests/test_e2e_wildcard_cache.py` | Create both wildcard node graphs, connect text/seed providers, convert a provider to a subgraph, Run, native Export and reload | Seeded outputs, cache reuse for fixed text and identical populate results, reproduce-to-populate transition, linked execution, and widget names/order/values under both restoration modes |
 | `tests/e2e_dd_compat.py` | Load the prepared Detailer graph through the frontend file input, click Run | Successful SEGSDetailer inference with DifferentialDiffusion, output image loaded in the UI, non-degenerate image, changed pixels by mean/std comparison |
 | `tests/test_e2e_switches.py` | Create and wire nodes; import prepared graphs and UI-created JSON, reload Chrome, delete connected nodes, reconnect and click Run | Slot identities and typed links, list order, non-default selection and routing, fresh output after reconnecting, output preservation when a Reroute disconnects during loading |
 | `tests/test_e2e_widget_catalog.py` | Load the UI-created catalog and pan to each node at 100% zoom | Every registered node exactly once; live widget names, order, and values match the fixed baseline under positional and named restoration |
@@ -67,9 +67,16 @@ wildcard_cache_limit_mb = 50
 The wildcard cache suite additionally runs `ImpactWildcardEncode` with a local
 checkpoint, defaulting to `v1-5-pruned-emaonly-fp16.safetensors`. Override it with
 `IMPACT_E2E_CHECKPOINT`; missing models fail rather than skip. No models are
-downloaded. Use an isolated server with caching enabled (`--cache-classic`).
+downloaded. Start a fresh isolated server with caching enabled (`--cache-classic`)
+for each suite invocation, so initial cache-miss assertions start with empty caches.
 The tests check history cache events and displayed text, then native widget
 values and serialized `widgets_values` before/after positional and named loading.
+Different seeds that expand to the same text must reuse both wildcard and
+downstream caches; a different expansion must invalidate them. Linked text is
+produced by a real core Concatenate Text node, also inside a UI-created subgraph,
+and linked seeds come from ImpactInt. Native exports preserve links and subgraph
+definitions as well as widgets. A restored reproduce workflow uses its saved
+expansion once, then populates from its source with the saved original seed.
 Encode runs use real CLIP encoding, but do not sample images or benchmark speed.
 
 ```bash
@@ -77,8 +84,23 @@ IMPACT_E2E_BASE_URL=http://127.0.0.1:18188 \
 IMPACT_E2E_ARTIFACTS=/tmp/impact-wildcard-cache \
 python -m pytest tests/test_e2e_wildcard_cache.py -v
 python tests/integration_wildcard_cache.py --comfy-root /path/to/ComfyUI --report /tmp/wildcard-cache.json
+python tests/integration_wildcard_cache_tensors.py --comfy-root /path/to/ComfyUI --report /tmp/wildcard-tensors.json
 ```
 
+For the core LRU cache, restart the isolated server with `--cache-lru 10` instead
+of `--cache-classic`, then run the same cache assertions with
+`python -m pytest tests/test_e2e_wildcard_cache.py -k 'plain_text or populate_reuses'`.
+
+The tensor integration runs on CPU with installed
+`v1-5-pruned-emaonly-fp16.safetensors` and `loras/SD1.5/etc/add_detail.safetensors`.
+It compares all conditioning tensors and metadata, MODEL/CLIP LoRA patches and
+strengths with independent core CLIPTextEncode/LoraLoader/ConditioningConcat
+execution, using explicit reference selections and prompt sections. Plain,
+populate, fixed multiline/nested, reproduce, LoRA and BREAK cases are covered;
+input models must remain unmodified. Only HTTP routes/feedback are substituted.
+Missing models fail without downloading or skipping. These checks do not cover
+other model architectures or external LoRA loader plugins.
+
 The separate CPU integration uses real prompt hooks, parser, node methods and
 ComfyUI cache keys, with HTTP feedback and the CLIP model substituted. Fixed
 expectations from pre-change Main cover dynamic syntax, legacy modes, linked

tests/README.md

@@ -34,10 +34,15 @@ See [E2E setup and artifacts](../docs/E2E_TEST_STRATEGY.md). These scripts need
 a running test server and are separate from the curl-based API integration tests.
 `python3 -m pytest tests/test_e2e_wildcard_cache.py -v` checks both wildcard nodes'
 seeded output, real downstream cache reuse and native widget restoration in both
-modes. Encode cases need a local checkpoint; see the E2E guide. The separate
+modes. It also checks equal/different populate results, restored reproduce mode,
+and actual linked text/seed execution through a native subgraph. Encode cases
+need a local checkpoint; see the E2E guide. The separate
 `python3 tests/integration_wildcard_cache.py --comfy-root /path/to/ComfyUI` runs on
 CPU with real prompt hooks/parser/cache keys and a substituted CLIP model.
 It covers legacy modes, linked inputs, dynamic syntax and saved seed positions.
+`python3 tests/integration_wildcard_cache_tensors.py --comfy-root /path/to/ComfyUI`
+uses installed SD1.5/LoRA weights on CPU to compare conditioning and MODEL/CLIP
+patches against core-node references, including seeded expansion and BREAK.
 `python3 -m pytest tests/test_e2e_legacy_sam.py -v` checks the legacy SAM menu
 and Clipspace button without importing the removed `ClipspaceDialog` module.
 Set `IMPACT_E2E_SAM_IMAGE` and install the configured SAM model to include real

tests/integration_wildcard_cache_tensors.py

@@ -0,0 +1,109 @@
+"""Compare wildcard Encode outputs with independent core CLIP/LoRA execution.
+
+Run separately from pytest's stubbed tests. Uses installed SD1.5 checkpoint and
+LoRA weights on CPU; downloads nothing. Only unused HTTP routes/feedback are
+substituted. Expected selections and reference text are explicit test data.
+"""
+import argparse
+import json
+from pathlib import Path
+import sys
+from types import SimpleNamespace
+import unittest
+
+import torch
+
+from integration_lora_wildcard_tensors import assert_lora_patches_equal
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--comfy-root", type=Path, required=True)
+    parser.add_argument("--report", type=Path)
+    args = parser.parse_args()
+    root = Path(__file__).resolve().parents[1]
+    sys.path[:0] = [str(args.comfy_root.resolve()), str(root / "modules")]
+    from comfy.cli_args import args as comfy_args
+    comfy_args.cpu = True
+    torch.set_num_threads(2)
+    from aiohttp import web
+    from server import PromptServer
+    PromptServer.instance = SimpleNamespace(
+        routes=web.RouteTableDef(), add_on_prompt_handler=lambda handler: None,
+        send_sync=lambda *args: None,
+    )
+    import nodes
+    from impact import impact_server, impact_pack
+
+    model, clip, _ = nodes.CheckpointLoaderSimple().load_checkpoint("v1-5-pruned-emaonly-fp16.safetensors")
+    lora = "SD1.5/etc/add_detail.safetensors"
+
+    class EncodeTensorTests(unittest.TestCase):
+        def compare(self, *, seed, text, expected, mode="fixed", source="", sections=None, with_lora=False):
+            inputs = dict(wildcard_text=source, populated_text=text, mode=mode, seed=seed)
+            prompt = {"prompt": {"1": {"class_type": "ImpactWildcardEncode", "inputs": inputs}}}
+            impact_server.onprompt_populate_wildcards(prompt)
+            actual_model, actual_clip, actual_conditioning, actual_text = impact_pack.ImpactWildcardEncode().doit(
+                model=model, clip=clip, **inputs)
+            self.assertEqual(actual_text, expected)
+            reference_model, reference_clip = model, clip
+            if with_lora:
+                reference_model, reference_clip = nodes.LoraLoader().load_lora(model, clip, lora, 0.6, 0.4)
+                self.assertTrue(actual_model.patches)
+                self.assertTrue(actual_clip.patcher.patches)
+            else:
+                self.assertIs(actual_model, model)
+                self.assertIs(actual_clip, clip)
+            reference_conditioning = None
+            for section in sections or [expected]:
+                encoded = nodes.CLIPTextEncode().encode(reference_clip, section)[0]
+                reference_conditioning = encoded if reference_conditioning is None else (
+                    nodes.ConditioningConcat().concat(reference_conditioning, encoded)[0])
+            torch.testing.assert_close(actual_conditioning, reference_conditioning, rtol=0, atol=0)
+            assert_lora_patches_equal(actual_model.patches, reference_model.patches)
+            assert_lora_patches_equal(actual_clip.patcher.patches, reference_clip.patcher.patches)
+            self.assertEqual(model.patches, {})
+            self.assertEqual(clip.patcher.patches, {})
+
+        def test_plain_text(self):
+            for seed in (2, 5):
+                with self.subTest(seed=seed):
+                    self.compare(seed=seed, text="a red flower", expected="a red flower")
+
+        def test_populate_selections(self):
+            for seed, color in ((0, "green"), (1, "green"), (2, "red"), (5, "blue")):
+                with self.subTest(seed=seed):
+                    self.compare(seed=seed, text="old result", source="{red|green|blue}",
+                                 mode="populate", expected=color)
+
+        def test_fixed_multiline_and_nested(self):
+            for text in ("{red|\ngreen|blue}", "{red|{green}|blue}"):
+                for seed, color in ((2, "red"), (5, "blue")):
+                    with self.subTest(text=text, seed=seed):
+                        self.compare(seed=seed, text=text, expected=color)
+
+        def test_reproduce_ignores_source(self):
+            self.compare(seed=5, text="red", source="{red|green|blue}", mode="reproduce", expected="red")
+
+        def test_lora_and_break(self):
+            tag = "<lora:SD1.5/etc/add_detail:0.6:0.4>"
+            for seed, color in ((2, "red"), (5, "blue")):
+                with self.subTest(seed=seed):
+                    self.compare(seed=seed, text="", mode="populate",
+                                 source=tag + " {red|green|blue} BREAK a flower",
+                                 expected=tag + f" {color} BREAK a flower",
+                                 sections=[color, "a flower"], with_lora=True)
+
+    result = unittest.TextTestRunner(verbosity=2).run(unittest.defaultTestLoader.loadTestsFromTestCase(EncodeTensorTests))
+    if args.report:
+        args.report.write_text(json.dumps({
+            "tests_run": result.testsRun, "successful": result.wasSuccessful(),
+            "failures": [{"test": str(test), "detail": detail} for test, detail in result.failures],
+            "errors": [{"test": str(test), "detail": detail} for test, detail in result.errors],
+            "skipped": result.skipped,
+        }, indent=2) + "\n")
+    return 0 if result.wasSuccessful() else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tests/test_e2e_wildcard_cache.py

@@ -72,6 +72,144 @@ def run(ui, preview_id, expected, label):
     return entry
 
 
+def native_roundtrip(ui, node_id):
+    """Compare rendered widget names/order/values and the native serialization."""
+    show_node(ui, node_id)
+    before = widget_state(ui, node_id)
+    trusted = json.loads((ROOT / "tests/workflows/all-impact-nodes.widgets.json").read_text())
+    reference = next(node for node in trusted["nodes"] if node["type"] == before["type"])
+    assert [w["name"] for w in before["widgets"]] == [w["name"] for w in reference["widgets"]]
+    workflow = ui.artifacts / "saved.json"
+    export_workflow(ui, workflow)
+    saved = json.loads(workflow.read_text())
+    ui.new_workflow()
+    ui.load_workflow(workflow, len(saved["nodes"]))
+    full_zoom(ui)
+    show_node(ui, node_id)
+    assert widget_state(ui, node_id) == before
+    roundtrip = ui.artifacts / "roundtrip.json"
+    export_workflow(ui, roundtrip)
+    restored = json.loads(roundtrip.read_text())
+    assert {str(n["id"]): n.get("widgets_values", []) for n in restored["nodes"]} == {
+        str(n["id"]): n.get("widgets_values", []) for n in saved["nodes"]}
+    # Native loading can reorder link records; their IDs and endpoints must
+    # remain identical. Widget arrays above deliberately retain their order.
+    assert sorted(restored["links"]) == sorted(saved["links"])
+    assert restored.get("definitions") == saved.get("definitions")
+    (ui.artifacts / "widget-state.json").write_text(json.dumps(before, indent=2) + "\n")
+    return before
+
+
+def input_slot(ui, node_type, name):
+    return ui.app.evaluate("""(app, {type, name}) =>
+        app.graph._nodes.find(n => n.type === type).inputs.findIndex(i => i.name === name)
+    """, {"type": node_type, "name": name})
+
+
+@pytest.mark.parametrize("restore_by_name", [False, True], ids=["positional", "named"])
+@pytest.mark.parametrize("node_type", NODE_TYPES)
+def test_linked_inputs_and_subgraph_native_queue(ui, node_type, restore_by_name):
+    setting(ui, "Restore widget values", "Comfy.Workflow.NamedValuesRestore", restore_by_name)
+    node_id, preview_id = graph(ui, node_type)
+    ui.choose(node_type, "mode", "fixed")
+    source_id = ui.add_node("Concatenate Text", "StringConcatenate", 100, 650)
+    show_node(ui, source_id)
+    # Neither fragment contains a complete choice, so scanning upstream input
+    # strings cannot determine the seed dependence of their combined output.
+    ui.page.get_by_placeholder("string_a", exact=True).fill("{red|")
+    ui.page.get_by_placeholder("string_b", exact=True).fill("green|blue}")
+    show_node(ui, node_id)
+    connect(ui, "StringConcatenate", node_type, target_slot=input_slot(ui, node_type, "populated_text"))
+    for seed, expected in ((2, "red"), (5, "blue")):
+        ui.set_number(node_type, "seed", seed)
+        entry = run(ui, preview_id, expected, f"linked-text-{seed}")
+        assert entry["prompt"][2][node_id]["inputs"]["populated_text"] == [source_id, 0]
+        assert entry["prompt"][2][node_id]["inputs"]["seed"] == seed
+
+    seed_id = ui.add_node("ImpactInt", "ImpactInt", 850, 180)
+    show_node(ui, seed_id)
+    ui.set_number("ImpactInt", "value", 2)
+    show_node(ui, node_id)
+    connect(ui, "ImpactInt", node_type, target_slot=input_slot(ui, node_type, "seed"))
+    entry = run(ui, preview_id, "red", "linked-seed")
+    assert entry["prompt"][2][node_id]["inputs"]["seed"] == [seed_id, 0]
+
+    # Convert the real upstream node using the frontend menu. No graph mutation
+    # or hand-written workflow JSON is used to construct the subgraph.
+    show_node(ui, source_id)
+    title = ui.page.evaluate("""({app, id}) => {
+        const n = app.graph.getNodeById(id);
+        return app.canvasPosToClientPos([n.pos[0] + 40, n.pos[1] - 15]);
+    }""", {"app": ui.app, "id": source_id})
+    ui.page.mouse.click(*title)
+    ui.page.mouse.click(*title, button="right")
+    ui.page.get_by_role("menuitem", name="Convert to Subgraph", exact=True).click()
+    ui.page.wait_for_function("""({app, id}) => !app.graph.getNodeById(id)
+    """, arg={"app": ui.app, "id": source_id})
+    native_roundtrip(ui, node_id)
+    saved = json.loads((ui.artifacts / "saved.json").read_text())
+    assert any(n["type"] == "StringConcatenate"
+               for subgraph in saved["definitions"]["subgraphs"] for n in subgraph["nodes"])
+    for seed, expected in ((5, "blue"), (2, "red")):
+        show_node(ui, seed_id)
+        ui.set_number("ImpactInt", "value", seed)
+        entry = run(ui, preview_id, expected, f"restored-subgraph-{seed}")
+        inputs = entry["prompt"][2][node_id]["inputs"]
+        assert inputs["seed"] == [seed_id, 0]
+        assert isinstance(inputs["populated_text"], list)
+        upstream_id = inputs["populated_text"][0]
+        assert entry["prompt"][2][upstream_id]["class_type"] == "StringConcatenate"
+
+
+@pytest.mark.parametrize("node_type", NODE_TYPES)
+def test_populate_reuses_cache_only_for_same_expansion(ui, node_type):
+    node_id, preview_id = graph(ui, node_type)
+    prefix = "populate cache "
+    ui.page.get_by_placeholder("Wildcard Prompt (User input)", exact=True).fill(prefix + "{red|green|blue}")
+    # Trusted pre-change selections: 0 and 1 -> green; 2 -> red.
+    for seed, color in ((0, "green"), (1, "green"), (2, "red")):
+        ui.set_number(node_type, "seed", seed)
+        entry = run(ui, preview_id, prefix + color, f"populate-cache-{seed}")
+        if seed == 1:
+            assert {node_id, preview_id} <= cached(entry)
+        else:
+            assert {node_id, preview_id}.isdisjoint(cached(entry))
+        assert entry["prompt"][2][node_id]["inputs"]["seed"] == 0
+        assert next(w["value"] for w in widget_state(ui, node_id)["widgets"] if w["name"] == "seed") == seed
+
+
+@pytest.mark.parametrize("restore_by_name", [False, True], ids=["positional", "named"])
+@pytest.mark.parametrize("node_type", NODE_TYPES)
+def test_reproduce_native_workflow_then_populate(ui, node_type, restore_by_name):
+    setting(ui, "Restore widget values", "Comfy.Workflow.NamedValuesRestore", restore_by_name)
+    node_id, preview_id = graph(ui, node_type)
+    ui.page.get_by_placeholder("Wildcard Prompt (User input)", exact=True).fill("{red|green|blue}")
+    ui.set_number(node_type, "seed", 2)
+    initial = run(ui, preview_id, "red", "initial-populate")
+    # The queued native metadata also preserves the original seed and stores
+    # expanded text in reproduce mode, as image/workflow reproduction requires.
+    saved_node = next(n for n in initial["prompt"][3]["extra_pnginfo"]["workflow"]["nodes"]
+                      if str(n["id"]) == node_id)
+    assert saved_node["widgets_values"][:3] == ["{red|green|blue}", "red", "reproduce"]
+    seed_index = 3 if node_type == "ImpactWildcardProcessor" else 5
+    assert saved_node["widgets_values"][seed_index] == 2
+    ui.choose(node_type, "mode", "reproduce")
+    ui.set_number(node_type, "seed", 5)
+    before = native_roundtrip(ui, node_id)
+    values = {w["name"]: w["value"] for w in before["widgets"]}
+    assert (values["wildcard_text"], values["populated_text"], values["mode"], values["seed"]) == (
+        "{red|green|blue}", "red", "reproduce", 5)
+    run(ui, preview_id, "red", "restored-reproduce")
+    ui.page.wait_for_function("""({app, id}) =>
+        app.graph.getNodeById(id).widgets.find(w => w.name === 'mode').value === 'populate'
+    """, arg={"app": ui.app, "id": node_id})
+    run(ui, preview_id, "blue", "next-populate")
+    assert widget_state(ui, node_id)["widgets"] == [
+        dict(w, value="populate") if w["name"] == "mode" else
+        dict(w, value="blue") if w["name"] == "populated_text" else w for w in before["widgets"]
+    ]
+
+
 @pytest.mark.parametrize("node_type", NODE_TYPES)
 def test_plain_text_reuses_downstream_cache(ui, node_type):
     node_id, preview_id = graph(ui, node_type)
@@ -120,24 +258,7 @@ def test_seeded_output_and_native_widgets(ui, node_type, restore_by_name):
     ui.set_number(node_type, "seed", 731)
     ui.choose(node_type, "control_after_generate", "increment")
     rendered(ui)
-    before = widget_state(ui, node_id)
-    trusted = json.loads((ROOT / "tests/workflows/all-impact-nodes.widgets.json").read_text())
-    reference = next(node for node in trusted["nodes"] if node["type"] == node_type)
-    assert [w["name"] for w in before["widgets"]] == [w["name"] for w in reference["widgets"]]
-    workflow = ui.artifacts / "saved.json"
-    export_workflow(ui, workflow)
-    saved = json.loads(workflow.read_text())
-    ui.new_workflow()
-    ui.load_workflow(workflow, len(saved["nodes"]))
-    full_zoom(ui)
-    show_node(ui, node_id)
-    assert widget_state(ui, node_id) == before
-    roundtrip = ui.artifacts / "roundtrip.json"
-    export_workflow(ui, roundtrip)
-    restored = json.loads(roundtrip.read_text())
-    saved_widgets = {str(node["id"]): node.get("widgets_values", []) for node in saved["nodes"]}
-    assert {str(node["id"]): node.get("widgets_values", []) for node in restored["nodes"]} == saved_widgets
-    (ui.artifacts / "widget-state.json").write_text(json.dumps(before, indent=2) + "\n")
+    before = native_roundtrip(ui, node_id)
     run(ui, preview_id, "saved plain α", "restored")
     assert widget_state(ui, node_id)["widgets"] == [
         dict(widget, value=732) if widget["name"] == "seed" else widget for widget in before["widgets"]