diff --git a/docs/E2E_TEST_STRATEGY.md b/docs/E2E_TEST_STRATEGY.md index 4d07fa24..5a290650 100644 --- a/docs/E2E_TEST_STRATEGY.md +++ b/docs/E2E_TEST_STRATEGY.md @@ -12,7 +12,7 @@ that setup does not extend the supported import scope. | Script | Setup through the UI | Assertions | | --- | --- | --- | | `tests/e2e_wildcards.py` | Create a blank workflow, search for and place ImpactWildcardProcessor and Preview as Text, drag a connection, enter text, select modes, click Run | UTF-8 file expansion, populate/fixed/reproduce behavior, editable/disabled text fields, websocket feedback, preview text | -| `tests/test_e2e_wildcard_cache.py` | Create both wildcard node graphs, edit seeds/text, Run, native Export and reload | Seeded outputs, actual downstream cache hits, invalidation on text changes, and widget names/order/values under both restoration modes | +| `tests/test_e2e_wildcard_cache.py` | Create both wildcard node graphs, connect text/seed providers, convert a provider to a subgraph, Run, native Export and reload | Seeded outputs, cache reuse for fixed text and identical populate results, reproduce-to-populate transition, linked execution, and widget names/order/values under both restoration modes | | `tests/e2e_dd_compat.py` | Load the prepared Detailer graph through the frontend file input, click Run | Successful SEGSDetailer inference with DifferentialDiffusion, output image loaded in the UI, non-degenerate image, changed pixels by mean/std comparison | | `tests/test_e2e_switches.py` | Create and wire nodes; import prepared graphs and UI-created JSON, reload Chrome, delete connected nodes, reconnect and click Run | Slot identities and typed links, list order, non-default selection and routing, fresh output after reconnecting, output preservation when a Reroute disconnects during loading | | `tests/test_e2e_widget_catalog.py` | Load the UI-created catalog and pan to each node at 100% zoom | Every registered node exactly once; live widget names, order, and values match the fixed baseline under positional and named restoration | @@ -67,9 +67,16 @@ wildcard_cache_limit_mb = 50 The wildcard cache suite additionally runs `ImpactWildcardEncode` with a local checkpoint, defaulting to `v1-5-pruned-emaonly-fp16.safetensors`. Override it with `IMPACT_E2E_CHECKPOINT`; missing models fail rather than skip. No models are -downloaded. Use an isolated server with caching enabled (`--cache-classic`). +downloaded. Start a fresh isolated server with caching enabled (`--cache-classic`) +for each suite invocation, so initial cache-miss assertions start with empty caches. The tests check history cache events and displayed text, then native widget values and serialized `widgets_values` before/after positional and named loading. +Different seeds that expand to the same text must reuse both wildcard and +downstream caches; a different expansion must invalidate them. Linked text is +produced by a real core Concatenate Text node, also inside a UI-created subgraph, +and linked seeds come from ImpactInt. Native exports preserve links and subgraph +definitions as well as widgets. A restored reproduce workflow uses its saved +expansion once, then populates from its source with the saved original seed. Encode runs use real CLIP encoding, but do not sample images or benchmark speed. ```bash @@ -77,8 +84,23 @@ IMPACT_E2E_BASE_URL=http://127.0.0.1:18188 \ IMPACT_E2E_ARTIFACTS=/tmp/impact-wildcard-cache \ python -m pytest tests/test_e2e_wildcard_cache.py -v python tests/integration_wildcard_cache.py --comfy-root /path/to/ComfyUI --report /tmp/wildcard-cache.json +python tests/integration_wildcard_cache_tensors.py --comfy-root /path/to/ComfyUI --report /tmp/wildcard-tensors.json ``` +For the core LRU cache, restart the isolated server with `--cache-lru 10` instead +of `--cache-classic`, then run the same cache assertions with +`python -m pytest tests/test_e2e_wildcard_cache.py -k 'plain_text or populate_reuses'`. + +The tensor integration runs on CPU with installed +`v1-5-pruned-emaonly-fp16.safetensors` and `loras/SD1.5/etc/add_detail.safetensors`. +It compares all conditioning tensors and metadata, MODEL/CLIP LoRA patches and +strengths with independent core CLIPTextEncode/LoraLoader/ConditioningConcat +execution, using explicit reference selections and prompt sections. Plain, +populate, fixed multiline/nested, reproduce, LoRA and BREAK cases are covered; +input models must remain unmodified. Only HTTP routes/feedback are substituted. +Missing models fail without downloading or skipping. These checks do not cover +other model architectures or external LoRA loader plugins. + The separate CPU integration uses real prompt hooks, parser, node methods and ComfyUI cache keys, with HTTP feedback and the CLIP model substituted. Fixed expectations from pre-change Main cover dynamic syntax, legacy modes, linked diff --git a/tests/README.md b/tests/README.md index 3b6d5e6f..c5f5aad9 100644 --- a/tests/README.md +++ b/tests/README.md @@ -34,10 +34,15 @@ See [E2E setup and artifacts](../docs/E2E_TEST_STRATEGY.md). These scripts need a running test server and are separate from the curl-based API integration tests. `python3 -m pytest tests/test_e2e_wildcard_cache.py -v` checks both wildcard nodes' seeded output, real downstream cache reuse and native widget restoration in both -modes. Encode cases need a local checkpoint; see the E2E guide. The separate +modes. It also checks equal/different populate results, restored reproduce mode, +and actual linked text/seed execution through a native subgraph. Encode cases +need a local checkpoint; see the E2E guide. The separate `python3 tests/integration_wildcard_cache.py --comfy-root /path/to/ComfyUI` runs on CPU with real prompt hooks/parser/cache keys and a substituted CLIP model. It covers legacy modes, linked inputs, dynamic syntax and saved seed positions. +`python3 tests/integration_wildcard_cache_tensors.py --comfy-root /path/to/ComfyUI` +uses installed SD1.5/LoRA weights on CPU to compare conditioning and MODEL/CLIP +patches against core-node references, including seeded expansion and BREAK. `python3 -m pytest tests/test_e2e_legacy_sam.py -v` checks the legacy SAM menu and Clipspace button without importing the removed `ClipspaceDialog` module. Set `IMPACT_E2E_SAM_IMAGE` and install the configured SAM model to include real diff --git a/tests/integration_wildcard_cache_tensors.py b/tests/integration_wildcard_cache_tensors.py new file mode 100644 index 00000000..d45a4841 --- /dev/null +++ b/tests/integration_wildcard_cache_tensors.py @@ -0,0 +1,109 @@ +"""Compare wildcard Encode outputs with independent core CLIP/LoRA execution. + +Run separately from pytest's stubbed tests. Uses installed SD1.5 checkpoint and +LoRA weights on CPU; downloads nothing. Only unused HTTP routes/feedback are +substituted. Expected selections and reference text are explicit test data. +""" +import argparse +import json +from pathlib import Path +import sys +from types import SimpleNamespace +import unittest + +import torch + +from integration_lora_wildcard_tensors import assert_lora_patches_equal + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--comfy-root", type=Path, required=True) + parser.add_argument("--report", type=Path) + args = parser.parse_args() + root = Path(__file__).resolve().parents[1] + sys.path[:0] = [str(args.comfy_root.resolve()), str(root / "modules")] + from comfy.cli_args import args as comfy_args + comfy_args.cpu = True + torch.set_num_threads(2) + from aiohttp import web + from server import PromptServer + PromptServer.instance = SimpleNamespace( + routes=web.RouteTableDef(), add_on_prompt_handler=lambda handler: None, + send_sync=lambda *args: None, + ) + import nodes + from impact import impact_server, impact_pack + + model, clip, _ = nodes.CheckpointLoaderSimple().load_checkpoint("v1-5-pruned-emaonly-fp16.safetensors") + lora = "SD1.5/etc/add_detail.safetensors" + + class EncodeTensorTests(unittest.TestCase): + def compare(self, *, seed, text, expected, mode="fixed", source="", sections=None, with_lora=False): + inputs = dict(wildcard_text=source, populated_text=text, mode=mode, seed=seed) + prompt = {"prompt": {"1": {"class_type": "ImpactWildcardEncode", "inputs": inputs}}} + impact_server.onprompt_populate_wildcards(prompt) + actual_model, actual_clip, actual_conditioning, actual_text = impact_pack.ImpactWildcardEncode().doit( + model=model, clip=clip, **inputs) + self.assertEqual(actual_text, expected) + reference_model, reference_clip = model, clip + if with_lora: + reference_model, reference_clip = nodes.LoraLoader().load_lora(model, clip, lora, 0.6, 0.4) + self.assertTrue(actual_model.patches) + self.assertTrue(actual_clip.patcher.patches) + else: + self.assertIs(actual_model, model) + self.assertIs(actual_clip, clip) + reference_conditioning = None + for section in sections or [expected]: + encoded = nodes.CLIPTextEncode().encode(reference_clip, section)[0] + reference_conditioning = encoded if reference_conditioning is None else ( + nodes.ConditioningConcat().concat(reference_conditioning, encoded)[0]) + torch.testing.assert_close(actual_conditioning, reference_conditioning, rtol=0, atol=0) + assert_lora_patches_equal(actual_model.patches, reference_model.patches) + assert_lora_patches_equal(actual_clip.patcher.patches, reference_clip.patcher.patches) + self.assertEqual(model.patches, {}) + self.assertEqual(clip.patcher.patches, {}) + + def test_plain_text(self): + for seed in (2, 5): + with self.subTest(seed=seed): + self.compare(seed=seed, text="a red flower", expected="a red flower") + + def test_populate_selections(self): + for seed, color in ((0, "green"), (1, "green"), (2, "red"), (5, "blue")): + with self.subTest(seed=seed): + self.compare(seed=seed, text="old result", source="{red|green|blue}", + mode="populate", expected=color) + + def test_fixed_multiline_and_nested(self): + for text in ("{red|\ngreen|blue}", "{red|{green}|blue}"): + for seed, color in ((2, "red"), (5, "blue")): + with self.subTest(text=text, seed=seed): + self.compare(seed=seed, text=text, expected=color) + + def test_reproduce_ignores_source(self): + self.compare(seed=5, text="red", source="{red|green|blue}", mode="reproduce", expected="red") + + def test_lora_and_break(self): + tag = "" + for seed, color in ((2, "red"), (5, "blue")): + with self.subTest(seed=seed): + self.compare(seed=seed, text="", mode="populate", + source=tag + " {red|green|blue} BREAK a flower", + expected=tag + f" {color} BREAK a flower", + sections=[color, "a flower"], with_lora=True) + + result = unittest.TextTestRunner(verbosity=2).run(unittest.defaultTestLoader.loadTestsFromTestCase(EncodeTensorTests)) + if args.report: + args.report.write_text(json.dumps({ + "tests_run": result.testsRun, "successful": result.wasSuccessful(), + "failures": [{"test": str(test), "detail": detail} for test, detail in result.failures], + "errors": [{"test": str(test), "detail": detail} for test, detail in result.errors], + "skipped": result.skipped, + }, indent=2) + "\n") + return 0 if result.wasSuccessful() else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_e2e_wildcard_cache.py b/tests/test_e2e_wildcard_cache.py index 602b8e67..50acfd21 100644 --- a/tests/test_e2e_wildcard_cache.py +++ b/tests/test_e2e_wildcard_cache.py @@ -72,6 +72,144 @@ def run(ui, preview_id, expected, label): return entry +def native_roundtrip(ui, node_id): + """Compare rendered widget names/order/values and the native serialization.""" + show_node(ui, node_id) + before = widget_state(ui, node_id) + trusted = json.loads((ROOT / "tests/workflows/all-impact-nodes.widgets.json").read_text()) + reference = next(node for node in trusted["nodes"] if node["type"] == before["type"]) + assert [w["name"] for w in before["widgets"]] == [w["name"] for w in reference["widgets"]] + workflow = ui.artifacts / "saved.json" + export_workflow(ui, workflow) + saved = json.loads(workflow.read_text()) + ui.new_workflow() + ui.load_workflow(workflow, len(saved["nodes"])) + full_zoom(ui) + show_node(ui, node_id) + assert widget_state(ui, node_id) == before + roundtrip = ui.artifacts / "roundtrip.json" + export_workflow(ui, roundtrip) + restored = json.loads(roundtrip.read_text()) + assert {str(n["id"]): n.get("widgets_values", []) for n in restored["nodes"]} == { + str(n["id"]): n.get("widgets_values", []) for n in saved["nodes"]} + # Native loading can reorder link records; their IDs and endpoints must + # remain identical. Widget arrays above deliberately retain their order. + assert sorted(restored["links"]) == sorted(saved["links"]) + assert restored.get("definitions") == saved.get("definitions") + (ui.artifacts / "widget-state.json").write_text(json.dumps(before, indent=2) + "\n") + return before + + +def input_slot(ui, node_type, name): + return ui.app.evaluate("""(app, {type, name}) => + app.graph._nodes.find(n => n.type === type).inputs.findIndex(i => i.name === name) + """, {"type": node_type, "name": name}) + + +@pytest.mark.parametrize("restore_by_name", [False, True], ids=["positional", "named"]) +@pytest.mark.parametrize("node_type", NODE_TYPES) +def test_linked_inputs_and_subgraph_native_queue(ui, node_type, restore_by_name): + setting(ui, "Restore widget values", "Comfy.Workflow.NamedValuesRestore", restore_by_name) + node_id, preview_id = graph(ui, node_type) + ui.choose(node_type, "mode", "fixed") + source_id = ui.add_node("Concatenate Text", "StringConcatenate", 100, 650) + show_node(ui, source_id) + # Neither fragment contains a complete choice, so scanning upstream input + # strings cannot determine the seed dependence of their combined output. + ui.page.get_by_placeholder("string_a", exact=True).fill("{red|") + ui.page.get_by_placeholder("string_b", exact=True).fill("green|blue}") + show_node(ui, node_id) + connect(ui, "StringConcatenate", node_type, target_slot=input_slot(ui, node_type, "populated_text")) + for seed, expected in ((2, "red"), (5, "blue")): + ui.set_number(node_type, "seed", seed) + entry = run(ui, preview_id, expected, f"linked-text-{seed}") + assert entry["prompt"][2][node_id]["inputs"]["populated_text"] == [source_id, 0] + assert entry["prompt"][2][node_id]["inputs"]["seed"] == seed + + seed_id = ui.add_node("ImpactInt", "ImpactInt", 850, 180) + show_node(ui, seed_id) + ui.set_number("ImpactInt", "value", 2) + show_node(ui, node_id) + connect(ui, "ImpactInt", node_type, target_slot=input_slot(ui, node_type, "seed")) + entry = run(ui, preview_id, "red", "linked-seed") + assert entry["prompt"][2][node_id]["inputs"]["seed"] == [seed_id, 0] + + # Convert the real upstream node using the frontend menu. No graph mutation + # or hand-written workflow JSON is used to construct the subgraph. + show_node(ui, source_id) + title = ui.page.evaluate("""({app, id}) => { + const n = app.graph.getNodeById(id); + return app.canvasPosToClientPos([n.pos[0] + 40, n.pos[1] - 15]); + }""", {"app": ui.app, "id": source_id}) + ui.page.mouse.click(*title) + ui.page.mouse.click(*title, button="right") + ui.page.get_by_role("menuitem", name="Convert to Subgraph", exact=True).click() + ui.page.wait_for_function("""({app, id}) => !app.graph.getNodeById(id) + """, arg={"app": ui.app, "id": source_id}) + native_roundtrip(ui, node_id) + saved = json.loads((ui.artifacts / "saved.json").read_text()) + assert any(n["type"] == "StringConcatenate" + for subgraph in saved["definitions"]["subgraphs"] for n in subgraph["nodes"]) + for seed, expected in ((5, "blue"), (2, "red")): + show_node(ui, seed_id) + ui.set_number("ImpactInt", "value", seed) + entry = run(ui, preview_id, expected, f"restored-subgraph-{seed}") + inputs = entry["prompt"][2][node_id]["inputs"] + assert inputs["seed"] == [seed_id, 0] + assert isinstance(inputs["populated_text"], list) + upstream_id = inputs["populated_text"][0] + assert entry["prompt"][2][upstream_id]["class_type"] == "StringConcatenate" + + +@pytest.mark.parametrize("node_type", NODE_TYPES) +def test_populate_reuses_cache_only_for_same_expansion(ui, node_type): + node_id, preview_id = graph(ui, node_type) + prefix = "populate cache " + ui.page.get_by_placeholder("Wildcard Prompt (User input)", exact=True).fill(prefix + "{red|green|blue}") + # Trusted pre-change selections: 0 and 1 -> green; 2 -> red. + for seed, color in ((0, "green"), (1, "green"), (2, "red")): + ui.set_number(node_type, "seed", seed) + entry = run(ui, preview_id, prefix + color, f"populate-cache-{seed}") + if seed == 1: + assert {node_id, preview_id} <= cached(entry) + else: + assert {node_id, preview_id}.isdisjoint(cached(entry)) + assert entry["prompt"][2][node_id]["inputs"]["seed"] == 0 + assert next(w["value"] for w in widget_state(ui, node_id)["widgets"] if w["name"] == "seed") == seed + + +@pytest.mark.parametrize("restore_by_name", [False, True], ids=["positional", "named"]) +@pytest.mark.parametrize("node_type", NODE_TYPES) +def test_reproduce_native_workflow_then_populate(ui, node_type, restore_by_name): + setting(ui, "Restore widget values", "Comfy.Workflow.NamedValuesRestore", restore_by_name) + node_id, preview_id = graph(ui, node_type) + ui.page.get_by_placeholder("Wildcard Prompt (User input)", exact=True).fill("{red|green|blue}") + ui.set_number(node_type, "seed", 2) + initial = run(ui, preview_id, "red", "initial-populate") + # The queued native metadata also preserves the original seed and stores + # expanded text in reproduce mode, as image/workflow reproduction requires. + saved_node = next(n for n in initial["prompt"][3]["extra_pnginfo"]["workflow"]["nodes"] + if str(n["id"]) == node_id) + assert saved_node["widgets_values"][:3] == ["{red|green|blue}", "red", "reproduce"] + seed_index = 3 if node_type == "ImpactWildcardProcessor" else 5 + assert saved_node["widgets_values"][seed_index] == 2 + ui.choose(node_type, "mode", "reproduce") + ui.set_number(node_type, "seed", 5) + before = native_roundtrip(ui, node_id) + values = {w["name"]: w["value"] for w in before["widgets"]} + assert (values["wildcard_text"], values["populated_text"], values["mode"], values["seed"]) == ( + "{red|green|blue}", "red", "reproduce", 5) + run(ui, preview_id, "red", "restored-reproduce") + ui.page.wait_for_function("""({app, id}) => + app.graph.getNodeById(id).widgets.find(w => w.name === 'mode').value === 'populate' + """, arg={"app": ui.app, "id": node_id}) + run(ui, preview_id, "blue", "next-populate") + assert widget_state(ui, node_id)["widgets"] == [ + dict(w, value="populate") if w["name"] == "mode" else + dict(w, value="blue") if w["name"] == "populated_text" else w for w in before["widgets"] + ] + + @pytest.mark.parametrize("node_type", NODE_TYPES) def test_plain_text_reuses_downstream_cache(ui, node_type): node_id, preview_id = graph(ui, node_type) @@ -120,24 +258,7 @@ def test_seeded_output_and_native_widgets(ui, node_type, restore_by_name): ui.set_number(node_type, "seed", 731) ui.choose(node_type, "control_after_generate", "increment") rendered(ui) - before = widget_state(ui, node_id) - trusted = json.loads((ROOT / "tests/workflows/all-impact-nodes.widgets.json").read_text()) - reference = next(node for node in trusted["nodes"] if node["type"] == node_type) - assert [w["name"] for w in before["widgets"]] == [w["name"] for w in reference["widgets"]] - workflow = ui.artifacts / "saved.json" - export_workflow(ui, workflow) - saved = json.loads(workflow.read_text()) - ui.new_workflow() - ui.load_workflow(workflow, len(saved["nodes"])) - full_zoom(ui) - show_node(ui, node_id) - assert widget_state(ui, node_id) == before - roundtrip = ui.artifacts / "roundtrip.json" - export_workflow(ui, roundtrip) - restored = json.loads(roundtrip.read_text()) - saved_widgets = {str(node["id"]): node.get("widgets_values", []) for node in saved["nodes"]} - assert {str(node["id"]): node.get("widgets_values", []) for node in restored["nodes"]} == saved_widgets - (ui.artifacts / "widget-state.json").write_text(json.dumps(before, indent=2) + "\n") + before = native_roundtrip(ui, node_id) run(ui, preview_id, "saved plain α", "restored") assert widget_state(ui, node_id)["widgets"] == [ dict(widget, value=732) if widget["name"] == "seed" else widget for widget in before["widgets"]