Vito
39877202b2
* feat(inspector): use recorded run history when the graph alone can't name the prompt (#386) When a file's own graph leaves the prompt open, both inspect routes ask recorded run history about the file (one run reported it and it was written during that run). The save node that run reported it from anchors the walk like a file-name match, so dataset branches that share one saver prefix show their own prompt, labelled "Recorded run". Still open at one sampler whose PromptManager text was recorded at execution (a language model's output), the recorded text shows as "Execution recorded". Results the file resolves alone never change; a failed lookup judges the file alone. * fix(inspector): recorded texts follow history's saver; conflicts anchor nothing (#386) Review follow-ups: the texts PromptManager recorded are looked up at the sampler history's saver leads to, never at a sampler the unanchored walk named; a file name naming a different saver than history anchors nothing; outputs and texts are read in one transaction; "Recorded run" ranks below every workflow-inferred source; unexpected lookup errors log at error level. * fix(history): look up recorded runs by history's own spelling of the file (#386) The history route inspects a case-normalized target on Windows, while history matches file names exactly, so the occurrence's reference names the file. * refactor(inspector): share the inspector's resolve as inspection_resolution inspect_prompt_evidence now calls inspection_resolution, which returns the resolution to use, the anchored result and whether the recorded saver chose it. Ingestion can call the same helper for files the default profile cannot decide (#247). * feat(ingestion): one per-file prompt decision for the scan and the monitor utils/file_prompt.py holds decide_file_prompt, the ladder the output scan runs today plus the inspector's rule (inspection profile, then the recorded saver) before the legacy parser, with a SCAN and a MONITOR policy. Junk text (blank, numbers, LoRA tags only, model file names) is never stored. graph_prompt_text and parse_comfyui_prompt move here; the parser accepts parsed metadata (#247). * feat(ingestion): store a file's prompt in one transaction, one prompt per file database/scanned_prompts.py links a file to the prompt of one text, creating it when new, inside one BEGIN IMMEDIATE transaction: a file linked to another prompt is a conflict, a failed link leaves no prompt behind and two writers store one prompt. pair_ingestion gains file_link_rows, link_conflict and Marks/mark_created; ingest_resolved_pair takes optional marks for the components it creates. recorded_runs and named_prompt in run_attribution become public (#247). * feat(scan): decide each scanned file with the shared ladder (#247) _ingest_scanned_file calls decide_file_prompt (scan policy) and stores the result: pairs through ingest_resolved_pair and positives through store_scanned_prompt, both with the scan marks. Files the default profile leaves undecided now get the inspector's resolve, with the saver recorded history names, before the legacy parser, so each branch of a shared-prefix dataset run links its own prompt. Junk text is never stored, a file linked to another prompt is a conflict instead of a second link, and a failed link leaves no prompt behind. _BatchHistory reads the stored prompt and the recorded saver from one recorded_runs pass. _parse_comfyui_prompt calls the moved parse_comfyui_prompt. TESTING.md documents the decision. * fix(ingestion): keep the file-name saver anchor out of the ladder (#390) The #390 ruling queues a file that only its saver's file name resolves for Needs review, never linking it. inspection_resolution gains name_anchor; the ladder passes False, so such a file falls to the legacy floor in the scan (and to Needs review) and is never stored by the monitor, while the recorded saver still decides. The inspector keeps the name anchor. * fix(ingestion): no joined PromptManager text from the legacy floor; junk keeps its step's status The legacy parser returns prepend + text + append, so its answer is refused when a PromptManager-family node has a prepend or append part (#343). Junk found at the undecided-graph step keeps that step's unlinked status and review, so an ambiguous file is still queued for Needs review. The monitor policy's ambiguous file no longer asks for the recorded run, and the file size may be read lazily. * fix(ingestion): keep the legacy answer that is a PromptManager node's own text The joined-text refusal also refused the #170 parity files whose PromptManager prepend is a link the browser parser does not follow: its answer is then the node's literal text, which is what the node stores. Refuse only an answer that is not such a node's own text. * fix(scan): one prompt per file for recorded-run links; writes survive a library move The recorded run's stored prompt now links in one transaction with the file's link rows, so a file linked to another prompt is a conflict instead of a second link. The scan's stores run through survive_library_switch. A single file asks history once (a one-file _BatchHistory), the scan passes in_output=False and reads the file size only for files with prompt metadata. A conflict row reads "Kept: already linked to a different prompt". TESTING.md records the rules. * fix(ingestion): match a file's link rows under its resolved spelling too The scan stores realpath spellings and the monitor the watcher's raw path, so a file reached through a symlinked or junction output folder could get a second generated_images row. file_link_rows now queries the path and its realpath (each normalized and raw) and dedupes by row id. TESTING.md notes the reverse order is matched only when the raw spelling is the realpath. * fix(ingestion): a switched-off PromptManager node behind a pass-through stores nothing (#90) The #90 graph walk follows only prompt-like inputs, so a PromptManagerText with Save to library off reaching the encoder through another input name (easy showAnything.anything, a custom node's value) was invisible to it, and the inspector's resolve stored its text. The ladder now checks the positive every step names: a resolved or inspected positive whose path passes a node that is off, or a legacy answer that is such a node's own text, makes the file not_saved. * fix(ingestion): any switched-off node upstream of a positive stores nothing (#90) The ladder walks every input upstream of each sampler or guider positive (not negative, model, clip, latent or image inputs) for PromptManager nodes whose Save to library is off or wired from an unreadable node, so the sampler-less step, an off node with a prepend part and a PromptManagerPair behind a pass-through no longer store its text. The legacy-floor guard reads a node's resolved text. Every guard reports not_saved for a switch that is off and not_saved_wired for one it cannot read. * fix(ingestion): the switch walk skips negative inputs by #90's own rule The upstream walk treated only names containing "negative" as negative, so an off node on a neg, neg_text or text_neg input blocked an enabled positive. prompt_graph gains is_negative_input (the _NEGATIVE_KEY rule, also used by follows_input) and the walk uses it. The guards accept a node with null inputs and a side whose provenance is not an object. * fix(ingestion): the switch walk also skips any input named with "negative" #90's rule matches only whole underscore-separated words, so negative2, negativeprompt and NegativePrompt counted as positive inputs and an off node wired only to one of them blocked the file. The walk skips those too; follows_input is unchanged.
275 lines
10 KiB
Python
275 lines
10 KiB
Python
"""Storing a file's prompt from its text: one transaction, one prompt each (#247)."""
|
|
|
|
import os
|
|
import sqlite3
|
|
import tempfile
|
|
import threading
|
|
import unittest
|
|
from concurrent.futures import ThreadPoolExecutor
|
|
from pathlib import Path
|
|
from unittest import mock
|
|
|
|
try:
|
|
from tests.open_handles import assert_closed
|
|
from tests import test_execution_history_database as history_tests
|
|
except ImportError: # discovered with tests/ as the top-level directory
|
|
from open_handles import assert_closed
|
|
import test_execution_history_database as history_tests
|
|
|
|
from database.operations import PromptDatabase
|
|
from database.pair_ingestion import (
|
|
RUN_MARKS,
|
|
SCAN_MARKS,
|
|
Marks,
|
|
ingest_resolved_pair,
|
|
)
|
|
from database.scanned_prompts import store_scanned_prompt, store_scanned_prompt_or_raise
|
|
from utils.hashing import generate_prompt_hash
|
|
|
|
META = {"prompt": {"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "x"}}}}
|
|
|
|
|
|
def resolved(positive="cat", negative="blur"):
|
|
return {
|
|
"state": "resolved",
|
|
"positive": {"text": positive, "provenance": {"source": "api"}},
|
|
"negative": {"text": negative, "provenance": {"source": "api"}},
|
|
}
|
|
|
|
|
|
class StoreCase(unittest.TestCase):
|
|
def setUp(self):
|
|
self.temp = tempfile.TemporaryDirectory()
|
|
self.db = PromptDatabase(str(Path(self.temp.name) / "test.db"))
|
|
self.path = str(Path(self.temp.name) / "gen.png")
|
|
|
|
def tearDown(self):
|
|
self.db.close_all()
|
|
assert_closed(self, self.temp.name)
|
|
self.temp.cleanup()
|
|
|
|
def prompts(self):
|
|
rows = self.db.model.get_connection().execute(
|
|
"SELECT id, text FROM prompts ORDER BY id"
|
|
)
|
|
return [tuple(row) for row in rows]
|
|
|
|
def links(self):
|
|
rows = self.db.model.get_connection().execute(
|
|
"SELECT prompt_id, image_path FROM generated_images ORDER BY id"
|
|
)
|
|
return [tuple(row) for row in rows]
|
|
|
|
def save(self, text, **fields):
|
|
return self.db.save_prompt(
|
|
text, prompt_hash=generate_prompt_hash(text), **fields
|
|
)
|
|
|
|
|
|
class TestStoreScannedPrompt(StoreCase):
|
|
def test_a_new_text_is_created_with_the_scan_marks_and_linked(self):
|
|
status, prompt_id = store_scanned_prompt_or_raise(
|
|
self.db, self.path, META, " a cat ", SCAN_MARKS
|
|
)
|
|
self.assertEqual(status, "added")
|
|
prompt = self.db.get_prompt_by_id(prompt_id)
|
|
self.assertEqual(prompt["text"], "a cat")
|
|
self.assertEqual(prompt["category"], "scanned")
|
|
self.assertEqual(prompt["tags"], ["auto-scanned"])
|
|
self.assertEqual(prompt["notes"], "Auto-scanned from gen.png")
|
|
self.assertEqual(prompt["run_count"], 0)
|
|
self.assertEqual(self.links(), [(prompt_id, self.path)])
|
|
|
|
def test_a_run_created_prompt_names_the_workflow_run(self):
|
|
_status, prompt_id = store_scanned_prompt_or_raise(
|
|
self.db, self.path, META, "a cat", RUN_MARKS
|
|
)
|
|
prompt = self.db.get_prompt_by_id(prompt_id)
|
|
self.assertEqual(prompt["notes"], "Saved from a workflow run: gen.png")
|
|
self.assertEqual(prompt["tags"], ["auto-scanned"])
|
|
|
|
def test_a_stored_text_is_reused_by_hash_and_keeps_its_fields(self):
|
|
existing = self.save("A Cat", category="mine", notes="kept")
|
|
status, prompt_id = store_scanned_prompt_or_raise(
|
|
self.db, self.path, META, "a cat ", SCAN_MARKS
|
|
)
|
|
self.assertEqual((status, prompt_id), ("linked", existing))
|
|
prompt = self.db.get_prompt_by_id(existing)
|
|
self.assertEqual((prompt["category"], prompt["notes"]), ("mine", "kept"))
|
|
self.assertEqual(prompt["tags"], [])
|
|
self.assertEqual(len(self.prompts()), 1)
|
|
|
|
def test_a_repeat_is_unchanged(self):
|
|
first = store_scanned_prompt_or_raise(
|
|
self.db, self.path, META, "a cat", SCAN_MARKS
|
|
)
|
|
again = store_scanned_prompt_or_raise(
|
|
self.db, self.path, META, "a cat", SCAN_MARKS
|
|
)
|
|
self.assertEqual(again, ("unchanged", first[1]))
|
|
self.assertEqual(len(self.links()), 1)
|
|
|
|
def test_a_file_linked_to_another_prompt_is_a_conflict(self):
|
|
other = self.save("a dog")
|
|
# Linked under another spelling of the same file.
|
|
spelled = os.path.join(self.temp.name, ".", "gen.png")
|
|
self.db.link_image_to_prompt(other, spelled)
|
|
result = store_scanned_prompt_or_raise(
|
|
self.db, self.path, META, "a cat", SCAN_MARKS
|
|
)
|
|
self.assertEqual(result, ("conflict", None))
|
|
self.assertEqual(self.prompts(), [(other, "a dog")])
|
|
self.assertEqual(self.links(), [(other, spelled)])
|
|
|
|
def test_a_failed_link_raises_and_leaves_no_prompt_behind(self):
|
|
with mock.patch.object(
|
|
self.db, "_insert_image_link", side_effect=sqlite3.OperationalError("x")
|
|
):
|
|
with self.assertRaises(sqlite3.OperationalError):
|
|
store_scanned_prompt_or_raise(
|
|
self.db, self.path, META, "a cat", SCAN_MARKS
|
|
)
|
|
with self.assertLogs("prompt_manager", "ERROR"):
|
|
result = store_scanned_prompt(
|
|
self.db, self.path, META, "a cat", SCAN_MARKS
|
|
)
|
|
self.assertEqual(result, ("found", None))
|
|
self.assertEqual(self.prompts(), [])
|
|
self.assertEqual(self.links(), [])
|
|
|
|
def test_two_concurrent_writers_store_one_prompt_and_one_link(self):
|
|
hold = history_tests.FirstWriterHold()
|
|
outcome = self.db._pair_component_outcome
|
|
started = threading.Barrier(2)
|
|
|
|
def held_open(*args, **kwargs):
|
|
hold()
|
|
return outcome(*args, **kwargs)
|
|
|
|
def write(_index):
|
|
started.wait(timeout=30)
|
|
try:
|
|
return store_scanned_prompt_or_raise(
|
|
self.db, self.path, META, "a cat", SCAN_MARKS
|
|
)[0]
|
|
finally:
|
|
self.db.model.close()
|
|
|
|
with (
|
|
mock.patch.object(
|
|
self.db, "_pair_component_outcome", side_effect=held_open
|
|
),
|
|
ThreadPoolExecutor(max_workers=2) as pool,
|
|
):
|
|
statuses = sorted(pool.map(write, range(2)))
|
|
self.assertEqual(statuses, ["added", "unchanged"])
|
|
self.assertEqual(len(self.prompts()), 1)
|
|
self.assertEqual(len(self.links()), 1)
|
|
|
|
def test_a_library_move_reruns_the_store(self):
|
|
from database import scanned_prompts
|
|
|
|
real = scanned_prompts.store_scanned_prompt_or_raise
|
|
calls = []
|
|
|
|
def moved_once(*args, **kwargs):
|
|
if not calls:
|
|
calls.append(1)
|
|
self.db.model._generation += 1
|
|
raise sqlite3.OperationalError("closed")
|
|
return real(*args, **kwargs)
|
|
|
|
with mock.patch.object(
|
|
scanned_prompts, "store_scanned_prompt_or_raise", side_effect=moved_once
|
|
):
|
|
status, _id = store_scanned_prompt(
|
|
self.db, self.path, META, "a cat", SCAN_MARKS
|
|
)
|
|
self.assertEqual((status, calls), ("added", [1]))
|
|
|
|
def _linked_folder(self):
|
|
"""``(out_real, out_link)``: a symlink to the same output folder."""
|
|
from tests.test_api_duplicate_files import TestReview
|
|
|
|
if not TestReview._can_symlink(self.temp.name): # pragma: no cover
|
|
self.skipTest("symlinks are not available")
|
|
real = Path(self.temp.name) / "out_real"
|
|
real.mkdir()
|
|
link = Path(self.temp.name) / "out_link"
|
|
os.symlink(str(real), str(link))
|
|
(real / "a.png").write_bytes(b"png")
|
|
return os.path.realpath(str(real)), str(link)
|
|
|
|
def test_a_file_linked_through_its_real_folder_is_found_through_a_link(self):
|
|
"""The scan stores realpath spellings, the monitor the watcher's."""
|
|
real, link = self._linked_folder()
|
|
other = self.save("a dog")
|
|
self.db.link_image_to_prompt(other, os.path.join(real, "a.png"))
|
|
through_link = os.path.join(link, "a.png")
|
|
self.assertEqual(
|
|
store_scanned_prompt_or_raise(
|
|
self.db, through_link, META, "a cat", SCAN_MARKS
|
|
),
|
|
("conflict", None),
|
|
)
|
|
self.assertEqual(
|
|
store_scanned_prompt_or_raise(
|
|
self.db, through_link, META, "a dog", SCAN_MARKS
|
|
),
|
|
("unchanged", other),
|
|
)
|
|
pair = ingest_resolved_pair(self.db, through_link, META, resolved())
|
|
self.assertEqual(pair["status"], "conflict")
|
|
self.assertEqual(len(self.links()), 1)
|
|
|
|
|
|
class TestPairMarks(StoreCase):
|
|
def test_marks_reach_only_created_components(self):
|
|
kept = self.save("cat", category="mine")
|
|
result = ingest_resolved_pair(
|
|
self.db, self.path, META, resolved(), marks=RUN_MARKS
|
|
)
|
|
self.assertEqual(result["status"], "added")
|
|
positive = self.db.get_prompt_by_id(kept)
|
|
self.assertEqual((positive["category"], positive["tags"]), ("mine", []))
|
|
negative_id = next(pid for pid, text in self.prompts() if text == "blur")
|
|
negative = self.db.get_prompt_by_id(negative_id)
|
|
self.assertEqual(negative["category"], "scanned")
|
|
self.assertEqual(negative["tags"], ["auto-scanned"])
|
|
self.assertEqual(negative["notes"], "Saved from a workflow run: gen.png")
|
|
|
|
def test_without_marks_created_components_stay_bare(self):
|
|
"""Regression guard (passes on the base): the node path is unchanged."""
|
|
ingest_resolved_pair(self.db, self.path, META, resolved())
|
|
for prompt_id, _text in self.prompts():
|
|
prompt = self.db.get_prompt_by_id(prompt_id)
|
|
self.assertIsNone(prompt["category"])
|
|
self.assertEqual(prompt["tags"], [])
|
|
|
|
def test_a_blank_negative_creates_only_the_positive(self):
|
|
result = ingest_resolved_pair(
|
|
self.db, self.path, META, resolved(negative=""), marks=SCAN_MARKS
|
|
)
|
|
self.assertEqual(result["status"], "added")
|
|
self.assertEqual([text for _id, text in self.prompts()], ["cat"])
|
|
|
|
def test_marks_are_a_frozen_value(self):
|
|
self.assertEqual(
|
|
SCAN_MARKS, Marks("scanned", ("auto-scanned",), SCAN_MARKS.note)
|
|
)
|
|
with self.assertRaises(AttributeError):
|
|
SCAN_MARKS.category = "x"
|
|
|
|
|
|
class TestPublicRunAttribution(unittest.TestCase):
|
|
def test_recorded_runs_and_named_prompt_are_public(self):
|
|
from database import run_attribution
|
|
|
|
self.assertTrue(callable(run_attribution.recorded_runs))
|
|
self.assertTrue(callable(run_attribution.named_prompt))
|
|
self.assertFalse(hasattr(run_attribution, "_recorded_runs"))
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|