perf(images): read prompt links in one SQLite step (#354) (#418)

* perf(images): read prompt links in one SQLite step (#354)

get_image_link_matches streamed one sqlite3_step per linked row, and
Python's sqlite3 releases and retakes the GIL around every step. While
ComfyUI generates, each retake waits on the busy thread, so the
first-page link read of the workspace Images page stretched from tens of
milliseconds to seconds.

SQLite now returns the unchanged filter SELECT's rows as one JSON value
(json_group_array), decoded with json.loads: a constant two or three GIL
hand-backs instead of one per row. Nothing is cached. Any
OperationalError from the wrapped statement, or a BLOB file_path or
rating, makes that read stream as before with one warning per process; a
missing JSON function stops the attempt for the rest of the process.

tests/test_database_link_matches.py proves the result equals the
streamed read on every filter combination and on NULL ratings, quoted,
backslashed, non-ASCII and multi-line paths, ambiguous and pair links,
an empty database and both fallbacks; the structural budget test also
checks the wrapped query plan.

* fix(images): keep the maintenance link reader, refine the one-step fallback (#354)

The one-step reader was named _link_rows, which shadowed
MaintenanceOperations._link_rows on PromptDatabase and broke the Missing
links preview and apply; it is now _one_step_link_rows, and a test pins
that the maintenance reader is not shadowed.

An interrupted read (Move library retiring the connection) is raised as
before instead of falling back and spending the one warning; a JSON text
over SQLite's length limit (DataError) streams like the other value
failures. The equivalence test compares value types per path rather than
by row order, and a test checks that the read leaves no WAL snapshot
pinned.

* docs(testing): name every one-step link read fallback (#354)

database/operations.py

@@ -243,7 +243,8 @@ WHERE gi.image_path IN ({marks}) OR gi.file_path IN ({marks}){fallback}"""
 # writer since the 3.2.4 migration stores file_path, so it is the path key.
 # The prompt-level conditions are uncorrelated IN sets SQLite builds once, and
 # the CASE evaluates the per-row ambiguity probe only for admitted rows.
-_LINK_FILTER_SQL = """SELECT gi.file_path, p.rating, {hit}, p.id
+_LINK_FILTER_SQL = """SELECT gi.file_path AS file_path, p.rating AS rating,
+       {hit} AS matched, p.id AS positive
 FROM generated_images gi
 LEFT JOIN prompt_pairs pp ON pp.id = (
     SELECT CASE WHEN COUNT(DISTINCT ip.pair_id) = 1 THEN MIN(ip.pair_id) END
@@ -252,6 +253,19 @@ JOIN prompts p ON p.id = COALESCE(pp.positive_prompt_id, gi.prompt_id)
 WHERE gi.file_path IS NOT NULL AND CASE WHEN {admit} THEN NOT EXISTS (
     SELECT 1 FROM generated_images other
     WHERE other.file_path = gi.file_path AND other.id <> gi.id) ELSE 0 END"""
+# _LINK_FILTER_SQL's rows as one JSON array, built by SQLite in one step (#354).
+# Python's sqlite3 releases and retakes the GIL around every step, so a read
+# that steps once per linked row waits on a busy ComfyUI thread thousands of
+# times. A BLOB file_path or rating (no writer stores one) makes the value NULL
+# (or json_array refuses it), and the read streams instead.
+_LINK_MATCHES_JSON_SQL = """SELECT CASE
+    WHEN max(typeof(file_path) = 'blob' OR typeof(rating) = 'blob') THEN NULL
+    ELSE json_group_array(json_array(file_path, rating, matched, positive)) END
+FROM ({select})"""
+# Set when this SQLite has no JSON functions: every later read streams.
+_one_step_unsupported = False
+# One warning per process when a read falls back to streaming.
+_one_step_warned = False
 # The linked positive of each given path (one unambiguous link, pair scope as
 # in _LINK_FILTER_SQL) with its category; {hit} as in link_filter_query. Used
 # for the few files whose current image tags meet a Tags filter (#249 B3).
@@ -310,6 +324,25 @@ def link_filter_query(
     return sql, params
 
 
+def link_matches_json_query(sql: str) -> str:
+    """*sql* from link_filter_query, wrapped to return its rows as one JSON value."""
+    return _LINK_MATCHES_JSON_SQL.format(select=sql)
+
+
+def _streamed_link_matches(
+    conn: sqlite3.Connection,
+    sql: str,
+    params: Dict[str, Any],
+    wanted: Optional[Set[str]],
+) -> Dict[str, Tuple[bool, Optional[int], int]]:
+    """The link read stepped once per row (the fallback of the one-step read)."""
+    matches: Dict[str, Tuple[bool, Optional[int], int]] = {}
+    for path, rating, matched, positive in conn.execute(sql, params):
+        if wanted is None or path in wanted:
+            matches[path] = (bool(matched), rating, positive)
+    return matches
+
+
 _ASCII_FOLD = str.maketrans("ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")
 
 
@@ -2558,17 +2591,65 @@ class PromptDatabase(
         names its negative variant and is not searched.
         Without category or tags only hits are returned, unless *every_link*
         asks for every linked image (the rating sort and the positive count).
-        The positive id is the card's linked positive. One SELECT whose rows
-        are streamed; with *wanted* (the listing's normalized paths) only those
-        paths are kept, so memory is bounded by the listing, not the database.
+        The positive id is the card's linked positive.
+
+        One SELECT, returned as one JSON value (#354): SQLite steps once, so
+        the read takes the GIL back a constant two or three times instead of
+        once per linked row while ComfyUI generates. Peak memory therefore
+        grows with the database's matching linked rows (about 530 bytes each,
+        freed on return); the result is bounded by *wanted* (the listing's
+        normalized paths) when given. Where the JSON step fails (no JSON
+        functions, a BLOB value, a JSON text over SQLite's length limit) the
+        rows are streamed as before.
         """
         sql, params = link_filter_query(q, category, tags, every_link=every_link)
-        matches: Dict[str, Tuple[bool, Optional[int], int]] = {}
         with self.model.get_connection() as conn:
-            for path, rating, matched, positive in conn.execute(sql, params):
-                if wanted is None or path in wanted:
-                    matches[path] = (bool(matched), rating, positive)
-        return matches
+            rows = None
+            if not _one_step_unsupported:
+                rows = self._one_step_link_rows(conn, sql, params)
+            if rows is None:
+                return _streamed_link_matches(conn, sql, params, wanted)
+        return {
+            path: (bool(matched), rating, positive)
+            for path, rating, matched, positive in rows
+            if wanted is None or path in wanted
+        }
+
+    def _one_step_link_rows(
+        self, conn: sqlite3.Connection, sql: str, params: Dict[str, Any]
+    ) -> Optional[List[List[Any]]]:
+        """*sql*'s rows from one JSON value, or None when this read must stream.
+
+        An interrupted read (Move library retiring the connection) is raised
+        as before, never retried row by row.
+        """
+        global _one_step_unsupported, _one_step_warned
+        error: Optional[str] = None
+        try:
+            cursor = conn.execute(link_matches_json_query(sql), params)
+            try:
+                ((document,),) = cursor.fetchall()
+            finally:
+                cursor.close()  # resets the statement: its snapshot ends now
+        except (sqlite3.OperationalError, sqlite3.DataError) as failure:
+            # DataError: the JSON text is longer than SQLITE_LIMIT_LENGTH.
+            error = str(failure)
+            if error == "interrupted":
+                raise
+            if error.startswith("no such function"):
+                _one_step_unsupported = True
+        else:
+            if document is not None:
+                return json.loads(document)
+            error = "a BLOB value"
+        if not _one_step_warned:
+            _one_step_warned = True
+            self.logger.warning(
+                "Reading image links row by row (slower while ComfyUI "
+                "generates); the one-step read failed: %s",
+                error,
+            )
+        return None
 
     def get_image_link_facts(
         self, paths: List[str], q: str = ""