feat(activity): scan new and unresolved files by default, and download a job report (#273)

* feat(activity): scan new and unresolved files by default, all files on request

Scan for metadata now skips, before reading any metadata, the files the
Library already links with a resolved link (one generated_images row with
exactly one pair link), found with one streamed query that keeps only the
scanned paths and compares paths with os.path.normcase on both sides.
Skipped files count as Already known ("Skipped · already linked").
Unlinked, fallback-linked, ambiguous and unresolved files are read again.

POST /prompt_manager/jobs/scan takes {scope: "new" | "all"} (default
"new", anything else is a 400). The review dialog shows the concept's
"New and unresolved files" and adds an "All files" switch (D58). The
legacy /prompt_manager/scan route keeps scanning all files with unchanged
SSE payloads.

* feat(activity): download a job report

GET /prompt_manager/jobs/{id}/report returns the summary, counters, every
outcome group's listed rows with truncated flags, the scope and the log as a
JSON attachment (prompt-manager-<kind>-<YYYYMMDD-HHMM>.json), with no absolute
paths. The job view's Download report saves it through the api-client URL, and
the scan log opens at its newest line and follows new lines only while the
reader is at the bottom.

* fix(activity): re-read known files whose size changed, and review follow-ups

The new scope now skips a known, resolved file only while its stored
generated_images.file_size equals its size on disk; a NULL stored size is
re-read. A file replaced at the same path is read again and its conflict
shows under Needs review. The scan stores each file's size when it ingests
it, and both sides of the known-set match go through normalize_image_path.

A failed known-set read logs and falls back to reading every file, and an
empty request body asks for the default scope. The All files switch is
described by its hint, a lost job's Download report is disabled, the report
toast is skipped once the view is gone, and the scan log opens at its
newest line every time. Fixtures use the real scope shape.

* test(activity): wait for the job view's settled detail read, not a fixed delay

The 200-row cap test now starts counting detail reads once one read has
returned the full listing and none is in flight. The job view re-reads after
an in-flight read only when the summary still lists more than that read
returned, so no redundant read is left due after opening.
This commit is contained in:
Vito
2026-10-21 18:07:16 -07:00
committed by GitHub
parent a6f6697c50
commit 2cecfcf0ac
16 changed files with 1068 additions and 71 deletions
+61
View File
@@ -181,6 +181,21 @@ AND p.id NOT IN (SELECT prompt_id FROM live_hint_prompts)
"""
# The scan's known set (#249 Activity-2): every image row, with its stored file
# size and whether its pair link is resolved: exactly one distinct pair, the rule
# _LINK_SUMMARY_SQL and get_image_prompt_info use. A file is known and resolved
# when it has exactly one image row, that row is resolved, and the row's stored
# size equals the file's size now; several rows (one file linked to more than
# one prompt) are ambiguous, and an unknown (NULL) stored size is not known. The
# rows are streamed in fetches of KNOWN_PATHS_FETCH_SIZE and only the scanned
# paths are kept, so memory follows the scan (at most MAX_SCAN_FILES paths), not
# the table.
_KNOWN_PATHS_SQL = """SELECT gi.file_path, gi.image_path, gi.file_size,
(SELECT COUNT(DISTINCT ip.pair_id) FROM image_prompt_pairs ip
WHERE ip.image_id = gi.id) = 1
FROM generated_images gi"""
KNOWN_PATHS_FETCH_SIZE = 1000
# One SELECT reads the link summaries of at most this many paths: three bound
# parameters per path stays under SQLite's historical 999-variable limit.
LINK_SUMMARY_BATCH = 200
@@ -2280,6 +2295,52 @@ class PromptDatabase(
"parameters": row[8] if len(row) > 8 else None,
}
def get_known_resolved_paths(self, files: Dict[str, Optional[int]]) -> Set[str]:
"""The paths in ``files`` the Library already links with a resolved link.
``files`` maps each scanned path to its size on disk (None when it could
not be read). A path qualifies when exactly one ``generated_images`` row
claims the file, that row has exactly one pair link
(``image_prompt_pairs``, written only by resolved ingestion), and its
stored ``file_size`` is not NULL and equals the given size, so a file
replaced at the same path is read again. Files linked without a pair
(the monitor's fallback and FIFO links, the legacy positive-only
import), linked to several prompts or pairs, or not linked at all do
not qualify. Both sides go through ``normalize_image_path`` (absolute,
normalized, ``os.path.normcase``), so a Windows row stored in another
case or spelling still matches. One query, streamed; returns the
matching keys of ``files``.
"""
wanted: Dict[str, List[Tuple[str, Optional[int]]]] = {}
for path, size in files.items():
key = normalize_image_path(str(path))
wanted.setdefault(key, []).append((str(path), size))
if not wanted:
return set()
rows_seen: Dict[str, int] = {}
stored_sizes: Dict[str, int] = {}
with self.model.get_connection() as conn:
cursor = conn.execute(_KNOWN_PATHS_SQL)
while True:
rows = cursor.fetchmany(KNOWN_PATHS_FETCH_SIZE)
if not rows:
break
for file_path, image_path, file_size, is_resolved in rows:
# image_path is NOT NULL; file_path is NULL only on old rows.
key = normalize_image_path(file_path or image_path)
if key not in wanted:
continue
rows_seen[key] = rows_seen.get(key, 0) + 1
if is_resolved and file_size is not None:
stored_sizes[key] = file_size
return {
path
for key, stored_size in stored_sizes.items()
if rows_seen[key] == 1
for path, size in wanted[key]
if size is not None and size == stored_size
}
def get_image_link_summaries(
self, paths: List[str], allow_filename_fallback: bool = True
) -> Dict[str, Dict[str, Any]]: