From 69de20a427ad551ddf69c3685b1cf07eca17943f Mon Sep 17 00:00:00 2001 From: Vito Date: Fri, 23 Oct 2026 13:15:06 -0700 Subject: [PATCH 1/2] feat(system): Sources & scans (#309) * feat(system): report configured, effective and watched sources GET /prompt_manager/settings adds settings.monitoring_enabled, a sources block (configured roots with valid/code/reason, effective roots with their listing index and watch state, auto_detected) and the monitor state (present, observer_alive, scheduled). Every path is a public path. A source is "watching" only while a live observer scheduled it, "manual" when monitoring is off or the process has no monitor, and otherwise "restart_pending". POST accepts monitoring_enabled as a strict bool, saved under gallery.monitoring.enabled; restart_required_fields names it when the running observer does not already match. A save that changes the sources runs as a "Source change" records hold, so it is refused with 409 busy while a scan runs. Case variants of one folder are saved once. POST /prompt_manager/settings/validate-source {path} checks a path lexically (no "..", no control characters, native path forms only) and then against the gallery-root rules, answering a code and reason, and writes nothing. * fix(system): no existence oracle in source validation A gallery root's containment in the allowed ComfyUI folders is now decided before its existence and type, so a path outside them answers "outside-allowed" whether or not it exists. validate-source also folds the filesystem-root and home-directory codes into "outside-allowed", with one reason, and logs the detail server-side only. Paths inside the allowed folders still say "not-found" or "not-directory". The legacy settings save shares the same check. * feat(scan): scan one selected source POST /prompt_manager/jobs/scan accepts roots: indexes into the effective sources, strict non-negative ints (no bools or floats), deduplicated, at most 64, non-empty when present. An index past the sources is a 400 "unknown-source" and starts nothing. The job walks only the selected sources, resolved when it was requested, and keeps each file's listing root_index. Its scope records files, roots (null for every source) and output_folders, the number of folders walked, which the log's first line counts. The legacy /scan route still scans every source, and attaching to a scoped job keeps its payload keys. * feat(system): manage sources and start scans natively System > Sources & scans renders the concept's section on the server's sources: one row per effective source with its watch state and a scan scoped to that source, the 3.2.0 update notice with the legacy dismissal, Add source checked by validate-source, Remove, monitoring and worker threads as host drafts behind the one Save row, and Refresh files. A save refused by a running scan names the job. Thumbnail generation hands off to the legacy gallery until S5. Deviations D98, D113 and D-NEW-S4-1 are recorded in TESTING.md. * fix(security): decide source containment as text before any filesystem access Every caller (validate-source, POST /settings, config.json at load) now refuses a source through one check, sources.source_problem: the form check first, then GalleryConfig.gallery_root_problem, which compares the normalised spelling with the allowed folders before anything resolves, stats or opens the path. A UNC share, the \\?\ and \\.\ namespaces, a device or any outside folder is refused with no SMB request, no executor stall and no existence or timing oracle; only then is the path resolved and contained again, which still catches a symlink out. The filesystem root and the home directory answer outside-allowed with the same reason as any outside path, so no caller learns the home path. Also: - POST /settings runs the form checks and parses the roots off the loop; an unusable source is a coded 400 invalid-source naming its public path (as text) and its index. - ComfyUI folders outside the base directory are path anchors, so their sources round-trip and never collide by folder name. - A per-source scan sends each index with its public path; a stale or shifted index is 409 sources-changed and starts nothing. - A source still watched after monitoring was switched off is "stopping". - Roots are compared as folders, not spellings, so an unchanged legacy resend takes no source-change hold. - Tests: no realpath/stat/lstat of an outside path (native and Windows forms with ntpath fakes), the alike answers with /, ~ and a link to ~, an event instead of a timer wait, and a teardown that drains jobs before the temporary database goes. * fix(system): say what Sources saved, refused and still watches - A per-source scan sends the source's public path; when the sources changed since they were read the toast says the server's reason and the list and saved values are read again. - "watching until restart" for a source the running monitor still watches after monitoring was saved off, and the status adds "Restart ComfyUI to apply: Watch for new outputs." from restart_required_fields. - A save refused for one unusable source marks that row and the status is the server's message naming it. - A valid configured source the effective list holds under another spelling reads "same folder as another source". - The busy toast names the job the 409 named (ApiError keeps a validated running job and refused source), falling back to the list. - The Add source dialog's listeners live on the page scope and Enter adds; its markup is compared with the concept (D127). - TESTING.md: D-NEW-S4-1 is D127; D-NEW-S4-2 records the new copy; the Sources contract describes the lexical containment, invalid-source and sources-changed. * fix(security): guard the Windows form probes on a token of their own The no-filesystem guard matched "C:", which on a Windows runner is also the temporary ComfyUI tree the save may resolve for its trusted first entry. The probe paths now carry a token no trusted directory holds. * docs(system): number the Sources states deviation D128 --- TESTING.md | 118 ++- py/api/admin.py | 52 +- py/api/job_routes.py | 110 +- py/api/settings_routes.py | 277 ++++- py/api/sources.py | 280 +++++ py/config.py | 146 ++- tests/browser/system-concept.spec.mjs | 243 +++++ .../browser/workspace-system-sources.spec.mjs | 401 ++++++++ tests/browser/workspace-system.spec.mjs | 5 +- tests/browser/workspace.spec.mjs | 10 +- tests/browser/workspace_system_fixture.py | 78 +- tests/js/api-client.test.js | 17 + tests/js/workspace-system-sources.test.js | 145 +++ tests/js/workspace-system-state.test.js | 48 + tests/test_api_admin.py | 18 + tests/test_gallery_roots.py | 21 +- tests/test_scan_scope.py | 235 ++++- tests/test_settings_sources.py | 959 ++++++++++++++++++ tools/vendor/frontend-coverage-baseline.json | 12 +- web/js/api-client.js | 15 + web/js/workspace/activity-scan.mjs | 30 +- web/js/workspace/system-sources.mjs | 395 ++++++++ web/js/workspace/system-state.mjs | 37 +- web/js/workspace/system.mjs | 31 +- web/js/workspace/workspace.css | 2 + 25 files changed, 3545 insertions(+), 140 deletions(-) create mode 100644 py/api/sources.py create mode 100644 tests/browser/workspace-system-sources.spec.mjs create mode 100644 tests/js/workspace-system-sources.test.js create mode 100644 tests/test_settings_sources.py create mode 100644 web/js/workspace/system-sources.mjs diff --git a/TESTING.md b/TESTING.md index 0ea8452..12adaec 100644 --- a/TESTING.md +++ b/TESTING.md @@ -1929,6 +1929,56 @@ cache off selecting the shown file again re-reads it (on, the one current record reused); paths off leaves out only the Path row of File & provenance; the viewer reads the video defaults. +Sources & scans (`system-sources.mjs`, S4) is the concept's "sources" body on the +server's sources. `GET /prompt_manager/settings` reports them (public paths only): +`sources.configured` (each `gallery_root_paths` entry with `valid`, `code` and the +server's `reason`), `sources.effective` (what is scanned and served, with its +`root_index` and `watch`: `watching` only when the image monitor's observer is alive, +scheduled that folder and monitoring is on, `stopping` when it still watches it but +monitoring was switched off, `manual` when monitoring is off or no monitor runs, else +`restart_pending`), `sources.auto_detected`, `monitor` and `settings.monitoring_enabled`. +`POST /prompt_manager/settings/validate-source {path}` checks one folder and writes +nothing. Every caller (validate-source, the settings save, config.json at load) refuses +a source through the same checks (`sources.source_problem`): its form first, then +containment decided as text (`normcase(normpath)` against the allowed folders' own +spellings), before anything resolves, stats or opens the path, so a UNC share, the +`\\?\` and `\\.\` namespaces, a device or any folder outside the allowed ones never +reaches the filesystem or the network (no SMB request, no existence or timing oracle). +Only then is the path resolved and contained again, which catches a symlink out. The +filesystem root and the home directory answer `outside-allowed` with the same reason as +any outside path. ComfyUI folders outside its base directory (`--output-directory`) +are path anchors too, so their sources keep a relative public path +(`renders/portraits`) that round-trips and never collides by folder name. A save whose +list holds a source the server cannot use is a 400 `invalid-source` naming its public +path (`source`, as text: a refused path is never resolved to name it), its `index` and +`reason`, and saves nothing. A save that changes the sources (compared as folders, not +spellings) holds the registry as `records` ("Source change"), so it is refused with +409 `busy` while a scan runs. `POST /prompt_manager/jobs/scan {scope, roots, paths}` +scans only the given effective `root_index`es (strict ints, at most 64), each with the +public path the page read for it; an index past the list or one that now names another +folder is 409 `sources-changed` and starts nothing (the page toasts the server's +reason and reads the sources again); the legacy `/scan` is unchanged. The section +lists one row per source (" · watching | watching until restart | manual scan | +restart pending" with "Scan changes" for a watched source, else "Scan folder"), which +opens the shared "Review metadata scan" for that one source ("1 output folder") and +starts the session job over it; the job opens in Activity, and the toast only says it +started. Enrich +existing metadata and the notice's Review scan review every source. The notice shows +until this browser dismissed the 3.2.0 update notice (`pm_update_notice_dismissed`, +the legacy admin page's key); Later dismisses it without scanning, and Review scan +dismisses it as the legacy Scan does. "+ Add source" asks validate-source, then adds +the folder to the host draft of `gallery_root_paths`; Remove takes one out; "Watch for +new outputs" (`monitoring_enabled`) and Worker threads (`worker_threads`) are host +drafts too, and Save writes them with the revision. A save the server refuses because a +scan runs toasts "Another job is running: ." with a link to that job (D57) and +keeps the drafts; a save refused for one source says the server's message as the status +and marks that row. After a save the source list is read again; a monitoring change +waits for a restart (the running observer keeps watching until then, and the status +says "Restart ComfyUI to apply: Watch for new outputs."). Thumbnail generation's +Review options opens the legacy gallery in a new tab (D94) until its review dialog (S5). +A list that cannot be read shows "Sources could not be loaded. Your saved data has not +changed." and Retry, never an empty list. + Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per visit through `latestOnly()` on the section scope; a late answer after the section left paints nothing. Until it answers, the section shows "Loading the overview…"; a failure @@ -1956,7 +2006,7 @@ the connection's `total_changes` is 0 at close and the data reads back unchanged `-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no reader can keep it byte-identical (`test_health_check_never_writes`). -Recorded deviations (plan D60–D65 renumbered D92–D97; D121, D122, D123, D124, D125 and D126 assigned): +Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121–D128 assigned): - **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server @@ -2023,6 +2073,47 @@ Recorded deviations (plan D60–D65 renumbered D92–D97; D121, D122, D123, D124 defaults" ("Restore this browser’s display defaults, then save. Other settings are kept.", button "Reset"), which drafts the gallery's defaults for the browser rows only. The node search timeout shows the server's real value, not the sample 30. CSS under D97. +- **D98** (Sources): each configured source's row has a plain Remove after its scan + button; nothing changes until Save. A source added this session reads "<path> · not + saved yet" with Remove only until Save; a configured source the server cannot use + reads "<path> · <server reason>" with Remove only. A line under the list says when the + list is not the whole story: "No source is configured. After saving, ComfyUI’s output + folder is used." (the last one removed), "No source is configured; ComfyUI’s output + folder is used." and "No configured source is available; ComfyUI’s output folder is + used." (auto-detected rows, listed without Remove). "restart pending" is the third + watch state. The operation "Refresh files" ("Read every source again to find files + added or removed outside ComfyUI.", button "Refresh") closes the surface: it calls the + legacy gallery's `/gallery/rescan` and toasts "Files refreshed. N files in M output + folders." or "Files could not be refreshed. Try again.". The concept's "sample path" + note is prototype copy and is dropped. +- **D113** (Sources): the Worker threads range ends at the server's `cpu_count` (the + concept's fixed `max=16`) and shows the server's real `worker_threads` (the concept's + sample 4). +- **D127** (Sources): the concept's `add-source` `simpleInput` with production + copy: the field is "Folder path" (the concept's "(sample only)" dropped), it starts + empty (not the demo "/images/new-collection"), and the action is "Add" (not "Apply in + prototype"); Enter in the field adds too. `add source dialog matches the concept` + (system-concept.spec) compares the open dialog with these three mapped, and `add + source comparison detects mutations` proves the mapping is the only difference. A + folder the server refuses keeps the dialog open with the field + `aria-invalid` and the server's reason in a `p.small[role=alert]` above the actions; + a folder the server names as one already listed says "This folder is already a + source.", and a check that cannot finish says "The folder could not be checked. Try + again."; neither adds anything. + The concept dialog does no check, so `invalid source shows the server reason` + (workspace-system-sources.spec) pins it. +- **D128** (Sources): states the concept's sample rows never show. A source the + running monitor still watches after "Watch for new outputs" was saved off reads + "<path> · watching until restart" (still "Scan changes"), and the save's status adds + "Restart ComfyUI to apply: Watch for new outputs." from `restart_required_fields`. A + configured source the effective list holds under another spelling reads "<path> · + same folder as another source" with only Remove. A save refused for one source + (`invalid-source`) marks that row (`aria-invalid`, outlined like a D92 setting) and + the status is the server's "<path> can't be used: <reason>. Remove it or reconnect it, + then save." ("Remove it, then save." when reconnecting cannot help). Pinned by + `Watch for new outputs saves monitoring_enabled; a restart applies it` and `a save + refused for one unusable source names it and marks its row` + (workspace-system-sources.spec) and the system-state and system-sources node tests. - **D124** (inspector): with "Load metadata on selection" off, the Images panel and the viewer's details show "Metadata loads when you ask for it." and a "Load metadata" button instead of reading the file. The concept has no such state, so no concept @@ -2074,9 +2165,16 @@ renames a metric label, swaps the `--line` token, recolours a chip, renames a ch removes the totals table and renames a caption. `appearance matches the concept` maps only D97 and the real timeout (unmapped, exactly those paths differ), and `appearance comparison detects mutations` removes a row, renames a label and an option and swaps -the `--line` token. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`, +the `--line` token. `sources matches the concept with|without the notice` (both +themes and sizes; the fixture's monitor watches the first source, as the concept's +first sample is watched, and the concept's Later removes its notice) maps the real +sources for the two samples, D98's Remove and Refresh files, and D113; unmapped, +exactly those paths differ. `sources comparison detects mutations` removes the notice +and a source row, renames a label and the Add source button and swaps the `--line` +token. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`, `{concept,production}-{statistics,health}-{dark,light}.png`, -`{concept,production}-appearance-{dark,light}-{1600,390}.png`) are test artifacts; there +`{concept,production}-appearance-{dark,light}-{1600,390}.png`, +`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`) are test artifacts; there are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts, the health check and metric 4, a failed check, the thumbnail row against the status endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the @@ -2085,9 +2183,14 @@ visits versus the current view and a saved view, the embedded gallery keeping Gr metadata off and on request, the cache (counted up to a request barrier), hidden paths, the video defaults, the host fields with their revision, a stale one and a failed read, reload and a second tab, a second tab's save while System was closed, storage refused, -the D97 rows and reset. Its hooks are +the D97 rows and reset. `tests/browser/workspace-system-sources.spec.mjs` covers the +notice's Later and Review scan, a scoped scan from one source into Activity, Add source +with its server check and Save, a refused folder, Remove down to auto-detect, an absent +configured source, the three watch states, the monitoring and worker rows, Refresh +files, the thumbnail hand-off, a save refused by a running scan and a failed read. Its hooks are `tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected -failures, an absent source, a deleted linked file, unthumbnailed files) on the +failures, an absent source, a deleted linked file, unthumbnailed files, a fake image +monitor) on the concept-shaped fixture, with delays through the shared request holds; reset also restores the server's host settings and its `config.json`, which must be the one inside the server's temp root (`tests/test_workspace_system_fixture.py`). @@ -2098,8 +2201,11 @@ python3 -m unittest tests.test_database_health tests.test_system_api tests.test_ node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-controls.test.js \ tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js \ tests/js/workspace-system-appearance.test.js tests/js/workspace-preferences.test.js \ - tests/js/workspace-inspection-state.test.js tests/js/workspace-images-state.test.js + tests/js/workspace-inspection-state.test.js tests/js/workspace-images-state.test.js \ + tests/js/workspace-system-sources.test.js +python3 -m unittest tests.test_settings_sources tests.test_scan_scope -v npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \ + workspace-system-sources.spec.mjs \ --project chromium --project touch --project firefox --project webkit ``` diff --git a/py/api/admin.py b/py/api/admin.py index dc4e9bc..5b530ef 100644 --- a/py/api/admin.py +++ b/py/api/admin.py @@ -4,6 +4,7 @@ import asyncio import contextlib from contextlib import closing import datetime +import functools import hashlib import json import math @@ -1344,28 +1345,34 @@ class AdminRoutesMixin: } ) - def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE): + def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE, roots=None, output_dirs=None): """Register the scan job synchronously; its runner starts on the loop. - ``files`` is the scope (``SCAN_SCOPES``). Raises JobBusy while another - record-changing job runs. + ``files`` is the scope (``SCAN_SCOPES``). ``roots`` selects sources by + their index in ``output_dirs`` (the effective sources the caller + resolved); None scans every source, resolved when the job starts. + Raises JobBusy while another record-changing job runs. """ if files not in SCAN_SCOPES: raise ValueError(f"Unknown scan scope: {files!r}") + if roots is not None and ( + output_dirs is None or not roots or max(roots) >= len(output_dirs) + ): + raise ValueError("A scoped scan needs the sources its roots index") job = self._jobs.start( SCAN_JOB_KIND, SCAN_JOB_TITLE, record_changing=True, - scope={"files": files}, - runner=self._run_scan_job, + scope={"files": files, "roots": None if roots is None else list(roots)}, + runner=functools.partial(self._run_scan_job, output_dirs=output_dirs), groups=SCAN_GROUP_NAMES, ) self._scan_job = job return job - async def _run_scan_job(self, job): + async def _run_scan_job(self, job, output_dirs=None): final = None - async for payload in self._scan_images_events(job): + async for payload in self._scan_images_events(job, output_dirs): job.publish(payload) final = payload.get("type") state = SCAN_FINAL_STATES.get(final, "finished") @@ -1400,16 +1407,25 @@ class AdminRoutesMixin: ) return response - async def _scan_images_events(self, job): - """Yield progress payloads while scanning every configured output directory. + async def _scan_images_events(self, job, output_dirs=None): + """Yield progress payloads while scanning the output directories. - The payloads are the legacy SSE events, unchanged; ``job`` also gets - its status line, totals and one outcome per processed file. + Every effective source is walked, or only those ``job.scope["roots"]`` + selects (indexes into ``output_dirs``). The payloads are the legacy SSE + events, unchanged; ``job`` also gets its status line, totals and one + outcome per processed file. Rows keep each source's listing index. """ try: self.logger.info("Starting image scan operation") - output_dirs = self._get_all_output_dirs() - if not output_dirs: + if output_dirs is None: + output_dirs = self._get_all_output_dirs() + selected = job.scope.get("roots") + walked = ( + output_dirs + if selected is None + else [output_dirs[index] for index in selected] + ) + if not walked: self.logger.error("No output directories found") message = ( "No output directories found. " @@ -1419,18 +1435,18 @@ class AdminRoutesMixin: yield {"type": "error", "message": message} return - dir_names = [self._public_path(d) for d in output_dirs] - folders = _plural(len(output_dirs), "output folder") - job.scope["output_folders"] = len(output_dirs) + dir_names = [self._public_path(d) for d in walked] + folders = _plural(len(walked), "output folder") + job.scope["output_folders"] = len(walked) scope = SCAN_SCOPE_LOG[job.scope.get("files", "all")] job.log_line(f"Scan started · {folders} · {scope}") job.update(status_line=f"Looking for media files in {folders}") yield self._scan_progress( - 0, f"Scanning {len(output_dirs)} directory(ies) for media files..." + 0, f"Scanning {len(walked)} directory(ies) for media files..." ) media_files = await self._run_in_executor( - self._collect_output_media_sync, output_dirs + self._collect_output_media_sync, walked ) counts = dict.fromkeys(SCAN_COUNT_KEYS, 0) job.update( diff --git a/py/api/job_routes.py b/py/api/job_routes.py index f855660..83caa85 100644 --- a/py/api/job_routes.py +++ b/py/api/job_routes.py @@ -18,29 +18,74 @@ def _not_found(): return web.json_response({"success": False, "error": "Job not found"}, status=404) -def _bad_request(message): - return web.json_response({"success": False, "error": message}, status=400) +MAX_SCAN_ROOTS = 64 +_SCOPE_ERROR = 'The scan scope must be "new" or "all"' +_ROOTS_ERROR = ( + f"roots must be a non-empty list of up to {MAX_SCAN_ROOTS} source indexes, " + "with paths naming the public path of each" +) +SOURCES_CHANGED_ERROR = "The sources changed since they were loaded. Reload them." -async def _scan_scope(request): - """The requested scan scope ("new" or "all"), or None when the body is invalid. +def _bad_request(message, code=None): + body = {"success": False, "error": message} + if code is not None: + body["code"] = code + return web.json_response(body, status=400) - An empty body (also a chunked one with no bytes), or one without - ``scope``, asks for the default. + +def _scan_roots(body): + """``(roots, error)``: ``{index: public path}`` of the selected sources, or None. + + Absent ``roots`` means every source. When present it is a non-empty list + of at most MAX_SCAN_ROOTS non-negative ints (never bools or floats), and + ``paths`` the same-length list of the public path the caller read for + each (GET /settings), so a stale index never scans another folder. """ + if "roots" not in body: + return None, None + raw = body["roots"] + paths = body.get("paths") + if not isinstance(raw, list) or not raw or len(raw) > MAX_SCAN_ROOTS: + return None, _ROOTS_ERROR + if not isinstance(paths, list) or len(paths) != len(raw): + return None, _ROOTS_ERROR + selected = {} + for value, path in zip(raw, paths): + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + return None, _ROOTS_ERROR + if not isinstance(path, str) or selected.get(value, path) != path: + return None, _ROOTS_ERROR + selected[value] = path + return dict(sorted(selected.items())), None + + +async def _scan_request(request): + """``({files, roots}, None)`` for a scan start, or ``(None, message)``. + + ``files`` is the scope ("new" or "all"); ``roots`` the selected source + indexes or None for every source. An empty body (also a chunked one with + no bytes), or one without ``scope``, asks for the default scope. + """ + default = {"files": DEFAULT_SCAN_SCOPE, "roots": None} if not request.can_read_body: - return DEFAULT_SCAN_SCOPE + return default, None text = await request.text() if not text.strip(): # e.g. a chunked POST that sent no bytes - return DEFAULT_SCAN_SCOPE + return default, None try: body = json.loads(text) except ValueError: - return None + return None, _SCOPE_ERROR if not isinstance(body, dict): - return None - scope = body.get("scope", DEFAULT_SCAN_SCOPE) - return scope if scope in SCAN_SCOPES else None + return None, _SCOPE_ERROR + files = body.get("scope", DEFAULT_SCAN_SCOPE) + if files not in SCAN_SCOPES: + return None, _SCOPE_ERROR + roots, error = _scan_roots(body) + if error: + return None, error + return {"files": files, "roots": roots}, None def busy_response(busy: JobBusy): @@ -64,6 +109,13 @@ def _summary_event(job): class JobRoutesMixin: """Mixin providing the /prompt_manager/jobs endpoints.""" + def _roots_still_listed(self, selected, output_dirs): + """Whether each ``{index: public path}`` still names that source.""" + return all( + index < len(output_dirs) and self._public_path(output_dirs[index]) == path + for index, path in selected.items() + ) + def _register_job_routes(self, routes): @routes.get("/prompt_manager/jobs") async def list_jobs_route(request): @@ -132,13 +184,37 @@ class JobRoutesMixin: """Start the metadata scan as a job (202), or 409 naming the busy job. The body's ``scope`` is "new" (the default: new and unresolved files) - or "all"; anything else is a 400. + or "all". ``roots``, when given, selects sources by their index in + the effective sources (``_get_all_output_dirs``), and ``paths`` names + the public path read for each. An index past them, or one whose + source now has another public path (the list changed since it was + read), is a 409 ``sources-changed`` and starts nothing. The job walks + the sources resolved here, so a later source change cannot move it. """ - files = await _scan_scope(request) - if files is None: - return _bad_request('The scan scope must be "new" or "all"') + scan, error = await _scan_request(request) + if error: + return _bad_request(error) + output_dirs = None + roots = None + if scan["roots"] is not None: + output_dirs = await self._run_in_executor(self._get_all_output_dirs) + listed = await self._run_in_executor( + self._roots_still_listed, scan["roots"], output_dirs + ) + if not listed: + return web.json_response( + { + "success": False, + "code": "sources-changed", + "error": SOURCES_CHANGED_ERROR, + }, + status=409, + ) + roots = list(scan["roots"]) try: - job = self._start_scan_job(files) + job = self._start_scan_job( + scan["files"], roots=roots, output_dirs=output_dirs + ) except JobBusy as busy: return busy_response(busy) return web.json_response({"success": True, "job": job.summary()}, status=202) diff --git a/py/api/settings_routes.py b/py/api/settings_routes.py index 258ad2a..4760e08 100644 --- a/py/api/settings_routes.py +++ b/py/api/settings_routes.py @@ -4,8 +4,15 @@ A save validates the payload, commits it to config.json through the config store and publishes it at runtime inside that commit; a save that cannot be written changes nothing at runtime and says so. ``_config_commit_failure`` is shared with the LoRA integration's save. + +GET also reports the image sources and the monitor (``sources.py``). A save +that changes the sources runs as a ``source_change`` registry hold, so it is +refused while a record-changing job (a scan) runs. ``monitoring_enabled`` +takes effect at the next restart. ``POST /prompt_manager/settings/validate-source`` +checks one path and writes nothing. """ +import contextlib import os from aiohttp import web @@ -15,7 +22,10 @@ from ..config_store import ( RevisionConflict, get_config_store, ) +from . import sources from .admin import _current_image_monitor +from .job_routes import busy_response +from .jobs import JobBusy try: from ...utils.validators import validate_result_timeout, validate_worker_threads @@ -23,13 +33,75 @@ except ImportError: from utils.validators import validate_result_timeout, validate_worker_threads -# Settings owned by POST /settings: config attribute -> (section, key). +# Settings owned by POST /settings: config attribute -> its keys in config.json. _SETTINGS_FILE_KEYS = { "RESULT_TIMEOUT": ("web_ui", "result_timeout"), "WEBUI_DISPLAY_MODE": ("web_ui", "webui_display_mode"), "INFINITE_SCROLL": ("web_ui", "infinite_scroll"), "WORKER_THREADS": ("performance", "worker_threads"), + "MONITORING_ENABLED": ("gallery", "monitoring", "enabled"), } +# Attributes that live on GalleryConfig; the others on PromptManagerConfig. +_GALLERY_SETTINGS = frozenset({"MONITORING_ENABLED"}) +SOURCE_CHANGE_JOB_KIND = "source_change" +SOURCE_CHANGE_JOB_TITLE = "Source change" + + +def _source_identity(path): + """The case-folded spelling two roots are the same directory by (Windows).""" + return os.path.normcase(path) + + +def _stat_identity(path): + """``(st_dev, st_ino)`` of an existing directory, or None. + + Catches case variants on a case-insensitive filesystem whose spelling + normcase keeps (macOS). + """ + try: + stat = os.stat(path) + except (OSError, ValueError): + return None + return (stat.st_dev, stat.st_ino) if stat.st_ino else None + + +# Refusals that a reconnected drive or folder can lift. +_RECONNECTABLE = frozenset({"not-found", "not-directory"}) + + +def _root_key(root): + """The directory a configured root names (resolved), else its spelling.""" + from ..config import GalleryConfig + + try: + return GalleryConfig.resolve_gallery_root(root) + except (OSError, TypeError, ValueError, AttributeError): + return root + + +def _request_error(message): + return {"code": "invalid-request", "error": message} + + +def _invalid_source(index, entry, code, reason): + """The 400 body for the source at ``index`` that cannot be used. + + Names it by its public spelling (computed as text: a refused path is + never resolved), so the page can mark that row and say why. + """ + source = sources.public_spelling(entry) + advice = ( + "Remove it or reconnect it, then save." + if code in _RECONNECTABLE + else "Remove it, then save." + ) + return { + "code": "invalid-source", + "source": source, + "index": index, + "reason": reason, + "error": f"{source} can't be used: {reason}. {advice}", + } def _section(config, name): @@ -47,10 +119,8 @@ def _settings_mutation(updates, new_roots): def mutate(config): for name, value in updates.items(): - section, key = _SETTINGS_FILE_KEYS[name] - merged = _section(config, section) - merged[key] = value - config[section] = merged + *sections, key = _SETTINGS_FILE_KEYS[name] + _set_nested(config, sections, key, value) if new_roots is not None: gallery = _section(config, "gallery") monitoring = _section(gallery, "monitoring") @@ -62,6 +132,16 @@ def _settings_mutation(updates, new_roots): return mutate +def _set_nested(config, sections, key, value): + """``config[s1][s2]...[key] = value``, each section a fresh dict.""" + if not sections: + config[key] = value + return + merged = _section(config, sections[0]) + _set_nested(merged, sections[1:], key, value) + config[sections[0]] = merged + + async def _read_json(request): """Parse a JSON object body through the package's capped reader.""" from . import _read_json_body @@ -81,6 +161,10 @@ class SettingsRoutesMixin: async def save_settings_route(request): return await self.save_settings(request) + @routes.post("/prompt_manager/settings/validate-source") + async def validate_source_route(request): + return await self.validate_source(request) + async def get_settings(self, request): """Get current settings.""" try: @@ -90,11 +174,15 @@ class SettingsRoutesMixin: self._public_path(d) for d in GalleryConfig.MONITORING_DIRECTORIES ] revision = await self._run_in_executor(self._config_revision) + source_status, monitor = await self._run_in_executor(self._source_status) return web.json_response( { "success": True, **revision, + "sources": source_status, + "monitor": monitor, "settings": { + "monitoring_enabled": GalleryConfig.MONITORING_ENABLED, "result_timeout": PromptManagerConfig.RESULT_TIMEOUT, "webui_display_mode": PromptManagerConfig.WEBUI_DISPLAY_MODE, "infinite_scroll": PromptManagerConfig.INFINITE_SCROLL, @@ -115,6 +203,61 @@ class SettingsRoutesMixin: status=500, ) + def _source_status(self): + """``(sources, monitor)`` of GET /settings (blocking).""" + from ..config import GalleryConfig + + output_dirs, auto_detected = self._output_dirs_with_origin() + return sources.describe_sources( + configured=list(GalleryConfig.MONITORING_DIRECTORIES), + output_dirs=output_dirs, + auto_detected=auto_detected, + monitor=_current_image_monitor(), + enabled=GalleryConfig.MONITORING_ENABLED, + ) + + async def validate_source(self, request): + """Check one source path ``{path}``; writes nothing. + + 200 ``{success, valid, label, path, code, reason}``: a valid source + answers its label and public path, an invalid one its code and reason + only. A body without a string ``path`` is a 400 ``invalid-request``. + """ + data, error_response = await _read_json(request) + if error_response is not None: + too_large = error_response.status == 413 + return web.json_response( + { + "success": False, + "code": "too-large" if too_large else "invalid-request", + "error": ( + "Request body too large" + if too_large + else "Request body must be a JSON object" + ), + }, + status=error_response.status, + ) + path = data.get("path") + if not isinstance(path, str): + return web.json_response( + { + "success": False, + "code": "invalid-request", + "error": "path must be a string", + }, + status=400, + ) + try: + result = await self._run_in_executor(sources.validate_source, path) + except Exception: + self.logger.exception("Source validation failed") + return web.json_response( + {"success": False, "code": None, "error": "Failed to check the source"}, + status=500, + ) + return web.json_response({"success": True, **result}) + def _monitored_directories(self): """Directories the running image monitor watches, else the configured roots.""" from ..config import GalleryConfig @@ -151,9 +294,15 @@ class SettingsRoutesMixin: {"success": False, "error": str(ve)}, status=400 ) - new_roots, error = self._parse_gallery_roots(data) + # Off the loop: an accepted root is resolved (filesystem access). + new_roots, error = await self._run_in_executor( + self._parse_gallery_roots, data + ) if error: - return web.json_response({"success": False, "error": error}, status=400) + return web.json_response({"success": False, **error}, status=400) + roots_change = new_roots is not None and await self._run_in_executor( + self._roots_change, new_roots + ) published = {} @@ -162,15 +311,30 @@ class SettingsRoutesMixin: # they were written, so the runtime ends on the last one. published.update(self._publish_settings(updates, new_roots)) - try: - _saved, revision = await self._run_in_executor( - get_config_store().commit, - _settings_mutation(updates, new_roots), - expected_revision=expected_revision, - publish=publish, - ) - except (ConfigUnreadable, RevisionConflict, OSError) as exc: - return self._config_commit_failure(exc) + async with contextlib.AsyncExitStack() as stack: + hold = None + if roots_change: + try: + hold = await stack.enter_async_context( + self._jobs.hold( + SOURCE_CHANGE_JOB_KIND, + SOURCE_CHANGE_JOB_TITLE, + admission="records", + ) + ) + except JobBusy as busy: + return busy_response(busy) + try: + _saved, revision = await self._run_in_executor( + get_config_store().commit, + _settings_mutation(updates, new_roots), + expected_revision=expected_revision, + publish=publish, + ) + except (ConfigUnreadable, RevisionConflict, OSError) as exc: + if hold is not None: + hold.finish("failed") + return self._config_commit_failure(exc) return web.json_response( { @@ -196,10 +360,11 @@ class SettingsRoutesMixin: as ``runtime_applied: false`` with the fields that need a restart; nothing is retried or rolled back. Never raises. """ - from ..config import PromptManagerConfig + from ..config import GalleryConfig, PromptManagerConfig for name, value in updates.items(): - setattr(PromptManagerConfig, name, value) + owner = GalleryConfig if name in _GALLERY_SETTINGS else PromptManagerConfig + setattr(owner, name, value) restart_fields = [] runtime_applied = True if new_roots is not None: @@ -212,11 +377,40 @@ class SettingsRoutesMixin: ) runtime_applied = False restart_fields.append("gallery_root_paths") + if "MONITORING_ENABLED" in updates: + # The monitor starts with the server: a restart applies the setting + # unless the observer already matches it. + if updates["MONITORING_ENABLED"] != self._observer_alive(): + restart_fields.append("monitoring_enabled") return { "runtime_applied": runtime_applied, "restart_required_fields": restart_fields, } + @staticmethod + def _roots_change(new_roots): + """Whether a save changes the sources (blocking: it resolves them). + + Compared as directories, not spellings: the legacy page resends the + roots resolved, while config.json may hold them as hand-written + (relative, another case), and an unchanged resend is no change. + """ + from ..config import GalleryConfig + + if new_roots is None: + return False + return [_root_key(root) for root in new_roots] != [ + _root_key(root) for root in GalleryConfig.MONITORING_DIRECTORIES + ] + + @staticmethod + def _observer_alive(): + """Whether the image monitor's observer is running now. Never raises.""" + try: + return sources.monitor_state(_current_image_monitor())[1] + except Exception: + return False + def _config_commit_failure(self, exc): """The response for a config commit that wrote nothing.""" not_saved = {"success": False, "persisted": False, "runtime_applied": False} @@ -283,6 +477,10 @@ class SettingsRoutesMixin: if not isinstance(data["infinite_scroll"], bool): raise ValueError("infinite_scroll must be true or false") updates["INFINITE_SCROLL"] = data["infinite_scroll"] + if "monitoring_enabled" in data: + if not isinstance(data["monitoring_enabled"], bool): + raise ValueError("monitoring_enabled must be true or false") + updates["MONITORING_ENABLED"] = data["monitoring_enabled"] if "worker_threads" in data: updates["WORKER_THREADS"] = validate_worker_threads( data["worker_threads"], PromptManagerConfig.max_worker_threads() @@ -295,15 +493,21 @@ class SettingsRoutesMixin: Accepts ``gallery_root_paths`` (list, preferred) or the legacy single ``gallery_root_path`` string. Blank entries are dropped. + Each entry passes the same checks as validate-source + (``sources.source_problem``): the form, then containment as text, + before anything is resolved. Blocking (an accepted root is resolved). + Returns: (roots, error): ``roots`` is the validated list, or ``None`` when - the payload carries no gallery root key; ``error`` is a message - when validation failed (and ``roots`` is then ``None``). + the payload carries no gallery root key; ``error`` is the coded + body of the 400 when validation failed (``roots`` is then None). + An unusable source is ``invalid-source`` naming its public path + and its position in the list. """ if "gallery_root_paths" in data: raw = data["gallery_root_paths"] if not isinstance(raw, list): - return None, "gallery_root_paths must be a list" + return None, _request_error("gallery_root_paths must be a list") elif "gallery_root_path" in data: raw = [data["gallery_root_path"]] else: @@ -312,27 +516,38 @@ class SettingsRoutesMixin: from ..config import GalleryConfig roots = [] - for entry in raw: + seen = set() + for index, entry in enumerate(raw): if not isinstance(entry, str): - return None, "Gallery root paths must be strings" + return None, _request_error("Gallery root paths must be strings") entry = entry.strip() if not entry: continue - ok, reason = GalleryConfig.validate_gallery_root(entry) - if not ok: - return ( - None, - f"Invalid gallery root '{os.path.basename(entry)}': {reason}", - ) - roots.append(GalleryConfig.resolve_gallery_root(entry)) + code, reason = sources.source_problem(entry) + if code is not None: + return None, _invalid_source(index, entry, code, reason) + root = GalleryConfig.resolve_gallery_root(entry) + keys = {("path", _source_identity(root)), ("file", _stat_identity(root))} + keys.discard(("file", None)) + if keys & seen: + continue # another spelling of a root already listed + seen |= keys + roots.append(root) return roots, None def _apply_gallery_roots(self, roots): - """Install validated gallery roots; returns True when they changed.""" + """Install validated gallery roots; returns True when they changed. + + Another spelling of the same directories installs the resolved + spelling and is no change (nothing to restart for). + """ from ..config import GalleryConfig if roots == list(GalleryConfig.MONITORING_DIRECTORIES): return False + if not self._roots_change(roots): + GalleryConfig.MONITORING_DIRECTORIES = roots + return False GalleryConfig.MONITORING_DIRECTORIES = roots self._cached_output_dir = None self.reset_gallery_roots_cache() diff --git a/py/api/sources.py b/py/api/sources.py new file mode 100644 index 0000000..90fb18c --- /dev/null +++ b/py/api/sources.py @@ -0,0 +1,280 @@ +"""Image sources as System › Sources & scans shows them (#249 S4). + +- ``source_form_error`` is the lexical check a submitted source path passes + before any filesystem access: no control characters, no ``..`` part, and + only the native path forms (on Windows a drive or a UNC share, never the + ``\\\\?\\`` and ``\\\\.\\`` namespaces, a drive-relative path, an alternate + data stream, a device name or a part Windows would silently trim; on POSIX + no Windows drive or backslash form). +- ``source_problem`` is the one refusal every caller answers (validate-source + and the settings save): the form check, then + ``GalleryConfig.gallery_root_problem``, which decides containment as text + before it touches the filesystem. +- ``describe_sources`` (blocking: it resolves paths) reports the configured + sources with their validity, the effective sources with their listing + index and watch state, and the image monitor's state. Every path in it is + a public path (``_public_path``), never an absolute one. + +A source's watch state is ``watching`` when the monitor's observer is alive, +scheduled that directory and monitoring is on; ``stopping`` when it still +watches it but monitoring was switched off (it stops at the next restart); +``manual`` when monitoring is off or this process has no monitor; else +``restart_pending`` (a source added since the monitor started, monitoring +switched on, or an observer that stopped). +""" + +import logging +import ntpath +import os +import re + +logger = logging.getLogger("prompt_manager.api") + +MAX_SOURCE_PATH = 4096 # characters +FORM_REASONS = { + "empty": "Gallery root must be a non-empty path", + "invalid-form": "Gallery root is not a folder path this server can use", + "traversal": "Gallery root must not contain '..'", +} +_DRIVE = re.compile(r"[A-Za-z]:") +_POSIX_FOREIGN = re.compile(r"^[A-Za-z]:") +# Reserved device names: Windows opens the device for any folder of that name. +_DEVICE_NAME = re.compile(r"^(con|prn|aux|nul|com[0-9¹²³]|lpt[0-9¹²³])(\..*)?$", re.I) +# folder_paths getters and the label of the directory each one reports. +_COMFYUI_LABELS = ( + ("get_output_directory", "ComfyUI output"), + ("get_input_directory", "ComfyUI input"), + ("get_temp_directory", "ComfyUI temp"), + ("get_user_directory", "ComfyUI user"), +) +WATCH_STATES = ("watching", "stopping", "manual", "restart_pending") + + +def source_form_error(text, *, windows=None): + """The form code that refuses ``text`` as a source path, or None. + + ``windows`` selects the path rules (default: this OS). No filesystem + access happens here. + """ + if windows is None: + windows = os.name == "nt" + if not isinstance(text, str) or len(text) > MAX_SOURCE_PATH: + return "invalid-form" + if any(ord(char) < 32 or ord(char) == 127 for char in text): + return "invalid-form" + text = text.strip() + if not text: + return "empty" + return _windows_form_error(text) if windows else _posix_form_error(text) + + +def _posix_form_error(text): + if "\\" in text or _POSIX_FOREIGN.match(text): + return "invalid-form" # a Windows path names nothing on this server + return "traversal" if ".." in text.split("/") else None + + +def _windows_form_error(text): + portable = text.replace("/", "\\") + if portable.startswith(("\\\\?\\", "\\\\.\\")): + return "invalid-form" # verbatim and device namespaces + if portable.startswith("\\\\"): + drive, rest = ntpath.splitdrive(portable) + share = drive[2:].split("\\") + if len(share) != 2 or any(part in ("", ".", "..") for part in share): + return "invalid-form" + else: + drive, rest = ntpath.splitdrive(portable) + if drive and (not _DRIVE.fullmatch(drive) or not rest.startswith("\\")): + return "invalid-form" # "C:folder" is relative to that drive's cwd + if ":" in rest: + return "invalid-form" # an alternate data stream + parts = rest.split("\\") + if ".." in parts: + return "traversal" + for part in parts: + if part in ("", "."): + continue + if part[-1] in ". " or _DEVICE_NAME.match(part): + return "invalid-form" + return None + + +def source_problem(text): + """``(code, reason)`` that refuses ``text`` as a source, else ``(None, "")``. + + The form check runs first and touches nothing; then containment is + decided as text before any filesystem access (see + ``GalleryConfig.gallery_root_problem``). Every caller answers alike. + """ + from ..config import GalleryConfig + + code = source_form_error(text) + if code is not None: + return code, FORM_REASONS[code] + return GalleryConfig.gallery_root_problem(text) + + +def public_spelling(text): + """How a refused source is named in a response, computed as text only. + + Relative to the path anchor it lies under (its public path), else its + last part: the caller sent the path, so naming it reveals nothing, and a + refused path is never resolved to name it. + """ + from ..config import GalleryConfig, _lexical_path, _lexically_inside + + text = str(text).strip() + try: + lexical = os.path.normpath(text) + key = _lexical_path(text) # the same length as ``lexical`` + if not os.path.isabs(text): + return lexical.replace("\\", "/") + for anchor in GalleryConfig.path_anchors(): + if _lexically_inside(key, [anchor]): + rest = lexical[len(anchor.rstrip(os.sep)) :].lstrip(os.sep) + return rest.replace(os.sep, "/") or "." + return os.path.basename(lexical) or lexical + except (TypeError, ValueError): + return "" + + +def identity(path): + """The comparison key of a directory: resolved and case-folded (Windows).""" + return os.path.normcase(os.path.realpath(str(path))) + + +def _comfyui_labels(): + """``{identity: label}`` of the directories ComfyUI reports.""" + try: + import folder_paths + except ImportError: + return {} + labels = {} + for getter_name, label in _COMFYUI_LABELS: + getter = getattr(folder_paths, getter_name, None) + if getter is None: + continue + try: + directory = getter() + except Exception: + continue + if isinstance(directory, str) and directory: + labels.setdefault(identity(directory), label) + return labels + + +def source_label(path, labels=None): + """ "ComfyUI output" for ComfyUI's own directories, else the folder name.""" + if labels is None: + labels = _comfyui_labels() + label = labels.get(identity(path)) + if label: + return label + return os.path.basename(os.path.normpath(str(path))) or str(path) + + +def _configured(entry, labels, public_path): + from ..config import GalleryConfig + + code, reason = GalleryConfig.gallery_root_problem(entry) + try: + resolved = GalleryConfig.resolve_gallery_root(entry) + except (OSError, TypeError, ValueError): + resolved = str(entry) + return { + "label": source_label(resolved, labels), + "path": public_path(resolved), + "valid": code is None, + "code": code, + "reason": reason or None, + } + + +def monitor_state(monitor): + """``(present, observer_alive, scheduled directories)`` of the image monitor.""" + if monitor is None: + return False, False, [] + status = monitor.get_status() + scheduled = [str(d) for d in status.get("monitored_directories") or []] + return True, bool(status.get("observer_alive")), scheduled + + +def watch_state(path, *, present, alive, scheduled_keys, enabled): + """One of ``WATCH_STATES`` (see the module docstring).""" + if alive and identity(path) in scheduled_keys: + return "watching" if enabled else "stopping" + if not present or not enabled: + return "manual" + return "restart_pending" + + +def describe_sources(*, configured, output_dirs, auto_detected, monitor, enabled): + """``(sources, monitor)`` for GET /settings (blocking; public paths only). + + ``configured`` is the configured roots, ``output_dirs`` and + ``auto_detected`` what ``_output_dirs_with_origin`` returned, ``monitor`` + the running ImageMonitor or None, ``enabled`` the monitoring setting. + """ + from . import _public_path + + labels = _comfyui_labels() + present, alive, scheduled = monitor_state(monitor) + scheduled_keys = {identity(path) for path in scheduled} + effective = [ + { + "label": source_label(path, labels), + "path": _public_path(path), + "root_index": index, + "watch": watch_state( + path, + present=present, + alive=alive, + scheduled_keys=scheduled_keys, + enabled=enabled, + ), + } + for index, path in enumerate(output_dirs) + ] + return ( + { + "configured": [ + _configured(entry, labels, _public_path) for entry in configured + ], + "effective": effective, + "auto_detected": bool(auto_detected), + }, + { + "present": present, + "observer_alive": alive, + "scheduled": [_public_path(path) for path in scheduled], + }, + ) + + +def validate_source(text): + """``{valid, label, path, code, reason}`` for one submitted path (blocking). + + Writes nothing. An invalid path answers its code and reason with no + label or path, so a response never echoes what was typed. + """ + from ..config import GalleryConfig + from . import _public_path + + code, reason = source_problem(text) + if code is not None: + return { + "valid": False, + "label": None, + "path": None, + "code": code, + "reason": reason, + } + resolved = GalleryConfig.resolve_gallery_root(text) + return { + "valid": True, + "label": source_label(resolved), + "path": _public_path(resolved), + "code": None, + "reason": None, + } diff --git a/py/config.py b/py/config.py index a228524..c6fa719 100644 --- a/py/config.py +++ b/py/config.py @@ -31,7 +31,6 @@ routes = server_instance.routes extension_uri = None # Will be set in __init__.py import os # noqa: E402 -from pathlib import Path # noqa: E402 from typing import Any, Dict, List, Optional, Tuple # noqa: E402 # Import logging system @@ -51,6 +50,14 @@ config_logger = get_logger("prompt_manager.config") # may be used as gallery roots in addition to ComfyUI's own directories. EXTRA_GALLERY_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS" +# The one answer for every path outside the allowed folders. The filesystem +# root and the home directory answer it too, so no caller learns the server's +# layout (home path, user name) from a refusal (#249 S4). +OUTSIDE_ALLOWED_REASON = ( + "Gallery root must be inside a ComfyUI directory (output, input, temp, user) " + f"or one listed in {EXTRA_GALLERY_ROOTS_ENV}" +) + # Environment variable overriding where config.json is read from and saved to. CONFIG_PATH_ENV = "PROMPT_MANAGER_CONFIG_PATH" @@ -89,6 +96,29 @@ def write_private_json(path: str, data: Any) -> None: os.chmod(path, 0o600) +def _lexical_path(path: str) -> str: + """``path`` normalised for comparison without touching the filesystem. + + normpath collapses ``.`` and ``..`` as text and normcase folds case on + Windows; neither opens, stats nor resolves anything (no SMB request for a + UNC path). A symlink is only seen by :func:`_canonical_path` later. + """ + return os.path.normcase(os.path.normpath(path)) + + +def _lexically_inside(path: str, parents: List[str]) -> bool: + """Whether the normalised ``path`` is one of ``parents`` or below one. + + Pure text on normalised spellings (:func:`_lexical_path` or + :func:`_canonical_path`): no filesystem access. + """ + sep = os.path.sep + return any( + path == parent or path.startswith(parent.rstrip(sep) + sep) + for parent in parents + ) + + def _is_filesystem_root(path: str) -> bool: """True for '/' on POSIX and drive roots such as 'C:\\' on Windows.""" return os.path.dirname(path) == path @@ -268,7 +298,12 @@ class GalleryConfig: """ monitoring = new_config.get("monitoring", {}) if "enabled" in monitoring: - cls.MONITORING_ENABLED = monitoring["enabled"] + if isinstance(monitoring["enabled"], bool): + cls.MONITORING_ENABLED = monitoring["enabled"] + else: # a hand-edited "false" would otherwise switch it on + config_logger.warning( + "Ignoring gallery monitoring enabled: expected true or false" + ) if "directories" in monitoring: directories = cls._validated_directories(monitoring["directories"]) if directories is not None: @@ -337,31 +372,56 @@ class GalleryConfig: Filesystem roots are never allowed as parents, even when listed in the environment, because that would re-open the whole disk. """ - parents = [] + return [canonical for canonical, _spelling in cls._allowed_parent_pairs()] + + @classmethod + def _allowed_parent_pairs(cls) -> List[Tuple[str, str]]: + """``(canonical, lexical spelling)`` of each allowed parent. + + Only trusted directories (ComfyUI's own and the environment's) are + resolved here; the spelling is how ComfyUI or the environment wrote + the directory, so a submitted path can be compared without resolving it. + """ + pairs = [] + seen = set() for candidate in _comfyui_directories() + _extra_gallery_roots(): try: canonical = _canonical_path(candidate) + spelling = _lexical_path(os.path.abspath(candidate)) except (OSError, ValueError): continue if _is_filesystem_root(canonical) or not os.path.isdir(canonical): continue - if canonical not in parents: - parents.append(canonical) - return parents + if canonical not in seen: + seen.add(canonical) + pairs.append((canonical, spelling)) + return pairs @classmethod def path_anchors(cls) -> List[str]: """Canonical directories that relative gallery paths are resolved against. The ComfyUI base directory comes first, followed by the parent of each - directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so an extra - root is addressed by its own name (``gallery/sub``) rather than by an - absolute path. The same anchors drive the public (relative) form of - paths in API responses. + ComfyUI directory outside it (``--output-directory D:\\Renders``) and of + each directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so such a + directory is addressed by its own name (``Renders/sub``) rather than by + an absolute path, and two folders of one name stay apart. The same + anchors drive the public (relative) form of paths in API responses. """ anchors = [] base = _comfyui_base_directory() candidates = [base] if base else [] + try: + base_key = _canonical_path(base) if base else None + except (OSError, ValueError): + base_key = None + for directory in _comfyui_directories(): + try: + canonical = _canonical_path(directory) + except (OSError, ValueError): + continue + if base_key is None or not _lexically_inside(canonical, [base_key]): + candidates.append(os.path.dirname(canonical)) candidates.extend(os.path.dirname(root) for root in _extra_gallery_roots()) for candidate in candidates: try: @@ -399,33 +459,69 @@ class GalleryConfig: (True, "") when the directory lies inside one of :meth:`allowed_gallery_parents`; otherwise (False, reason). """ + code, reason = cls.gallery_root_problem(path) + return code is None, reason + + @classmethod + def gallery_root_problem(cls, path: Any) -> Tuple[Optional[str], str]: + """``(code, reason)`` why ``path`` may not be a gallery root, else (None, ""). + + Codes: ``empty``, ``unresolvable``, ``outside-allowed``, ``not-found`` + and ``not-directory``. No reason names the path. + + Containment is decided twice. First lexically, with no filesystem + access at all: a path (UNC, ``\\\\?\\``, a device, anything) whose + normalised spelling lies outside every allowed parent answers + ``outside-allowed`` before it is resolved, so the server never opens + or stats it (no SMB connection, no existence or timing oracle). Only + then is it resolved and checked again, which catches a symlink out. + The filesystem root and the home directory answer ``outside-allowed`` + too: one answer for every caller (validate-source, the settings save, + config.json), and the detail goes to the log only. + """ if not isinstance(path, str) or not path.strip(): - return False, "Gallery root must be a non-empty path" + return "empty", "Gallery root must be a non-empty path" + + pairs = cls._allowed_parent_pairs() + spellings = [spelling for pair in pairs for spelling in pair] + try: + lexical = cls._lexical_candidates(path) + except (TypeError, ValueError): + return "unresolvable", "Gallery root could not be resolved" + if not any(_lexically_inside(candidate, spellings) for candidate in lexical): + return "outside-allowed", OUTSIDE_ALLOWED_REASON try: canonical = cls.resolve_gallery_root(path) except (OSError, ValueError): - return False, "Gallery root could not be resolved" + return "unresolvable", "Gallery root could not be resolved" if _is_filesystem_root(canonical): - return False, "Gallery root cannot be a filesystem root" + config_logger.info("Gallery root refused: a filesystem root") + return "outside-allowed", OUTSIDE_ALLOWED_REASON if canonical == _canonical_path(os.path.expanduser("~")): - return False, "Gallery root cannot be the home directory" + config_logger.info("Gallery root refused: the home directory") + return "outside-allowed", OUTSIDE_ALLOWED_REASON + if not _lexically_inside(canonical, [canonical for canonical, _ in pairs]): + return "outside-allowed", OUTSIDE_ALLOWED_REASON if not os.path.exists(canonical): - return False, "Gallery root does not exist" + return "not-found", "Gallery root does not exist" if not os.path.isdir(canonical): - return False, "Gallery root is not a directory" + return "not-directory", "Gallery root is not a directory" + return None, "" - candidate = Path(canonical) - for parent in cls.allowed_gallery_parents(): - if candidate.is_relative_to(Path(parent)): - return True, "" + @classmethod + def _lexical_candidates(cls, path: str) -> List[str]: + """Every spelling ``path`` may resolve to, normalised as text (no IO). - return ( - False, - "Gallery root must be inside a ComfyUI directory " - f"(output, input, temp, user) or one listed in {EXTRA_GALLERY_ROOTS_ENV}", - ) + An absolute path is itself; a relative one is joined to each + :meth:`path_anchors` entry, as :meth:`resolve_gallery_root` tries them. + """ + path = path.strip() + if os.path.isabs(path): + return [_lexical_path(path)] + anchors = cls.path_anchors() + return [_lexical_path(os.path.join(anchor, path)) for anchor in anchors] class IntegrationConfig: diff --git a/tests/browser/system-concept.spec.mjs b/tests/browser/system-concept.spec.mjs index 761134b..9317a19 100644 --- a/tests/browser/system-concept.spec.mjs +++ b/tests/browser/system-concept.spec.mjs @@ -533,3 +533,246 @@ test('appearance comparison detects mutations', async ({ page, request }) => { await undoToken(); expect(diffInventory(expected, await measure())).toEqual([]); }); + +/** + * Runs in the page: Sources & scans (app.js 679–681): the optional notice, then the "Image sources" surface with its + * panel heading, the source list, the two setting rows and the operations, with the concept's geometry and tokens. + */ +function measureSources(mainSelector) { + const layout = document.querySelector(`${mainSelector} .settings-layout`); + const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null; + const name = node => [node.localName, ...[...node.classList].sort()].join('.'); + const text = node => node?.textContent.trim() ?? null; + const button = node => ({ text: text(node), className: node.className, + ...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') }); + const operation = row => ({ tree: [...row.children].map(name), title: text(row.querySelector('h3')), + description: text(row.querySelector('p')), actions: [...row.querySelectorAll('.row-actions > button')].map(button) }); + const notice = layout.querySelector(':scope > .notice'); + const surface = layout.querySelector(':scope > section.surface'); + const heading = surface.querySelector(':scope > .panel-heading'); + const list = surface.querySelector(':scope > #source-list'); + const control = row => { + const toggle = row.querySelector(':scope > * input[type="checkbox"]'); + if (toggle) return { kind: 'switch', id: toggle.id, wrapper: name(toggle.parentElement), checked: toggle.checked, + ...css(toggle.parentElement, 'width', 'height') }; + const input = row.querySelector(':scope > input'); + return { kind: `input.${input.type}`, id: input.id, min: input.min, max: input.max, value: input.value, + ...css(input, 'width', 'accentColor') }; + }; + const rows = [...surface.querySelectorAll(':scope > .setting-row')]; + return { + layout: [...layout.children].map(name), + notice: notice && { tree: [...notice.children].map(name), + text: [...notice.querySelectorAll(':scope > div > *')].map(node => [node.localName, text(node)]), + actions: [...notice.querySelectorAll(':scope > .row-actions > button')].map(button), + ...css(notice, 'display', 'alignItems', 'columnGap', 'borderTopColor', 'backgroundColor', 'paddingTop', 'marginBottom') }, + surface: { tree: [...surface.children].map(name), intro: text(surface.querySelector(':scope > p')), + ...css(surface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') }, + heading: { tree: [...heading.children].map(name), title: text(heading.querySelector('h2')), + action: button(heading.querySelector('button')), ...css(heading, 'display', 'alignItems', 'justifyContent', 'marginBottom'), + h2: css(heading.querySelector('h2'), 'fontSize', 'fontWeight', 'color') }, + sources: [...list.children].map(operation), + rows: rows.map(row => ({ tree: [...row.children].map(name), text: [...row.firstElementChild.children].map(node => [node.localName, text(node), + node.localName === 'label' ? node.htmlFor : null]), control: control(row) })), + operations: [...surface.querySelectorAll(':scope > .operation')].map(operation), + operation: css(list.querySelector('.operation'), 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', + 'paddingBottom', 'borderTopWidth', 'borderTopColor'), + h3: css(list.querySelector('.operation h3'), 'fontSize', 'fontWeight', 'color'), + p: css(list.querySelector('.operation p'), 'fontSize', 'color', 'marginTop'), + row: css(rows[0], 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', 'borderTopColor'), + pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth, + }; +} + +// D128: "watching until restart" after monitoring was switched off while the observer still runs. +const WATCH_TEXT = { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan', restart_pending: 'restart pending' }; +const WATCHED = new Set(['watching', 'stopping']); +// D98: the operation Sources adds after the concept's own, as production renders it. +const REFRESH_FILES = { title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.', + label: 'Refresh' }; + +/** The concept's Sources with the recorded deviations applied, given the server's real settings and sources. */ +function mapSourcesConcept(reference, { settings, sources }) { + const [sample] = reference.sources; + const scan = sample.actions[0]; + return { + ...reference, + // The real sources replace the two samples (without the "sample path" note). D98: each configured one has Remove, + // a plain button after its scan. + sources: sources.effective.map(source => ({ ...sample, title: source.label, + description: `${source.path} · ${WATCH_TEXT[source.watch]}`, + actions: [{ ...scan, text: WATCHED.has(source.watch) ? 'Scan changes' : 'Scan folder' }, { ...scan, text: 'Remove' }] })), + rows: reference.rows.map(row => { + if (row.control.id === 'monitoring') return { ...row, control: { ...row.control, checked: settings.monitoring_enabled } }; + // D113: the range ends at the server's core count (the concept's fixed 16) and shows the real worker count. + return { ...row, control: { ...row.control, max: String(settings.cpu_count), value: String(settings.worker_threads) } }; + }), + // D98: "Refresh files" closes the surface. + surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.operation'] }, + operations: [...reference.operations, { ...reference.operations[0], title: REFRESH_FILES.title, + description: REFRESH_FILES.description, actions: [{ ...reference.operations[0].actions[0], text: REFRESH_FILES.label }] }], + }; +} + +async function conceptSources(page, theme, { notice }) { + await openConcept(page, 'system', theme); + await page.locator('#main .section-nav').getByRole('button', { name: 'Sources & scans', exact: true }).click(); + await expect(page.locator('#main .panel-heading > h2')).toHaveText('Image sources'); + if (!notice) await page.locator('#main .notice').getByRole('button', { name: 'Later', exact: true }).click(); + await expect(page.locator('#main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0); + await page.mouse.move(0, 0); + return page.evaluate(measureSources, '#main'); +} + +async function productionSources(page, theme, { notice }) { + // about:blank has no storage; the workspace page reads the dismissal. + if (!notice) await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('pm_update_notice_dismissed', '3.2.0'); }); + await openProduction(page, 'system/sources', theme); + await expect(page.locator('#workspace-main #source-list .operation')).toHaveCount(2); + await expect(page.locator('#workspace-main #workers')).toBeEnabled(); + await expect(page.locator('#workspace-main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0); + await page.mouse.move(0, 0); + return page.evaluate(measureSources, '#workspace-main'); +} + +/** The fixture's sources as the concept shows its samples: the first one watched, the second not. */ +async function watchedSources(request) { + await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK(); + return serverJson(request, '/prompt_manager/settings'); +} + +for (const theme of THEMES) for (const [width, height] of SIZES) for (const notice of [true, false]) { + const variant = notice ? 'with' : 'without'; + test(`sources matches the concept ${variant} the notice ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => { + await page.setViewportSize({ width, height }); + const server = await watchedSources(request); + const concept = await conceptSources(page, theme, { notice }); + await page.screenshot({ path: testInfo.outputPath(`concept-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true }); + const production = await productionSources(page, theme, { notice }); + await page.screenshot({ path: testInfo.outputPath(`production-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true }); + expect(diffInventory(mapSourcesConcept(concept, server), production)).toEqual([]); + expect(production.pageScroll).toBe(false); + }); +} + +test('sources comparison detects mutations', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const server = await watchedSources(request); + const concept = await conceptSources(page, 'dark', { notice: true }); + const expected = mapSourcesConcept(concept, server); + const production = await productionSources(page, 'dark', { notice: true }); + expect(diffInventory(expected, production)).toEqual([]); + // Unmapped, the concept differs only in the real sources, the D98 operation and the D113 range. + expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.')) + .filter((path, index, all) => all.indexOf(path) === index).sort()) + .toEqual(['operations.2', 'operations.length', 'rows.1', 'sources.0', 'sources.1', 'surface.tree'].sort()); + const measure = () => page.evaluate(measureSources, '#workspace-main'); + const paths = async () => diffInventory(expected, await measure()).map(item => item.path); + // A removed notice and a removed source row. + const undoNotice = await mutate(page, { remove: ['#workspace-main .settings-layout > .notice'] }); + expect(await paths()).toEqual(expect.arrayContaining(['layout.length', 'notice.tree.length'])); + await undoNotice(); + const undoRow = await mutate(page, { remove: ['#workspace-main #source-list > .operation:nth-child(2)'] }); + expect(await paths()).toEqual(expect.arrayContaining(['sources.length'])); + await undoRow(); + // A changed label and a changed button. + await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for outputs'; }); + expect(await paths()).toEqual(['rows.0.text.0.1']); + await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for new outputs'; }); + await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = 'Add source'; }); + expect(await paths()).toEqual(['heading.action.text']); + await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = '+ Add source'; }); + // A swapped token. + const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' }); + expect(await paths()).toEqual(expect.arrayContaining(['operation.borderTopColor'])); + await undoToken(); + expect(diffInventory(expected, await measure())).toEqual([]); +}); + +/** + * Runs in the page: the open "Add image source" dialog (the concept's `simpleInput`, app.js 1138 and 1497): its title, + * the body's tree, the field's label and input, and the actions, with the concept's geometry and tokens. + */ +function measureAddSource() { + const dialog = document.querySelector('dialog#modal[open]'); + const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null; + const name = node => [node.localName, ...[...node.classList].sort()].join('.'); + const text = node => node?.textContent.trim() ?? null; + const body = dialog.querySelector('#modal-body'); + const field = body.querySelector(':scope > .field'); + const label = field?.querySelector(':scope > label'); + const input = field?.querySelector(':scope > input'); + const actions = body.querySelector(':scope > .dialog-actions'); + const rect = dialog.getBoundingClientRect(); + return { + title: text(dialog.querySelector('#modal-title')), + tree: [...body.children].map(name), + field: field && { tree: [...field.children].map(name), ...css(field, 'display', 'flexDirection', 'rowGap', 'marginBottom') }, + label: label && { text: text(label), for: label.htmlFor, ...css(label, 'fontSize', 'fontWeight', 'color') }, + input: input && { id: input.id, type: input.type, value: input.value, + ...css(input, 'height', 'paddingLeft', 'fontSize', 'color', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') }, + actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'), + buttons: [...actions.children].map(button => ({ text: text(button), className: button.className, + ...css(button, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) }, + width: Math.round(rect.width), + }; +} + +/** D127: the field is "Folder path", starts empty (no sample folder) and its action is "Add". */ +function mapAddSourceConcept(reference) { + const [cancel, apply] = reference.actions.buttons; + return { ...reference, label: { ...reference.label, text: 'Folder path' }, input: { ...reference.input, value: '' }, + actions: { ...reference.actions, buttons: [cancel, { ...apply, text: 'Add' }] } }; +} + +async function conceptAddSource(page, theme) { + await conceptSources(page, theme, { notice: false }); + await page.locator('#main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source'); + await page.mouse.move(0, 0); + return page.evaluate(measureAddSource); +} + +async function productionAddSource(page, theme) { + await productionSources(page, theme, { notice: false }); + await page.locator('#workspace-main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source'); + await page.mouse.move(0, 0); + return page.evaluate(measureAddSource); +} + +for (const theme of THEMES) for (const [width, height] of SIZES) { + test(`add source dialog matches the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => { + await page.setViewportSize({ width, height }); + await watchedSources(request); + const concept = await conceptAddSource(page, theme); + await page.screenshot({ path: testInfo.outputPath(`concept-add-source-${theme}-${width}.png`) }); + const production = await productionAddSource(page, theme); + await page.screenshot({ path: testInfo.outputPath(`production-add-source-${theme}-${width}.png`) }); + expect(diffInventory(mapAddSourceConcept(concept), production)).toEqual([]); + }); +} + +test('add source comparison detects mutations', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + await watchedSources(request); + const concept = await conceptAddSource(page, 'dark'); + const expected = mapAddSourceConcept(concept); + const production = await productionAddSource(page, 'dark'); + expect(diffInventory(expected, production)).toEqual([]); + // Unmapped, the concept differs only where D127 is recorded. + expect(diffInventory(concept, production).map(item => item.path).sort()) + .toEqual(['actions.buttons.1.text', 'input.value', 'label.text']); + const paths = async () => diffInventory(expected, await page.evaluate(measureAddSource)).map(item => item.path); + // A changed label, a swapped action order and a changed style. + await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder'; }); + expect(await paths()).toEqual(['label.text']); + await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder path'; }); + await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); }); + expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text'])); + await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); }); + const undoStyle = await mutate(page, { css: 'dialog#modal #action-input { border-color: rgb(255, 0, 0); } dialog#modal .dialog-actions { gap: 31px; }' }); + expect(await paths()).toEqual(expect.arrayContaining(['input.borderTopColor', 'actions.columnGap'])); + await undoStyle(); + expect(diffInventory(expected, await page.evaluate(measureAddSource))).toEqual([]); +}); diff --git a/tests/browser/workspace-system-sources.spec.mjs b/tests/browser/workspace-system-sources.spec.mjs new file mode 100644 index 0000000..6450344 --- /dev/null +++ b/tests/browser/workspace-system-sources.spec.mjs @@ -0,0 +1,401 @@ +import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs'; +import { test, expect } from './fixtures.mjs'; +import { seedConcept } from './concept-inventory.mjs'; + +// System › Sources & scans (#249 System S4 Task 11) on the concept-shaped real fixture, whose two sources are this +// server's `output` and `input` folders: the source list, monitoring and worker rows read GET /settings (S4 Task 9), +// Add/Remove edit the host draft behind the one Save row, and every scan starts the session job through the shared +// "Review metadata scan" (D55), scoped to one source from its row (S4 Task 10). The hooks are workspace_system_fixture.py's. +const workspace = '/prompt_manager/workspace'; +const SETTINGS_URL = '/prompt_manager/settings'; +const NOTICE_KEY = 'pm_update_notice_dismissed'; +const OUTSIDE_REASON = 'Gallery root must be inside a ComfyUI directory (output, input, temp, user) or one listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS'; + +test.beforeEach(async ({ page, request, browserName }) => { + await startCoverage(page, browserName); + await seedConcept(request); + await expect(await request.post('/__test__/system/reset')).toBeOK(); + await expect(await request.post('/__test__/jobs/restart')).toBeOK(); + page.systemErrors = []; + page.on('pageerror', error => page.systemErrors.push(error.message)); +}); + +test.afterEach(async ({ page, request, browserName }) => { + await saveCoverage(page, browserName); + await request.post('/__test__/release'); + await request.post('/__test__/jobs/restart'); + await request.post('/__test__/system/reset'); + expect(page.systemErrors).toEqual([]); +}); + +const toast = page => page.locator('#workspace-toast'); +const layoutOf = page => page.locator('#workspace-main .settings-layout'); +const sourceList = page => layoutOf(page).locator('#source-list'); +const statusOf = page => layoutOf(page).locator('#settings-status'); +const saveButton = page => layoutOf(page).getByRole('button', { name: 'Save preferences', exact: true }); +const dialog = page => page.locator('dialog#modal[open]'); + +async function serverJson(request, url) { + const response = await request.get(url); + await expect(response).toBeOK(); + return response.json(); +} + +/** Open Sources with the update notice dismissed (or shown, `notice: true`) once the source list has loaded. */ +async function openSources(page, { notice = false } = {}) { + if (!notice) await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY); + await page.goto(`${workspace}#/system/sources`); + await expect(sourceList(page).locator('.operation').first()).toBeVisible(); + await expect(layoutOf(page).locator('#workers')).toBeEnabled(); + return layoutOf(page); +} + +/** The source list's rows: `[h3, p, [buttons]]`, then its note line when there is one. */ +function rowsOf(page) { + return sourceList(page).evaluate(list => [...list.children].map(row => row.matches('.operation') + ? [row.querySelector('h3').textContent, row.querySelector('p').textContent, + [...row.querySelectorAll('.row-actions > button')].map(button => button.textContent)] + : [row.localName, row.className, row.textContent])); +} + +const sourceRow = (page, title) => sourceList(page).locator('.operation').filter({ has: page.getByRole('heading', { name: title, exact: true }) }); + +test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed) without scanning', async ({ page, request }) => { + const writes = []; + page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) writes.push(item.url()); }); + const layout = await openSources(page, { notice: true }); + const notice = layout.locator(':scope > .notice'); + await expect(notice.locator('h2')).toHaveText('Bring older images into your prompt history'); + await expect(notice.locator('p')).toHaveText('The schema upgrades automatically. An optional metadata scan links existing positives and negatives without regenerating all thumbnails.'); + await expect(notice.locator('.row-actions > button')).toHaveText(['Later', 'Review scan']); + await notice.getByRole('button', { name: 'Later', exact: true }).click(); + await expect(layout.locator(':scope > .notice')).toHaveCount(0); + expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0'); + await expect(dialog(page)).toHaveCount(0); + // The dismissal is the legacy admin page's own, so neither page shows the notice again. + await page.reload(); + await expect(sourceList(page).locator('.operation').first()).toBeVisible(); + await expect(layout.locator(':scope > .notice')).toHaveCount(0); + expect(writes).toEqual([]); + expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]); +}); + +test('Review scan opens the shared Review metadata scan with all sources', async ({ page, request }) => { + const layout = await openSources(page, { notice: true }); + // The notice's Review scan dismisses it, as the legacy page's does, and reviews every source. + await layout.locator(':scope > .notice').getByRole('button', { name: 'Review scan', exact: true }).click(); + await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan'); + await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']); + expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0'); + await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click(); + await expect(layout.locator(':scope > .notice')).toHaveCount(0); + // Enrich existing metadata is the same review. + const enrich = layout.locator('.operation').filter({ hasText: 'Enrich existing metadata' }); + await expect(enrich.locator('p')).toHaveText('Resolve both prompts, deduplicate recipes, preserve uncertain links.'); + await enrich.getByRole('button', { name: 'Review scan', exact: true }).click(); + await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']); + await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click(); + expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]); +}); + +test('Scan changes opens it with that one source ("1 output folder") and starts a scoped job that opens in Activity', async ({ page, request }) => { + await expect(await request.post('/__test__/system/monitor', { data: { watching: ['input'] } })).toBeOK(); + const bodies = []; + page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan')) bodies.push(item.postDataJSON()); }); + await openSources(page); + const input = sourceRow(page, 'ComfyUI input'); + await expect(input.locator('p')).toHaveText('input · watching'); + await input.getByRole('button', { name: 'Scan changes', exact: true }).click(); + await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan'); + await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['1 output folder', 'New and unresolved files']); + await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click(); + await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/); + // The toast says it started (or, for a scan this quick, that it already finished); Activity shows its progress. + await expect(toast(page)).toHaveText(/^Scan (started\.|finished · )/); + expect(bodies).toEqual([{ scope: 'new', roots: [1], paths: ['input'] }]); + const id = page.url().split('/').pop(); + // The job walks only that source (the count is recorded once the walk has listed it). + await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.scope) + .toEqual({ files: 'new', roots: [1], output_folders: 1 }); + await expect(page.locator('#workspace-main h1')).toHaveText('Scan for metadata'); +}); + +test('Add source validates on the server, then marks the host draft dirty; Save persists', async ({ page, request }) => { + // A folder inside ComfyUI's output folder that is not a source yet. + await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 1 } })).toBeOK(); + const layout = await openSources(page); + expect(await rowsOf(page)).toEqual([ + ['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']], + ['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']], + ]); + await expect(layout.locator('.panel-heading > h2')).toHaveText('Image sources'); + const add = layout.locator('.panel-heading').getByRole('button', { name: '+ Add source', exact: true }); + await add.click(); + await expect(dialog(page).locator('#modal-title')).toHaveText('Add image source'); + await expect(dialog(page).locator('label[for="action-input"]')).toHaveText('Folder path'); + await expect(dialog(page).locator('#action-input')).toHaveValue(''); + await expect(dialog(page).locator('.dialog-actions > button')).toHaveText(['Cancel', 'Add']); + await dialog(page).locator('#action-input').fill('output/unthumbnailed'); + const validated = page.waitForResponse(response => response.url().endsWith('/prompt_manager/settings/validate-source')); + await dialog(page).getByRole('button', { name: 'Add', exact: true }).click(); + expect(await (await validated).json()).toEqual({ success: true, valid: true, label: 'unthumbnailed', path: 'output/unthumbnailed', code: null, reason: null }); + await expect(dialog(page)).toHaveCount(0); + await expect(add).toBeFocused(); + // Validation wrote nothing: the draft is dirty and the server still has two sources. + expect(await rowsOf(page)).toEqual([ + ['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']], + ['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']], + ['unthumbnailed', 'output/unthumbnailed · not saved yet', ['Remove']], + ]); + await expect(statusOf(page)).toHaveText('Unsaved changes.'); + expect((await serverJson(request, SETTINGS_URL)).sources.effective).toHaveLength(2); + const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL)); + await saveButton(page).click(); + const body = (await saved).postDataJSON(); + expect(body.gallery_root_paths).toEqual(['output', 'input', 'output/unthumbnailed']); + expect(typeof body.expected_revision).toBe('string'); + await expect(statusOf(page)).toHaveText('Saved to the server configuration.'); + // The list is the server's again: three effective sources, each scannable. + await expect(sourceRow(page, 'unthumbnailed').locator('p')).toHaveText('output/unthumbnailed · manual scan'); + await expect(sourceRow(page, 'unthumbnailed').getByRole('button', { name: 'Scan folder', exact: true })).toBeVisible(); + expect((await serverJson(request, SETTINGS_URL)).sources.effective.map(source => source.path)).toEqual(['output', 'input', 'output/unthumbnailed']); +}); + +test('invalid source shows the server reason', async ({ page, request }) => { + const layout = await openSources(page); + await layout.getByRole('button', { name: '+ Add source', exact: true }).click(); + const field = dialog(page).locator('#action-input'); + for (const [path, reason, submit] of [ + ['output/not-there', 'Gallery root does not exist', 'click'], + // Enter in the field adds too (M6). + ['/etc', OUTSIDE_REASON, 'Enter'], + ]) { + await field.fill(path); + if (submit === 'Enter') await field.press('Enter'); + else await dialog(page).getByRole('button', { name: 'Add', exact: true }).click(); + await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText(reason); + await expect(field).toHaveAttribute('aria-invalid', 'true'); + await expect(field).toHaveValue(path); + } + // A folder already listed (the server's answer names it) is not added twice. + await field.fill('input'); + await dialog(page).getByRole('button', { name: 'Add', exact: true }).click(); + await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('This folder is already a source.'); + // A check that cannot finish says so and adds nothing. + await page.route('**/prompt_manager/settings/validate-source', route => route.fulfill({ status: 500, + json: { success: false, code: null, error: 'Failed to check the source' } })); + await field.fill('output/not-checked'); + await dialog(page).getByRole('button', { name: 'Add', exact: true }).click(); + await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('The folder could not be checked. Try again.'); + await page.unroute('**/prompt_manager/settings/validate-source'); + // Typing again clears the reason. + await field.fill('output/unthumbnailed'); + await expect(dialog(page).locator('p.small[role="alert"]')).toHaveCount(0); + await expect(field).not.toHaveAttribute('aria-invalid', 'true'); + await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click(); + await expect(statusOf(page)).toHaveText('No unsaved changes.'); + expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2); +}); + +test('Remove marks the draft; removing the last entry states auto-detect', async ({ page, request }) => { + await openSources(page); + await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click(); + expect(await rowsOf(page)).toEqual([['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']]]); + await expect(statusOf(page)).toHaveText('Unsaved changes.'); + await sourceRow(page, 'ComfyUI output').getByRole('button', { name: 'Remove', exact: true }).click(); + expect(await rowsOf(page)).toEqual([['p', 'small', 'No source is configured. After saving, ComfyUI’s output folder is used.']]); + await saveButton(page).click(); + await expect(statusOf(page)).toHaveText('Saved to the server configuration.'); + // Saved: ComfyUI's output folder is detected and listed without Remove. + await expect.poll(rowsOf.bind(null, page)).toEqual([ + ['ComfyUI output', 'output · manual scan', ['Scan folder']], + ['p', 'small', 'No source is configured; ComfyUI’s output folder is used.'], + ]); + const { sources } = await serverJson(request, SETTINGS_URL); + expect([sources.configured, sources.auto_detected]).toEqual([[], true]); +}); + +test('a configured source that does not exist says why and can be removed', async ({ page, request }) => { + await expect(await request.post('/__test__/system/missing-source')).toBeOK(); + await openSources(page); + // The fixture's unmounted drive is not inside a ComfyUI folder, so the server says that (it never probes outside). + expect((await rowsOf(page)).at(-1)).toEqual(['unmounted-drive', `unmounted-drive · ${OUTSIDE_REASON}`, ['Remove']]); + await sourceRow(page, 'unmounted-drive').getByRole('button', { name: 'Remove', exact: true }).click(); + await expect(sourceList(page).locator('.operation')).toHaveCount(2); + await saveButton(page).click(); + await expect(statusOf(page)).toHaveText('Saved to the server configuration.'); + expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output', 'input']); +}); + +test('watching only with a live observer; restart pending otherwise', async ({ page, request }) => { + // No image monitor in this process: every source is scanned by hand. + await openSources(page); + await expect(sourceList(page).locator('.operation p')).toHaveText(['output · manual scan', 'input · manual scan']); + // A live observer that scheduled output: output is watched; input was added since it started. + await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK(); + await page.reload(); + await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · restart pending']); + await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan folder']); + // An observer that died watches nothing. + await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'], alive: false } })).toBeOK(); + await page.reload(); + await expect(sourceList(page).locator('.operation p')).toHaveText(['output · restart pending', 'input · restart pending']); +}); + +test('Watch for new outputs saves monitoring_enabled; a restart applies it', async ({ page, request }) => { + await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'] } })).toBeOK(); + const layout = await openSources(page); + const monitoring = layout.getByRole('switch', { name: 'Watch for new outputs', exact: true }); + await expect(monitoring).toBeChecked(); + await expect(layout.locator('.setting-row').filter({ has: page.locator('#monitoring') }).locator('p')) + .toHaveText('New files enter the same library. Existing scans do not count as new runs.'); + await monitoring.uncheck(); + await expect(statusOf(page)).toHaveText('Unsaved changes.'); + const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL)); + await saveButton(page).click(); + const answer = await (await saved).json(); + expect(answer.restart_required_fields).toEqual(['monitoring_enabled']); + // D128: saved for the next start, and the status says a restart applies it. + await expect(statusOf(page)).toHaveText('Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.'); + // The running observer keeps watching until ComfyUI restarts, and the list says so. + const { settings, monitor } = await serverJson(request, SETTINGS_URL); + expect([settings.monitoring_enabled, monitor.observer_alive]).toEqual([false, true]); + await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching until restart', 'input · watching until restart']); + await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan changes']); + // Switched back on before the restart: the observer still runs, so nothing waits for one. + await monitoring.check(); + await saveButton(page).click(); + await expect(statusOf(page)).toHaveText('Saved to the server configuration.'); + await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · watching']); +}); + +test('worker range max is cpu_count', async ({ page, request }) => { + const { settings } = await serverJson(request, SETTINGS_URL); + const layout = await openSources(page); + const workers = layout.locator('#workers'); + await expect(layout.locator('label[for="workers"]')).toHaveText('Worker threads'); + await expect(workers).toHaveAttribute('type', 'range'); + await expect(workers).toHaveAttribute('min', '1'); + await expect(workers).toHaveAttribute('max', String(settings.cpu_count)); + await expect(workers).toHaveValue(String(settings.worker_threads)); + const next = settings.worker_threads === 1 ? settings.cpu_count : 1; + await workers.fill(String(next)); + await expect(statusOf(page)).toHaveText(next === settings.worker_threads ? 'No unsaved changes.' : 'Unsaved changes.'); + test.skip(next === settings.worker_threads, 'One core: the range has a single value'); + const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL)); + await saveButton(page).click(); + expect((await saved).postDataJSON().worker_threads).toBe(next); + await expect(statusOf(page)).toHaveText('Saved to the server configuration.'); + expect((await serverJson(request, SETTINGS_URL)).settings.worker_threads).toBe(next); +}); + +test('Refresh files calls /gallery/rescan and toasts its counts', async ({ page }) => { + const layout = await openSources(page); + const refresh = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Refresh files', exact: true }) }); + await expect(refresh.locator('p')).toHaveText('Read every source again to find files added or removed outside ComfyUI.'); + const answered = page.waitForResponse(response => response.url().endsWith('/prompt_manager/gallery/rescan')); + await refresh.getByRole('button', { name: 'Refresh', exact: true }).click(); + const { total, roots } = await (await answered).json(); + expect(roots).toBe(2); + await expect(toast(page)).toHaveText(`Files refreshed. ${total.toLocaleString('en-US')} files in 2 output folders.`); + // A failed refresh says so. + await page.route('**/prompt_manager/gallery/rescan', route => route.fulfill({ status: 500, json: { success: false, error: 'busy disk' } })); + await refresh.getByRole('button', { name: 'Refresh', exact: true }).click(); + await expect(toast(page)).toHaveText('Files could not be refreshed. Try again.'); +}); + +test('Thumbnail generation hands off to the existing gallery until its review dialog', async ({ page }) => { + const layout = await openSources(page); + const thumbnails = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) }); + await expect(thumbnails.locator('p')).toHaveText('Generate missing previews; rebuilding all is a separate explicit option.'); + const popup = page.waitForEvent('popup'); + await thumbnails.getByRole('button', { name: 'Review options', exact: true }).click(); + const gallery = await popup; + expect(new URL(gallery.url()).pathname).toBe('/prompt_manager/gallery'); + await gallery.close(); + await expect(statusOf(page)).toHaveText('No unsaved changes.'); +}); + +test('a save refused by a running scan shows the busy link toast', async ({ page, request }) => { + const running = await (await request.post('/__test__/jobs/start', { data: { title: 'Scan for metadata' } })).json(); + await openSources(page); + await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click(); + await saveButton(page).click(); + await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.'); + await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`); + await expect(statusOf(page)).toHaveText('Some changes were not saved. Review the highlighted settings.'); + // The draft is kept for the next Save, and the server still has both sources. + await expect(sourceList(page).locator('.operation')).toHaveCount(1); + expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2); + // An answer that does not name its job (an older server): the toast names the running one from the job list. + await page.route(`**${SETTINGS_URL}`, route => (route.request().method() === 'POST' + ? route.fulfill({ status: 409, json: { success: false, code: 'busy', error: 'Scan for metadata already running' } }) + : route.continue())); + await page.locator('#workspace-toast').evaluate(node => { node.textContent = ''; }); + await saveButton(page).click(); + await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.'); + await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`); + await page.unroute(`**${SETTINGS_URL}`); + await request.post(`/__test__/jobs/${running.job.id}/finish`, { data: { state: 'finished' } }); +}); + +test('a source list that cannot be read says so and retries', async ({ page }) => { + await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY); + let failing = true; + await page.route(`**${SETTINGS_URL}`, route => (failing && route.request().method() === 'GET' + ? route.fulfill({ status: 500, json: { success: false, error: 'Injected failure' } }) : route.continue())); + await page.goto(`${workspace}#/system/sources`); + await expect(sourceList(page).getByRole('alert')).toHaveText('Sources could not be loaded. Your saved data has not changed.'); + await expect(sourceList(page).locator('.operation')).toHaveCount(0); + // The server settings rows say so too, and stay disabled. + await expect(layoutOf(page).locator('#workers')).toBeDisabled(); + failing = false; + await sourceList(page).getByRole('button', { name: 'Retry', exact: true }).click(); + await expect(sourceList(page).locator('.operation')).toHaveCount(2); +}); + +test('a scan from a list that changed since it was read starts nothing, says why and reads the list again', async ({ page, request }) => { + await openSources(page); + // The legacy page (or another tab) removes ComfyUI output: the page's "input" is index 1, now past the end. + await expect(await request.post(SETTINGS_URL, { data: { gallery_root_paths: ['input'] } })).toBeOK(); + const refused = page.waitForResponse(response => response.url().endsWith('/prompt_manager/jobs/scan')); + await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click(); + await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click(); + const answer = await refused; + expect([answer.status(), (await answer.json()).code]).toEqual([409, 'sources-changed']); + await expect(toast(page)).toHaveText('The sources changed since they were loaded. Reload them.'); + // No scan started (the other save ran as its own source-change hold). + expect((await serverJson(request, '/prompt_manager/jobs')).jobs.map(job => job.kind)).toEqual(['source_change']); + // Read again: the one source is index 0 now, and its scan starts. + await expect.poll(rowsOf.bind(null, page)).toEqual([['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']]]); + await expect(statusOf(page)).toHaveText('No unsaved changes.'); + const started = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan')); + await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click(); + await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click(); + expect((await started).postDataJSON()).toEqual({ scope: 'new', roots: [0], paths: ['input'] }); + await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/); +}); + +test('a save refused for one unusable source names it and marks its row', async ({ page, request }) => { + await expect(await request.post('/__test__/system/missing-source', { data: { inside: true } })).toBeOK(); + await openSources(page); + await expect(sourceRow(page, 'unplugged').locator('p')).toHaveText('output/unplugged · manual scan'); + // The drive is unplugged after the page read the list; removing another source resends the whole list. + await expect(await request.post('/__test__/system/unplug')).toBeOK(); + await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click(); + const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL)); + await saveButton(page).click(); + const answer = await saved; + expect(answer.status()).toBe(400); + expect(await answer.json()).toMatchObject({ code: 'invalid-source', source: 'output/unplugged', index: 1 }); + await expect(statusOf(page)).toHaveText("output/unplugged can't be used: Gallery root does not exist. Remove it or reconnect it, then save."); + await expect(sourceRow(page, 'unplugged')).toHaveAttribute('aria-invalid', 'true'); + await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(1); + expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(3); + // Removed, the list saves. + await sourceRow(page, 'unplugged').getByRole('button', { name: 'Remove', exact: true }).click(); + await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(0); + await saveButton(page).click(); + await expect(statusOf(page)).toHaveText('Saved to the server configuration.'); + expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output']); +}); diff --git a/tests/browser/workspace-system.spec.mjs b/tests/browser/workspace-system.spec.mjs index ca2da6e..a11cfdf 100644 --- a/tests/browser/workspace-system.spec.mjs +++ b/tests/browser/workspace-system.spec.mjs @@ -237,10 +237,11 @@ test('out-of-order overview responses never repaint a newer section', async ({ p await nav.getByRole('button', { name: 'Appearance', exact: true }).click(); await nav.getByRole('button', { name: 'Overview', exact: true }).click(); await expect.poll(held).toBe(2); - await nav.getByRole('button', { name: 'Sources & scans', exact: true }).click(); + // Integrations still hands off (S10), so its body is one known panel. + await nav.getByRole('button', { name: 'Integrations', exact: true }).click(); await expect(await request.post('/__test__/release')).toBeOK(); await expect.poll(held).toBe(0); - // Both late answers reached a section that had left: Sources keeps its own body. + // Both late answers reached a section that had left: Integrations keeps its own body. await expect(layout.locator(':scope > section.handoff')).toBeVisible(); await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))); await expect(layout.locator('.summary-metrics')).toHaveCount(0); diff --git a/tests/browser/workspace.spec.mjs b/tests/browser/workspace.spec.mjs index 8d847a3..79313a3 100644 --- a/tests/browser/workspace.spec.mjs +++ b/tests/browser/workspace.spec.mjs @@ -425,8 +425,7 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept const SYSTEM_SECTION_LABELS = ['Overview', 'Appearance', 'Sources & scans', 'Integrations', 'Auto-tagging', 'Maintenance', 'Backup & export', 'Diagnostics & logs']; const SYSTEM_HASHES = ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics']; -const SYSTEM_HANDOFFS = { sources: ['/prompt_manager/admin', '/prompt_manager/gallery'], - maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] }; +const SYSTEM_HANDOFFS = { maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] }; test('System is native: the concept shell, section navigation, sections, hand-offs and the Save row', async ({ page }) => { const mutations = []; @@ -462,14 +461,17 @@ test('System is native: the concept shell, section navigation, sections, hand-of await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1); // The section changes in place: focus stays on the pressed section, not the page title. await expect(button).toBeFocused(); - // S2 Task 6: Overview is native (workspace-system.spec.mjs), and so is Appearance (S3 Task 8); Maintenance shows + // S2 Task 6: Overview is native (workspace-system.spec.mjs), and so are Appearance (S3 Task 8) and Sources & scans + // (S4 Task 11, workspace-system-sources.spec.mjs, its update notice shown in a fresh browser); Maintenance shows // its health-check notice above the hand-off (D123); every other section hands off. if (hash === 'overview') await expect(layout.locator(':scope > .summary-metrics')).toBeVisible(); + if (hash === 'sources') await expect(layout.locator('#source-list .operation')).toHaveCount(2); expect(await layout.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`))) .toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'], appearance: ['section.surface', 'div.save-row'], + sources: ['div.notice', 'section.surface', 'div.save-row'], maintenance: ['div.notice', 'section.surface handoff', 'div.save-row'] }[hash] ?? ['section.surface handoff', 'div.save-row']); - if (hash === 'overview' || hash === 'appearance') continue; + if (['overview', 'appearance', 'sources'].includes(hash)) continue; const handoff = layout.locator('section.handoff'); await expect(handoff.locator('h2')).toHaveText('Your current tools are ready'); expect(await handoff.locator('a.workspace-link').evaluateAll(links => links.map(link => link.getAttribute('href')))) diff --git a/tests/browser/workspace_system_fixture.py b/tests/browser/workspace_system_fixture.py index 2413c1f..95f3218 100644 --- a/tests/browser/workspace_system_fixture.py +++ b/tests/browser/workspace_system_fixture.py @@ -11,11 +11,18 @@ read. These per-server hooks only change what a System test needs on top: requests for ``path`` (a ``/prompt_manager/system/`` route) with the routes' 500 error envelope; - ``POST /__test__/system/missing-source`` configures one more source that - does not exist (an unmounted drive); + does not exist (an unmounted drive); with ``{"inside": true}`` it is an + existing folder inside the output folder (``output/unplugged``), which + ``POST /__test__/system/unplug`` then removes (a drive unplugged while the + page shows it); - ``POST /__test__/system/missing-file`` deletes the newest linked image's file from disk, leaving its database row (a missing-file link); - ``POST /__test__/system/unthumbnailed`` ``{count}`` writes ``count`` plain - PNGs with no thumbnail into the output folder. + PNGs with no thumbnail into the output folder; +- ``POST /__test__/system/monitor`` ``{watching, alive}`` stands in a fake + image monitor whose observer is ``alive`` (default true) and schedules the + named folders of this server's root (``["output"]``), so Sources reads + watching, manual scan and restart pending states (S4). Delays use the shared request holds (``request_hold.py``). """ @@ -45,13 +52,17 @@ def register_system_commands(app, routes, api, root: Path): f"PROMPT_MANAGER_CONFIG_PATH must be {config_path}, not {configured_path!r}" ) from py.config import GalleryConfig, PromptManagerConfig + from utils import image_monitor configured = list(GalleryConfig.MONITORING_DIRECTORIES) config_bytes = config_path.read_bytes() if config_path.exists() else None host_values = ( PromptManagerConfig.RESULT_TIMEOUT, PromptManagerConfig.WEBUI_DISPLAY_MODE, + PromptManagerConfig.WORKER_THREADS, + GalleryConfig.MONITORING_ENABLED, ) + monitor = getattr(image_monitor, "_monitor_instance", None) failures: dict[str, int] = {} @web.middleware @@ -66,13 +77,21 @@ def register_system_commands(app, routes, api, root: Path): app.middlewares.append(inject_failures) + unplugged = root / "output" / "unplugged" + def reset(): failures.clear() + if unplugged.is_dir(): + unplugged.rmdir() GalleryConfig.MONITORING_DIRECTORIES = list(configured) api._health_report = None - PromptManagerConfig.RESULT_TIMEOUT, PromptManagerConfig.WEBUI_DISPLAY_MODE = ( - host_values - ) + ( + PromptManagerConfig.RESULT_TIMEOUT, + PromptManagerConfig.WEBUI_DISPLAY_MODE, + PromptManagerConfig.WORKER_THREADS, + GalleryConfig.MONITORING_ENABLED, + ) = host_values + image_monitor._monitor_instance = monitor if config_bytes is None: config_path.unlink(missing_ok=True) else: @@ -103,14 +122,26 @@ def register_system_commands(app, routes, api, root: Path): @routes.post("/__test__/system/missing-source") async def missing_source(request): - absent = str(root / "unmounted-drive") - if os.path.exists(absent): - raise web.HTTPConflict() + data = await request.json() if request.can_read_body else {} + if data.get("inside") is True: + unplugged.mkdir(exist_ok=True) + absent = str(unplugged) + else: + absent = str(root / "unmounted-drive") + if os.path.exists(absent): + raise web.HTTPConflict() GalleryConfig.MONITORING_DIRECTORIES = [*configured, absent] return web.json_response( {"success": True, "configured": len(GalleryConfig.MONITORING_DIRECTORIES)} ) + @routes.post("/__test__/system/unplug") + async def unplug(request): + if not unplugged.is_dir(): + raise web.HTTPConflict() + unplugged.rmdir() + return web.json_response({"success": True}) + @routes.post("/__test__/system/missing-file") async def missing_file(request): with api.db.model.get_connection() as conn: @@ -141,4 +172,35 @@ def register_system_commands(app, routes, api, root: Path): api.invalidate_gallery_cache() return web.json_response({"success": True}) + @routes.post("/__test__/system/monitor") + async def fake_monitor(request): + data = await request.json() + watching, alive = data.get("watching", []), data.get("alive", True) + if ( + not isinstance(watching, list) + or not all(name in ("output", "input") for name in watching) + or not isinstance(alive, bool) + ): + raise web.HTTPBadRequest() + image_monitor._monitor_instance = _FakeMonitor( + [str(root / name) for name in watching], alive + ) + return web.json_response({"success": True}) + return reset + + +class _FakeMonitor: + """The image monitor's status surface (``get_status``) with no observer thread.""" + + def __init__(self, directories, alive): + self.monitored_directories = directories + self._alive = alive + + def get_status(self): + return { + "running": True, + "monitored_directories": list(self.monitored_directories), + "handler_active": True, + "observer_alive": self._alive, + } diff --git a/tests/js/api-client.test.js b/tests/js/api-client.test.js index df2fa21..e22878f 100644 --- a/tests/js/api-client.test.js +++ b/tests/js/api-client.test.js @@ -355,3 +355,20 @@ test("keepalive is forwarded only when asked, so a request can outlive its page" for (const call of fetch.calls) call.resolve(jsonResponse(200, { success: true })); await Promise.all(writes); }); + +test("ApiError keeps a validated running job on busy and a validated source on invalid-source", () => { + const body = { running: { id: "a".repeat(32), title: "Scan for metadata" }, source: "output/gone" }; + const busy = new ApiError(409, "busy", "busy", null, body); + assert.deepEqual([busy.running, busy.source], [{ id: "a".repeat(32), title: "Scan for metadata" }, null]); + assert.notEqual(busy.running, body.running, "server objects must be copied"); + const invalid = new ApiError(400, "refused", "invalid-source", null, body); + assert.deepEqual([invalid.running, invalid.source], [null, "output/gone"]); + for (const running of [null, "x", { id: 1, title: "t" }, { id: "a b", title: "t" }, { id: "a", title: "" }, + { id: "a", title: "x".repeat(201) }, { id: "a".repeat(65), title: "t" }]) { + assert.equal(new ApiError(409, "busy", "busy", null, { running }).running, null); + } + for (const source of [null, 7, "", "x".repeat(4097)]) { + assert.equal(new ApiError(400, "refused", "invalid-source", null, { source }).source, null); + } + assert.deepEqual([new ApiError(409, "old caller").running, new ApiError(409, "old caller").source], [null, null]); +}); diff --git a/tests/js/workspace-system-sources.test.js b/tests/js/workspace-system-sources.test.js new file mode 100644 index 0000000..53e4663 --- /dev/null +++ b/tests/js/workspace-system-sources.test.js @@ -0,0 +1,145 @@ +// System › Sources & scans (#249 System S4 Task 11): the concept's "Image sources" list (app.js 679–681) built from the +// server's configured, effective and watched sources (GET /settings), the host draft of `gallery_root_paths`, the +// 3.2.0 update notice's dismissal and the Refresh files toast (D98). +const test = require('node:test'); +const assert = require('node:assert/strict'); + +const sourcesModule = import('../../web/js/workspace/system-sources.mjs'); + +const STATUS = Object.freeze({ + configured: [ + { label: 'ComfyUI output', path: 'output', valid: true, code: null, reason: null }, + { label: 'ComfyUI input', path: 'input', valid: true, code: null, reason: null }, + ], + effective: [ + { label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'watching' }, + { label: 'ComfyUI input', path: 'input', root_index: 1, watch: 'manual' }, + ], + auto_detected: false, +}); + +test('each effective source is the concept row: label, "<path> · watching|manual scan", Scan changes or Scan folder', async () => { + const { sourceRows } = await sourcesModule; + const saved = ['output', 'input']; + const { rows, note } = sourceRows({ sources: STATUS, saved, current: saved }); + assert.equal(note, null); + assert.deepEqual(rows, [ + { key: 'output', title: 'ComfyUI output', description: 'output · watching', scan: { label: 'Scan changes', root: 0, path: 'output' }, + remove: 'output' }, + { key: 'input', title: 'ComfyUI input', description: 'input · manual scan', scan: { label: 'Scan folder', root: 1, path: 'input' }, + remove: 'input' }, + ]); +}); + +test('watching only with a live observer; restart pending otherwise (D98)', async () => { + const { sourceRows, WATCH_COPY } = await sourcesModule; + assert.deepEqual(WATCH_COPY, { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan', + restart_pending: 'restart pending' }); + const sources = { ...STATUS, effective: STATUS.effective.map(source => ({ ...source, watch: 'restart_pending' })) }; + const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] }); + assert.deepEqual(rows.map(row => [row.description, row.scan.label]), + [['output · restart pending', 'Scan folder'], ['input · restart pending', 'Scan folder']]); +}); + +test('a configured source that cannot be used shows the server reason and only Remove', async () => { + const { sourceRows } = await sourcesModule; + const sources = { ...STATUS, configured: [...STATUS.configured, + { label: 'unmounted-drive', path: 'unmounted-drive', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] }; + const saved = ['output', 'input', 'unmounted-drive']; + const { rows } = sourceRows({ sources, saved, current: saved }); + assert.deepEqual(rows.at(-1), { key: 'unmounted-drive', title: 'unmounted-drive', + description: 'unmounted-drive · Gallery root does not exist', scan: null, remove: 'unmounted-drive' }); +}); + +test('an added source is listed as not saved yet with its validated label; a removed one leaves the list', async () => { + const { sourceRows } = await sourcesModule; + const labels = new Map([['output/archive', 'archive']]); + const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['input', 'output/archive'], labels }); + assert.equal(note, null); + assert.deepEqual(rows.map(row => [row.title, row.description, row.scan?.label ?? null, row.remove]), [ + ['ComfyUI input', 'input · manual scan', 'Scan folder', 'input'], + ['archive', 'output/archive · not saved yet', null, 'output/archive'], + ]); + // Without a recorded label (a reload of System), the folder name stands in. + assert.equal(sourceRows({ sources: STATUS, saved: [], current: ['output/new-collection'] }).rows[0].title, 'new-collection'); +}); + +test('removing the last entry states auto-detect (D98)', async () => { + const { sourceRows, NOTES } = await sourcesModule; + const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: [] }); + assert.deepEqual(rows, []); + assert.equal(note, NOTES.removedAll); + assert.equal(NOTES.removedAll, 'No source is configured. After saving, ComfyUI’s output folder is used.'); +}); + +test('auto-detected sources are listed without Remove, with the line that says how they were chosen', async () => { + const { sourceRows, NOTES } = await sourcesModule; + const auto = { configured: [], effective: [{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'manual' }], auto_detected: true }; + assert.deepEqual(sourceRows({ sources: auto, saved: [], current: [] }), { rows: [ + { key: 'auto:0', title: 'ComfyUI output', description: 'output · manual scan', scan: { label: 'Scan folder', root: 0, path: 'output' }, + remove: null }, + ], note: NOTES.autoDetected }); + // Every configured source unusable: ComfyUI's output folder is used instead. + const fallback = { ...auto, configured: [{ label: 'gone', path: 'gone', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] }; + const result = sourceRows({ sources: fallback, saved: ['gone'], current: ['gone'] }); + assert.deepEqual(result.rows.map(row => [row.title, row.remove]), [['gone', 'gone'], ['ComfyUI output', null]]); + assert.equal(result.note, NOTES.fallback); + assert.equal(sourceRows({ sources: { configured: [], effective: [], auto_detected: false }, saved: [], current: [] }).note, NOTES.none); +}); + +test('a path listed twice in the draft is one row', async () => { + const { sourceRows } = await sourcesModule; + assert.equal(sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['output', 'output'] }).rows.length, 1); +}); + +test('roots equal to the saved list are the saved list itself, so the draft is clean again', async () => { + const { draftRoots } = await sourcesModule; + const saved = ['output', 'input']; + assert.equal(draftRoots(['output', 'input'], saved), saved); + assert.deepEqual(draftRoots(['input'], saved), ['input']); + assert.notEqual(draftRoots(['input', 'output'], saved), saved); + assert.deepEqual(draftRoots([], undefined), []); +}); + +test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed); storage that throws keeps it for this view', async () => { + const { noticeDismissed, dismissNotice, NOTICE_KEY, NOTICE_VERSION } = await sourcesModule; + assert.deepEqual([NOTICE_KEY, NOTICE_VERSION], ['pm_update_notice_dismissed', '3.2.0']); + const values = new Map(); + const storage = { getItem: key => values.get(key) ?? null, setItem: (key, value) => values.set(key, String(value)) }; + assert.equal(noticeDismissed(() => storage), false); + values.set(NOTICE_KEY, '3.1.0'); + assert.equal(noticeDismissed(() => storage), false); + assert.equal(dismissNotice(() => storage), true); + assert.equal(values.get(NOTICE_KEY), '3.2.0'); + assert.equal(noticeDismissed(() => storage), true); + const denied = () => { throw new Error('SecurityError'); }; + assert.equal(noticeDismissed(denied), false); + assert.equal(dismissNotice(denied), false); + assert.equal(dismissNotice(() => ({ setItem() { throw new Error('QuotaExceededError'); } })), false); +}); + +test('Refresh files toasts the rescan counts, or says it finished when they are missing', async () => { + const { refreshedText } = await sourcesModule; + assert.equal(refreshedText({ total: 1234, roots: 2 }), 'Files refreshed. 1,234 files in 2 output folders.'); + assert.equal(refreshedText({ total: 1, roots: 1 }), 'Files refreshed. 1 file in 1 output folder.'); + assert.equal(refreshedText({ total: -1, roots: 1 }), 'Files refreshed.'); + assert.equal(refreshedText(null), 'Files refreshed.'); +}); + +test('monitoring switched off: a source still watched says so until the restart (D128)', async () => { + const { sourceRows } = await sourcesModule; + const sources = { ...STATUS, effective: [{ ...STATUS.effective[0], watch: 'stopping' }, STATUS.effective[1]] }; + const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] }); + assert.deepEqual(rows.map(row => [row.description, row.scan.label]), + [['output · watching until restart', 'Scan changes'], ['input · manual scan', 'Scan folder']]); +}); + +test('a valid configured source missing from the effective list is another spelling of a listed folder (D128)', async () => { + const { sourceRows, DUPLICATE_SOURCE } = await sourcesModule; + assert.equal(DUPLICATE_SOURCE, 'same folder as another source'); + const sources = { ...STATUS, configured: [...STATUS.configured, + { label: 'output', path: 'output/.', valid: true, code: null, reason: null }] }; + const saved = ['output', 'input', 'output/.']; + assert.deepEqual(sourceRows({ sources, saved, current: saved }).rows.at(-1), { key: 'output/.', title: 'output', + description: 'output/. · same folder as another source', scan: null, remove: 'output/.' }); +}); diff --git a/tests/js/workspace-system-state.test.js b/tests/js/workspace-system-state.test.js index af60efb..f7e611e 100644 --- a/tests/js/workspace-system-state.test.js +++ b/tests/js/workspace-system-state.test.js @@ -192,6 +192,7 @@ test('status text per outcome', async () => { partial: 'Some changes were not saved. Review the highlighted settings.', tabOnly: 'Saved for this tab only; browser storage is unavailable.', conflict: 'Settings changed elsewhere. Reload this section to see them.', + restart: 'Restart ComfyUI to apply:', }); const clean = { browser: 'clean', host: 'clean', runtime: 'clean' }; assert.equal(statusText(clean), 'No unsaved changes.'); @@ -316,3 +317,50 @@ test('host read state says whether the server settings could be read', async () assert.equal(await drafts.reloadHost(), true); assert.equal(drafts.hostState(), 'loaded'); }); + +test('a host save names the settings that apply after a restart (D128)', async () => { + const { statusText, RESTART_LABELS } = await source; + assert.deepEqual(RESTART_LABELS, { monitoring_enabled: 'Watch for new outputs' }); + const saved = { browser: 'clean', host: 'saved', runtime: 'clean' }; + assert.equal(statusText({ ...saved, restart: ['monitoring_enabled'] }), + 'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.'); + // A field without a row label (the sources say "restart pending" per row) adds nothing. + assert.equal(statusText({ ...saved, restart: ['gallery_root_paths'] }), 'Saved to the server configuration.'); + assert.equal(statusText({ ...saved, restart: [] }), 'Saved to the server configuration.'); + + const domains = adapters({ host: { monitoring_enabled: true } }); + const drafts = await loaded(domains); + domains.outcome.host = () => ({ success: true, persisted: true, revision: 'sha256:two', + restart_required_fields: ['monitoring_enabled', 7] }); + drafts.set('host', 'monitoring_enabled', false); + assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'saved', runtime: 'clean', restart: ['monitoring_enabled'] }); + assert.equal(drafts.statusText(), 'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.'); + // The next edit is unsaved again. + drafts.set('host', 'monitoring_enabled', true); + assert.equal(drafts.statusText(), 'Unsaved changes.'); +}); + +test('a host save refused for one source says the server reason and names it until the next edit', async () => { + const domains = adapters({ host: { gallery_root_paths: ['output', 'output/gone'] } }); + const drafts = await loaded(domains); + const message = "output/gone can't be used: Gallery root does not exist. Remove it or reconnect it, then save."; + domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source', source: 'output/gone' }); }; + drafts.set('host', 'gallery_root_paths', ['output/gone']); + const outcome = await drafts.save(); + assert.deepEqual(outcome, { browser: 'clean', host: 'failed', runtime: 'clean', + invalid: { source: 'output/gone', message } }); + assert.equal(drafts.statusText(), message); + assert.equal(drafts.invalidSource(), 'output/gone'); + assert.deepEqual([...drafts.failed('host')], ['gallery_root_paths']); + // Any other failure keeps the generic copy and names no source. + domains.outcome.host = () => { throw Object.assign(new Error('busy'), { code: 'busy' }); }; + await drafts.save(); + assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.'); + assert.equal(drafts.invalidSource(), null); + domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source' }); }; + await drafts.save(); + assert.equal(drafts.invalidSource(), null); + drafts.set('host', 'gallery_root_paths', ['output']); + assert.equal(drafts.invalidSource(), null); + assert.equal(drafts.statusText(), 'Unsaved changes.'); +}); diff --git a/tests/test_api_admin.py b/tests/test_api_admin.py index b34c0df..8718199 100644 --- a/tests/test_api_admin.py +++ b/tests/test_api_admin.py @@ -52,6 +52,7 @@ class AdminAPITestCase(AioHTTPTestCase): self.outside_dir = Path(self.tmpdir) / "outside" self.outside_dir.mkdir() + self.addAsyncCleanup(self._drain_and_remove) self.config_path = os.path.join(self.tmpdir, "config.json") self._orig_env = { key: os.environ.pop(key, None) for key in (CONFIG_PATH_ENV, EXTRA_ROOTS_ENV) @@ -89,6 +90,23 @@ class AdminAPITestCase(AioHTTPTestCase): sys.modules.pop("folder_paths", None) else: sys.modules["folder_paths"] = self._orig_folder_paths + + async def _drain_and_remove(self): + """The last cleanup: no job outlives the temporary database. + + Registered first, so it runs after every cleanup a test added (which + may release its own gated jobs). A job still running then, such as a + scan a failed assertion never awaited, is cancelled and awaited + before the database and the tree are removed. + """ + api = getattr(self, "api", None) + jobs = api._jobs.list() if api is not None else [] + tasks = [ + job.task for job in jobs if job.task is not None and not job.task.done() + ] + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) PromptModel.close_all_instances() # restore may have swapped api.db assert_closed(self, self.tmpdir) shutil.rmtree(self.tmpdir, ignore_errors=True) diff --git a/tests/test_gallery_roots.py b/tests/test_gallery_roots.py index 5ae871b..b16ffed 100644 --- a/tests/test_gallery_roots.py +++ b/tests/test_gallery_roots.py @@ -23,7 +23,11 @@ sys.modules.setdefault("server", _mock_server) sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from py.config import GalleryConfig, PromptManagerConfig # noqa: E402 +from py.config import ( # noqa: E402 + OUTSIDE_ALLOWED_REASON, + GalleryConfig, + PromptManagerConfig, +) EXTRA_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS" @@ -158,9 +162,10 @@ class GalleryRootTestCase(unittest.TestCase): class TestValidateGalleryRootRejects(GalleryRootTestCase): def test_filesystem_root_rejected(self): + """Refused as outside the allowed folders: no reason names the layout.""" ok, reason = GalleryConfig.validate_gallery_root(os.path.abspath(os.sep)) self.assertFalse(ok) - self.assertIn("root", reason.lower()) + self.assertEqual(reason, OUTSIDE_ALLOWED_REASON) @unittest.skipUnless(os.name == "nt", "drive roots only exist on Windows") def test_windows_drive_root_rejected(self): @@ -170,7 +175,17 @@ class TestValidateGalleryRootRejects(GalleryRootTestCase): def test_home_directory_rejected(self): ok, reason = GalleryConfig.validate_gallery_root(os.path.expanduser("~")) self.assertFalse(ok) - self.assertIn("home", reason.lower()) + self.assertEqual(reason, OUTSIDE_ALLOWED_REASON) + + def test_home_directory_inside_an_allowed_parent_is_still_refused(self): + """Even when the environment opens the home's parent, home answers + like any outside path.""" + home = os.path.expanduser("~") + os.environ[EXTRA_ROOTS_ENV] = os.path.dirname(home) + self.assertEqual( + GalleryConfig.gallery_root_problem(home), + ("outside-allowed", OUTSIDE_ALLOWED_REASON), + ) def test_directory_outside_comfyui_rejected(self): outside = Path(self.tmpdir) / "elsewhere" diff --git a/tests/test_scan_scope.py b/tests/test_scan_scope.py index 6044714..f5f085f 100644 --- a/tests/test_scan_scope.py +++ b/tests/test_scan_scope.py @@ -260,7 +260,7 @@ class TestScanScope(ScanScopeTestCase): async def test_empty_read_body_asks_for_the_default_scope(self): """A chunked POST with no bytes can read a body: still the default.""" - from py.api.job_routes import _scan_scope + from py.api.job_routes import _scan_request class Request: can_read_body = True @@ -271,10 +271,21 @@ class TestScanScope(ScanScopeTestCase): async def text(self): return self._text - self.assertEqual(await _scan_scope(Request("")), "new") - self.assertEqual(await _scan_scope(Request(" \n")), "new") - self.assertEqual(await _scan_scope(Request('{"scope": "all"}')), "all") - self.assertIsNone(await _scan_scope(Request("{"))) + default = ({"files": "new", "roots": None}, None) + self.assertEqual(await _scan_request(Request("")), default) + self.assertEqual(await _scan_request(Request(" \n")), default) + self.assertEqual( + await _scan_request( + Request( + '{"scope": "all", "roots": [2, 0, 2],' + ' "paths": ["output/c", "output/a", "output/c"]}' + ) + ), + ({"files": "all", "roots": {0: "output/a", 2: "output/c"}}, None), + ) + request, error = await _scan_request(Request("{")) + self.assertIsNone(request) + self.assertIsInstance(error, str) async def test_invalid_scope_is_refused(self): for body in ({"scope": "everything"}, {"scope": 1}, ["new"]): @@ -292,6 +303,220 @@ class TestScanScope(ScanScopeTestCase): self.assertIsNone(self.api._scan_job) +class TestPerSourceScope(ScanScopeTestCase): + """``roots``: scan only the selected effective sources (#249 S4 Task 10).""" + + async def asyncSetUp(self): + await super().asyncSetUp() + self.a = self.output_dir / "a" + self.b = self.output_dir / "b" + self.a.mkdir() + self.b.mkdir() + _write_png(self.a / "in_a.png", prompt_text="a prompt in a") + _write_png(self.b / "in_b.png", prompt_text="a prompt in b") + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)] + + async def _refused(self, body): + resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body) + data = await resp.json() + self.assertEqual(resp.status, 400, data) + self.assertFalse(data["success"]) + self.assertNotIn(self.tmpdir, json.dumps(data)) + self.assertIsNone(self.api._scan_job) + self.assertEqual(self.api._jobs.list(), []) + return data + + async def test_scoped_scan_only_walks_selected_root(self): + walked = [] + original = self.api._collect_output_media_sync + + def spy(output_dirs): + walked.append([os.path.basename(str(d)) for d in output_dirs]) + return original(output_dirs) + + self.api._collect_output_media_sync = spy + reads = self._spy_reads() + + job = await self._scan({"scope": "all", "roots": [1], "paths": ["output/b"]}) + + self.assertEqual(walked, [["b"]]) + self.assertEqual(reads, ["in_b.png"]) + self.assertEqual( + job["scope"], {"files": "all", "roots": [1], "output_folders": 1} + ) + self.assertEqual( + job["outcomes"]["added"], + [ + { + "path": "in_b.png", + "root": "output/b", + "root_index": 1, + "detail": job["outcomes"]["added"][0]["detail"], + } + ], + ) + self.assertEqual(job["counters"]["added"], 1) + self.assertEqual(self.api._scan_job.last_event["directories"], ["output/b"]) + + async def test_duplicate_roots_are_one_selection(self): + job = await self._scan( + {"roots": [1, 0, 1], "paths": ["output/b", "output/a", "output/b"]} + ) + self.assertEqual(job["scope"]["roots"], [0, 1]) + self.assertEqual(job["scope"]["output_folders"], 2) + self.assertEqual(job["counters"]["added"], 2) + + async def test_scope_line_counts_selected_folders(self): + job = await self._scan({"roots": [0], "paths": ["output/a"]}) + self.assertTrue( + job["log"][0].endswith( + "Scan started · 1 output folder · new and unresolved files" + ), + job["log"][0], + ) + self.assertEqual(job["scope"]["output_folders"], 1) + + unscoped = await self._scan({"scope": "all"}) + self.assertEqual( + unscoped["scope"], {"files": "all", "roots": None, "output_folders": 2} + ) + self.assertIn("Scan started · 2 output folders · all files", unscoped["log"][0]) + + async def _stale(self, body): + resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body) + data = await resp.json() + self.assertEqual(resp.status, 409, data) + self.assertEqual(data["code"], "sources-changed") + self.assertFalse(data["success"]) + self.assertNotIn(self.tmpdir, json.dumps(data)) + self.assertIsNone(self.api._scan_job) + self.assertEqual(self.api._jobs.list(), []) + return data + + async def test_index_past_the_sources_is_409_without_job(self): + await self._stale({"roots": [2], "paths": ["output/c"]}) + await self._stale({"roots": [0, 64], "paths": ["output/a", "output/x"]}) + + async def test_stale_index_never_scans_another_source(self): + """The list changed after the page read it: source 0 was removed, so + old index 1 (b) is past the end and old index 0 now names b.""" + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.b)] + await self._stale({"roots": [1], "paths": ["output/b"]}) + await self._stale({"roots": [0], "paths": ["output/a"]}) + + job = await self._scan({"roots": [0], "paths": ["output/b"]}) + self.assertEqual(job["outcomes"]["added"][0]["root"], "output/b") + + async def test_roots_need_their_paths(self): + for body in ( + {"roots": [0]}, + {"roots": [0], "paths": None}, + {"roots": [0], "paths": "output/a"}, + {"roots": [0], "paths": []}, + {"roots": [0, 1], "paths": ["output/a"]}, + {"roots": [0], "paths": [0]}, + {"roots": [0, 0], "paths": ["output/a", "output/b"]}, + ): + with self.subTest(body=body): + await self._refused(body) + + async def test_bool_and_float_roots_rejected(self): + for roots in ( + [True], + [False], + [0.0], + [1.5], + ["0"], + [None], + [-1], + [0, None], + [], + None, + 0, + "0", + {"0": 0}, + list(range(65)), + ): + with self.subTest(roots=roots): + paths = ["output/a"] * len(roots) if isinstance(roots, list) else [] + await self._refused({"roots": roots, "paths": paths}) + + async def test_legacy_scan_unscoped_unchanged(self): + from py.api.admin import SCAN_COUNT_KEYS + + resp = await self.client.request("POST", "/prompt_manager/scan", data="{}") + payloads = _sse_payloads(await resp.text()) + + final = payloads[-1] + self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"}) + self.assertEqual(final["directories"], ["output/a", "output/b"]) + self.assertEqual(final["added"], 2) + self.assertEqual( + self.api._scan_job.scope, + {"files": "all", "roots": None, "output_folders": 2}, + ) + + async def test_legacy_scan_attaches_to_a_scoped_job_as_is(self): + import threading + + from py.api.admin import SCAN_COUNT_KEYS + + gate = threading.Event() + original = self.api._collect_output_media_sync + + def gated(output_dirs): + gate.wait(5) + return original(output_dirs) + + self.api._collect_output_media_sync = gated + # The legacy route has decided to attach once it streams the job. + attached = asyncio.Event() + stream = self.api._stream_scan_job + + async def attaching(request, job): + attached.set() + return await stream(request, job) + + self.api._stream_scan_job = attaching + resp = await self.client.request( + "POST", + "/prompt_manager/jobs/scan", + json={"roots": [0], "paths": ["output/a"]}, + ) + self.assertEqual(resp.status, 202) + started = (await resp.json())["job"] + legacy = asyncio.ensure_future( + self.client.request("POST", "/prompt_manager/scan", data="{}") + ) + try: + await asyncio.wait_for(attached.wait(), 10) + finally: + gate.set() + payloads = _sse_payloads(await (await legacy).text()) + await asyncio.wait_for(self.api._scan_job.task, 10) + + self.assertEqual(self.api._scan_job.id, started["id"]) + self.assertEqual(len(self.api._jobs.list()), 1) + final = payloads[-1] + self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"}) + self.assertEqual(final["directories"], ["output/a"]) + self.assertEqual(final["added"], 1) + + async def test_scoped_scan_uses_the_sources_it_was_started_with(self): + """Sources saved after the start never change what the job walks.""" + resolved = [] + original = self.api._get_all_output_dirs + + def counting(): + resolved.append(1) + return original() + + self.api._get_all_output_dirs = counting + job = await self._scan({"roots": [1], "paths": ["output/b"]}) + self.assertEqual(len(resolved), 1) # the request's own resolution only + self.assertEqual(job["outcomes"]["added"][0]["root_index"], 1) + + class TestKnownResolvedPaths(ScanScopeTestCase): """``PromptDatabase.get_known_resolved_paths``: one streamed query.""" diff --git a/tests/test_settings_sources.py b/tests/test_settings_sources.py new file mode 100644 index 0000000..e4958bd --- /dev/null +++ b/tests/test_settings_sources.py @@ -0,0 +1,959 @@ +"""System › Sources & scans backend (#249 S4 Task 9). + +``GET /prompt_manager/settings`` reports the configured, effective and +watched sources and the monitoring setting; ``POST`` saves +``monitoring_enabled`` (a restart applies it) and refuses a source change +while a record-changing job runs; ``POST /prompt_manager/settings/validate-source`` +checks one path and writes nothing. No response carries an absolute path. +""" + +import asyncio +import functools +import json +import ntpath +import os +import sys +import types +import unittest +from contextlib import contextmanager +from unittest.mock import patch + +try: + from tests.test_api_admin import AdminAPITestCase +except ImportError: # discovered with tests/ as the top-level directory + from test_api_admin import AdminAPITestCase + +from py import config as config_module # noqa: E402 +from py.api import settings_routes, sources # noqa: E402 +from py.config import GalleryConfig # noqa: E402 + +VALIDATE = "/prompt_manager/settings/validate-source" +SETTINGS = "/prompt_manager/settings" + + +@contextmanager +def no_filesystem_access_for(marker, test): + """Fail ``test`` if anything resolves, stats or opens a path holding ``marker``. + + Trusted directories (ComfyUI's, the anchors) may still be resolved. + """ + real = { + "realpath": os.path.realpath, + "stat": os.stat, + "lstat": os.lstat, + } + + def guard(name): + def call(path, *args, **kwargs): + if isinstance(path, (str, bytes)) and marker in os.fsdecode(path): + test.fail(f"{name}({path!r}) touched an outside path") + return real[name](path, *args, **kwargs) + + return call + + with ( + patch("os.path.realpath", guard("realpath")), + patch("os.stat", guard("stat")), + patch("os.lstat", guard("lstat")), + ): + yield + + +class FakeMonitor: + """The image monitor as get_settings sees it: status and scheduled roots.""" + + def __init__(self, scheduled, alive=True): + self.monitored_directories = [str(path) for path in scheduled] + self.alive = alive + + def get_status(self): + return { + "running": True, + "monitored_directories": self.monitored_directories, + "handler_active": True, + "observer_alive": self.alive, + } + + +class SourcesTestCase(AdminAPITestCase): + async def asyncSetUp(self): + await super().asyncSetUp() + self.a = self.output_dir / "a" + self.b = self.output_dir / "b" + self.a.mkdir() + self.b.mkdir() + enabled = GalleryConfig.MONITORING_ENABLED + self.addCleanup(setattr, GalleryConfig, "MONITORING_ENABLED", enabled) + GalleryConfig.MONITORING_ENABLED = True + self._install_monitor(None) + + def _install_monitor(self, monitor): + import utils.image_monitor as im_mod + + orig = im_mod._monitor_instance + im_mod._monitor_instance = monitor + self.addCleanup(setattr, im_mod, "_monitor_instance", orig) + + async def _get(self): + resp = await self.client.request("GET", "/prompt_manager/settings") + body = await resp.text() + self.assertEqual(resp.status, 200, body) + self.assertNotIn(self.tmpdir, body) + return json.loads(body) + + async def _post(self, url, payload): + resp = await self.client.request("POST", url, json=payload) + body = await resp.text() + self.assertNotIn(self.tmpdir, body) + return resp.status, json.loads(body) + + def _config_bytes(self): + if not os.path.exists(self.config_path): + return None + with open(self.config_path, "rb") as f: + return f.read() + + def _states(self, data): + return {row["path"]: row["watch"] for row in data["sources"]["effective"]} + + +class TestSourceStatus(SourcesTestCase): + async def test_sources_list_configured_effective_and_auto_detected(self): + missing = self.output_dir / "gone" + missing.mkdir() + GalleryConfig.MONITORING_DIRECTORIES = [ + self._root(self.a), + self._root(missing), + ] + missing.rmdir() # an unmounted drive: configured, no longer a directory + + data = await self._get() + + self.assertEqual( + data["sources"]["configured"], + [ + { + "label": "a", + "path": "output/a", + "valid": True, + "code": None, + "reason": None, + }, + { + "label": "gone", + "path": "output/gone", + "valid": False, + "code": "not-found", + "reason": "Gallery root does not exist", + }, + ], + ) + self.assertEqual( + data["sources"]["effective"], + [{"label": "a", "path": "output/a", "root_index": 0, "watch": "manual"}], + ) + self.assertFalse(data["sources"]["auto_detected"]) + # The legacy keys are unchanged. + self.assertEqual( + data["settings"]["gallery_root_paths"], ["output/a", "output/gone"] + ) + + async def test_empty_list_means_auto_detect(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)] + self._write_config({"gallery": {"monitoring": {"directories": ["x"]}}}) + # Detection (the upward search for a ComfyUI tree) finds the output. + self.api._find_comfyui_output_dir = lambda: str(self.output_dir) + + status, saved = await self._post( + "/prompt_manager/settings", {"gallery_root_paths": []} + ) + + self.assertEqual(status, 200, saved) + self.assertEqual( + self._read_config()["gallery"]["monitoring"]["directories"], [] + ) + data = await self._get() + self.assertEqual(data["sources"]["configured"], []) + self.assertTrue(data["sources"]["auto_detected"]) + self.assertEqual( + data["sources"]["effective"], + [ + { + "label": "ComfyUI output", + "path": "output", + "root_index": 0, + "watch": "manual", + } + ], + ) + + async def test_watching_requires_live_observer_with_root(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)] + + # No monitor in this process: nothing is watched, a restart cannot help. + data = await self._get() + self.assertEqual( + data["monitor"], + {"present": False, "observer_alive": False, "scheduled": []}, + ) + self.assertEqual( + self._states(data), {"output/a": "manual", "output/b": "manual"} + ) + + # A dead observer watches nothing, even the roots it scheduled. + self._install_monitor(FakeMonitor([self.a], alive=False)) + data = await self._get() + self.assertEqual( + data["monitor"], + {"present": True, "observer_alive": False, "scheduled": ["output/a"]}, + ) + self.assertEqual( + self._states(data), + {"output/a": "restart_pending", "output/b": "restart_pending"}, + ) + + # A live observer watches its scheduled root; a root configured since + # it started waits for a restart. + self._install_monitor(FakeMonitor([self.a])) + data = await self._get() + self.assertTrue(data["monitor"]["observer_alive"]) + self.assertEqual( + self._states(data), {"output/a": "watching", "output/b": "restart_pending"} + ) + + # Monitoring switched off: the live observer still watches until the + # restart ("stopping"), and nothing else is pending. + GalleryConfig.MONITORING_ENABLED = False + data = await self._get() + self.assertFalse(data["settings"]["monitoring_enabled"]) + self.assertEqual( + self._states(data), {"output/a": "stopping", "output/b": "manual"} + ) + + async def test_scheduled_root_matches_through_its_spelling(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)] + spelled = os.path.join(str(self.output_dir), "b", "..", "a") + self._install_monitor(FakeMonitor([spelled])) + + data = await self._get() + + self.assertEqual(self._states(data), {"output/a": "watching"}) + self.assertEqual(data["monitor"]["scheduled"], ["output/a"]) + + async def test_status_failure_is_500_without_paths(self): + def boom(*args, **kwargs): + raise OSError(2, "No such file", f"{self.tmpdir}/secret") + + with patch.object(sources, "describe_sources", boom): + resp = await self.client.request("GET", "/prompt_manager/settings") + self.assertEqual(resp.status, 500) + self.assertNotIn(self.tmpdir, await resp.text()) + + +class TestMonitoringSetting(SourcesTestCase): + async def test_monitoring_enabled_strict_bool_and_restart_flag(self): + self._write_config({"custom": {"keep": True}}) + before = self._config_bytes() + for value in ("false", 0, 1, None, [], {"on": True}): + with self.subTest(value=value): + status, data = await self._post( + "/prompt_manager/settings", {"monitoring_enabled": value} + ) + self.assertEqual(status, 400, data) + self.assertFalse(data["success"]) + self.assertEqual(self._config_bytes(), before) + self.assertTrue(GalleryConfig.MONITORING_ENABLED) + + self._install_monitor(FakeMonitor([self.output_dir])) + status, data = await self._post( + "/prompt_manager/settings", {"monitoring_enabled": False} + ) + self.assertEqual(status, 200, data) + self.assertTrue(data["persisted"]) + self.assertTrue(data["restart_required"]) + self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"]) + saved = self._read_config() + self.assertIs(saved["gallery"]["monitoring"]["enabled"], False) + self.assertEqual(saved["custom"], {"keep": True}) + self.assertFalse(GalleryConfig.MONITORING_ENABLED) + self.assertFalse((await self._get())["settings"]["monitoring_enabled"]) + + # Back on while the observer still runs: nothing waits for a restart. + status, data = await self._post( + "/prompt_manager/settings", {"monitoring_enabled": True} + ) + self.assertEqual(status, 200, data) + self.assertEqual(data["restart_required_fields"], []) + self.assertIs(self._read_config()["gallery"]["monitoring"]["enabled"], True) + + async def test_enabling_without_a_live_observer_needs_a_restart(self): + GalleryConfig.MONITORING_ENABLED = False + status, data = await self._post( + "/prompt_manager/settings", {"monitoring_enabled": True} + ) + self.assertEqual(status, 200, data) + self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"]) + self.assertTrue(GalleryConfig.MONITORING_ENABLED) + + def test_config_file_enabled_must_be_a_bool(self): + GalleryConfig.MONITORING_ENABLED = True + with self.assertLogs("prompt_manager.config", level="WARNING"): + GalleryConfig.update_config({"monitoring": {"enabled": "false"}}) + self.assertIs(GalleryConfig.MONITORING_ENABLED, True) + GalleryConfig.update_config({"monitoring": {"enabled": False}}) + self.assertIs(GalleryConfig.MONITORING_ENABLED, False) + + +class TestSourceChangeHold(SourcesTestCase): + async def test_root_save_refused_while_scan_job_runs(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)] + self._write_config({"gallery": {"monitoring": {"directories": ["kept"]}}}) + before = self._config_bytes() + gate = asyncio.Event() + + async def runner(job): + await gate.wait() + + scan = self.api._jobs.start( + "scan", "Scan for metadata", record_changing=True, runner=runner + ) + try: + status, data = await self._post( + "/prompt_manager/settings", + {"gallery_root_paths": [str(self.b)]}, + ) + self.assertEqual(status, 409, data) + self.assertEqual(data["code"], "busy") + self.assertEqual( + data["running"], {"id": scan.id, "title": "Scan for metadata"} + ) + self.assertEqual(self._config_bytes(), before) + self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [self._root(self.a)]) + + # A save without sources, or one resending the current sources + # (the legacy page always does), is not a source change. + status, data = await self._post( + "/prompt_manager/settings", {"result_timeout": 9} + ) + self.assertEqual(status, 200, data) + status, data = await self._post( + "/prompt_manager/settings", + {"result_timeout": 8, "gallery_root_paths": [str(self.a)]}, + ) + self.assertEqual(status, 200, data) + finally: + gate.set() + await asyncio.wait_for(scan.task, 5) + + async def test_root_save_runs_as_a_source_change_hold(self): + seen = [] + original = settings_routes.get_config_store + + def store(): + seen.extend( + (job.kind, job.title, job.admission, job.cancellable) + for job in self.api._jobs.list() + if job.running + ) + return original() + + with patch.object(settings_routes, "get_config_store", store): + status, data = await self._post( + "/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]} + ) + self.assertEqual(status, 200, data) + self.assertIn(("source_change", "Source change", "records", False), seen) + self.assertFalse(any(job.running for job in self.api._jobs.list())) + + async def test_failed_root_save_ends_the_hold_failed(self): + os.makedirs(self.config_path) # config.json is a directory: unreadable + status, data = await self._post( + "/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]} + ) + self.assertGreaterEqual(status, 400, data) + held = [j for j in self.api._jobs.list() if j.kind == "source_change"] + self.assertEqual([job.state for job in held], ["failed"]) + + async def test_mixed_case_windows_paths_dedupe(self): + """Two spellings of one Windows folder are saved once (normcase).""" + with ( + patch.object(sources, "source_problem", return_value=(None, "")), + patch.object( + GalleryConfig, "resolve_gallery_root", side_effect=lambda path: path + ), + patch.object(settings_routes, "_source_identity", ntpath.normcase), + patch.object(self.api, "_apply_gallery_roots", return_value=True), + ): + roots, error = self.api._parse_gallery_roots( + { + "gallery_root_paths": [ + "C:\\ComfyUI\\Output\\Renders", + "c:/comfyui/output/renders", + "C:\\ComfyUI\\output\\other", + ] + } + ) + self.assertIsNone(error) + self.assertEqual( + roots, ["C:\\ComfyUI\\Output\\Renders", "C:\\ComfyUI\\output\\other"] + ) + + async def test_case_variant_of_one_folder_is_saved_once_natively(self): + variant = str(self.a)[:-1] + "A" + if not os.path.isdir(variant): + self.skipTest("case-sensitive filesystem") + status, data = await self._post( + "/prompt_manager/settings", {"gallery_root_paths": [str(self.a), variant]} + ) + self.assertEqual(status, 200, data) + self.assertEqual(len(GalleryConfig.MONITORING_DIRECTORIES), 1) + + +class TestValidateSource(SourcesTestCase): + async def test_validate_source_writes_nothing(self): + self._write_config({"gallery": {"monitoring": {"directories": []}}}) + before = self._config_bytes() + roots = list(GalleryConfig.MONITORING_DIRECTORIES) + + status, data = await self._post(VALIDATE, {"path": str(self.a)}) + + self.assertEqual(status, 200, data) + self.assertEqual( + data, + { + "success": True, + "valid": True, + "label": "a", + "path": "output/a", + "code": None, + "reason": None, + }, + ) + self.assertEqual(self._config_bytes(), before) + self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, roots) + self.assertEqual(self.api._jobs.list(), []) + + async def test_relative_path_resolves_against_comfyui(self): + status, data = await self._post(VALIDATE, {"path": "output/b"}) + self.assertEqual((status, data["valid"], data["path"]), (200, True, "output/b")) + status, data = await self._post(VALIDATE, {"path": "output"}) + self.assertEqual(data["label"], "ComfyUI output") + + async def test_invalid_sources_answer_a_code(self): + a_file = self.output_dir / "file.png" + a_file.write_bytes(b"x") + cases = { + "": "empty", + " ": "empty", + str(self.output_dir / "missing"): "not-found", + str(a_file): "not-directory", + str(self.outside_dir): "outside-allowed", + os.path.abspath(os.sep): "outside-allowed", + os.path.expanduser("~"): "outside-allowed", + os.path.join(str(self.output_dir), "b", "..", "a"): "traversal", + "output/../output/a": "traversal", + str(self.a) + "\x00": "invalid-form", + "\\\\?\\C:\\ComfyUI\\output": "invalid-form", + "x" * 5000: "invalid-form", + } + if os.name != "nt": + cases["C:\\ComfyUI\\output"] = "invalid-form" + for path, code in cases.items(): + with self.subTest(code=code, path=path[:40]): + status, data = await self._post(VALIDATE, {"path": path}) + self.assertEqual(status, 200, data) + self.assertFalse(data["valid"]) + self.assertEqual(data["code"], code) + self.assertIsInstance(data["reason"], str) + self.assertIsNone(data["label"]) + self.assertIsNone(data["path"]) + self.assertNotIn(os.path.expanduser("~"), json.dumps(data)) + + async def test_outside_paths_answer_alike_whether_they_exist(self): + """No existence oracle: outside the allowed folders every path looks + the same, existing or not, file or folder.""" + a_file = self.outside_dir / "file.txt" + a_file.write_bytes(b"x") + answers = [] + for path in ( + self.outside_dir, + self.outside_dir / "missing", + a_file, + os.path.join(self.tmpdir, "no", "such", "tree"), + os.path.abspath(os.sep), + os.path.expanduser("~"), + ): + status, data = await self._post(VALIDATE, {"path": str(path)}) + answers.append((status, data)) + self.assertEqual(answers[0][1]["code"], "outside-allowed") + self.assertEqual([answer for answer in answers], [answers[0]] * len(answers)) + + async def test_settings_save_refuses_outside_paths_alike(self): + """The legacy save answers every outside path alike too: the + filesystem root, the home directory and a link to it included.""" + (self.outside_dir / "x").mkdir() + link = os.path.join(self.tmpdir, "link-home") + try: + os.symlink(os.path.expanduser("~"), link, target_is_directory=True) + except (OSError, NotImplementedError): + link = None + paths = [ + str(self.outside_dir / "x"), + self.tmpdir + "/nothing/x", + os.path.abspath(os.sep), + os.path.expanduser("~"), + ] + if link: + paths.append(link) + answers = [] + for path in paths: + resp = await self.client.request( + "POST", SETTINGS, json={"gallery_root_paths": [path]} + ) + body = await resp.text() + self.assertNotIn(os.path.expanduser("~"), body) + data = json.loads(body) + # Only the name the caller sent may differ. + answers.append( + ( + resp.status, + {k: v for k, v in data.items() if k not in ("source", "error")}, + ) + ) + self.assertEqual( + data["error"], + f"{data['source']} can't be used: " + f"{config_module.OUTSIDE_ALLOWED_REASON}. Remove it, then save.", + ) + self.assertEqual( + answers[0], + ( + 400, + { + "success": False, + "code": "invalid-source", + "index": 0, + "reason": config_module.OUTSIDE_ALLOWED_REASON, + }, + ), + ) + self.assertEqual(answers, [answers[0]] * len(answers)) + self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, []) + + def test_every_caller_folds_root_and_home_alike(self): + for path in (os.path.abspath(os.sep), os.path.expanduser("~")): + with self.subTest(path=path): + self.assertEqual( + GalleryConfig.gallery_root_problem(path), + ("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON), + ) + self.assertEqual( + GalleryConfig.validate_gallery_root(path), + (False, config_module.OUTSIDE_ALLOWED_REASON), + ) + self.assertEqual( + sources.validate_source(path)["code"], "outside-allowed" + ) + + async def test_outside_path_is_never_resolved_or_stat(self): + """Containment is decided as text first: no realpath, stat or lstat of + an outside path, existing or not, by any caller (I2).""" + probe = self.outside_dir / "probe-x" + probe.mkdir() + (probe / "deep").mkdir() + cases = [str(probe), str(probe / "deep"), str(probe / "missing")] + with no_filesystem_access_for("probe-x", self): + for path in cases: + with self.subTest(path=path): + self.assertEqual( + GalleryConfig.gallery_root_problem(path)[0], "outside-allowed" + ) + self.assertEqual( + sources.validate_source(path)["code"], "outside-allowed" + ) + _roots, error = self.api._parse_gallery_roots( + {"gallery_root_paths": [path]} + ) + self.assertEqual(error["code"], "invalid-source") + status, data = await self._post(VALIDATE, {"path": path}) + self.assertEqual(data["code"], "outside-allowed") + status, data = await self._post( + SETTINGS, {"gallery_root_paths": [path]} + ) + self.assertEqual((status, data["code"]), (400, "invalid-source")) + # A traversal that normalises outside is refused as text too. + sneaky = os.path.join( + str(self.output_dir), "..", "..", "outside", "probe-x" + ) + self.assertEqual( + GalleryConfig.gallery_root_problem(sneaky)[0], "outside-allowed" + ) + + async def test_settings_save_applies_the_form_checks(self): + """POST /settings refuses what validate-source refuses, by its form.""" + for path, code in { + str(self.a) + "\x00": "invalid-form", + os.path.join(str(self.output_dir), "b", "..", "a"): "traversal", + }.items(): + with self.subTest(code=code): + status, data = await self._post( + SETTINGS, {"gallery_root_paths": [path]} + ) + self.assertEqual((status, data["code"]), (400, "invalid-source")) + self.assertEqual(data["reason"], sources.FORM_REASONS[code]) + windows = functools.partial(sources.source_form_error, windows=True) + with ( + patch.object(sources, "source_form_error", windows), + # The probes hold a token no trusted directory does (on Windows the + # temporary ComfyUI tree itself sits on C:). + no_filesystem_access_for("PMProbe", self), + ): + for path in ( + "\\\\?\\C:\\PMProbe\\output", + "\\\\.\\pipe\\PMProbe", + "C:\\PMProbe\\output\\CON", + "C:\\PMProbe\\output\\a:stream", + "C:PMProbe", + ): + with self.subTest(path=path): + _roots, error = self.api._parse_gallery_roots( + {"gallery_root_paths": ["output/a", path]} + ) + self.assertEqual(error["code"], "invalid-source") + self.assertEqual(error["index"], 1) + self.assertEqual( + error["reason"], sources.FORM_REASONS["invalid-form"] + ) + + async def test_inside_paths_say_why_they_are_not_usable(self): + a_file = self.output_dir / "file.png" + a_file.write_bytes(b"x") + status, missing = await self._post( + VALIDATE, {"path": str(self.output_dir / "missing")} + ) + status, not_dir = await self._post(VALIDATE, {"path": str(a_file)}) + self.assertEqual( + (missing["code"], not_dir["code"]), ("not-found", "not-directory") + ) + + async def test_malformed_request_is_400(self): + for body in ({}, {"path": 42}, {"path": None}, ["x"]): + with self.subTest(body=body): + status, data = await self._post(VALIDATE, body) + self.assertEqual(status, 400, data) + self.assertEqual(data["code"], "invalid-request") + self.assertFalse(data["success"]) + + +class TestSourceEdges(SourcesTestCase): + async def test_validation_failure_is_500_without_paths(self): + def boom(path): + raise OSError(13, "Permission denied", f"{self.tmpdir}/secret") + + with patch.object(sources, "validate_source", boom): + status, data = await self._post(VALIDATE, {"path": str(self.a)}) + self.assertEqual(status, 500) + self.assertEqual(data["error"], "Failed to check the source") + + async def test_oversized_body_is_413_too_large(self): + resp = await self.client.request( + "POST", VALIDATE, data=json.dumps({"path": "x" * (2 << 20)}) + ) + self.assertEqual(resp.status, 413) + self.assertEqual((await resp.json())["code"], "too-large") + + async def test_unreadable_monitor_counts_as_no_live_observer(self): + class Broken: + def get_status(self): + raise RuntimeError("observer gone") + + self._install_monitor(Broken()) + status, data = await self._post( + "/prompt_manager/settings", {"monitoring_enabled": True} + ) + self.assertEqual(status, 200, data) + self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"]) + + def test_labels_skip_getters_that_fail_or_are_missing(self): + import sys + import types + + def fails(): + raise RuntimeError("no input dir") + + fake = types.SimpleNamespace( + get_output_directory=lambda: str(self.output_dir), + get_input_directory=fails, + get_temp_directory=lambda: None, + ) + with patch.dict(sys.modules, {"folder_paths": fake}): + self.assertEqual(sources.source_label(self.output_dir), "ComfyUI output") + self.assertEqual(sources.source_label(self.a), "a") + with patch.dict(sys.modules, {"folder_paths": None}): + self.assertEqual(sources.source_label(self.output_dir), "output") + + +class TestOutsideBaseDirectories(SourcesTestCase): + """ComfyUI folders outside the base directory (``--output-directory``): + their sources round-trip and two folders of one name stay apart (I3).""" + + async def asyncSetUp(self): + await super().asyncSetUp() + self.renders = self.outside_dir / "renders" + self.input_dir = self.comfy_dir / "input" + (self.renders / "portraits").mkdir(parents=True) + (self.input_dir / "portraits").mkdir(parents=True) + sys.modules["folder_paths"] = types.SimpleNamespace( + base_path=str(self.comfy_dir), + get_output_directory=lambda: str(self.renders), + get_input_directory=lambda: str(self.input_dir), + ) + + async def test_source_outside_base_round_trips(self): + status, rendered = await self._post( + VALIDATE, {"path": str(self.renders / "portraits")} + ) + self.assertEqual( + (rendered["valid"], rendered["path"]), (True, "renders/portraits") + ) + status, data = await self._post( + SETTINGS, {"gallery_root_paths": [rendered["path"]]} + ) + self.assertEqual(status, 200, data) + self.assertEqual( + GalleryConfig.MONITORING_DIRECTORIES, + [self._root(self.renders / "portraits")], + ) + configured = (await self._get())["sources"]["configured"] + self.assertEqual( + [(row["path"], row["valid"]) for row in configured], + [("renders/portraits", True)], + ) + # The page resends what it read: still the same folder, no change. + status, data = await self._post( + SETTINGS, {"gallery_root_paths": ["renders/portraits"]} + ) + self.assertEqual((status, data["restart_required_fields"]), (200, [])) + + async def test_same_name_folders_stay_apart(self): + answers = [] + for path in (self.renders / "portraits", self.input_dir / "portraits"): + status, data = await self._post(VALIDATE, {"path": str(path)}) + answers.append(data["path"]) + self.assertEqual(answers, ["renders/portraits", "input/portraits"]) + status, data = await self._post(SETTINGS, {"gallery_root_paths": answers}) + self.assertEqual(status, 200, data) + sources_ = (await self._get())["sources"] + self.assertEqual([row["path"] for row in sources_["configured"]], answers) + self.assertEqual([row["path"] for row in sources_["effective"]], answers) + + +class TestUnusableSourceSave(SourcesTestCase): + async def test_unusable_source_is_named_and_saves_nothing(self): + """A configured source that went missing blocks the save and is named.""" + gone = self.output_dir / "gone" + gone.mkdir() + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(gone)] + self._write_config({"custom": 1}) + before = self._config_bytes() + gone.rmdir() # an unplugged drive + + status, data = await self._post( + SETTINGS, + { + "gallery_root_paths": ["output/a", "output/gone", "output/b"], + "worker_threads": 2, + }, + ) + + self.assertEqual(status, 400, data) + self.assertEqual( + data, + { + "success": False, + "code": "invalid-source", + "source": "output/gone", + "index": 1, + "reason": "Gallery root does not exist", + "error": "output/gone can't be used: Gallery root does not exist. " + "Remove it or reconnect it, then save.", + }, + ) + self.assertEqual(self._config_bytes(), before) + + async def test_malformed_roots_are_coded_invalid_request(self): + for roots in ("output/a", [1]): + with self.subTest(roots=roots): + status, data = await self._post(SETTINGS, {"gallery_root_paths": roots}) + self.assertEqual((status, data["code"]), (400, "invalid-request")) + + +class TestUnchangedRootsTakeNoHold(SourcesTestCase): + async def test_resent_spelling_of_configured_roots_is_no_change(self): + """config.json holds a hand-written spelling; the legacy page resends + it resolved: no source change, so no hold and no 409 during a scan (M1).""" + GalleryConfig.MONITORING_DIRECTORIES = ["output/a"] + gate = asyncio.Event() + + async def runner(job): + await gate.wait() + job.finish("finished") + + self.api._jobs.start( + "scan", "Metadata scan", record_changing=True, runner=runner + ) + try: + status, data = await self._post( + SETTINGS, {"gallery_root_paths": [str(self.a)], "result_timeout": 30} + ) + self.assertEqual(status, 200, data) + self.assertEqual(data["restart_required_fields"], []) + self.assertEqual([j.kind for j in self.api._jobs.list()], ["scan"]) + status, data = await self._post( + SETTINGS, {"gallery_root_paths": [str(self.b)]} + ) + self.assertEqual((status, data["code"]), (409, "busy")) + finally: + gate.set() + + +class TestWindowsContainment(unittest.TestCase): + """Windows path forms, with ntpath fakes: anything outside the allowed + folders (a UNC share, the verbatim and device namespaces, another drive) + is refused as text, before any path is resolved or opened (I2).""" + + ALLOWED = "c:\\comfyui\\output" + NAS = "\\\\nas\\share\\renders" + + def setUp(self): + self.touched = [] + + def realpath(path, *args, **kwargs): + self.touched.append(path) + return ntpath.normcase(ntpath.normpath(path)) + + def probe(path): + self.touched.append(path) + return True + + names = ("normcase", "normpath", "isabs", "join", "dirname", "basename", "sep") + fake_path = types.SimpleNamespace( + **{name: getattr(ntpath, name) for name in names}, + realpath=realpath, + exists=probe, + isdir=probe, + expanduser=lambda path: "c:\\users\\someone", + ) + fake_os = types.SimpleNamespace( + path=fake_path, environ={}, pathsep=";", name="nt" + ) + for target, value in ( + (patch.object(config_module, "os", fake_os), None), + ( + patch.object( + GalleryConfig, + "_allowed_parent_pairs", + classmethod( + lambda cls: [(self.ALLOWED, self.ALLOWED), (self.NAS, self.NAS)] + ), + ), + None, + ), + ( + patch.object( + GalleryConfig, + "path_anchors", + classmethod(lambda cls: ["c:\\comfyui", "\\\\nas\\share"]), + ), + None, + ), + ): + target.start() + self.addCleanup(target.stop) + + def test_outside_forms_are_refused_untouched(self): + for path in ( + "\\\\attacker\\share\\x", + "//attacker/share/x", + "\\\\?\\C:\\ComfyUI\\output\\a", + "\\\\?\\UNC\\attacker\\share", + "\\\\.\\pipe\\x", + "\\\\.\\C:\\ComfyUI\\output", + "D:\\Renders", + "C:\\Windows", + "C:\\ComfyUI\\output\\..\\..\\Windows", + "C:\\ComfyUI\\output2", + "\\\\nas\\share\\other", + "..\\..\\Windows", + ): + with self.subTest(path=path): + self.touched.clear() + self.assertEqual( + GalleryConfig.gallery_root_problem(path), + ("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON), + ) + self.assertEqual(self.touched, []) + + def test_allowed_spellings_are_accepted(self): + for path in ( + "C:\\ComfyUI\\output\\a", + "c:/comfyui/OUTPUT/a", + "output\\a", + "\\\\NAS\\share\\renders\\a", # a UNC spelling of an allowed extra root + ): + with self.subTest(path=path): + self.assertEqual(GalleryConfig.gallery_root_problem(path), (None, "")) + + +class TestSourceFormError(unittest.TestCase): + """The lexical check that runs before any filesystem access.""" + + def test_windows_forms(self): + err = sources.source_form_error + for path in ( + "C:\\ComfyUI\\output", + "C:/ComfyUI/output", + "\\\\server\\share\\output", + "D:\\images", + ): + with self.subTest(path=path): + self.assertIsNone(err(path, windows=True)) + for path, code in { + "C:output": "invalid-form", # drive-relative + "C:\\out\\a:stream": "invalid-form", # alternate data stream + "C:\\out\\CON": "invalid-form", + "C:\\out\\nul.txt": "invalid-form", + "C:\\out\\folder.": "invalid-form", + "C:\\out\\folder \\sub": "invalid-form", + "\\\\?\\C:\\out": "invalid-form", + "//?/C:/out": "invalid-form", + "\\\\.\\pipe\\x": "invalid-form", + "\\\\server": "invalid-form", + "C:\\out\\..\\in": "traversal", + "output\\..\\input": "traversal", + }.items(): + with self.subTest(path=path): + self.assertEqual(err(path, windows=True), code) + + def test_posix_forms(self): + err = sources.source_form_error + for path in ("/srv/comfy/output", "output/renders", "out/CON", "a b/c."): + with self.subTest(path=path): + self.assertIsNone(err(path, windows=False)) + for path, code in { + "C:\\ComfyUI\\output": "invalid-form", + "C:/ComfyUI/output": "invalid-form", + "\\\\server\\share": "invalid-form", + "out\\renders": "invalid-form", + "/srv/../etc": "traversal", + "../output": "traversal", + "/srv/out\x00": "invalid-form", + "/srv/out\n": "invalid-form", + "": "empty", + }.items(): + with self.subTest(path=path): + self.assertEqual(err(path, windows=False), code) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/vendor/frontend-coverage-baseline.json b/tools/vendor/frontend-coverage-baseline.json index 177df6b..75926eb 100644 --- a/tools/vendor/frontend-coverage-baseline.json +++ b/tools/vendor/frontend-coverage-baseline.json @@ -7,7 +7,7 @@ }, "api-client.js": { "lines": 98, - "branches": 95, + "branches": 96, "functions": 98, "statements": 98 }, @@ -163,7 +163,7 @@ }, "workspace/activity-scan.mjs": { "lines": 96, - "branches": 73, + "branches": 78, "functions": 98, "statements": 96 }, @@ -467,6 +467,12 @@ "functions": 98, "statements": 96 }, + "workspace/system-sources.mjs": { + "lines": 98, + "branches": 88, + "functions": 95, + "statements": 98 + }, "workspace/system-state.mjs": { "lines": 98, "branches": 94, @@ -476,7 +482,7 @@ "workspace/system.mjs": { "lines": 98, "branches": 92, - "functions": 85, + "functions": 86, "statements": 98 }, "workspace/theme.mjs": { diff --git a/web/js/api-client.js b/web/js/api-client.js index b334467..ee21d36 100644 --- a/web/js/api-client.js +++ b/web/js/api-client.js @@ -40,6 +40,21 @@ Number.isSafeInteger(current.tag_revision) && current.tag_revision >= 0) { this.current = { tags: current.tags.slice(), tag_revision: current.tag_revision }; } + // A busy answer names the running job that holds the slot; an + // invalid-source answer names the source it refused. Only + // well-formed values are kept. + this.running = null; + const running = current && current.running; + if (this.code === "busy" && running && typeof running.id === "string" && + /^[A-Za-z0-9_-]{1,64}$/.test(running.id) && typeof running.title === "string" && + running.title.length > 0 && running.title.length <= 200) { + this.running = { id: running.id, title: running.title }; + } + this.source = null; + if (this.code === "invalid-source" && current && typeof current.source === "string" && + current.source.length > 0 && current.source.length <= 4096) { + this.source = current.source; + } } } diff --git a/web/js/workspace/activity-scan.mjs b/web/js/workspace/activity-scan.mjs index 8ec2c87..f92af7b 100644 --- a/web/js/workspace/activity-scan.mjs +++ b/web/js/workspace/activity-scan.mjs @@ -17,6 +17,8 @@ export const REPORT_DONE = 'Job report downloaded.'; export const REPORT_FAILED = 'The report could not be downloaded. Try again.'; const JOBS_REPORT_BASE = '/prompt_manager/jobs'; export const START_FAILED = 'The scan could not start. Try again.'; +// A per-source scan whose sources changed since they were read (S4): the server's 409 says so. +const SOURCES_CHANGED_CODE = 'sources-changed'; const FALLBACK_NAME = 'prompts_backup.db'; const REPORT_FALLBACK_NAME = 'prompt-manager-report.json'; @@ -81,25 +83,37 @@ export function busyToast(running) { /** * Start the scan job over `files` ('new': new and unresolved files, or 'all') and open its view; a refused or failed - * start says why and stays on the page. + * start says why and stays on the page. `roots` (System › Sources, S4) limits it to those effective source indexes, and + * `paths` names the public path read for each: when the sources changed since (409 `sources-changed`) nothing starts, + * the toast says the server's reason and `onStale` (which every caller with `roots` passes) reads them again. */ -export async function startMetadataScan({ store, scope, files = 'new', notify = showWorkspaceToast, navigate = navigateTo }) { +export async function startMetadataScan({ store, scope, files = 'new', roots = null, paths = null, onStale, + notify = showWorkspaceToast, navigate = navigateTo }) { try { - const result = await store.start('scan', { scope: files }); + const result = await store.start('scan', roots ? { scope: files, roots, paths } : { scope: files }); if (scope.signal.aborted) return; if (result.busy !== undefined) { notify(busyToast(result.busy)); return; } navigate(`#/activity/${result.job.id}`); - } catch { - if (!scope.signal.aborted) notify(START_FAILED); + } catch (error) { + if (scope.signal.aborted) return; + if (error?.code === SOURCES_CHANGED_CODE) { + notify(error.message); + onStale(); + return; + } + notify(START_FAILED); } } -/** Open "Review metadata scan" for `sources` output folders. */ -export function openMetadataScan({ scope, store, sources, trigger = null }) { +/** + * Open "Review metadata scan" for `sources` output folders; with `roots` and `paths` (one source's index and public + * path, System › Sources) the scan reads only those, and `onStale` runs when they changed since they were read. + */ +export function openMetadataScan({ scope, store, sources, roots = null, paths = null, onStale, trigger = null }) { openScanReviewDialog({ scope, sources, trigger, onBackup: () => downloadDatabaseBackup(), - onStart: ({ scope: files }) => startMetadataScan({ store, scope, files }) }); + onStart: ({ scope: files }) => startMetadataScan({ store, scope, files, roots, paths, onStale }) }); } diff --git a/web/js/workspace/system-sources.mjs b/web/js/workspace/system-sources.mjs new file mode 100644 index 0000000..a204f07 --- /dev/null +++ b/web/js/workspace/system-sources.mjs @@ -0,0 +1,395 @@ +/** + * System › Sources & scans (#249 System S4 Task 11): the concept's `renderSystem` "sources" body (app.js 679–681). + * + * - The optional `.notice` "Bring older images into your prompt history" shows until this browser dismissed the 3.2.0 + * update notice (`pm_update_notice_dismissed`, the legacy admin page's own key). Later dismisses it without scanning; + * Review scan dismisses it, as the legacy page's Scan does, and reviews every source. + * - "Image sources" lists the server's sources (GET /settings, S4 Task 9) in `#source-list`, one `.operation` each: + * its label, "<path> · watching | watching until restart | manual scan | restart pending" and "Scan changes" + * (watched) or "Scan folder", which opens the shared "Review metadata scan" (D55) for that one source ("1 output + * folder") and starts the session job over it (`roots` with its public path, S4 Task 10). The job opens in Activity; + * the toast only says it started. When the sources changed since they were read, nothing starts: the toast says so + * and the list is read again. + * - "+ Add source" asks the server whether a folder can be a source (`validate-source`, which writes nothing), then adds + * it to the host draft of `gallery_root_paths`; Remove takes one out. Both wait for the one Save row. + * - "Watch for new outputs" (`monitoring_enabled`) and "Worker threads" (`worker_threads`, a range up to the server's + * `cpu_count`, D113) are host drafts too. "Enrich existing metadata" reviews a scan of every source. + * - "Thumbnail generation → Review options" opens the existing gallery's thumbnail tool until its review dialog (S5), + * the D94 hand-off for the one capability not native yet. + * + * D98: per-source Remove; an added source reads "not saved yet" until Save; the line under the list says when + * ComfyUI's output folder is used instead; "restart pending" for a source the running monitor does not watch yet; the + * "Refresh files" operation (`/gallery/rescan`) closes the surface. D127: the add dialog's field is "Folder path", + * starts empty and says the server's reason in place; its action is "Add", and Enter adds too. + * D128: "watching until restart" for a source the running monitor still watches after monitoring was switched off; + * "same folder as another source" for a configured source the effective list already holds under another spelling; a + * save refused for one source (`invalid-source`) marks that row and the status says the server's reason. + */ +import { createButton } from '../shared/toolkit/core.js'; +import { closeWorkspaceDialog, field, openContentDialog } from './workspace-dialogs.mjs'; +import { openMetadataScan } from './activity-scan.mjs'; +import { settingRow, operationRow } from './system-controls.mjs'; +import { HOST_UNAVAILABLE } from './system-appearance.mjs'; +import { showWorkspaceToast } from './toast.mjs'; +import { noun } from './plural.mjs'; + +const SETTINGS_URL = '/prompt_manager/settings'; +const VALIDATE_URL = '/prompt_manager/settings/validate-source'; +const RESCAN_URL = '/prompt_manager/gallery/rescan'; +const GALLERY_URL = '/prompt_manager/gallery'; +const ROOTS_KEY = 'gallery_root_paths'; + +export const NOTICE_KEY = 'pm_update_notice_dismissed'; +export const NOTICE_VERSION = '3.2.0'; +export const WATCH_COPY = Object.freeze({ watching: 'watching', stopping: 'watching until restart', manual: 'manual scan', + restart_pending: 'restart pending' }); +// The states whose observer runs now: their scan only has to catch up on changes. +const WATCHED = new Set(['watching', 'stopping']); +const NOT_SAVED = 'not saved yet'; +export const DUPLICATE_SOURCE = 'same folder as another source'; +/** D98: the line under the source list when it does not say everything by itself. */ +export const NOTES = Object.freeze({ + removedAll: 'No source is configured. After saving, ComfyUI’s output folder is used.', + autoDetected: 'No source is configured; ComfyUI’s output folder is used.', + fallback: 'No configured source is available; ComfyUI’s output folder is used.', + none: 'No source is available.', +}); +export const LOADING_TEXT = 'Loading sources…'; +export const LOAD_FAILED_TEXT = 'Sources could not be loaded. Your saved data has not changed.'; +export const DUPLICATE_TEXT = 'This folder is already a source.'; +export const VALIDATE_FAILED_TEXT = 'The folder could not be checked. Try again.'; +export const REFRESH_FAILED_TEXT = 'Files could not be refreshed. Try again.'; + +// Labels of the sources added in this page session (validate-source answers them), for the rows not saved yet. +const addedLabels = new Map(); + +const defaultStorage = () => window.localStorage; + +/** Whether this browser dismissed the 3.2.0 notice; storage that cannot be read shows it. */ +export function noticeDismissed(getStorage = defaultStorage) { + try { + return getStorage().getItem(NOTICE_KEY) === NOTICE_VERSION; + } catch { + return false; + } +} + +/** Dismiss the notice for this browser; false when storage refused (it stays dismissed for this view). */ +export function dismissNotice(getStorage = defaultStorage) { + try { + getStorage().setItem(NOTICE_KEY, NOTICE_VERSION); + return true; + } catch { + return false; + } +} + +const sameRoots = (a, b) => Array.isArray(a) && Array.isArray(b) && a.length === b.length && a.every((path, index) => path === b[index]); + +/** The roots to draft: the saved list itself when they are equal, so the draft is clean again. */ +export function draftRoots(next, saved) { + return sameRoots(next, saved) ? saved : [...next]; +} + +const folderName = path => path.split('/').filter(Boolean).at(-1) ?? path; + +function effectiveRow(source, remove) { + const watched = WATCHED.has(source.watch); + return { key: remove ?? `auto:${source.root_index}`, title: source.label, + description: `${source.path} · ${WATCH_COPY[source.watch] ?? WATCH_COPY.manual}`, + scan: { label: watched ? 'Scan changes' : 'Scan folder', root: source.root_index, path: source.path }, remove }; +} + +/** + * The source list for `sources` (GET /settings' `sources`), the saved `gallery_root_paths` and the drafted ones + * (`current`): `{rows: [{key, title, description, scan: {label, root, path} | null, remove: path | null}], note}`. + * A valid configured source missing from the effective list is another spelling of a folder already listed. + */ +export function sourceRows({ sources, saved, current, labels = addedLabels }) { + const configured = sources.configured ?? []; + const effective = sources.effective ?? []; + const savedRoots = saved ?? configured.map(source => source.path); + const roots = current ?? savedRoots; + const dirty = !sameRoots(roots, savedRoots); + const rows = []; + for (const path of new Set(roots)) { + const entry = configured.find(source => source.path === path); + const live = entry?.valid ? effective.find(source => source.path === path) : undefined; + if (live) rows.push(effectiveRow(live, path)); + else if (entry) { + const why = entry.valid ? DUPLICATE_SOURCE : entry.reason ?? WATCH_COPY.manual; + rows.push({ key: path, title: entry.label, description: `${path} · ${why}`, scan: null, remove: path }); + } else rows.push({ key: path, title: labels.get(path) ?? folderName(path), description: `${path} · ${NOT_SAVED}`, scan: null, remove: path }); + } + if (dirty) return { rows, note: roots.length ? null : NOTES.removedAll }; + if (sources.auto_detected) { + rows.push(...effective.map(source => effectiveRow(source, null))); + return { rows, note: configured.length ? NOTES.fallback : NOTES.autoDetected }; + } + return { rows, note: rows.length ? null : NOTES.none }; +} + +/** The Refresh files toast for the rescan's `{total, roots}`. */ +export function refreshedText(result) { + const { total, roots } = result ?? {}; + if (![total, roots].every(value => Number.isSafeInteger(value) && value >= 0)) return 'Files refreshed.'; + return `Files refreshed. ${total.toLocaleString('en-US')} ${noun(total, 'file')} in ${roots.toLocaleString('en-US')} output ${noun(roots, 'folder')}.`; +} + +function el(tag, text, className) { + const node = document.createElement(tag); + if (text !== undefined) node.textContent = text; + if (className) node.className = className; + return node; +} + +/** The concept's `.notice` with Later and Review scan. */ +function noticeNode({ scope, onLater, onReview }) { + const notice = el('div', undefined, 'notice'); + const text = el('div'); + text.append(el('h2', 'Bring older images into your prompt history'), + el('p', 'The schema upgrades automatically. An optional metadata scan links existing positives and negatives without regenerating all thumbnails.')); + const actions = el('div', undefined, 'row-actions'); + const later = createButton('Later'); + const review = createButton('Review scan', 'ca-primary'); + scope.listen(later, 'click', onLater); + scope.listen(review, 'click', onReview); + actions.append(later, review); + notice.append(text, actions); + return notice; +} + +/** + * "Add image source" (the concept's `simpleInput`, D127): the server checks the folder; a refused one keeps the + * dialog open with the server's reason under the field. Enter in the field adds, as the Add button does. + */ +function openAddSource({ api, scope, trigger, listed, onAdd }) { + const input = el('input'); + const reason = el('p', undefined, 'small'); + reason.setAttribute('role', 'alert'); + const cancel = createButton('Cancel'); + const add = createButton('Add', 'ca-primary'); + let open = true; + const refuse = text => { + reason.textContent = text; + if (!reason.isConnected) actions.before(reason); + input.setAttribute('aria-invalid', 'true'); + input.focus(); + }; + scope.listen(input, 'input', () => { + reason.remove(); + input.removeAttribute('aria-invalid'); + }); + scope.listen(input, 'keydown', event => { + if (event.key !== 'Enter' || event.isComposing) return; + event.preventDefault(); + submit(); + }); + scope.listen(cancel, 'click', () => closeWorkspaceDialog()); + scope.listen(add, 'click', () => submit()); + async function submit() { + const path = input.value.trim(); + if (!path) { input.focus(); return; } + if (add.getAttribute('aria-disabled') === 'true') return; + add.setAttribute('aria-disabled', 'true'); + try { + const result = await api.post(VALIDATE_URL, { path }, { signal: scope.signal }); + if (!open) return; + if (!result.valid) refuse(result.reason ?? VALIDATE_FAILED_TEXT); + else if (listed().includes(result.path)) refuse(DUPLICATE_TEXT); + else { + closeWorkspaceDialog(); + onAdd(result); + } + } catch { + if (open) refuse(VALIDATE_FAILED_TEXT); + } finally { + add.removeAttribute('aria-disabled'); + } + } + const actions = el('div', undefined, 'dialog-actions'); + actions.append(cancel, add); + openContentDialog({ scope, title: 'Add image source', trigger, nodes: [field('Folder path', input, 'action-input'), actions], + onClose: () => { open = false; } }); + input.focus(); +} + +/** The row's control (`data-setting`), wherever the Toolkit switch nests it. */ +const controlOf = (row, id) => row.querySelector(`[data-setting="${id}"]`); + +/** + * Mount Sources in `container` (System's `.settings-layout`); returns the section's nodes. `activity` is the page's + * Activity store (scans start through it), `drafts` the System drafts. + */ +export function mountSources(container, { api, activity, scope, drafts, notify = showWorkspaceToast, storage = defaultStorage, + openWindow = (url) => window.open(url, '_blank', 'noopener'), latestOnly = window.ApiClient.latestOnly, + STALE = window.ApiClient.STALE }) { + const statusRead = latestOnly({ lifetime: scope.signal }); + // undefined while read, null when the read failed, else GET /settings' `sources`. + let sources; + const nodes = []; + const scanAll = trigger => openMetadataScan({ scope, store: activity, trigger, sources: sources?.effective?.length ?? 0 }); + + if (!noticeDismissed(storage)) { + const notice = noticeNode({ scope, + onLater: () => { dismissNotice(storage); notice.remove(); }, + onReview: () => { + dismissNotice(storage); + // The pressed button leaves with the notice, so focus returns to the list's Add source. + notice.remove(); + scanAll(addButton); + } }); + nodes.push(notice); + } + + const surface = el('section', undefined, 'surface'); + const heading = el('div', undefined, 'panel-heading'); + const addButton = createButton('+ Add source'); + heading.append(el('h2', 'Image sources'), addButton); + const list = el('div'); + list.id = 'source-list'; + surface.append(heading, el('p', 'The same folders power browsing, monitoring and metadata enrichment.'), list); + + const roots = () => drafts.get('host', ROOTS_KEY) ?? sources?.configured?.map(source => source.path) ?? []; + const setRoots = next => drafts.set('host', ROOTS_KEY, draftRoots(next, drafts.saved('host')[ROOTS_KEY])); + scope.listen(addButton, 'click', () => openAddSource({ api, scope, trigger: addButton, listed: roots, + onAdd: ({ path, label }) => { + addedLabels.set(path, label); + setRoots([...roots(), path]); + } })); + + const monitoring = settingRow({ id: 'monitoring', label: 'Watch for new outputs', + description: 'New files enter the same library. Existing scans do not count as new runs.', control: { type: 'toggle' }, + scope, onChange: value => drafts.set('host', 'monitoring_enabled', value) }); + const workers = settingRow({ id: 'workers', label: 'Worker threads', + description: 'Balance background scans and thumbnail work against generation.', control: { type: 'range', value: 4 }, + scope, onChange: value => drafts.set('host', 'worker_threads', value) }); + const hostRows = [[monitoring, 'monitoring', 'monitoring_enabled'], [workers, 'workers', 'worker_threads']] + .map(([row, id, key]) => ({ input: controlOf(row, id), key, text: row.children[0], note: null })); + for (const row of hostRows) row.input.dataset.key = row.key; + surface.append(monitoring, workers, + operationRow({ scope, title: 'Enrich existing metadata', description: 'Resolve both prompts, deduplicate recipes, preserve uncertain links.', + label: 'Review scan', onClick: scanAll }), + // D94: the gallery keeps thumbnail generation until its review dialog (S5). + operationRow({ scope, title: 'Thumbnail generation', description: 'Generate missing previews; rebuilding all is a separate explicit option.', + label: 'Review options', onClick: () => openWindow(window.ApiClient.resolveUrl(GALLERY_URL)) }), + // D98: the legacy gallery's Refresh, which rereads every source's file list. + operationRow({ scope, title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.', + label: 'Refresh', onClick: refresh })); + nodes.push(surface); + + async function refresh(button) { + if (button.getAttribute('aria-disabled') === 'true') return; + button.setAttribute('aria-disabled', 'true'); + try { + const result = await api.post(RESCAN_URL, {}, { signal: scope.signal }); + if (!scope.signal.aborted) notify(refreshedText(result)); + } catch { + if (!scope.signal.aborted) notify(REFRESH_FAILED_TEXT); + } finally { + button.removeAttribute('aria-disabled'); + } + } + + function sourceNode(row) { + const actions = []; + if (row.scan) { + const scan = createButton(row.scan.label); + scope.listen(scan, 'click', () => openMetadataScan({ scope, store: activity, trigger: scan, sources: 1, + roots: [row.scan.root], paths: [row.scan.path], onStale: reread })); + actions.push(scan); + } + if (row.remove !== null) { + // D98: a plain button; nothing is removed until Save. + const remove = createButton('Remove'); + scope.listen(remove, 'click', () => setRoots(roots().filter(path => path !== row.remove))); + actions.push(remove); + } + const node = el('div', undefined, 'operation'); + // D128: the source the last save was refused for; the status says why. + if (row.remove !== null && row.remove === drafts.invalidSource?.()) node.setAttribute('aria-invalid', 'true'); + const text = el('div'); + text.append(el('h3', row.title), el('p', row.description)); + const group = el('div', undefined, 'row-actions'); + group.append(...actions); + node.append(text, group); + return node; + } + + function paintList() { + if (sources === undefined) { + const loading = el('p', LOADING_TEXT, 'small'); + loading.setAttribute('role', 'status'); + list.replaceChildren(loading); + return; + } + if (sources === null) { + const alert = el('p', LOAD_FAILED_TEXT); + alert.setAttribute('role', 'alert'); + const retry = createButton('Retry'); + scope.listen(retry, 'click', () => { + load(); + if (drafts.hostState?.() === 'failed') drafts.reloadHost(); + }); + list.replaceChildren(alert, retry); + return; + } + const { rows, note } = sourceRows({ sources, saved: drafts.saved('host')[ROOTS_KEY], current: drafts.get('host', ROOTS_KEY) }); + list.replaceChildren(...rows.map(sourceNode), ...(note ? [el('p', note, 'small')] : [])); + } + + function paintControls() { + const unavailable = drafts.hostState?.() === 'failed'; + for (const row of hostRows) { + const value = drafts.get('host', row.key); + row.input.disabled = value === undefined; + if (row.key === 'worker_threads') { + // D113: the range ends at the server's core count (the concept's fixed 16). + const cpus = drafts.get('host', 'cpu_count'); + if (Number.isSafeInteger(cpus) && cpus >= 1) row.input.max = String(cpus); + if (value !== undefined) row.input.value = String(value); + } else if (value !== undefined) row.input.checked = value === true; + if (unavailable && value === undefined) { + if (!row.note) { row.note = el('p', HOST_UNAVAILABLE, 'small'); row.text.append(row.note); } + } else if (row.note) { row.note.remove(); row.note = null; } + } + } + + // The sources changed since this page read them (a refused per-source scan): read them and the saved values again. + function reread() { + load(); + drafts.reloadHost?.(); + } + + async function load() { + sources = undefined; + paintList(); + let answer; + try { + answer = await statusRead(signal => api.get(SETTINGS_URL, { signal, cache: 'no-store' })); + } catch { + answer = null; + } + if (answer === STALE || scope.signal.aborted) return; + sources = answer?.sources ?? null; + paintList(); + } + + // A save (or a reread after a conflict) changed the saved sources or monitoring: read their state again. The first + // host read on mount is not a change. + const savedState = () => { + const saved = drafts.saved('host'); + return saved[ROOTS_KEY] === undefined ? undefined : JSON.stringify([saved[ROOTS_KEY], saved.monitoring_enabled]); + }; + let state = savedState(); + scope.cleanup(drafts.subscribe(() => { + const previous = state; + state = savedState(); + paintControls(); + if (previous !== undefined && state !== previous) load(); + else if (sources) paintList(); + })); + paintControls(); + load(); + return nodes; +} diff --git a/web/js/workspace/system-state.mjs b/web/js/workspace/system-state.mjs index b827dc3..a0d29ff 100644 --- a/web/js/workspace/system-state.mjs +++ b/web/js/workspace/system-state.mjs @@ -17,8 +17,10 @@ * Adapters: `readBrowser()` and `readRuntime()` return the values; `readHost()` returns `{values, revision}`. * `writeBrowser(changes)` returns `{persisted}` (false: kept for this tab only) and throws an error with `code: * 'settings-changed'` when another page changed a drafted value (a conflict: the browser values are read again), - * `writeHost(changes, {expected_revision})` the settings response (`{success, persisted, revision}`), `writeRuntime(changes)` anything; - * a write that throws failed. + * `writeHost(changes, {expected_revision})` the settings response (`{success, persisted, revision}`, and the + * `restart_required_fields` the status names), `writeRuntime(changes)` anything; a write that throws failed. A host + * write refused as `invalid-source` (its error's `source` and server message) makes that message the status and + * `invalidSource()` that source, so Sources marks its row (S4). */ export { SYSTEM_SECTIONS, isSystemSection } from './state.mjs'; @@ -32,21 +34,35 @@ export const SETTINGS_STATUS = Object.freeze({ partial: 'Some changes were not saved. Review the highlighted settings.', tabOnly: 'Saved for this tab only; browser storage is unavailable.', conflict: 'Settings changed elsewhere. Reload this section to see them.', + // D128: a saved setting the running server applies only after a restart. + restart: 'Restart ComfyUI to apply:', }); +/** D128: the settings a host save can report in `restart_required_fields`, by their row label. */ +export const RESTART_LABELS = Object.freeze({ monitoring_enabled: 'Watch for new outputs' }); + export const DRAFT_DOMAINS = Object.freeze(['browser', 'host', 'runtime']); const CONFLICT_CODE = 'settings-changed'; +const INVALID_SOURCE_CODE = 'invalid-source'; const ADAPTERS = ['readBrowser', 'readHost', 'readRuntime', 'writeBrowser', 'writeHost', 'writeRuntime']; -/** The status copy of one `save()` outcome (`{browser, host, runtime}`, each clean/saved/tab-only/applied/failed/conflict). */ +/** + * The status copy of one `save()` outcome (`{browser, host, runtime}`, each clean/saved/tab-only/applied/failed/conflict). + * `restart` lists the host fields the server applies at its next start, `invalid` the source a refused host save named + * (`{source, message}`): its message is the status (D128). + */ export function statusText(outcome) { const results = DRAFT_DOMAINS.map(domain => outcome?.[domain]); if (results.includes('conflict')) return SETTINGS_STATUS.conflict; - if (results.includes('failed')) return SETTINGS_STATUS.partial; + if (results.includes('failed')) return outcome?.invalid?.message || SETTINGS_STATUS.partial; const saved = []; if (outcome?.browser === 'saved') saved.push(SETTINGS_STATUS.browser); if (outcome?.browser === 'tab-only') saved.push(SETTINGS_STATUS.tabOnly); - if (outcome?.host === 'saved') saved.push(SETTINGS_STATUS.host); + if (outcome?.host === 'saved') { + saved.push(SETTINGS_STATUS.host); + const restart = (outcome.restart ?? []).map(field => RESTART_LABELS[field]).filter(Boolean); + if (restart.length) saved.push(`${SETTINGS_STATUS.restart} ${restart.join(', ')}.`); + } if (outcome?.runtime === 'applied') saved.push(SETTINGS_STATUS.runtime); return saved.join(' ') || SETTINGS_STATUS.clean; } @@ -68,6 +84,8 @@ export function createSystemDrafts(adapters) { let hostRead = 'pending'; let outcome = null; let saving = null; + // What the last host save said beyond its result: the restart fields, or the source it refused. + let hostDetail = {}; const notify = () => { for (const listener of [...listeners]) listener(); }; const changes = domain => Object.fromEntries(drafts[domain]); @@ -150,6 +168,8 @@ export function createSystemDrafts(adapters) { if (result?.code === CONFLICT_CODE) { fail(domain, written); return 'conflict'; } if (!result?.success || result.persisted !== true) { fail(domain, written); return 'failed'; } if (typeof result.revision === 'string') revision = result.revision; + const restart = result.restart_required_fields; + if (Array.isArray(restart)) hostDetail.restart = restart.filter(field => typeof field === 'string'); commit(domain, written); return 'saved'; } @@ -158,6 +178,9 @@ export function createSystemDrafts(adapters) { return 'applied'; } catch (error) { fail(domain, written); + if (domain === 'host' && error?.code === INVALID_SOURCE_CODE && typeof error.message === 'string') { + hostDetail.invalid = { source: typeof error.source === 'string' ? error.source : null, message: error.message }; + } return error?.code === CONFLICT_CODE ? 'conflict' : 'failed'; } } @@ -172,8 +195,10 @@ export function createSystemDrafts(adapters) { function save() { saving ??= (async () => { for (const domain of DRAFT_DOMAINS) failedKeys[domain].clear(); + hostDetail = {}; const result = {}; for (const domain of DRAFT_DOMAINS) result[domain] = await saveDomain(domain); + Object.assign(result, hostDetail); // Settings changed elsewhere: read them (and their revision), keeping the drafts (Q10). if (result.host === 'conflict') await reloadHost(); if (result.browser === 'conflict') await load('browser').catch(() => {}); @@ -228,6 +253,8 @@ export function createSystemDrafts(adapters) { changes, saving: () => saving !== null, failed: domain => { requireDomain(domain); return new Set(failedKeys[domain]); }, + /** The source the last save was refused for (its public path), until the next edit; else null. */ + invalidSource: () => outcome?.invalid?.source ?? null, /** The Save row's text: the last save's outcome until the next edit, else unsaved or clean. */ statusText: () => (outcome ? statusText(outcome) : dirty() ? SETTINGS_STATUS.dirty : SETTINGS_STATUS.clean), subscribe(listener) { diff --git a/web/js/workspace/system.mjs b/web/js/workspace/system.mjs index 27e1a54..1b4b870 100644 --- a/web/js/workspace/system.mjs +++ b/web/js/workspace/system.mjs @@ -4,8 +4,8 @@ * `aria-pressed`; a press navigates to its hash), then `.settings-layout` with the section's body and the Save row * (`span.small#settings-status` + "Save preferences"). D92: no "Sample configuration" chip, the status copy is * truthful and "Save preferences" is aria-disabled while nothing is unsaved. A section not delivered yet shows the - * per-section hand-off to its legacy tool (D94), never mock placeholders. Overview (system-overview.mjs) and - * Appearance (system-appearance.mjs) are native. + * per-section hand-off to its legacy tool (D94), never mock placeholders. Overview (system-overview.mjs), + * Appearance (system-appearance.mjs) and Sources & scans (system-sources.mjs) are native. * * Switching sections keeps the shell (and focus on the pressed button) and disposes the old section's scope, so its * listeners and timers never outlive it. The drafts are module-level (Q10): they live for the page session. @@ -16,6 +16,9 @@ import { createSystemDrafts, DRAFT_DOMAINS } from './system-state.mjs'; import { sectionNav } from './system-controls.mjs'; import { healthNotice, mountOverview } from './system-overview.mjs'; import { mountAppearance } from './system-appearance.mjs'; +import { mountSources } from './system-sources.mjs'; +import { busyToast } from './activity-scan.mjs'; +import { showWorkspaceToast } from './toast.mjs'; import { displayPreferences, readPreferences, reconcilePreferences, subscribePreferences, writePreferences } from './preferences.mjs'; const SETTINGS_URL = '/prompt_manager/settings'; @@ -27,7 +30,6 @@ const GALLERY = Object.freeze({ href: '/prompt_manager/gallery', label: 'Open ex * settings, thumbnails, duplicates and cache tools. */ export const SECTION_HANDOFFS = Object.freeze({ - sources: [ADMIN, GALLERY], integrations: [ADMIN], tagging: [ADMIN], maintenance: [ADMIN, GALLERY], @@ -37,6 +39,15 @@ export const SECTION_HANDOFFS = Object.freeze({ export const NOT_DELIVERED = 'This setting is not available here yet.'; let drafts = null; +// The Activity store of the System page shown last: a refused host save that does not name its job reads the list. +let jobs = null; + +/** The running job a 409 `busy` named, else the first running one from a fresh list read (the load never throws). */ +async function runningJob(error) { + if (error?.running) return error.running; + await jobs.load(); + return jobs.running()[0]; +} function el(tag, text, className) { const node = document.createElement(tag); @@ -71,7 +82,15 @@ export function systemDrafts(api) { // The store reads a missing value as none and a missing revision as unreadable (no host save). return { values: result.settings, revision: result.revision }; }, - writeHost: (changes, { expected_revision }) => api.post(SETTINGS_URL, { ...changes, expected_revision }), + async writeHost(changes, { expected_revision }) { + try { + return await api.post(SETTINGS_URL, { ...changes, expected_revision }); + } catch (error) { + // A sources change waits for a running record-changing job (S4): name it, as a refused start does (D57). + if (error?.code === 'busy') showWorkspaceToast(busyToast(await runningJob(error))); + throw error; + } + }, }); return drafts; } @@ -100,12 +119,13 @@ function markFailed(layout, store) { } /** - * The sections delivered so far: Overview (S2 Task 6), Appearance (S3 Task 8), and Maintenance's notice with "Run health checks" above the + * The sections delivered so far: Overview (S2 Task 6), Appearance (S3 Task 8), Sources & scans (S4 Task 11), and Maintenance's notice with "Run health checks" above the * hand-off to the maintenance tools that are not native yet (D123). */ export const DELIVERED_SECTIONS = Object.freeze({ overview: mountOverview, appearance: mountAppearance, + sources: mountSources, maintenance: (container, { api, activity, scope }) => [healthNotice({ api, activity, scope }), ...handoff('maintenance')], }); @@ -123,6 +143,7 @@ export const DELIVERED_SECTIONS = Object.freeze({ */ export function mountSystem(main, api, scope, { section, activity, sections = DELIVERED_SECTIONS, navigate = navigateTo }) { const store = systemDrafts(api); + jobs = activity; store.reloadHost(); // Another tab or the legacy gallery changed the display defaults: clean values follow it; a draft it overtook stays // and is flagged (D92's conflict status). `next` is null when the values cannot be read (no gallery-settings.js). diff --git a/web/js/workspace/workspace.css b/web/js/workspace/workspace.css index ace96b0..2de1f88 100644 --- a/web/js/workspace/workspace.css +++ b/web/js/workspace/workspace.css @@ -89,6 +89,8 @@ main:focus-visible { outline: 2px solid var(--ca-focus); outline-offset: 4px; } not save is highlighted, as its "Review the highlighted settings" status says. */ .save-row button[aria-disabled="true"] { opacity: .55; cursor: not-allowed; /* D92 */ } .setting-row [aria-invalid="true"] { outline: 2px solid var(--red); outline-offset: 2px; /* D92 */ } +/* D92: a source the last save was refused for (System › Sources) is highlighted the same way. */ +#source-list > .operation[aria-invalid="true"] { outline: 2px solid var(--red); outline-offset: 2px; /* D92 */ } /* D93: an Overview row whose action is not available yet (the review queue, Q6) keeps its button, disabled. */ .operation .row-actions button[aria-disabled="true"] { opacity: .55; cursor: not-allowed; /* D93 */ } /* D97: Appearance's legacy display rows. "Show image details on cards" off keeps a card's Info and Tag at the end of From 3f11ea6168243e539d1d8b172e37e1c76075429b Mon Sep 17 00:00:00 2001 From: Vito <vito.sansevero@gmail.com> Date: Fri, 23 Oct 2026 14:19:11 -0700 Subject: [PATCH 2/2] feat(annotations): Needs review queue backend (#284) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(review): add review queue schema with restore validation Add the Needs review schema group (media_review_metadata, media_review_items, media_review_decisions) beside the image annotation group it cites. review_schema_error validates the catalog, columns, foreign keys, row values and canonical 8 MiB evidence read-only; migrate_review_state installs it in a SAVEPOINT after the annotation group, and verify_database_file requires it under require_library_state. A review group without a valid annotation group is refused. Decisions null their pair and image navigation through indexes when those are deleted. Old backups and annotation-era backups restore and gain the group; browser fixtures clear the new tables on reset. * feat(review): persist ambiguous sampler evidence from scans and new outputs The output scan and the live monitor now queue every file the ingestion resolver finds ambiguous for Needs review. The file is captured, its metadata is read again from the capture and resolved with the default profile, and only an ambiguous result is stored, bound to the capture's annotation byte record and fenced by its storage epoch. Evidence holds one hashed projection per selectable candidate; none, more than 16 or over 8 MiB is a limited item. Recording is idempotent per bytes and evidence and never touches a decision. Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the job scope gains review_items/review_errors and the log a Needs review line. The monitor records after its fallback link, best effort. * feat(review): explicit association decisions with compatible linking Needs review gains its queue reads and decisions: - GET /prompt_manager/review/items lists pending items oldest first with the exact pending total and an HMAC keyset cursor bound to the storage epoch and the review index revision. - GET /prompt_manager/review/items/{id} adds the candidates (role texts, label, selectability), the dependency fingerprint and the digest. - POST /prompt_manager/review/items/{id}/decision applies a candidate only for a capture of the item's bytes that the current file still hashes to and whose evidence is unchanged, through the ingester's compatible-only rules in the same transaction as the decision row; a conflict writes nothing. Leave unresolved records the decision and links nothing. ingest_resolved_pair keeps its contract; its body moves into _ingest_resolved_pair_in, which runs in the caller's transaction. Items keep the file's recorded spelling so targets keep their case on Windows, and a rescan skips the copy and hash of a file whose size and mtime match the bytes an item already cites. * fix(review): build the spelling validation test from native paths On Windows the locator is case-folded and uses backslashes, so the fixed POSIX pair /out/a.png no longer folded to itself and the new spelling check refused the good case. The test now spells a mixed-case native path and folds it with os.path.normcase, and also refuses a locator of another file. * fix(review): Apply sees the file under every spelling and rechecks only its choice Apply now finds the legacy rows of the same physical file however its root is reached. The roots are real paths while the monitor links under the directory it watches, so a symlinked or junctioned root, subst or a mapped drive used to hide the monitor's row and Apply added a second, differently linked row. Same-name rows are resolved to their physical identity (normcase(realpath), else samefile); the dependency fingerprint covers all of them, two rows refuse the choice, one row is enriched under the ingester's compatible-only rules, and a same-name row the request did not classify is a changed association. Apply re-verifies only what the decision depends on: the bytes are still ambiguous and still offer the chosen candidate. A resolver change that rewords the reason or reorders candidates leaves pending items applicable. The resolver version is one constant in prompt_pair_resolver, shared by ingestion provenance and Needs review. The transaction's byte check receives the re-hashed digest (or the body's for Leave unresolved), every Apply refusal branch is tested with zero writes, and the root listing and realpath calls run on a worker thread. * fix(review): validate stored review evidence once per startup Startup used to run the full review validation twice (the early guard and the install) and the annotation group's row checks inside each, so every evidence document was parsed and re-encoded twice. The early guard is now catalog-only (a review group without annotation tables), and the install validates the group once, trusting the annotation group it has just validated. The evidence predicate no longer pattern-scans every document in SQL, because the streamed canonical pass proves it, and the codec is built once. Measured on a database with 40,000 items of 4 KiB evidence: startup open went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen. Restore and verify_database_file keep the full validation. * fix(review): keep one capture worker for interactive requests Needs review recording from the scan and the monitor shared the capture store's two workers with the Tag dialog and Apply, so a busy scan could answer a user's preview or decision with "Too many previews are open". Background captures now never take the last worker or capture slot, and wait up to two seconds for room before counting the file as not added. Interactive captures are admitted or refused at once, as before. * fix(review): refresh the recorded spelling when a file is seen under a new one A case-only rename on Windows or macOS keeps size and mtime_ns, so the rescan skipped the file and the item kept the old spelling. Its target was then formatted from a name the directory no longer lists, the capture refused it, and the item stayed pending with no way to apply it. The skip path and a duplicate insert now record the current spelling for every item at the locator and move the review index revision; item revisions do not change, so an open decision stays valid. Tested under case-folding semantics on every platform. * fix(review): final capture store close and recoverable stale cursors Closing the capture store at shutdown now runs under the store lock and is final: a monitor recording or scan batch still in flight gets a capture error instead of creating a new temp directory and restore subscription after the cleanup. A well-formed queue cursor this process cannot verify (opened before a ComfyUI restart, whose cursor secret is per process) now answers 409 review-conflict with the current total, like any other stale page, instead of an uncoded 400. Malformed cursors stay 400. * fix(review): make only the shutdown cleanup final The previous commit made every close of the capture store final, which also refused the explicit closes that swap in a fresh store. Only the host's on_cleanup hook is final now; an observe after it answers a coded capture error instead of an unhandled 500. * refactor(admin): move Needs review recording into its own mixin _record_scanned_review, _record_review_sync, record_review_file and _log_review_items move unchanged from py/api/admin.py to py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from growing further past the size ceiling. No behaviour change. * fix(review): record the review fix round and the skip's residual risks in TESTING.md The Needs review paragraphs now state the rescan skip's residual risks (an in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted or moved file leaves its pending item counted), the physical identity rule for Apply, the candidate-only recheck and shared resolver version, the spelling refresh, background capture admission, the final shutdown close, the stale-cursor answer and the measured startup validation cost. * fix(review): refuse monitor recording beside record-changing jobs System S1 (#289) gives every record change a registry admission class. The scan already records Needs review items under its own records job. The monitor's recording lasts one file and is not a job, so it now checks the records admission: while a records or guarded job or hold runs it records nothing, logs that the file waits for the next scan, and answers deferred. The fallback link stands and the next "new" scan reads the ambiguous file again, because a fallback link is not a resolved one. JobRegistry.blocker and active iterate a snapshot of the jobs, since the monitor's worker thread now asks them too. * test(backup): wait for the restore, not a 5 s timer, in the thread test test_database_is_usable_from_every_thread_after_restore let its worker wait 5 s for the restore. On a stalled Windows runner (py3.11, where the previous restore test also took 25 s instead of 0.3 s) the restore took longer, the wait timed out, and the worker read the old database before the restore ran. The worker now waits for the event under a 120 s bound that only a hung test reaches, reports a timeout as an error, and the main thread releases it even when the restore raises. With a restore delayed by 6 s the old test fails and the new one passes. * fix(review): store non-finite evidence as null and count pending items canonical_evidence now follows main's non-finite policy (#303): NaN and ±Infinity are written as null instead of raising, so a NaN-stamped graph never turns a review item limited. The System Overview reports the review queue's pending total in needs_review.pending, read in the same snapshot as the Library counts. * fix(review): show the pending count on the System Overview review row The Overview's review row now reads "N ambiguous image links" with the concept's line, from needs_review.pending. Its Review metadata action stays aria-disabled until the Needs review page ships. The system specs and their concept comparison follow the real count. * fix(system): answer the Overview and Statistics only after every read ended asyncio.gather answered at the first failed read while its sibling executor reads were still inside SQLite. In the suite the test teardown then deleted the database under the running read, and py3.10 on Ubuntu segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope). Both routes now wait for all their reads before answering. * fix(review): queue only ambiguous files that nothing links, and heal the queue The maintainer's live scan (19,908 files, scope "new") reported 4,677 files under Needs review, but the queue held 10,232 items: every file the strict resolver found ambiguous was queued, including the ones the legacy browser-parser fallback then linked (all sampled "added" rows) and the ones already linked to a known prompt (135 of 200 sampled "known" rows). A file is now queued only when its final outcome is ambiguous and unlinked: the resolver finds it ambiguous, the fallback links nothing and no saved row links it already. An ambiguous file that a saved row links is reported as unchanged ("Already linked to its prompt"), not as Needs review. The queue heals itself. After each batch, the scan removes the pending items of every listed file (read, or skipped as known) that a saved row now links. The monitor does the same for each ambiguous output it links, and never queues one. Only an undecided item is removed (pending, revision 0, no decision row), so a decision is never deleted or changed and the schema needs no new state. The scan job's scope.review_items now counts this scan's files that are in the pending queue after it, new or already there, so it agrees with the review group's ambiguous rows and with the queue total. review_added and review_settled are new, and the log gains "already in Needs review" and "left Needs review · now linked" lines. A duplicate whose item is decided is reported as "decided" and is not counted. * fix(review): never queue an ambiguous file with nothing to choose The maintainer's live queue held 39 items whose candidates all read "unresolved" ("No positive / No negative"). An ambiguous result is now reviewable only when at least two candidates are resolved with non-blank positive text. With none the file could not be traced; with exactly one, the untraced branches may still have made the file, so that text is not certain either. Both are reported unresolved, never linked and never queued (also the A1111 two-delimiter case, which has no candidates). A rescan removes the pending items of such files whose own evidence offers no choice, with the same safe delete as the linked settle (pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two candidates is kept. The job scope gains review_untraced and the log "= N images left Needs review · nothing to choose". * test(scan): pin only the per-source scope keys beside the review counters * test(system): match the scan scope keys beside the review counters --- TESTING.md | 170 ++- __init__.py | 15 + database/library.py | 7 +- database/models.py | 23 +- database/operations.py | 4 + database/pair_ingestion.py | 238 ++-- database/review.py | 541 ++++++++ database/review_decisions.py | 287 ++++ database/review_schema.py | 507 +++++++ py/api/__init__.py | 9 + py/api/admin.py | 97 +- py/api/annotations.py | 48 +- py/api/jobs.py | 7 +- py/api/review.py | 689 ++++++++++ py/api/review_recording.py | 241 ++++ py/api/system.py | 9 +- tests/browser/server.py | 4 + tests/browser/system-concept.spec.mjs | 9 +- .../browser/workspace-system-sources.spec.mjs | 2 +- tests/browser/workspace-system.spec.mjs | 11 +- tests/browser/workspace_concept_fixture.py | 2 + .../workspace_library_editor_fixture.py | 2 + tests/browser/workspace_tagging_fixture.py | 2 + tests/browser/workspace_visual_fixture.py | 2 + tests/browser/workspace_vocabulary_fixture.py | 2 + tests/test_annotation_schema.py | 13 +- tests/test_api_admin.py | 13 +- tests/test_api_annotations.py | 10 +- tests/test_api_core.py | 13 + tests/test_api_review.py | 1161 +++++++++++++++++ tests/test_database_backup.py | 67 + tests/test_library.py | 1 + tests/test_media_capture.py | 88 ++ tests/test_pair_ingestion.py | 66 + tests/test_pair_scan.py | 6 +- tests/test_pair_schema_restore.py | 4 + tests/test_review.py | 1053 +++++++++++++++ tests/test_review_choosable.py | 335 +++++ tests/test_review_outcome.py | 374 ++++++ tests/test_review_schema.py | 918 +++++++++++++ tests/test_scan_scope.py | 13 +- tests/test_system_api.py | 2 +- utils/image_monitor.py | 40 +- utils/media_capture.py | 62 +- utils/prompt_pair_resolver.py | 6 + utils/review_capture.py | 299 +++++ web/js/workspace/system-overview.mjs | 2 +- 47 files changed, 7291 insertions(+), 183 deletions(-) create mode 100644 database/review.py create mode 100644 database/review_decisions.py create mode 100644 database/review_schema.py create mode 100644 py/api/review.py create mode 100644 py/api/review_recording.py create mode 100644 tests/test_api_review.py create mode 100644 tests/test_review.py create mode 100644 tests/test_review_choosable.py create mode 100644 tests/test_review_outcome.py create mode 100644 tests/test_review_schema.py create mode 100644 utils/review_capture.py diff --git a/TESTING.md b/TESTING.md index 12adaec..9117fe4 100644 --- a/TESTING.md +++ b/TESTING.md @@ -2019,8 +2019,10 @@ Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121 and active saved negatives and the indexed `generated_images` rows (Q2). "Database integrity" is "Not checked" until a check ran this session, then "N issues" (checks that found issues, even beside a check that could not run), "Passed" or "Incomplete". - "Ambiguous image links" reads "Not available yet." with its Review metadata action - `aria-disabled` (CSS under D93) until the Needs review queue reports a count (Q6). + The review row is "N ambiguous image links" (the Needs review queue's pending count, + `needs_review.pending`) with the concept's line; its Review metadata action is + `aria-disabled` (CSS under D93) until the Needs review page ships (#288). A count + that is not an integer reads "Ambiguous image links" / "Not available yet." (Q6). "Backup status" reads "No retained backup record." until verified backups exist (S8). The thumbnail row is "Checking thumbnails…" until its own read answers, then "No missing thumbnails", "N missing thumbnails" (the pending count of @@ -2383,6 +2385,162 @@ inspection's own stable stat (`tests/test_api_workspace_images.py` browser fixtures clear `media_image_tags`, `media_annotation_bytes` and `media_annotation_files` on reset and never the metadata row. +Review queue schema (#249, `database/review_schema.py`, +`tests/test_review_schema.py`): the group `media_review_metadata` (one row: +schema version, `review_index_revision`), `media_review_items` (one ambiguous +resolver result per `(byte_id, resolver_version, evidence_hash)`, citing an +annotation byte record `ON DELETE RESTRICT`, with the file's recorded +`spelled_path` (the locator is case-folded on Windows, a listing target is +not), 0..16 candidates, `limited`, +`state` pending/selected/unresolved and a revision) and `media_review_decisions` +(one row per item decision revision; `selected` exactly when a +`candidate_hash` is present; `pair_id`/`image_id` navigation is set to NULL, +never cascaded, when the pair or image is deleted, through +`idx_media_review_decisions_pair`/`_image`). Evidence is canonical JSON +(`canonical_evidence`: one object, sorted keys, no whitespace, ASCII only, no +NaN/infinity, at most 8 MiB) and `evidence_hash` is its SHA-256. The group is +valid only beside a valid annotation group (a review group without annotation +tables is refused, catalog only, before the annotation install), installs like +the annotation group (optional in old backups, required once current under +`require_library_state`, SAVEPOINT install validated before and after), and +validation re-canonicalizes every evidence document in one streamed pass. A +startup runs that pass once (`test_startup_validates_stored_evidence_once`; the +install trusts the annotation group it has just validated, and the SQL row +predicates check type and size only, because the streamed pass proves the +rest): 2.5 s to open a database of 40,000 items with 4 KiB of evidence each, +was 7.7 s with two passes. Restore and `verify_database_file` keep the full +validation, annotation group included. The browser fixtures clear +`media_review_decisions` and `media_review_items` before the annotation tables, +never the metadata row. + +Needs review items (#249, `utils/review_capture.py`, `database/review.py`, +`tests/test_review.py`, `tests/test_review_outcome.py`, +`tests/test_review_choosable.py`): a file is queued only when its final outcome +is ambiguous and unlinked and there is something to choose. Only scans produce +items; the monitor links new outputs and clears pending items it resolves +(maintainer rulings, 2026-10-23). The output scan (Activity's +Scan for metadata and the legacy `/scan`) hands `record_ambiguous_file` a file +the ingestion resolver finds `ambiguous` only when the browser-parser fallback +then links nothing (status `ambiguous`) and no saved row links the file +already; an ambiguous file a saved row links is reported `unchanged` ("Already +linked to its prompt"), not Needs review. A file the fallback links, an exact +pair the ingester stores, and `conflict` and `unresolved` files are not +queued. Neither is an ambiguous file with nothing to choose: fewer than two of +its candidates are `resolved` with non-blank positive text (`reviewable`). With +none the file could not be traced; with exactly one, the untraced branches may +still have made the file, so that text is not certain either. Both report +`unresolved` ("could not be traced"), are never linked, and their pending, +undecided items whose own evidence offers no choice (a limited item counts as +a choice when it holds at least two candidates) leave the queue on the next +scan or monitor pass (`settle_untraced_review_items`). The maintainer's +second live check found 39 items whose 11 candidates were all "unresolved". The queue heals itself: after each batch, the scan removes the pending +items of every listed file (read or skipped as known) that a saved row now +links, and the monitor does the same for each ambiguous output it links +(`settle_linked_review_items`; the review index revision moves). Only an +undecided item goes (`pending`, revision 0, no decision row): a decision is +never deleted or changed, so no schema state was added. The monitor never +queues an output it linked; an unlinked file passed to `record_review_file` +follows the scan's rule (a file the fallback would link is `skipped`). The +maintainer's live check of #284/#288 found the queue at 10,232 against a review +group of 4,677: every fallback-linked ambiguous file had been queued. It captures +the file (scope `needs-review`), reads its metadata again from the capture, +resolves it with the default profile and stores only an ambiguous result, +bound to the capture's byte record (created or refreshed like an observe) and +fenced by the capture's storage epoch. Evidence is the resolver reason plus one +`{sampler_id, state, reason, positive, negative, hash}` per selectable +candidate (nested ambiguous stages flattened, duplicates dropped; each side +`{text, storage_text, provenance}`; `hash` = SHA-256 of the canonical candidate +without it). More than 16 candidates, or evidence over 8 MiB, is a `limited` +item without candidates. Recording is idempotent per `(byte_id, +resolver_version, evidence_hash)` and never touches an existing item or its +decisions; changed bytes add a new pending item and keep the old one. The +capture is released before the write; no model is loaded and no pixel is +decoded. Scan statuses, counters, outcome rows and the legacy SSE events are +unchanged apart from the already-linked and nothing-to-choose cases above; the job's `scope` gains +`review_items` (this scan's files that are in the pending queue after it, new +or already there: the review group's ambiguous rows, less files a capture could +not read or a decision already settled), `review_added` (new items), +`review_settled` (items removed because their file is linked now), +`review_untraced` (items removed because they offer nothing to choose) and +`review_errors`. The log ends with `? N images added to Needs review`, +`? N images already in Needs review`, `= N images left Needs review · now +linked`, `= N images left Needs review · nothing to choose` and `! N images could not be added to Needs review` (failures, e.g. +capture capacity), each only when N > 0. Recording captures in the background: it never takes the store's +last worker or capture slot, which stay free for an interactive Tag dialog or +Apply, and waits up to 2 s for room before counting the file as not added +(`test_background_recording_leaves_a_worker_for_the_tag_dialog`). The monitor's +pass runs after its own link, best effort: a failure is logged and the link +stays. Recording follows the registry's `records` +admission (#289): the scan's recording runs under its own `records` job, and +the monitor's (one file, not a job) is refused while a `records` or `guarded` +job or hold runs. It then records nothing, logs that the file waits for the +next scan, and answers `deferred`; the next "new" scan reads the ambiguous file +again (`test_monitor_recording_is_refused_beside_record_changing_jobs`). A new item moves `review_index_revision`. A +rescan does not copy or hash an ambiguous file whose current size and +`mtime_ns` equal the verified fingerprint of bytes an item already cites (a +stat compare); the file still stays in the default "new" scope, because it is +not a known image. The skip and a duplicate insert record the current spelling +of the locator on its items (`refresh_review_spelling`, the review index +revision moves, item revisions do not), so a case-only rename on Windows or +macOS keeps its item applicable. Residual risks of the skip, like S1's: an +in-place rewrite that keeps both size and `mtime_ns` is never re-queued (Apply +still re-hashes, so it cannot link those bytes), and a file that is deleted or +moved leaves its pending item counted in `total` (its target then formats as +`unavailable` or fails to open, and Leave unresolved clears it). + +Needs review decisions and queue reads (#249, `py/api/review.py`, +`database/review.py`, `database/review_decisions.py`, +`tests/test_api_review.py`): `GET /prompt_manager/review/items?limit=1..40&cursor=` +answers `{success, library_epoch, total, items, next_cursor}` with pending items +oldest first; `total` is the exact pending count. The cursor is an HMAC keyset +on `(created_at, id)` bound to the storage epoch and `review_index_revision`: a +malformed one is 400, a well-formed one this process cannot verify (tampered, +or opened before a ComfyUI restart: the secret is per process) is 409 +`review-conflict` with the current `total`, one from before a restore is 409 +`storage-changed`, one from before any queue change (a decision, a new item, a +refreshed spelling) is 409 `review-conflict`. +Targets are formatted from `spelled_path` against the current roots (the +roots' real paths are the only file access); a file under no current root is +`unavailable` with a null target. Every read and decision lists the roots and +resolves real paths on a worker thread, never on the event loop. `GET .../items/{id}` adds `candidates` +(`hash`, `sampler_id`, `state` resolved/unresolved, `reason`, `label` "<positive +title> / <negative title>" from the saved prompt's Library title else the +first line, and the role texts), `dependency_fingerprint` (SHA-256 of the +file's legacy rows under every spelling of the same physical file, positive +hashes and pair links with their negative hashes, as the ingester reads them) and `sha256`; a detail over 8 MiB, or a +limited item, has no candidates. `POST .../items/{id}/decision +{expected_epoch, expected_revision, capture_token, sha256, dependency_fingerprint, +choice}` with a candidate hash (Apply) requires a capture of the item's file +whose bytes are the item's, re-hashes the current file (`file-changed` +otherwise, also for a fresh capture of new bytes), re-reads the capture and +requires that its bytes are still ambiguous and still offer the chosen +candidate (`review-conflict` otherwise; a resolver change that rewords the +reason or reorders candidates leaves the item applicable, and +`RESOLVER_VERSION` in `utils/prompt_pair_resolver.py` is the one version +ingestion provenance and review share), then in one `_pair_transaction` +checks the epoch, the item revision and state (`review-conflict` with the +current item for a repeat), the re-hashed digest, the dependency fingerprint +and the public ingester's compatible-only rules through +`_ingest_resolved_pair_in` (any refusal is 409 `association-conflict` with +zero writes, explained inline; Apply never reparents a link). Physical +identity: the roots are real paths while the monitor links under the +directory it watches, so a symlinked or junctioned root, `subst` or a mapped +drive spells the same file otherwise. Every legacy row whose spelling ends with +the file's name (the link summaries' unindexed suffix lookup) is resolved by +`normcase(realpath)` against the locator, else `samefile`; two rows of the +file refuse the choice, one is enriched under the spelling it was linked +with, and a same-name row the request did not classify is a changed +association (`TestPhysicalIdentity`, symlinked root and a portable fake +junction), and writes +the link with `kind:"user-selection"` provenance (decision id, reviewed digest, +candidate hash, resolver version, role provenance), the decision row, the +item's new state and revision and `review_index_revision`. `"unresolved"` +records the decision only and accepts `capture_token: null`. A successful +decision releases its capture. No decision changes run or usage counts or +calls an execution writer; a Tag save never changes a review item. Errors carry +fixed sentences: `review-failed` / `decision-failed` 500s never echo exception +text or paths. + Capture store (#249, `utils/media_capture.py`, `tests/test_media_capture.py`): one explicit action captures one rooted media file into an owner-only `prompt-manager-captures-*` temp directory, streaming it in 1 MiB chunks while @@ -2399,8 +2557,12 @@ directories are `CaptureDenied`; the locator is `normcase(join(realpath(root), relative))`, so a symlinked root shares the listing's identity. Fixed limits: 256 MiB per file (stat and stream), 8 completed captures, 512 MiB reserved, 2 capture/verify workers, 900 s idle, -1800 s hard; admission is nonblocking (`CaptureCapacity`) and lifetimes are -tested with an injected clock. An acquired capture outlives expiry and release +1800 s hard; interactive admission is nonblocking (`CaptureCapacity`), a +`background=True` capture (Needs review recording) never takes the last worker +or capture slot and waits up to `BACKGROUND_WAIT_SECONDS` (2 s) for room, and +lifetimes are tested with an injected clock. The host's on_cleanup close is +final: a recording still in flight gets a capture error instead of creating a +new temp directory after it. An acquired capture outlives expiry and release until its last use ends. Header-only dimensions (4096 px per side, 16,777,216 px) and video mark Generate unsupported; `verify` re-hashes the current file without a copy. Every handle closes before unlink (a patched diff --git a/__init__.py b/__init__.py index 140f7ea..6a67386 100644 --- a/__init__.py +++ b/__init__.py @@ -122,6 +122,21 @@ try: _global_prompt_tracker = get_prompt_tracker(_global_db) _global_image_monitor = get_image_monitor(_global_db, _global_prompt_tracker) + # The Needs review pass for ambiguous new outputs runs through the API's + # capture store (the one the scan and the Tag dialog use): a linked output + # is never queued and its pending items leave the queue. It stops before + # that store is closed at shutdown. + _review_api = globals().get("api") + if _review_api is not None: + _global_image_monitor.set_review_recorder(_review_api.record_review_file) + _review_host = getattr(config.server_instance, "app", None) + if _review_host is not None and hasattr(_review_host, "on_cleanup"): + + async def _stop_review_recording(app, monitor=_global_image_monitor): + monitor.set_review_recorder(None) + + _review_host.on_cleanup.insert(0, _stop_review_recording) + # Start monitoring _global_image_monitor.start_monitoring() _init_logger.info("Global image monitoring system started successfully") diff --git a/database/library.py b/database/library.py index 7393e6d..d91677b 100644 --- a/database/library.py +++ b/database/library.py @@ -637,7 +637,8 @@ class LibraryOperations: pairs (every pair's positive is eligible). ``images`` counts indexed ``generated_images`` rows, not files on disk. An image has no rating of its own (it shows its linked prompt's), so ``rated_images`` counts - the indexed images whose linked prompt is rated. + the indexed images whose linked prompt is rated. ``review_pending`` is + the Needs review queue's pending total. """ with self.library_snapshot() as (conn, _epoch, _ceiling): row = conn.execute( @@ -645,13 +646,15 @@ class LibraryOperations: "(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0)," "(SELECT COUNT(*) FROM generated_images)," "(SELECT COUNT(*) FROM generated_images gi JOIN prompts r " - "ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL)" + "ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL)," + "(SELECT COUNT(*) FROM media_review_items WHERE state='pending')" ).fetchone() return { "positive_prompts": row[0], "negative_variants": row[1], "images": row[2], "rated_images": row[3], + "review_pending": row[4], } def usage_by_category(self) -> list[dict]: diff --git a/database/models.py b/database/models.py index 5268577..3724465 100644 --- a/database/models.py +++ b/database/models.py @@ -18,6 +18,11 @@ from typing import Optional, Set, Tuple from .library_schema import library_schema_error, migrate_library_state from .image_favorites import image_favorites_schema_error, migrate_image_favorites from .annotation_schema import annotation_schema_error, migrate_annotation_state +from .review_schema import ( + migrate_review_state, + review_schema_error, + review_without_annotations_error, +) from .saved_views import migrate_saved_views, saved_views_schema_error from .execution_history_schema import ( history_schema_error, @@ -303,7 +308,14 @@ class PromptModel: migrate_library_state(conn) migrate_image_favorites(conn) migrate_saved_views(conn) + # A review group without its annotation group is refused before the + # annotation install could make it look valid (catalog only; the rows + # are validated once, in migrate_review_state). + review_error = review_without_annotations_error(conn) + if review_error: + raise sqlite3.DatabaseError(review_error) migrate_annotation_state(conn) + migrate_review_state(conn) def _migrate_prompt_pairs(self, conn: sqlite3.Connection) -> None: conn.execute("SAVEPOINT prompt_pair_schema") @@ -1023,9 +1035,9 @@ class PromptModel: Args: path: File to inspect require_library_state: Also require the tables every current - database has (Library state, image favorites, saved views - and image annotations), as a migrated restore stage must; - old backups may lack them. + database has (Library state, image favorites, saved views, + image annotations and the review queue), as a migrated + restore stage must; old backups may lack them. Returns: (ok, reason): reason is "ok" when the file is acceptable, otherwise @@ -1095,6 +1107,11 @@ class PromptModel: ) if annotation_error: return False, annotation_error + review_error = review_schema_error( + conn, allow_missing=not require_library_state + ) + if review_error: + return False, review_error return True, "ok" except sqlite3.Error as e: return False, f"Invalid SQLite database: {e}" diff --git a/database/operations.py b/database/operations.py index 024dc86..cf9cec1 100644 --- a/database/operations.py +++ b/database/operations.py @@ -14,6 +14,8 @@ from .pairs import PairDependencyError, PairOperations from .library import LibraryOperations, titled from .image_favorites import ImageFavoriteOperations from .annotations import AnnotationOperations +from .review import ReviewOperations +from .review_decisions import ReviewDecisionOperations from .saved_views import SavedViewOperations from .library_commands import LibraryCommandOperations from .vocabulary import VocabularyOperations @@ -345,6 +347,8 @@ class PromptDatabase( HistoryReadOperations, ImageFavoriteOperations, AnnotationOperations, + ReviewOperations, + ReviewDecisionOperations, SavedViewOperations, HealthOperations, ): diff --git a/database/pair_ingestion.py b/database/pair_ingestion.py index 2778198..23c7044 100644 --- a/database/pair_ingestion.py +++ b/database/pair_ingestion.py @@ -9,9 +9,11 @@ from .operations import _coerce_file_info try: from ..utils.hashing import generate_prompt_hash from ..utils.json_strict import dumps_strict + from ..utils.prompt_pair_resolver import RESOLVER_VERSION except ImportError: from utils.hashing import generate_prompt_hash from utils.json_strict import dumps_strict + from utils.prompt_pair_resolver import RESOLVER_VERSION def ingest_resolved_pair( @@ -20,117 +22,143 @@ def ingest_resolved_pair( state = resolution.get("state", "unresolved") if state != "resolved": return {"status": state, "pairs_added": 0, "links_added": 0} + with db._pair_transaction() as conn: + return _ingest_resolved_pair_in(conn, db, image_path, metadata, resolution) + + +def _conflict() -> Dict[str, Any]: + return {"status": "conflict", "pairs_added": 0, "links_added": 0} + + +def _ingest_resolved_pair_in( + conn: Any, + db: Any, + image_path: str, + metadata: Dict[str, Any], + resolution: Dict[str, Any], + *, + provenance: Dict[str, Any] | None = None, +) -> Dict[str, Any]: + """``ingest_resolved_pair`` inside the caller's open ``_pair_transaction``. + + The compatible-only rules are the public ingester's: an image with more + than one exact row, another positive, or a prior pair with another + negative is a ``conflict`` and nothing is written. The caller owns the + transaction, so its own later failure rolls the link back too. + ``provenance`` replaces the link's default resolver provenance (Needs + review records the user's selection there). + """ positive, negative = resolution["positive"], resolution["negative"] positive_text = positive.get("storage_text", positive["text"]) negative_text = negative.get("storage_text", negative["text"]) normalized = normalize_image_path(image_path) - provenance = {"resolver_version": 1, "positive": positive, "negative": negative} + if provenance is None: + provenance = { + "resolver_version": RESOLVER_VERSION, + "positive": positive, + "negative": negative, + } encoded = dumps_strict(provenance, sort_keys=True, separators=(",", ":")) - with db._pair_transaction() as conn: - images = conn.execute( - "SELECT gi.*,p.hash AS positive_hash FROM generated_images gi " - "JOIN prompts p ON p.id=gi.prompt_id " - "WHERE gi.file_path=? OR gi.image_path=?", - (normalized, image_path), + images = conn.execute( + "SELECT gi.*,p.hash AS positive_hash FROM generated_images gi " + "JOIN prompts p ON p.id=gi.prompt_id " + "WHERE gi.file_path=? OR gi.image_path=?", + (normalized, image_path), + ).fetchall() + if len(images) > 1 or ( + images and images[0]["positive_hash"] != generate_prompt_hash(positive_text) + ): + return _conflict() + existing = images[0] if images else None + if existing: + prior = conn.execute( + "SELECT n.hash AS negative_hash FROM image_prompt_pairs ip " + "JOIN prompt_pairs pp ON pp.id=ip.pair_id " + "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?", + (existing["id"],), ).fetchall() - if len(images) > 1 or ( - images and images[0]["positive_hash"] != generate_prompt_hash(positive_text) - ): - return {"status": "conflict", "pairs_added": 0, "links_added": 0} - existing = images[0] if images else None - if existing: - prior = conn.execute( - "SELECT n.hash AS negative_hash FROM image_prompt_pairs ip " - "JOIN prompt_pairs pp ON pp.id=ip.pair_id " - "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?", - (existing["id"],), - ).fetchall() - expected = ( - generate_prompt_hash(negative_text) if negative_text.strip() else None - ) - if any(row["negative_hash"] != expected for row in prior): - return {"status": "conflict", "pairs_added": 0, "links_added": 0} - prior_pair = conn.execute( - "SELECT pp.id FROM prompt_pairs pp " - "JOIN prompts p ON p.id=pp.positive_prompt_id " - "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id " - "WHERE p.hash=? AND n.hash IS ?", - ( - generate_prompt_hash(positive_text), - generate_prompt_hash(negative_text) if negative_text.strip() else None, - ), - ).fetchone() - pair_id = db._save_pair(conn, positive_text, negative_text) - pair = conn.execute( - "SELECT * FROM prompt_pairs WHERE id=?", (pair_id,) - ).fetchone() - pairs_added = int(prior_pair is None) - file_size, width, height, image_format = _coerce_file_info(metadata) - # ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON. - snapshots = { - column: dumps_strict(metadata[key], sort_keys=True) - for column, key in ( - ("workflow_data", "workflow"), - ("prompt_metadata", "prompt"), - ("parameters", "parameters"), - ) - if key in metadata - } - info = { - "file_size": file_size, - "width": width, - "height": height, - "format": image_format, - } - snapshots.update( - {key: value for key, value in info.items() if value is not None} + expected = ( + generate_prompt_hash(negative_text) if negative_text.strip() else None ) - changed = False - if existing: - image_id = existing["id"] - updates = { - key: value for key, value in snapshots.items() if existing[key] != value - } - if updates: - conn.execute( - "UPDATE generated_images SET " - + ",".join(f"{key}=?" for key in updates) - + " WHERE id=?", - [*updates.values(), image_id], - ) - changed = True - else: - columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots] - values = [ - pair["positive_prompt_id"], - image_path, - os.path.basename(image_path), - normalized, - *snapshots.values(), - ] - image_id = conn.execute( - "INSERT INTO generated_images(" - + ",".join(columns) - + ") VALUES(" - + ",".join("?" for _ in columns) - + ")", - values, - ).lastrowid - linked = conn.execute( - "SELECT 1 FROM image_prompt_pairs WHERE pair_id=? " - "AND image_id=? AND provenance=?", - (pair_id, image_id, encoded), - ).fetchone() - db._link_image_to_pair(conn, pair_id, image_id, provenance) - status = ( - "added" - if existing is None - else "enriched" if changed or not linked else "unchanged" + if any(row["negative_hash"] != expected for row in prior): + return _conflict() + prior_pair = conn.execute( + "SELECT pp.id FROM prompt_pairs pp " + "JOIN prompts p ON p.id=pp.positive_prompt_id " + "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id " + "WHERE p.hash=? AND n.hash IS ?", + ( + generate_prompt_hash(positive_text), + generate_prompt_hash(negative_text) if negative_text.strip() else None, + ), + ).fetchone() + pair_id = db._save_pair(conn, positive_text, negative_text) + pair = conn.execute("SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)).fetchone() + pairs_added = int(prior_pair is None) + file_size, width, height, image_format = _coerce_file_info(metadata) + # ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON. + snapshots = { + column: dumps_strict(metadata[key], sort_keys=True) + for column, key in ( + ("workflow_data", "workflow"), + ("prompt_metadata", "prompt"), + ("parameters", "parameters"), ) - return { - "status": status, - "pairs_added": pairs_added, - "links_added": int(not linked), - "pair_id": pair_id, - "image_id": image_id, + if key in metadata + } + info = { + "file_size": file_size, + "width": width, + "height": height, + "format": image_format, + } + snapshots.update({key: value for key, value in info.items() if value is not None}) + changed = False + if existing: + image_id = existing["id"] + updates = { + key: value for key, value in snapshots.items() if existing[key] != value } + if updates: + conn.execute( + "UPDATE generated_images SET " + + ",".join(f"{key}=?" for key in updates) + + " WHERE id=?", + [*updates.values(), image_id], + ) + changed = True + else: + columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots] + values = [ + pair["positive_prompt_id"], + image_path, + os.path.basename(image_path), + normalized, + *snapshots.values(), + ] + image_id = conn.execute( + "INSERT INTO generated_images(" + + ",".join(columns) + + ") VALUES(" + + ",".join("?" for _ in columns) + + ")", + values, + ).lastrowid + linked = conn.execute( + "SELECT 1 FROM image_prompt_pairs WHERE pair_id=? " + "AND image_id=? AND provenance=?", + (pair_id, image_id, encoded), + ).fetchone() + db._link_image_to_pair(conn, pair_id, image_id, provenance) + status = ( + "added" + if existing is None + else "enriched" if changed or not linked else "unchanged" + ) + return { + "status": status, + "pairs_added": pairs_added, + "links_added": int(not linked), + "pair_id": pair_id, + "image_id": image_id, + } diff --git a/database/review.py b/database/review.py new file mode 100644 index 0000000..40780e8 --- /dev/null +++ b/database/review.py @@ -0,0 +1,541 @@ +"""Needs review items (#249): persist and read capture-bound ambiguous evidence. + +An item is one ambiguous resolver result for one annotation byte record (the +whole-file SHA-256 of one physical file at one locator), keyed by +``(byte_id, resolver_version, evidence_hash)``. Recording is idempotent: the +same evidence for the same bytes is a duplicate and never touches an existing +item, its state, revision or decisions. Changed bytes are a new byte record +and therefore a new pending item; the old one stays. + +Callers pass verified facts (the capture's digest, size and ``mtime_ns``); +nothing here reads the filesystem. Writes run in one ``_pair_transaction`` +fenced by the storage epoch the capture was taken under. +""" + +import hashlib +import json +import os +import re +import sqlite3 +from typing import Iterator + +try: + from ..utils.hashing import generate_prompt_hash +except ImportError: + from utils.hashing import generate_prompt_hash + +from .annotation_schema import ( + BYTES_TABLE, + FILES_TABLE, + METADATA_TABLE as ANNOTATION_METADATA_TABLE, +) +from .annotations import _new_id, _validate_locator, _validate_target +from .models import normalize_image_path, utc_now_iso +from .review_schema import ( + DECISIONS_TABLE, + ITEMS_TABLE, + MAX_CANDIDATES, + MAX_SPELLED_PATH_BYTES, + METADATA_TABLE, + canonical_evidence, + evidence_digest, + evidence_offers_choice, +) + +# Only this module's table constants are interpolated into the SQL below. +_INSERT_ITEM = ( + f"INSERT OR IGNORE INTO {ITEMS_TABLE}(id,byte_id,spelled_path," # nosec B608 + "resolver_version,evidence_hash,evidence,candidate_count,limited,state," + "revision,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?,'pending',0,?,?)" +) +_BUMP_REVIEW_INDEX = ( + f"UPDATE {METADATA_TABLE} SET " # nosec B608 + "review_index_revision=review_index_revision+1 WHERE singleton_id=1" +) +# A later spelling of the same locator (a case-only rename on a case-folding +# filesystem) replaces the recorded one, so targets keep formatting. +_REFRESH_SPELLING = ( + f"UPDATE {ITEMS_TABLE} SET spelled_path=? " # nosec B608 + f"WHERE spelled_path<>? AND byte_id IN (SELECT b.id FROM {BYTES_TABLE} b " + f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE f.locator=?)" +) +_REVIEW_INDEX = ( + f"SELECT review_index_revision FROM {METADATA_TABLE} " # nosec B608 + "WHERE singleton_id=1" +) +_BUMP_IMAGE_INDEX = ( + f"UPDATE {ANNOTATION_METADATA_TABLE} SET " # nosec B608 + "image_index_revision=image_index_revision+1 WHERE singleton_id=1" +) +_ITEMS_OF_LOCATOR = ( + "SELECT i.id,i.byte_id,i.resolver_version,i.evidence_hash," # nosec B608 + "i.evidence,i.candidate_count,i.limited,i.state,i.revision,i.created_at," + f"i.updated_at,b.sha256 FROM {FILES_TABLE} f " + f"JOIN {BYTES_TABLE} b ON b.file_id=f.id " + f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id " + "WHERE f.locator=? ORDER BY i.created_at,i.id" +) +_QUEUED = ( + f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608 + f"JOIN {BYTES_TABLE} b ON b.file_id=f.id " + f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id " + "WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? " + "LIMIT 1" +) +_PENDING_AT = ( + f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608 + f"JOIN {BYTES_TABLE} b ON b.file_id=f.id " + f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id " + "WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? " + "AND i.state='pending' LIMIT 1" +) +# Pending items of one file: the candidates a link settles. +_PENDING_OF_LOCATOR = ( + f"SELECT i.id FROM {FILES_TABLE} f " # nosec B608 + f"JOIN {BYTES_TABLE} b ON b.file_id=f.id " + f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id " + "WHERE f.locator=? AND i.state='pending'" +) +# Pending items of one file with their evidence: an untraced file's settle. +_PENDING_EVIDENCE_OF_LOCATOR = ( + f"SELECT i.id,i.evidence FROM {FILES_TABLE} f " # nosec B608 + f"JOIN {BYTES_TABLE} b ON b.file_id=f.id " + f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id " + "WHERE f.locator=? AND i.state='pending'" +) +# Only an undecided item goes: a decision row (RESTRICT) is never deleted. +_SETTLE_ITEM = ( + f"DELETE FROM {ITEMS_TABLE} WHERE id=? AND state='pending' " # nosec B608 + f"AND revision=0 AND NOT EXISTS (SELECT 1 FROM {DECISIONS_TABLE} d " + f"WHERE d.item_id={ITEMS_TABLE}.id)" +) +_EPOCH = "SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id=1" +_PENDING_COUNT = ( + f"SELECT COUNT(*) FROM {ITEMS_TABLE} WHERE state='pending'" # nosec B608 +) +_QUEUE_COLUMNS = ( + "id", + "state", + "revision", + "created_at", + "candidate_count", + "limited", + "spelled_path", + "locator", +) +_QUEUE_SELECT = ( + "SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608 + f"i.limited,i.spelled_path,f.locator FROM {ITEMS_TABLE} i " + f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id " + f"JOIN {FILES_TABLE} f ON f.id=b.file_id " +) +_QUEUE_FIRST = _QUEUE_SELECT + ( + "WHERE i.state='pending' ORDER BY i.created_at,i.id LIMIT ?" +) +_QUEUE_AFTER = _QUEUE_SELECT + ( + "WHERE i.state='pending' AND (i.created_at,i.id)>(?,?) " + "ORDER BY i.created_at,i.id LIMIT ?" +) +_DETAIL_COLUMNS = _QUEUE_COLUMNS + ( + "byte_id", + "resolver_version", + "evidence_hash", + "evidence", + "updated_at", + "sha256", +) +_DETAIL = ( + "SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608 + "i.limited,i.spelled_path,f.locator,i.byte_id,i.resolver_version," + "i.evidence_hash,i.evidence,i.updated_at,b.sha256 " + f"FROM {ITEMS_TABLE} i JOIN {BYTES_TABLE} b ON b.id=i.byte_id " + f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE i.id=?" +) +# The legacy rows and pair links the ingester's compatibility rules read. +_DEPENDENT_IMAGES = ( + "SELECT gi.id,p.hash,gi.image_path FROM generated_images gi " + "JOIN prompts p ON p.id=gi.prompt_id " + "WHERE gi.file_path=? OR gi.image_path=? ORDER BY gi.id" +) +_DEPENDENT_LINKS = ( + "SELECT DISTINCT ip.pair_id,n.hash FROM image_prompt_pairs ip " + "JOIN prompt_pairs pp ON pp.id=ip.pair_id " + "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id " + "WHERE ip.image_id=? ORDER BY ip.pair_id" +) +# Rows whose spelling ends with the file's name: the candidates for another +# spelling of the same physical file (the link summaries' filename fallback). +_SAME_NAME_ROWS = ( + "SELECT id,image_path FROM generated_images " + "WHERE image_path LIKE ? ESCAPE '\\' ORDER BY id" +) +_TITLES = ( + "SELECT p.hash,s.title FROM prompts p " + "JOIN prompt_library_state s ON s.prompt_id=p.id " + "WHERE p.hash IN ({marks}) AND s.title IS NOT NULL ORDER BY p.id" +) +_ITEM_ID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}") +_ITEM_COLUMNS = ( + "id", + "byte_id", + "resolver_version", + "evidence_hash", + "evidence", + "candidate_count", + "limited", + "state", + "revision", + "created_at", + "updated_at", + "sha256", +) + + +def _validate_spelling(spelled_path: object, locator: str) -> None: + if ( + not isinstance(spelled_path, str) + or not spelled_path + or len(spelled_path.encode("utf-8", "surrogatepass")) > MAX_SPELLED_PATH_BYTES + or os.path.normcase(spelled_path) != locator + ): + raise ValueError("spelled_path must be the file's spelling of its locator") + + +def dependent_images(conn: sqlite3.Connection, spellings) -> list: + """``(id, image_path, positive_hash)`` of every legacy row of the file. + + ``spellings`` are the paths the file is linked under (the Apply path and + every alias of the same physical file); each is matched as the ingester + matches one path. Rows are distinct and ordered by id. + """ + found = {} + for spelled in dict.fromkeys(spellings): + for row in conn.execute( + _DEPENDENT_IMAGES, (normalize_image_path(spelled), spelled) + ): + found[row[0]] = (row[0], row[2], row[1]) + return [found[key] for key in sorted(found)] + + +def same_name_rows(conn: sqlite3.Connection, name: str) -> list: + """``(id, image_path)`` of rows whose spelling may end with ``name``.""" + # Imported here: operations composes this module's mixin. + from .operations import escape_like + + return [ + (row[0], row[1]) + for row in conn.execute(_SAME_NAME_ROWS, (f"%{escape_like(name)}",)) + ] + + +def dependency_fingerprint( + conn: sqlite3.Connection, image_path: str, aliases=() +) -> str: + """SHA-256 of what Apply's compatibility depends on for ``image_path``. + + The exact legacy rows of the file under ``image_path`` and every alias + spelling of it, with their positive hashes, and each row's pair links + with the negative hash, read as the ingester reads them. Unrelated prompt + changes (tags, titles) do not move it. + """ + rows = [] + for image_id, _spelled, positive_hash in dependent_images( + conn, (image_path, *aliases) + ): + links = conn.execute(_DEPENDENT_LINKS, (image_id,)).fetchall() + rows.append([image_id, positive_hash, [list(link) for link in links]]) + text = canonical_evidence({"v": 1, "images": rows}) + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +class ReviewOperations: + """PromptDatabase mixin: insert and read Needs review items.""" + + def record_review_item( + self, + *, + locator: str, + spelled_path: str, + sha256: str, + size: int, + mtime_ns: int, + epoch: str, + resolver_version: int, + evidence: dict, + candidate_count: int, + limited: bool, + ) -> bool: + """Store one ambiguous result for these exact bytes; True when new. + + The byte record is created (or its verified fingerprint refreshed) + like an observe. ``spelled_path`` is the file's own spelling (it must + fold to ``locator``); targets are formatted from it. Raises + ``LibraryCommandError("storage-changed")`` when the storage epoch is + no longer ``epoch`` (a restore since the capture) and ``ValueError`` + for invalid facts or evidence. + """ + _validate_target(locator, sha256, size, mtime_ns) + _validate_spelling(spelled_path, locator) + if type(resolver_version) is not int or resolver_version < 1: + raise ValueError("resolver_version must be a positive integer") + if ( + type(candidate_count) is not int + or not 0 <= candidate_count <= MAX_CANDIDATES + ): + raise ValueError(f"candidate_count must be between 0 and {MAX_CANDIDATES}") + if not isinstance(limited, bool) or (limited and candidate_count): + raise ValueError("A limited item carries no candidates") + text = canonical_evidence(evidence) + evidence_hash = evidence_digest(text) + with self._pair_transaction() as conn: + self._library_epoch(conn, epoch) + now = utc_now_iso() + file_id = self._annotation_owner(conn, locator, now) + byte_id, _revision, created, moved = self._annotation_byte( + conn, file_id, sha256, size, mtime_ns, now + ) + if created or moved: + conn.execute(_BUMP_IMAGE_INDEX) + cursor = conn.execute( + _INSERT_ITEM, + ( + _new_id(), + byte_id, + spelled_path, + resolver_version, + evidence_hash, + text, + candidate_count, + int(limited), + now, + now, + ), + ) + inserted = cursor.rowcount > 0 + refreshed = _refresh_spelling(conn, locator, spelled_path) + if inserted or refreshed: + conn.execute(_BUMP_REVIEW_INDEX) + return inserted + + def refresh_review_spelling(self, locator: str, spelled_path: str) -> bool: + """Record ``spelled_path`` as the spelling of every item at ``locator``. + + The rescan's skip calls it for a file it does not copy again: a + case-only rename keeps size and ``mtime_ns``, and a target formatted + from the old spelling would be refused. True when a spelling changed + (the review index revision moves; item revisions do not, so an open + decision stays valid). + """ + _validate_locator(locator) + _validate_spelling(spelled_path, locator) + with self._pair_transaction() as conn: + refreshed = _refresh_spelling(conn, locator, spelled_path) + if refreshed: + conn.execute(_BUMP_REVIEW_INDEX) + return refreshed + + def review_index_revision(self) -> int: + """The queue's revision (moved by every queue change).""" + row = self.model.get_connection().execute(_REVIEW_INDEX).fetchone() + if row is None: + raise sqlite3.DatabaseError("Missing review queue metadata") + return row[0] + + def review_items_for(self, locator: str) -> list: + """Every item of the file at ``locator``, oldest first, evidence parsed.""" + _validate_locator(locator) + return list(self._iter_review_items(locator)) + + def _iter_review_items(self, locator: str) -> Iterator[dict]: + conn = self.model.get_connection() + for row in conn.execute(_ITEMS_OF_LOCATOR, (locator,)): + item = dict(zip(_ITEM_COLUMNS, tuple(row))) + item["evidence"] = json.loads(item["evidence"]) + item["limited"] = bool(item["limited"]) + yield item + + # -- queue reads (#249 E3) ------------------------------------------------- + + def review_fingerprint_queued( + self, locator: str, size: int, mtime_ns: int, resolver_version: int + ) -> bool: + """True when an item cites a byte record verified at this exact stat. + + A rescan uses it to skip the copy and hash of a file it already + queued: the same ``(size, mtime_ns)`` the bytes were verified with. + """ + row = ( + self.model.get_connection() + .execute(_QUEUED, (locator, size, mtime_ns, resolver_version)) + .fetchone() + ) + return row is not None + + def review_pending_at( + self, locator: str, size: int, mtime_ns: int, resolver_version: int + ) -> bool: + """True when a *pending* item cites bytes verified at this exact stat. + + ``review_fingerprint_queued`` also matches decided items; a file whose + only item is decided is not in the queue. + """ + row = ( + self.model.get_connection() + .execute(_PENDING_AT, (locator, size, mtime_ns, resolver_version)) + .fetchone() + ) + return row is not None + + def review_file_linked(self, spellings) -> bool: + """True when a saved row links the file under any of ``spellings``.""" + with self._annotation_read() as conn: + return bool(dependent_images(conn, spellings)) + + def settle_linked_review_items(self, files) -> int: + """Remove the pending, undecided items of files that are now linked. + + ``files`` holds ``(locator, spellings)`` per file. An item leaves the + queue only when it is still ``pending`` at revision 0 with no decision + row and a saved row links its file under one of ``spellings``; a + decided item is never touched. Files without a pending item cost one + indexed read and open no write transaction. Returns how many items + were removed (the review index revision moves when any was). + """ + files = [(locator, tuple(spellings)) for locator, spellings in files] + for locator, _spellings in files: + _validate_locator(locator) + conn = self.model.get_connection() + candidates = [ + (locator, spellings) + for locator, spellings in files + if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None + ] + if not candidates: + return 0 + removed = 0 + with self._pair_transaction() as conn: + for locator, spellings in candidates: + if not dependent_images(conn, spellings): + continue + for (item_id,) in conn.execute( + _PENDING_OF_LOCATOR, (locator,) + ).fetchall(): + removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount + if removed: + conn.execute(_BUMP_REVIEW_INDEX) + return removed + + def settle_untraced_review_items(self, locators) -> int: + """Remove the pending, undecided items that offer nothing to choose. + + ``locators`` are files the caller found ambiguous without a choice + (fewer than two candidates carry a prompt). Of their items, one leaves + the queue only when its own stored evidence offers no choice either + (``evidence_offers_choice``) and it is still ``pending`` at revision 0 + with no decision row, the same safe delete as + ``settle_linked_review_items``. Files without a pending item cost one + indexed read and open no write transaction. Returns how many items + were removed (the review index revision moves when any was). + """ + locators = list(locators) + for locator in locators: + _validate_locator(locator) + conn = self.model.get_connection() + pending = [ + locator + for locator in locators + if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None + ] + if not pending: + return 0 + removed = 0 + with self._pair_transaction() as conn: + for locator in pending: + rows = conn.execute(_PENDING_EVIDENCE_OF_LOCATOR, (locator,)).fetchall() + for item_id, evidence in rows: + if evidence_offers_choice(json.loads(evidence)): + continue + removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount + if removed: + conn.execute(_BUMP_REVIEW_INDEX) + return removed + + def review_queue_page(self, *, limit: int, after: tuple | None = None) -> dict: + """One page of pending items, oldest first, from one snapshot. + + ``after`` is the ``(created_at, id)`` keyset of the last item already + shown. Returns ``{epoch, revision, total, items, more}``; ``total`` is + the exact pending count and ``items`` carry their ``spelled_path`` and + ``locator`` (server-side only). + """ + if type(limit) is not int or limit < 1: + raise ValueError("limit must be a positive integer") + with self._annotation_read() as conn: + epoch = conn.execute(_EPOCH).fetchone() + revision = conn.execute(_REVIEW_INDEX).fetchone() + if epoch is None or revision is None: + raise sqlite3.DatabaseError("Missing review queue metadata") + total = conn.execute(_PENDING_COUNT).fetchone()[0] + if after is None: + rows = conn.execute(_QUEUE_FIRST, (limit + 1,)).fetchall() + else: + rows = conn.execute(_QUEUE_AFTER, (*after, limit + 1)).fetchall() + items = [_queue_row(row, _QUEUE_COLUMNS) for row in rows[:limit]] + return { + "epoch": epoch[0], + "revision": revision[0], + "total": total, + "items": items, + "more": len(rows) > limit, + } + + def review_item(self, item_id: str) -> dict | None: + """One item with its parsed evidence and the pending total, or None.""" + if not isinstance(item_id, str) or _ITEM_ID.fullmatch(item_id) is None: + return None + with self._annotation_read() as conn: + epoch = conn.execute(_EPOCH).fetchone() + if epoch is None: + raise sqlite3.DatabaseError("Missing execution history metadata") + row = conn.execute(_DETAIL, (item_id,)).fetchone() + if row is None: + return None + item = _queue_row(row, _DETAIL_COLUMNS) + item["evidence"] = json.loads(item["evidence"]) + item["epoch"] = epoch[0] + item["total"] = conn.execute(_PENDING_COUNT).fetchone()[0] + return item + + def review_dependency_fingerprint(self, image_path: str, aliases=()) -> str: + with self._annotation_read() as conn: + return dependency_fingerprint(conn, image_path, aliases) + + def review_same_name_rows(self, name: str) -> list: + """``(id, image_path)`` of legacy rows that may spell a file ``name``.""" + with self._annotation_read() as conn: + return same_name_rows(conn, name) + + def stored_prompt_titles(self, texts) -> dict: + """Library titles of saved prompts by text (dedup hash), when set.""" + hashes = {generate_prompt_hash(text): text for text in texts if text.strip()} + if not hashes: + return {} + marks = ",".join("?" * len(hashes)) + found: dict[str, tuple] = {} + conn = self.model.get_connection() + for prompt_hash, title in conn.execute( + _TITLES.format(marks=marks), list(hashes) + ): + text = hashes[prompt_hash] + found[text] = found.get(text, ()) + (title,) + return found + + +def _refresh_spelling(conn: sqlite3.Connection, locator: str, spelled: str) -> bool: + return conn.execute(_REFRESH_SPELLING, (spelled, spelled, locator)).rowcount > 0 + + +def _queue_row(row, columns) -> dict: + item = dict(zip(columns, tuple(row))) + item["limited"] = bool(item["limited"]) + return item diff --git a/database/review_decisions.py b/database/review_decisions.py new file mode 100644 index 0000000..3b5b1ff --- /dev/null +++ b/database/review_decisions.py @@ -0,0 +1,287 @@ +"""Needs review decisions (#249 E3): explicit, auditable and compatible-only. + +A decision is one row of ``media_review_decisions`` for one item revision. +``selected`` links the chosen candidate's pair to the file through the public +ingester's compatible-only rules (``_ingest_resolved_pair_in``) in the same +``_pair_transaction`` as the decision row, the item's new state and revision +and the review index revision; any conflict or later failure rolls all of it +back. ``unresolved`` records the decision and links nothing. No decision +counts a run, touches usage or calls an execution writer. + +Callers verify the file first (the capture must be the item's bytes and the +current file must still hash to them); nothing here reads the filesystem. +""" + +import os +import sqlite3 + +from .annotation_schema import BYTES_TABLE +from .annotations import _new_id +from .library_commands import LibraryCommandError +from .models import utc_now_iso +from .review import ( + _BUMP_REVIEW_INDEX, + _DETAIL, + _DETAIL_COLUMNS, + _PENDING_COUNT, + _queue_row, + dependency_fingerprint, + dependent_images, + same_name_rows, +) +from .review_schema import DECISIONS_TABLE, ITEMS_TABLE, canonical_evidence + +DECISION_KIND = "user-selection" +_INSERT_DECISION = ( + f"INSERT INTO {DECISIONS_TABLE}(id,item_id,decision_revision," # nosec B608 + "created_at,outcome,candidate_hash,evidence,pair_id,image_id) " + "VALUES(?,?,?,?,?,?,?,?,?)" +) +_ADVANCE_ITEM = ( + f"UPDATE {ITEMS_TABLE} SET state=?,revision=revision+1," # nosec B608 + "updated_at=? WHERE id=? AND revision=?" +) +_ITEM_BYTES = ( + f"SELECT b.sha256 FROM {ITEMS_TABLE} i " # nosec B608 + f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id WHERE i.id=?" +) + + +class ReviewCommandError(LibraryCommandError): + """A decision lost a race or would rewrite an association. + + ``code`` is ``review-conflict`` (the item was decided or changed since it + was read), ``association-conflict`` (the choice is not compatible with + the file's saved links, or those links changed), ``file-changed`` (the + item is not about these bytes) or ``review-missing``. ``item`` is the + current item read inside the transaction, ``total`` the pending count and + ``fingerprint`` the current dependency fingerprint, when known; + ``changed`` marks an association conflict caused by links that changed + since the reviewer read them (rather than an incompatible choice). + """ + + def __init__( + self, code, epoch, *, item=None, total=None, fingerprint=None, changed=False + ): + super().__init__(code, epoch) + self.item = item + self.total = total + self.fingerprint = fingerprint + self.changed = changed + # The public view of ``item``, set by the API when it formats one. + self.view = None + + +class _Rollback(Exception): + """Raised inside the transaction so the context manager rolls back.""" + + def __init__(self, error: ReviewCommandError): + super().__init__(error.code) + self.error = error + + +class ReviewDecisionOperations: + """PromptDatabase mixin: record one decision on one Needs review item.""" + + def decide_review_item( + self, + *, + item_id: str, + expected_epoch: str, + expected_revision: int, + sha256: str, + outcome: str, + candidate: dict | None = None, + image_path: str | None = None, + metadata: dict | None = None, + expected_fingerprint: str | None = None, + aliases: tuple = (), + known_rows: frozenset | None = None, + ) -> dict: + """Record the decision; return ``{item, total, pair}``. + + ``selected`` needs the evidence ``candidate`` (with its ``hash``), the + ``image_path`` the scan and the monitor link under, the captured + ``metadata`` and the ``expected_fingerprint`` the reviewer saw. + ``aliases`` are the other spellings legacy rows of the same physical + file use (a symlinked, junctioned or mapped root) and ``known_rows`` + the ids of every same-name row the caller classified; a same-name row + it did not see is a changed association. + Raises ``LibraryCommandError("storage-changed")`` after a restore and + :class:`ReviewCommandError` for every refused decision, with zero + writes either way. + """ + if outcome not in ("selected", "unresolved"): + raise ValueError("outcome must be selected or unresolved") + if outcome == "selected" and ( + not isinstance(candidate, dict) or not image_path + ): + raise ValueError("A selected decision needs a candidate and a file") + try: + with self._pair_transaction() as conn: + return self._decide_in( + conn, + item_id=item_id, + expected_epoch=expected_epoch, + expected_revision=expected_revision, + sha256=sha256, + outcome=outcome, + candidate=candidate, + image_path=image_path, + metadata=metadata or {}, + expected_fingerprint=expected_fingerprint, + aliases=tuple(aliases), + known_rows=known_rows, + ) + except _Rollback as rollback: + raise rollback.error from None + + def _decide_in(self, conn: sqlite3.Connection, **command) -> dict: + epoch = self._library_epoch(conn, command["expected_epoch"]) + item_id = command["item_id"] + current = _current_item(conn, item_id) + if current is None: + raise _Rollback(ReviewCommandError("review-missing", epoch)) + total = conn.execute(_PENDING_COUNT).fetchone()[0] + if ( + current["state"] != "pending" + or current["revision"] != command["expected_revision"] + ): + raise _Rollback( + ReviewCommandError("review-conflict", epoch, item=current, total=total) + ) + if conn.execute(_ITEM_BYTES, (item_id,)).fetchone()[0] != command["sha256"]: + raise _Rollback(ReviewCommandError("file-changed", epoch)) + decision_id = _new_id() + pair = None + candidate = command["candidate"] if command["outcome"] == "selected" else None + if candidate is not None: + pair = self._apply_candidate( + conn, epoch, current, decision_id, total, **command + ) + evidence = { + "kind": DECISION_KIND, + "decision_id": decision_id, + "outcome": command["outcome"], + "reviewed_digest": command["sha256"], + "item_evidence_hash": current["evidence_hash"], + "resolver_version": current["resolver_version"], + "candidate": candidate, + "dependency_fingerprint": command["expected_fingerprint"], + "pair": pair, + } + now = utc_now_iso() + self._insert_review_decision( + conn, + ( + decision_id, + item_id, + current["revision"] + 1, + now, + command["outcome"], + candidate["hash"] if candidate is not None else None, + canonical_evidence(evidence), + pair["id"] if pair else None, + pair["image_id"] if pair else None, + ), + ) + conn.execute( + _ADVANCE_ITEM, (command["outcome"], now, item_id, current["revision"]) + ) + conn.execute(_BUMP_REVIEW_INDEX) + return { + "item": _current_item(conn, item_id), + "total": conn.execute(_PENDING_COUNT).fetchone()[0], + "pair": pair, + "decision_id": decision_id, + } + + def _apply_candidate( + self, conn, epoch, current, decision_id, total, **command + ) -> dict: + """Link the candidate's pair, compatible-only; raise on any conflict.""" + # Imported here: pair_ingestion imports operations, which composes this. + from .pair_ingestion import _ingest_resolved_pair_in + + image_path = command["image_path"] + aliases = command["aliases"] + fingerprint = dependency_fingerprint(conn, image_path, aliases) + known = command["known_rows"] + unseen = known is not None and any( + row_id not in known + for row_id, _path in same_name_rows(conn, os.path.basename(image_path)) + ) + if unseen or fingerprint != command["expected_fingerprint"]: + raise _Rollback( + ReviewCommandError( + "association-conflict", + epoch, + item=current, + total=total, + fingerprint=fingerprint, + changed=True, + ) + ) + rows = dependent_images(conn, (image_path, *aliases)) + if len(rows) > 1: + # One file linked under several spellings: no single row to enrich. + raise _Rollback( + ReviewCommandError( + "association-conflict", + epoch, + item=current, + total=total, + fingerprint=fingerprint, + ) + ) + if rows: + # Enrich the one row of this file under the spelling it was + # linked with, so the ingester's compatibility rules see it. + image_path = rows[0][1] + candidate = command["candidate"] + positive, negative = candidate["positive"], candidate["negative"] + provenance = { + "kind": DECISION_KIND, + "decision_id": decision_id, + "reviewed_digest": command["sha256"], + "candidate_hash": candidate["hash"], + "resolver_version": current["resolver_version"], + "positive": positive, + "negative": negative, + } + result = _ingest_resolved_pair_in( + conn, + self, + image_path, + command["metadata"], + {"state": "resolved", "positive": positive, "negative": negative}, + provenance=provenance, + ) + if result["status"] == "conflict": + raise _Rollback( + ReviewCommandError( + "association-conflict", + epoch, + item=current, + total=total, + fingerprint=fingerprint, + ) + ) + return { + "id": result["pair_id"], + "image_id": result["image_id"], + "status": result["status"], + } + + @staticmethod + def _insert_review_decision(conn: sqlite3.Connection, values: tuple) -> None: + conn.execute(_INSERT_DECISION, values) + + +def _current_item(conn: sqlite3.Connection, item_id: str) -> dict | None: + row = conn.execute(_DETAIL, (item_id,)).fetchone() + if row is None: + return None + item = _queue_row(row, _DETAIL_COLUMNS) + item.pop("evidence") + return item diff --git a/database/review_schema.py b/database/review_schema.py new file mode 100644 index 0000000..0370a5e --- /dev/null +++ b/database/review_schema.py @@ -0,0 +1,507 @@ +"""Needs review schema group (#249): capture-bound ambiguous items and decisions. + +The group installs like ``image_annotations``: optional in old backups, +required once a database is current, and valid only beside a valid image +annotation group, because every item cites one annotation byte record. It owns + +- ``media_review_metadata``: one row, the group's schema version and the + review index revision (moved by every queue change); +- ``media_review_items``: one ambiguous resolver result for one byte record, + keyed by ``(byte_id, resolver_version, evidence_hash)``, with the file's + spelling when it was recorded (the locator is case-folded on Windows, a + target is not), its canonical evidence, candidate count, ``limited`` flag, + state and revision; +- ``media_review_decisions``: the auditable decision history of an item, one + row per decision revision, with optional navigation to the pair and image + the decision linked (nulled, never cascaded, when those are deleted). + +Evidence is canonical JSON (:func:`canonical_evidence`): one object, sorted +keys, no whitespace, ASCII only (so characters, bytes and the 8 MiB bound +agree and lone surrogates survive), no NaN or infinity. ``evidence_hash`` is +the SHA-256 of the item's evidence text. + +Installation never backfills anything and never touches the filesystem. +Validation reads only: catalog objects compared after whitespace compaction, +columns and foreign keys through the PRAGMAs, persisted values in single +bounded SQL passes, then one streamed pass that re-canonicalizes each evidence +document (one row in memory at a time). +""" + +import hashlib +import json +import re +import sqlite3 +from typing import Any, Optional + +from .annotation_schema import BYTES_TABLE +from .annotation_schema import TABLES as ANNOTATION_TABLES +from .annotation_schema import _catalog as _annotation_catalog +from .annotation_schema import annotation_schema_error + +try: + from ..utils.json_strict import dumps_strict +except ImportError: + from utils.json_strict import dumps_strict + +SCHEMA_VERSION = 1 +METADATA_TABLE = "media_review_metadata" +ITEMS_TABLE = "media_review_items" +DECISIONS_TABLE = "media_review_decisions" +TABLES = (METADATA_TABLE, ITEMS_TABLE, DECISIONS_TABLE) +MAX_EVIDENCE_BYTES = 8 * 1024 * 1024 +MAX_CANDIDATES = 16 +# The file's own spelling when it was recorded (bytes of UTF-8 text). Locators +# are case-folded on Windows, so a target is formatted from this instead. +MAX_SPELLED_PATH_BYTES = 16384 +ITEM_STATES = ("pending", "selected", "unresolved") +DECISION_OUTCOMES = ("selected", "unresolved") +_TIMESTAMP_BYTES = 64 +_HEX64 = "length({c})=64 AND {c} NOT GLOB '*[^0-9a-f]*'" +_EVIDENCE_CHECK = f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES})" + +_TABLE_DDL = { + METADATA_TABLE: ( + f"CREATE TABLE {METADATA_TABLE} (singleton_id INTEGER PRIMARY KEY " + "CHECK(singleton_id=1), schema_version INTEGER NOT NULL " + f"CHECK(schema_version={SCHEMA_VERSION}), " + "review_index_revision INTEGER NOT NULL DEFAULT 0 " + "CHECK(review_index_revision>=0))" + ), + ITEMS_TABLE: ( + f"CREATE TABLE {ITEMS_TABLE} (id TEXT PRIMARY KEY CHECK(length(id)=36), " + f"byte_id TEXT NOT NULL REFERENCES {BYTES_TABLE}(id) ON DELETE RESTRICT, " + "spelled_path TEXT NOT NULL CHECK(length(CAST(spelled_path AS BLOB)) " + f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}), " + "resolver_version INTEGER NOT NULL, evidence_hash TEXT NOT NULL " + f"CHECK({_HEX64.format(c='evidence_hash')}), " + f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, " + "candidate_count INTEGER NOT NULL " + f"CHECK(candidate_count BETWEEN 0 AND {MAX_CANDIDATES}), " + "limited INTEGER NOT NULL CHECK(limited IN (0,1)), " + "state TEXT NOT NULL CHECK(state IN ('pending','selected','unresolved')), " + "revision INTEGER NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, " + "updated_at TEXT NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))" + ), + DECISIONS_TABLE: ( + f"CREATE TABLE {DECISIONS_TABLE} (id TEXT PRIMARY KEY " + "CHECK(length(id)=36), item_id TEXT NOT NULL " + f"REFERENCES {ITEMS_TABLE}(id) ON DELETE RESTRICT, " + "decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, " + "outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), " + "candidate_hash TEXT CHECK(candidate_hash IS NULL OR " + f"({_HEX64.format(c='candidate_hash')})), " + f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, " + "pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, " + "image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, " + "UNIQUE(item_id, decision_revision), " + "CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))" + ), +} +_INDEX_DDL = { + # The queue reads pending items oldest first with a stable tiebreak. + "idx_media_review_items_queue": ( + "CREATE INDEX idx_media_review_items_queue ON " + f"{ITEMS_TABLE}(state, created_at, id)" + ), + # Deleting a pair or an image nulls decision navigation by index, not scan. + "idx_media_review_decisions_pair": ( + "CREATE INDEX idx_media_review_decisions_pair ON " f"{DECISIONS_TABLE}(pair_id)" + ), + "idx_media_review_decisions_image": ( + "CREATE INDEX idx_media_review_decisions_image ON " + f"{DECISIONS_TABLE}(image_id)" + ), +} +# ``sqlite_autoindex_<table>_<n>``: the TEXT primary key and the UNIQUE key. +_AUTOINDEXES = {ITEMS_TABLE: 2, DECISIONS_TABLE: 2} +# (name, declared type, not null, default, primary key position) +_COLUMNS = { + METADATA_TABLE: ( + ("singleton_id", "INTEGER", 0, None, 1), + ("schema_version", "INTEGER", 1, None, 0), + ("review_index_revision", "INTEGER", 1, "0", 0), + ), + ITEMS_TABLE: ( + ("id", "TEXT", 0, None, 1), + ("byte_id", "TEXT", 1, None, 0), + ("spelled_path", "TEXT", 1, None, 0), + ("resolver_version", "INTEGER", 1, None, 0), + ("evidence_hash", "TEXT", 1, None, 0), + ("evidence", "TEXT", 1, None, 0), + ("candidate_count", "INTEGER", 1, None, 0), + ("limited", "INTEGER", 1, None, 0), + ("state", "TEXT", 1, None, 0), + ("revision", "INTEGER", 1, None, 0), + ("created_at", "TEXT", 1, None, 0), + ("updated_at", "TEXT", 1, None, 0), + ), + DECISIONS_TABLE: ( + ("id", "TEXT", 0, None, 1), + ("item_id", "TEXT", 1, None, 0), + ("decision_revision", "INTEGER", 1, None, 0), + ("created_at", "TEXT", 1, None, 0), + ("outcome", "TEXT", 1, None, 0), + ("candidate_hash", "TEXT", 0, None, 0), + ("evidence", "TEXT", 1, None, 0), + ("pair_id", "INTEGER", 0, None, 0), + ("image_id", "INTEGER", 0, None, 0), + ), +} +# (column, parent table, parent column, on delete) +_FOREIGN_KEYS = { + METADATA_TABLE: set(), + ITEMS_TABLE: {("byte_id", BYTES_TABLE, "id", "RESTRICT")}, + DECISIONS_TABLE: { + ("item_id", ITEMS_TABLE, "id", "RESTRICT"), + ("pair_id", "prompt_pairs", "id", "SET NULL"), + ("image_id", "generated_images", "id", "SET NULL"), + }, +} + + +class _NonCanonical(ValueError): + pass + + +def _reject_constant(name: str) -> Any: + raise _NonCanonical(f"JSON constant {name} is not allowed") + + +# Built once: validation re-encodes every stored document at startup. +_ENCODER = json.JSONEncoder( + sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False +) +_DECODER = json.JSONDecoder(parse_constant=_reject_constant) + + +def canonical_evidence(value: Any) -> str: + """Serialize one evidence object canonically; ``ValueError`` when invalid. + + The text is a JSON object with sorted keys, no whitespace, ASCII only and + no NaN/infinity, at most ``MAX_EVIDENCE_BYTES`` bytes. A non-finite number + (ComfyUI stamps ``NaN`` into graphs) is written as ``null``, the store's + policy everywhere (``utils.json_strict.dumps_strict``). + """ + if not isinstance(value, dict): + raise ValueError("Review evidence must be a JSON object") + try: + try: + text = _ENCODER.encode(value) + except ValueError as error: + if "Out of range float" not in str(error): + raise + text = dumps_strict( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ) + except (TypeError, RecursionError) as error: + raise ValueError(f"Review evidence is not serializable: {error}") from None + if len(text) > MAX_EVIDENCE_BYTES: + raise ValueError("Review evidence is larger than 8 MiB") + return text + + +def choosable(candidate: Any) -> bool: + """True when a candidate offers a prompt to choose: it is ``resolved`` and + its positive side carries non-blank text. The page disables any other + candidate, so it is never something to choose. + """ + if not isinstance(candidate, dict) or candidate.get("state") != "resolved": + return False + positive = candidate.get("positive") + if not isinstance(positive, dict): + return False + text = positive.get("text") + return isinstance(text, str) and bool(text.strip()) + + +# An item is worth a decision only when at least this many candidates offer +# a prompt; with fewer there is nothing to choose between (maintainer test of +# #284/#288: 39 items whose candidates were all "unresolved"). +MIN_CHOOSABLE = 2 + + +def evidence_offers_choice(evidence: Any) -> bool: + """False when stored evidence provably offers nothing to choose. + + Unlimited evidence must list ``MIN_CHOOSABLE`` choosable candidates. A + limited item stores none, only ``candidate_total``; it may offer a choice + (more than ``MAX_CANDIDATES`` prompts) unless that total is too small. + """ + if not isinstance(evidence, dict): + return False + if evidence.get("limited"): + total = evidence.get("candidate_total") + return type(total) is int and total >= MIN_CHOOSABLE + candidates = evidence.get("candidates") + if not isinstance(candidates, list): + return False + return sum(1 for c in candidates if choosable(c)) >= MIN_CHOOSABLE + + +def evidence_digest(text: str) -> str: + """SHA-256 of canonical evidence text (ASCII, hence UTF-8 identical).""" + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def _is_canonical(text: Any) -> bool: + if not isinstance(text, str): + return False + try: + value = _DECODER.decode(text) + return canonical_evidence(value) == text + except (ValueError, RecursionError): + return False + + +def _compact(sql: str) -> str: + return re.sub(r"\s+", "", sql).lower() + + +def _uuid(column: str) -> str: + """SQL: *column* is a lowercase hyphenated 36-character UUID string.""" + return ( + f"typeof({column}) = 'text' AND length({column}) = 36 " + f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' " + f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' " + f"AND length(replace({column}, '-', '')) = 32 " + f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'" + ) + + +def _timestamp(column: str) -> str: + return ( + f"typeof({column}) = 'text' " + f"AND length(CAST({column} AS BLOB)) <= {_TIMESTAMP_BYTES}" + ) + + +def _non_negative(column: str) -> str: + return f"typeof({column}) = 'integer' AND {column} >= 0" + + +def _hex64(column: str) -> str: + return f"typeof({column}) = 'text' AND " + _HEX64.format(c=column) + + +def _evidence(column: str) -> str: + # Type and size only: the streamed pass below proves the text canonical + # (which implies printable ASCII), so a pattern scan here would read every + # document a second time. + return ( + f"typeof({column}) = 'text' " + f"AND length(CAST({column} AS BLOB)) BETWEEN 2 AND {MAX_EVIDENCE_BYTES}" + ) + + +def _nullable_integer(column: str) -> str: + return f"({column} IS NULL OR typeof({column}) = 'integer')" + + +def _in(column: str, values: tuple) -> str: + return f"typeof({column}) = 'text' AND {column} IN ({repr(values)[1:-1]})" + + +# Every row of a table must satisfy its predicate; one pass each, LIMIT 1. +_ROW_PREDICATES = { + ITEMS_TABLE: " AND ".join( + ( + _uuid("id"), + _uuid("byte_id"), + "typeof(spelled_path) = 'text' AND length(CAST(spelled_path AS BLOB)) " + f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}", + "typeof(resolver_version) = 'integer'", + _hex64("evidence_hash"), + _evidence("evidence"), + _non_negative("candidate_count"), + f"candidate_count <= {MAX_CANDIDATES}", + "typeof(limited) = 'integer' AND limited IN (0, 1)", + _in("state", ITEM_STATES), + _non_negative("revision"), + _timestamp("created_at"), + _timestamp("updated_at"), + ) + ), + DECISIONS_TABLE: " AND ".join( + ( + _uuid("id"), + _uuid("item_id"), + _non_negative("decision_revision"), + _timestamp("created_at"), + _in("outcome", DECISION_OUTCOMES), + "(CASE WHEN outcome = 'selected' THEN " + f"{_hex64('candidate_hash')} ELSE candidate_hash IS NULL END)", + _evidence("evidence"), + _nullable_integer("pair_id"), + _nullable_integer("image_id"), + ) + ), +} + + +def _expected_catalog() -> set: + objects = {("table", name, _compact(sql)) for name, sql in _TABLE_DDL.items()} + objects |= {("index", name, _compact(sql)) for name, sql in _INDEX_DDL.items()} + objects |= { + ("index", f"sqlite_autoindex_{table}_{n}", "") + for table, count in _AUTOINDEXES.items() + for n in range(1, count + 1) + } + return objects + + +def _catalog(conn: sqlite3.Connection) -> list: + names = TABLES + tuple(_INDEX_DDL) + marks = ",".join("?" * len(names)) + table_marks = ",".join("?" * len(TABLES)) + return conn.execute( # only "?" placeholders are interpolated + "SELECT type,name,sql FROM sqlite_master " + f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608 + names + TABLES, + ).fetchall() + + +def _structure_error(conn: sqlite3.Connection) -> Optional[str]: + for table, expected in _COLUMNS.items(): + columns = tuple( + (row[1], row[2].upper(), row[3], row[4], row[5]) + for row in conn.execute(f"PRAGMA table_info({table})") + ) + if columns != expected: + return f"Review queue table {table} has invalid columns" + foreign_keys = { + (row[3], row[2], row[4], row[6]) + for row in conn.execute(f"PRAGMA foreign_key_list({table})") + } + if foreign_keys != _FOREIGN_KEYS[table]: + return f"Review queue table {table} has invalid foreign keys" + if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone(): + return f"Review queue table {table} contains foreign key violations" + return None + + +def _evidence_error(conn: sqlite3.Connection) -> Optional[str]: + """Stream every evidence document once; one row is held at a time.""" + for evidence_hash, text in conn.execute( + f"SELECT evidence_hash, evidence FROM {ITEMS_TABLE}" # nosec B608 + ): + if not _is_canonical(text) or evidence_digest(text) != evidence_hash: + return f"Review queue table {ITEMS_TABLE} contains invalid evidence" + for (text,) in conn.execute( + f"SELECT evidence FROM {DECISIONS_TABLE}" # nosec B608 + ): + if not _is_canonical(text): + return f"Review queue table {DECISIONS_TABLE} contains invalid evidence" + return None + + +def _value_error(conn: sqlite3.Connection) -> Optional[str]: + # Only this module's constants are interpolated into the SQL below. + rows = conn.execute( + "SELECT COUNT(*), MIN(typeof(singleton_id) = 'integer' " # nosec B608 + "AND singleton_id = 1 AND typeof(schema_version) = 'integer' " + f"AND schema_version = {SCHEMA_VERSION} " + f"AND {_non_negative('review_index_revision')}) FROM {METADATA_TABLE}" + ).fetchone() + if rows[0] != 1 or rows[1] != 1: + return "Review queue metadata is invalid" + for table, predicate in _ROW_PREDICATES.items(): + if conn.execute( + f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1" # nosec B608 + ).fetchone(): + return f"Review queue table {table} contains invalid values" + return _evidence_error(conn) + + +def review_without_annotations_error(conn: sqlite3.Connection) -> Optional[str]: + """Catalog only: a review group present without its annotation tables. + + Startup asks this before the annotation group installs, which would make + a review group copied without it look valid. It reads no rows; the full + validation runs once in :func:`migrate_review_state`. + """ + if not _catalog(conn): + return None + present = { + name for kind, name, _sql in _annotation_catalog(conn) if kind == "table" + } + if present != set(ANNOTATION_TABLES): + return ( + "Review queue schema requires a valid image annotation schema: " + "Missing image annotation schema" + ) + return None + + +def review_schema_error( + conn: sqlite3.Connection, + *, + allow_missing: bool = True, + annotations_valid: bool = False, +) -> Optional[str]: + """Reject partial or unsupported review schemas and invalid rows. + + Reads only: nothing is ever repaired. A database without any of the + group's objects is acceptable for backups that predate it unless + ``allow_missing`` is false. A present group requires a present, valid + image annotation group; ``annotations_valid`` says the caller has just + validated it (startup's annotation install), so only its presence is + checked again. + """ + objects = _catalog(conn) + if not objects: + return None if allow_missing else "Missing review queue schema" + if annotations_valid: + annotation_error = review_without_annotations_error(conn) + else: + annotation_error = annotation_schema_error(conn, allow_missing=False) + if annotation_error: + annotation_error = ( + "Review queue schema requires a valid image annotation schema: " + + annotation_error + ) + if annotation_error: + return annotation_error + present = {name for kind, name, _sql in objects if kind == "table"} + if present != set(TABLES): + return "Incomplete review queue schema" + actual = {(kind, name, _compact(sql or "")) for kind, name, sql in objects} + if actual != _expected_catalog(): + return "Review queue has an invalid schema definition" + return _structure_error(conn) or _value_error(conn) + + +def migrate_review_state(conn: sqlite3.Connection) -> None: + """Install the group atomically; validate before and after creating it. + + Runs after ``migrate_annotation_state`` (which has just validated the + annotation group): the group is only valid beside the annotation group it + cites. A present group is validated once, evidence included. + """ + conn.execute("SAVEPOINT media_review_schema") + try: + installed = bool(_catalog(conn)) + error = review_schema_error(conn, annotations_valid=True) + if error: + raise sqlite3.DatabaseError(error) + if installed: + # Already complete and valid: nothing to create, and the evidence + # pass is the expensive part of validation, so it runs once. + conn.execute("RELEASE media_review_schema") + return + for sql in _TABLE_DDL.values(): + conn.execute(sql.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1)) + for sql in _INDEX_DDL.values(): + conn.execute(sql.replace("INDEX", "INDEX IF NOT EXISTS", 1)) + conn.execute( + f"INSERT OR IGNORE INTO {METADATA_TABLE} " + "(singleton_id, schema_version, review_index_revision) VALUES (1, ?, 0)", + (SCHEMA_VERSION,), + ) + error = review_schema_error(conn, allow_missing=False, annotations_valid=True) + if error: + raise sqlite3.DatabaseError(error) + conn.execute("RELEASE media_review_schema") + except BaseException: + conn.execute("ROLLBACK TO media_review_schema") + conn.execute("RELEASE media_review_schema") + raise diff --git a/py/api/__init__.py b/py/api/__init__.py index 34ae5da..a37208f 100644 --- a/py/api/__init__.py +++ b/py/api/__init__.py @@ -8,6 +8,8 @@ Split into domain-specific mixins for maintainability: - AutotagRoutesMixin: auto-tagging model management and tagging - JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py) - ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs +- ReviewRoutesMixin: the Needs review queue, item detail and decisions +- ReviewRecordingMixin: Needs review recording for the scan and the monitor - SystemRoutesMixin: System health check, Overview and usage statistics """ @@ -30,6 +32,8 @@ from .pairs import PairRoutesMixin from .library import LibraryRoutesMixin from .vocabulary import VocabularyRoutesMixin from .annotations import AnnotationRoutesMixin +from .review import ReviewRoutesMixin +from .review_recording import ReviewRecordingMixin from .saved_views import SavedViewRoutesMixin from .history import HistoryRoutesMixin from .images import ImageRoutesMixin @@ -264,6 +268,8 @@ class PromptManagerAPI( LibraryRoutesMixin, VocabularyRoutesMixin, AnnotationRoutesMixin, + ReviewRoutesMixin, + ReviewRecordingMixin, SavedViewRoutesMixin, HistoryRoutesMixin, PromptRoutesMixin, @@ -304,6 +310,8 @@ class PromptManagerAPI( # created on first use with their restore subscription. self._captures = None self._captures_unsubscribe = None + # Set by close_annotation_captures (shutdown): no store is recreated. + self._captures_closed = False self._cached_output_dir = None # Lazy-cached by _find_comfyui_output_dir() self._html_cache = {} # Cached HTML file contents keyed by path self._gallery_cache = {} # dict: path_str -> (files, timestamp) @@ -484,6 +492,7 @@ class PromptManagerAPI( self._register_library_routes(routes) self._register_vocabulary_routes(routes) self._register_annotation_routes(routes) + self._register_review_routes(routes) self._register_saved_view_routes(routes) self._register_history_routes(routes) self._register_prompt_routes(routes) diff --git a/py/api/admin.py b/py/api/admin.py index 5b530ef..0b9b89c 100644 --- a/py/api/admin.py +++ b/py/api/admin.py @@ -45,6 +45,7 @@ try: parse_page_json, ) from ...utils.prompt_pair_resolver import resolve_prompt_pair + from ...utils.review_capture import reviewable except ImportError: from database.operations import PromptDatabase from database.pair_ingestion import ingest_resolved_pair @@ -55,6 +56,7 @@ except ImportError: parse_page_json, ) from utils.prompt_pair_resolver import resolve_prompt_pair + from utils.review_capture import reviewable try: @@ -212,6 +214,30 @@ def _time_left(done, total, batches, elapsed): return f" · about {minutes} min left" +def new_review_counts(): + """The scan's Needs review tally: ``added`` new items, ``queued`` files + in the queue after this scan (new or already there), ``settled`` items + removed because their file is linked now, ``untraced`` items removed + because they offer nothing to choose, and ``errors``.""" + return {"added": 0, "queued": 0, "settled": 0, "untraced": 0, "errors": 0} + + +def _review_scope(review): + """The scan job's ``scope`` keys for its Needs review tally. + + ``review_items`` is the number of this scan's files that are in the + pending queue after it: the review group's ambiguous files, less those a + capture could not read or a decision already settled. + """ + return { + "review_items": review["queued"], + "review_added": review["added"], + "review_settled": review["settled"], + "review_untraced": review["untraced"], + "review_errors": review["errors"], + } + + def _scan_status_line(done, total, batches, elapsed): """ "Processing file N of M", with an estimate once enough batches ran.""" line = f"Processing file {done:,} of {total:,}" @@ -1461,10 +1487,13 @@ class AdminRoutesMixin: known = set() if media_files and job.scope.get("files") != "all": known = await self._known_scan_files(job, media_files) + review = new_review_counts() + job.scope.update(_review_scope(review)) async for event in self._scan_batches( - media_files, counts, job, roots, known + media_files, counts, job, roots, known, review ): yield event + self._log_review_items(job, review) self.logger.info( f"Scan completed: processed={counts['processed']}, " @@ -1501,12 +1530,16 @@ class AdminRoutesMixin: sizes = {str(media_file): _file_size(media_file) for media_file in media_files} return self.db.get_known_resolved_paths(sizes) - async def _scan_batches(self, media_files, counts, job, roots=(), known=()): + async def _scan_batches( + self, media_files, counts, job, roots=(), known=(), review=None + ): """Process ``media_files`` in executor batches, updating ``counts`` in place. Records each file's outcome in ``job`` and yields one progress event per batch. Files in ``known`` (path strings) are not read: they count - as processed and unchanged and are recorded as skipped. + as processed and unchanged and are recorded as skipped. ``review`` + counts the Needs review items the batches record (``job.scope`` + ``review_items`` / ``review_errors`` follow it after every batch). """ total = len(media_files) started = time.monotonic() @@ -1529,9 +1562,16 @@ class AdminRoutesMixin: batch_results, roots=roots, outcomes=outcomes, + review=review, ) for key in counts: counts[key] += batch_counts[key] + if review is not None: + # After the batch's links: a file linked now leaves the queue. + await self._run_in_executor( + self._settle_scanned_reviews, batch, roots, review + ) + job.scope.update(_review_scope(review)) for group, row, log_line in outcomes: job.record(group, row, log_line) @@ -1581,12 +1621,20 @@ class AdminRoutesMixin: return map_parallel(extract, file_batch, PromptManagerConfig.WORKER_THREADS) - def _ingest_scan_batch_sync(self, batch_results, roots=(), outcomes=None): + def _ingest_scan_batch_sync( + self, batch_results, roots=(), outcomes=None, review=None + ): """Store the prompts found in a metadata batch (blocking DB work). Returns the legacy counts. When ``outcomes`` is a list, every processed file appends its ``(group, row, log_line)`` for the scan job, with the - path relative to its root in ``roots`` (see ``_scan_roots``). + path relative to its root in ``roots`` (see ``_scan_roots``). A file + whose final outcome is ambiguous and unlinked (the resolver found it + ambiguous and the fallback did not link it) is recorded for Needs + review (best effort, counted in ``review`` when given); one a saved + row already links is reported ``unchanged`` and never queued. An + ambiguous file with nothing to choose is reported ``unresolved`` and + its pending items without a choice leave the queue. """ counts = dict.fromkeys(SCAN_COUNT_KEYS, 0) @@ -1598,13 +1646,27 @@ class AdminRoutesMixin: if self._scan_job and self._scan_job.cancelled: break counts["processed"] += 1 + ambiguous, untraced = [], [] try: - outcome = self._ingest_scanned_file(media_file, metadata, detailed=True) + outcome = self._ingest_scanned_file( + media_file, + metadata, + detailed=True, + ambiguous=ambiguous, + untraced=untraced, + ) except Exception as e: self.logger.error(f"Error processing {media_file.name}: {e}") counts["failed"] += 1 note(media_file, "error") continue + if ambiguous and outcome == "ambiguous": + if self._review_file_linked(media_file, roots): + outcome = "unchanged" # already linked: not a review case + else: + self._record_scanned_review(media_file, roots, review) + elif untraced and outcome == "unresolved": + self._settle_untraced_reviews(untraced, roots, review) if outcome is None: note(media_file, None) continue @@ -1638,13 +1700,21 @@ class AdminRoutesMixin: return None return positive.strip() or None - def _ingest_scanned_file(self, media_file, metadata, detailed=False): + def _ingest_scanned_file( + self, media_file, metadata, detailed=False, ambiguous=None, untraced=None + ): """Enrich exact pairs; retain legacy positive-only import for absent pairs. Returns the file's scan status, or None without prompt metadata. With ``detailed``, a stored or linked file returns a dict instead: its ``status``, the prompt's display ``title`` and, for a resolved pair, - the pair counts. + the pair counts. When the resolver finds the file ambiguous (whether + or not the browser parser then links it) and ``ambiguous`` is a list, + ``media_file`` is appended to it; only a file that then returns + ``"ambiguous"`` (nothing linked it) is a Needs review case. An + ambiguous result with nothing to choose (``reviewable`` is False: fewer + than two candidates carry a prompt) that nothing links returns + ``"unresolved"`` instead, and ``media_file`` goes to ``untraced``. """ if not metadata: return None @@ -1660,6 +1730,13 @@ class AdminRoutesMixin: return None _with_file_size(canonical, media_file) resolution = resolve_prompt_pair(canonical) + # Ambiguous without a choice: unresolved unless something links it. + unchoosable = resolution["state"] == "ambiguous" and not reviewable(resolution) + if unchoosable: + if untraced is not None: + untraced.append(media_file) + elif resolution["state"] == "ambiguous" and ambiguous is not None: + ambiguous.append(media_file) if resolution["state"] == "resolved": result = ingest_resolved_pair( self.db, str(media_file), canonical, resolution @@ -1679,9 +1756,9 @@ class AdminRoutesMixin: # Graphs without a recognized sampler keep the legacy extractor below. prompt_text = self._graph_prompt_text(metadata) if not prompt_text: - return resolution["state"] + return "unresolved" if unchoosable else resolution["state"] elif resolution["state"] == "ambiguous": - return "ambiguous" + return "unresolved" if unchoosable else "ambiguous" elif resolution["state"] == "unresolved" and not isinstance( canonical.get("parameters"), str ): diff --git a/py/api/annotations.py b/py/api/annotations.py index 1e08e8c..8c799ff 100644 --- a/py/api/annotations.py +++ b/py/api/annotations.py @@ -13,6 +13,7 @@ import datetime import functools import os import re +import threading from pathlib import Path from aiohttp import web @@ -73,6 +74,9 @@ _HEX64 = re.compile(r"[0-9a-f]{64}") _UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}") # secrets.token_urlsafe(32) is 43 characters; the bound only rejects junk. _TOKEN = re.compile(r"[A-Za-z0-9_-]{1,128}") +# The store is created on first use by the event loop, the scan's executor or +# the monitor's worker (Needs review), so its creation is serialized. +_CAPTURE_STORE_LOCK = threading.Lock() _PREVIEW_TYPES = { ".png": "image/png", @@ -259,31 +263,46 @@ class AnnotationRoutesMixin: Created lazily so an API object that never observes a file (most tests, a server that never opens the Tag dialog) owns no temp directory. A restore of the database dooms every capture (``invalidate_all``). + Thread-safe: Needs review recording calls it off the event loop. + Raises ``CaptureError`` after ``close_annotation_captures``. """ - if self._captures is None: - self._captures = CaptureStore() - if self._captures_unsubscribe is None: - self._captures_unsubscribe = subscribe_history_restore( - self.db.model.db_path, self._invalidate_captures - ) - return self._captures + with _CAPTURE_STORE_LOCK: + if getattr(self, "_captures_closed", False): + # A recording still in flight at shutdown must not create a + # new temp directory after the cleanup removed the old one. + raise CaptureError("The capture store is closed") + if self._captures is None: + self._captures = CaptureStore() + if self._captures_unsubscribe is None: + self._captures_unsubscribe = subscribe_history_restore( + self.db.model.db_path, self._invalidate_captures + ) + return self._captures def _invalidate_captures(self) -> None: store = self._captures if store is not None: store.invalidate_all() - def close_annotation_captures(self) -> None: - """Remove every capture file and the owned directory. Idempotent.""" - unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None + def close_annotation_captures(self, *, final: bool = False) -> None: + """Remove every capture file and the owned directory. Idempotent. + + Taken under the store lock. ``final`` (the shutdown cleanup) also makes + ``_capture_store`` refuse to create another store, so a recording + still in flight cannot leak a new temp directory after it. + """ + with _CAPTURE_STORE_LOCK: + if final: + self._captures_closed = True + unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None + store, self._captures = self._captures, None if unsubscribe is not None: unsubscribe() - store, self._captures = self._captures, None if store is not None: store.close() async def cleanup_annotation_captures(self, app: web.Application) -> None: - self.close_annotation_captures() + self.close_annotation_captures(final=True) # -- shared request steps -------------------------------------------------- @@ -329,7 +348,10 @@ class AnnotationRoutesMixin: root, error = self._scoped_root(index, scope) if error is not None: return error - store = self._capture_store() + try: + store = self._capture_store() + except CaptureError as exc: # the server is shutting down + return _capture_error(exc) try: epoch = await self._run_in_executor(self.db.get_execution_epoch) capture = await _settled( diff --git a/py/api/jobs.py b/py/api/jobs.py index 09ab15a..b078503 100644 --- a/py/api/jobs.py +++ b/py/api/jobs.py @@ -435,7 +435,8 @@ class JobRegistry: if same_kind is not None: return same_kind refused = _REFUSED_BESIDE[resolve_admission(admission, record_changing)] - running = [job for job in self._jobs.values() if job.running] + # A snapshot: the monitor's worker thread asks too (Needs review). + running = [job for job in tuple(self._jobs.values()) if job.running] running.sort(key=lambda job: job._started_seq) for job in running: if job.admission in refused: @@ -466,8 +467,8 @@ class JobRegistry: return self._jobs.get(job_id) def active(self, kind): - """The running job of ``kind``, or None.""" - for job in self._jobs.values(): + """The running job of ``kind``, or None (safe off the event loop).""" + for job in tuple(self._jobs.values()): if job.kind == kind and job.running: return job return None diff --git a/py/api/review.py b/py/api/review.py new file mode 100644 index 0000000..2d3e425 --- /dev/null +++ b/py/api/review.py @@ -0,0 +1,689 @@ +"""Needs review routes (#249 E3): the pending queue, item detail, decisions. + +``GET /prompt_manager/review/items`` lists pending items oldest first behind +an HMAC keyset cursor bound to the storage epoch and the review index +revision (any decision, new item or restore makes an old cursor stale); +``total`` is the exact pending count. ``GET .../items/{id}`` adds the +candidates and the dependency fingerprint. ``POST .../items/{id}/decision`` +records one explicit decision: Apply links the chosen sampler's pair only +when the capture is the item's bytes, the current file still hashes to them, +the evidence they yield is unchanged and the ingester's compatible-only rules +allow it; Leave unresolved links nothing. + +Targets are formatted from the recorded spelling against the current roots +(no file access beyond the roots' real paths); a file under no current root +is ``unavailable``. Responses carry no server path, no locator and no +exception text. +""" + +import base64 +import binascii +import hashlib +import hmac +import json +import os +import re + +from aiohttp import web + +from .annotations import _HEX64, _TOKEN, _UUID, _capture_error, _error +from .prompts import bad_request + +try: + from ...database.library import display_title + from ...database.library_commands import LibraryCommandError + from ...database.review_decisions import ReviewCommandError + from ...utils.media_capture import CaptureError + from ...utils.review_capture import captured_review, review_evidence +except ImportError: + from database.library import display_title + from database.library_commands import LibraryCommandError + from database.review_decisions import ReviewCommandError + from utils.media_capture import CaptureError + from utils.review_capture import captured_review, review_evidence + +ITEMS_URL = "/prompt_manager/review/items" +DEFAULT_LIMIT = 40 +MAX_LIMIT = 40 +MAX_CURSOR = 4096 +MAX_BODY_BYTES = 64 * 1024 +# The serialized detail stays under this; past it the detail is ``limited``. +MAX_DETAIL_BYTES = 8 * 1024 * 1024 +UNRESOLVED = "unresolved" +_DECISION_FIELDS = frozenset( + { + "expected_epoch", + "expected_revision", + "capture_token", + "sha256", + "dependency_fingerprint", + "choice", + } +) +_CURSOR_SECRET = os.urandom(32) +# The real path of a spelling; module-level so a test can fake a junction. +_realpath = os.path.realpath +_LIMIT = re.compile(r"[0-9]{1,3}") +_NO_STORE = {"Cache-Control": "no-store"} + +_STATUS = { + "review-missing": 404, + "review-conflict": 409, + "association-conflict": 409, + "file-changed": 409, + "storage-changed": 409, + "source-changed": 409, + "capture-expired": 409, +} +_MESSAGES = { + "review-missing": "This review item no longer exists. Reload the queue.", + "review-conflict": ( + "This item was decided or changed since it was opened. Reload the queue." + ), + "association-conflict": ( + "This image is already linked to another prompt or negative, so this " + "sampler path cannot be attached safely. The existing links were " + "preserved." + ), + "dependencies-changed": ( + "The image's saved links changed since this item was opened. Reopen it " + "to see the current association." + ), + "file-changed": ( + "The file changed since it was reviewed. A metadata scan adds its new " + "bytes as a new item." + ), + "storage-changed": "Prompt storage was restored; reload the review queue.", + "source-changed": ( + "The source changed. Refresh the image collection before reviewing this " + "file." + ), + "capture-expired": "The preview expired. Open the item again.", + "other-file": "The preview belongs to another file. Open the item again.", + "evidence-changed": ( + "This file's metadata no longer gives the evidence that was reviewed. " + "Reload the queue." + ), +} + + +_DEFAULT_MESSAGE = "The decision could not be saved. Reload the queue." + + +def _coded(code: str, message_key: str | None = None, **extra) -> web.Response: + message = _MESSAGES.get(message_key or code, _DEFAULT_MESSAGE) + return _error(_STATUS.get(code, 409), code, message, **extra) + + +def _failed() -> web.Response: + return _error(500, "review-failed", "The review queue could not be read.") + + +def format_review_target(spelled_path, locator, real_roots, *, pathmod=os.path): + """``(root_index, relative)`` of the first root that lists the file, or None. + + ``spelled_path`` is the file's recorded spelling, ``locator`` its folded + identity and ``real_roots`` ``(index, realpath)`` pairs. The relative + path keeps the recorded spelling (a Windows locator is case-folded, a + listing target is not) and must fold back to the locator. + """ + sep = pathmod.sep + for index, real_root in real_roots: + head = spelled_path[: len(real_root)] + if pathmod.normcase(head) != pathmod.normcase(real_root): + continue + rest = spelled_path[len(real_root) :] + if not real_root.endswith(sep): + if not rest.startswith(sep): + continue + rest = rest[1:] + parts = rest.split(sep) + if any( + part in ("", ".", "..") or "/" in part or "\\" in part or ":" in part + for part in parts + ): + continue + if pathmod.normcase(pathmod.join(real_root, *parts)) != locator: + continue + return index, "/".join(parts) + return None + + +class _StaleCursor(ValueError): + """A well-formed cursor this process cannot verify (e.g. after a restart).""" + + +def _encode_cursor(epoch: str, revision: int, key: tuple) -> str: + payload = json.dumps( + {"v": 1, "e": epoch, "r": revision, "k": list(key)}, + separators=(",", ":"), + ensure_ascii=True, + ).encode() + signature = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest() + return base64.urlsafe_b64encode(signature + payload).decode().rstrip("=") + + +def _decode_cursor(token: str) -> tuple: + """``(epoch, revision, (created_at, id))`` of a cursor. + + ``ValueError`` when malformed; :class:`_StaleCursor` when well-formed but + signed by another secret (the per-process secret changes on every start, + so an open page's cursor is stale after a restart, not forged). + """ + try: + if len(token) > MAX_CURSOR: + raise ValueError() + raw = base64.b64decode( + token + "=" * (-len(token) % 4), altchars=b"-_", validate=True + ) + signature, payload = raw[:32], raw[32:] + if len(signature) != 32 or not payload: + raise ValueError() + expected = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest() + if not hmac.compare_digest(signature, expected): + raise _StaleCursor("This page of the queue is out of date") + data = json.loads(payload) + key = data["k"] + if ( + data["v"] != 1 + or not isinstance(data["e"], str) + or type(data["r"]) is not int + or not isinstance(key, list) + or len(key) != 2 + or not all(isinstance(part, str) for part in key) + ): + raise ValueError() + return data["e"], data["r"], tuple(key) + except _StaleCursor: + raise + except (ValueError, KeyError, TypeError, UnicodeError, binascii.Error): + raise ValueError("Invalid review cursor; reload the queue") from None + + +def _limit(value): + if value is None: + return DEFAULT_LIMIT + if _LIMIT.fullmatch(value) is None or not 1 <= int(value) <= MAX_LIMIT: + return None + return int(value) + + +def _side_text(side) -> str | None: + if isinstance(side, dict) and isinstance(side.get("text"), str): + return side["text"] + return None + + +def _workable(candidate: dict) -> bool: + return ( + candidate.get("state") == "resolved" + and _side_text(candidate.get("positive")) is not None + and _side_text(candidate.get("negative")) is not None + ) + + +def _side_label(titles: dict, side, empty: str) -> str: + """The Library title of the side's saved prompt, else its first line.""" + text = _side_text(side) + if text is None: + return empty + stored = side.get("storage_text") + stored = stored if isinstance(stored, str) else text + return display_title(titles.get(stored, ()), text) or empty + + +class ReviewRoutesMixin: + """The Needs review queue over E1's schema and E2's recorded items.""" + + def _register_review_routes(self, routes: web.RouteTableDef) -> None: + routes.get(ITEMS_URL)(self.list_review_items) + routes.get(ITEMS_URL + "/{item_id}")(self.get_review_item) + routes.post(ITEMS_URL + "/{item_id}/decision")(self.decide_review_item) + + # -- shared ---------------------------------------------------------------- + + def _review_roots(self): + """``(roots, real_roots)``: the current gallery roots and their realpaths. + + Blocking (it lists the roots and resolves them): call it on a worker. + """ + roots = list(self._get_all_output_dirs()) + return roots, [(i, _realpath(str(root))) for i, root in enumerate(roots)] + + def _review_view(self, row: dict, roots, real_roots) -> dict: + """The public ReviewItem of one stored row (no path, no locator).""" + found = format_review_target(row["spelled_path"], row["locator"], real_roots) + target = None + if found is not None: + index, relative = found + target = { + "root_index": index, + "root_scope": self._gallery_root_scope(roots[index], index), + "path": relative, + } + return { + "id": row["id"], + "state": row["state"], + "revision": row["revision"], + "created_at": row["created_at"], + "filename": os.path.basename(row["spelled_path"]), + "target": target, + "unavailable": target is None, + "candidate_count": row["candidate_count"], + "limited": row["limited"], + } + + @staticmethod + def _review_image_path(row: dict, roots, target) -> str: + """The path the scan and the monitor link this file under.""" + if target is None: + return row["spelled_path"] + root = roots[target["root_index"]] + return os.path.join(str(root), *target["path"].split("/")) + + # -- queue ----------------------------------------------------------------- + + async def list_review_items(self, request: web.Request) -> web.Response: + """GET ``?limit=1..40&cursor=``: one page of pending items.""" + limit = _limit(request.query.get("limit")) + if limit is None: + return bad_request(f"limit must be an integer from 1 to {MAX_LIMIT}") + bound = None + cursor = request.query.get("cursor") + if cursor: + try: + bound = _decode_cursor(cursor) + except _StaleCursor: + return await self._stale_cursor() + except ValueError as exc: + return bad_request(str(exc)) + try: + page, views = await self._run_in_executor( + self._review_page, limit, bound[2] if bound else None + ) + except Exception: + self.logger.exception("Review queue read failed") + return _failed() + if bound is not None and bound[0] != page["epoch"]: + return _coded("storage-changed", library_epoch=page["epoch"]) + if bound is not None and bound[1] != page["revision"]: + return _coded("review-conflict", total=page["total"]) + items = page["items"] + next_cursor = None + if page["more"] and items: + last = items[-1] + next_cursor = _encode_cursor( + page["epoch"], page["revision"], (last["created_at"], last["id"]) + ) + return web.json_response( + { + "success": True, + "library_epoch": page["epoch"], + "total": page["total"], + "items": views, + "next_cursor": next_cursor, + }, + headers=_NO_STORE, + ) + + async def _stale_cursor(self) -> web.Response: + """409 ``review-conflict`` with the current total: reload the queue.""" + try: + page = await self._run_in_executor(self.db.review_queue_page, limit=1) + except Exception: + self.logger.exception("Review queue read failed") + return _failed() + return _coded( + "review-conflict", total=page["total"], library_epoch=page["epoch"] + ) + + def _review_page(self, limit: int, after): + """Worker thread: one queue page and the public views of its items.""" + page = self.db.review_queue_page(limit=limit, after=after) + roots, real_roots = self._review_roots() + return page, [ + self._review_view(row, roots, real_roots) for row in page["items"] + ] + + # -- detail ---------------------------------------------------------------- + + async def get_review_item(self, request: web.Request) -> web.Response: + """GET one item with its candidates and dependency fingerprint.""" + try: + read = await self._run_in_executor( + self._review_detail, request.match_info["item_id"] + ) + except Exception: + self.logger.exception("Review item read failed") + return _failed() + if read is None: + return _coded("review-missing") + item, view, fingerprint, candidates = read + detail = { + "success": True, + "library_epoch": item["epoch"], + **view, + "candidates": candidates, + "dependency_fingerprint": fingerprint, + "sha256": item["sha256"], + "total": item["total"], + } + if len(json.dumps(detail).encode("utf-8")) > MAX_DETAIL_BYTES: + detail = {**detail, "candidates": [], "limited": True} + return web.json_response(detail, headers=_NO_STORE) + + def _review_detail(self, item_id: str): + """Worker thread: ``(item, view, fingerprint, candidates)`` or None.""" + item = self.db.review_item(item_id) + if item is None: + return None + roots, real_roots = self._review_roots() + view = self._review_view(item, roots, real_roots) + image_path = self._review_image_path(item, roots, view["target"]) + aliases, _known = physical_spellings(self.db, image_path, item["locator"]) + fingerprint = self.db.review_dependency_fingerprint(image_path, aliases) + return item, view, fingerprint, self._review_candidates(item) + + def _review_candidates(self, item: dict) -> list: + """The detail's candidates: role texts, a label and selectability.""" + if item["limited"]: + return [] + candidates = [ + c for c in item["evidence"].get("candidates", []) if isinstance(c, dict) + ] + texts = set() + for candidate in candidates: + for role in ("positive", "negative"): + side = candidate.get(role) + if isinstance(side, dict): + for key in ("text", "storage_text"): + if isinstance(side.get(key), str): + texts.add(side[key]) + titles = self.db.stored_prompt_titles(texts) + rows = [] + for candidate in candidates: + positive = _side_text(candidate.get("positive")) + negative = _side_text(candidate.get("negative")) + label = ( + _side_label(titles, candidate.get("positive"), "No positive") + + " / " + + _side_label(titles, candidate.get("negative"), "No negative") + ) + rows.append( + { + "hash": candidate.get("hash"), + "sampler_id": candidate.get("sampler_id"), + "state": "resolved" if _workable(candidate) else "unresolved", + "reason": candidate.get("reason") or "", + "label": label, + "positive": {"text": positive} if positive is not None else None, + "negative": {"text": negative} if negative is not None else None, + } + ) + return rows + + # -- decision -------------------------------------------------------------- + + async def decide_review_item(self, request: web.Request) -> web.Response: + """POST one explicit decision on one pending item.""" + data, error = await self._read_json_body(request, max_bytes=MAX_BODY_BYTES) + if error is not None: + return error + guards, error = _decision_guards(data) + if error is not None: + return error + try: + item = await self._run_in_executor( + self.db.review_item, request.match_info["item_id"] + ) + except Exception: + self.logger.exception("Review item read failed") + return _failed() + if item is None: + return _coded("review-missing") + choice = guards["choice"] + candidate = None + if choice != UNRESOLVED: + candidate = next( + ( + c + for c in item["evidence"].get("candidates", []) + if isinstance(c, dict) and c.get("hash") == choice + ), + None, + ) + if candidate is None or item["limited"]: + return bad_request("choice is not one of this item's candidates") + if not _workable(candidate): + return bad_request("This sampler path cannot be applied") + if guards["capture_token"] is None: + return bad_request("Apply needs a capture of the file") + verified = None + if guards["capture_token"] is not None: + verified, error = await self._verified_review_capture( + item, guards, need_metadata=candidate is not None + ) + if error is not None: + return error + elif guards["sha256"] != item["sha256"]: + return bad_request("sha256 does not match the item") + return await self._record_decision(item, guards, candidate, verified) + + async def _verified_review_capture(self, item, guards, *, need_metadata): + """``((capture, facts, image_path, metadata), None)`` or ``(None, error)``. + + The capture must be the item's file and bytes, under the expected + epoch and a current root; the current file must re-hash to the same + bytes; for Apply, the capture's metadata must still yield exactly the + reviewed evidence. + """ + store = self._captures + capture = store.get(guards["capture_token"]) if store is not None else None + if capture is None: + return None, _coded("capture-expired") + if capture.locator != item["locator"]: + return None, _coded("capture-expired", "other-file") + if guards["sha256"] != capture.sha256: + return None, bad_request("sha256 does not match the preview") + if capture.sha256 != item["sha256"]: + return None, _coded("file-changed") + if guards["expected_epoch"] != capture.epoch: + current = await self._run_in_executor(self.db.get_execution_epoch) + return None, _coded("storage-changed", library_epoch=current) + try: + root, error = await self._run_in_executor( + self._scoped_root, capture.root_index, capture.scope + ) + if error is not None: + return None, error + read = await self._run_in_executor( + _verify_and_read, store, capture, item, root, need_metadata + ) + except CaptureError as exc: + return None, _capture_error(exc) + if read is None: + return None, _coded("source-changed") + relative, facts, captured = read + if facts.locator != item["locator"]: + return None, _coded("capture-expired", "other-file") + if facts.sha256 != item["sha256"]: + return None, _coded("file-changed") + metadata = None + if need_metadata: + if captured is None or not _candidate_still_offered( + captured[1], guards["choice"] + ): + return None, _coded("review-conflict", "evidence-changed") + metadata = captured[0] + image_path = os.path.join(str(root), *relative.split("/")) + return (capture, facts, image_path, metadata), None + + def _decide_sync(self, item, guards, candidate, verified): + """Worker thread: resolve the file's spellings, decide, build the view. + + Returns ``(result, view)``; a refusal raises with ``exc.view`` set. + """ + _capture, facts, image_path, metadata = verified or (None, None, None, None) + aliases, known = (), None + if candidate is not None: + aliases, known = physical_spellings(self.db, image_path, item["locator"]) + try: + result = self.db.decide_review_item( + item_id=item["id"], + expected_epoch=guards["expected_epoch"], + expected_revision=guards["expected_revision"], + # The bytes the re-hash proved, or the body's for Leave unresolved. + sha256=facts.sha256 if facts is not None else guards["sha256"], + outcome="selected" if candidate is not None else UNRESOLVED, + candidate=candidate, + image_path=image_path, + metadata=metadata, + expected_fingerprint=guards["dependency_fingerprint"], + aliases=aliases, + known_rows=known, + ) + except ReviewCommandError as exc: + if exc.item is not None: + roots, real_roots = self._review_roots() + exc.view = self._review_view(exc.item, roots, real_roots) + raise + roots, real_roots = self._review_roots() + return result, self._review_view(result["item"], roots, real_roots) + + async def _record_decision(self, item, guards, candidate, verified): + capture = verified[0] if verified else None + try: + result, view = await self._run_in_executor( + self._decide_sync, item, guards, candidate, verified + ) + except ReviewCommandError as exc: + return self._decision_refused(exc) + except LibraryCommandError as exc: + return _coded(exc.code, library_epoch=exc.epoch) + except Exception: + self.logger.exception("Review decision failed") + return _error(500, "decision-failed", "The decision could not be saved.") + if capture is not None: + # The decision used the capture; the page returns to Images. + self._captures.release(capture.token) + body = { + "success": True, + "library_epoch": guards["expected_epoch"], + "item": view, + "total": result["total"], + } + if result["pair"] is not None: + body["pair"] = result["pair"] + return web.json_response(body, headers=_NO_STORE) + + def _decision_refused(self, exc: ReviewCommandError) -> web.Response: + extra = {"library_epoch": exc.epoch} + if exc.total is not None: + extra["total"] = exc.total + if exc.view is not None: + extra["item"] = exc.view + message = None + if exc.fingerprint is not None: + extra["dependency_fingerprint"] = exc.fingerprint + if exc.code == "association-conflict" and exc.changed: + message = "dependencies-changed" + return _coded(exc.code, message, **extra) + + +def _decision_guards(data: dict): + """The validated decision fields, or ``(None, error)``.""" + if set(data) != _DECISION_FIELDS: + return None, bad_request( + "Request must contain exactly " + ", ".join(sorted(_DECISION_FIELDS)) + ) + epoch, revision = data["expected_epoch"], data["expected_revision"] + if not isinstance(epoch, str) or _UUID.fullmatch(epoch) is None: + return None, bad_request("expected_epoch must be a canonical UUID") + if type(revision) is not int or revision < 0: + return None, bad_request("expected_revision must be a non-negative integer") + token = data["capture_token"] + if token is not None and ( + not isinstance(token, str) or _TOKEN.fullmatch(token) is None + ): + return None, bad_request("Invalid capture token") + for name in ("sha256", "dependency_fingerprint"): + value = data[name] + if not isinstance(value, str) or _HEX64.fullmatch(value) is None: + return None, bad_request( + f"{name} must be 64 lowercase hexadecimal characters" + ) + choice = data["choice"] + if not isinstance(choice, str) or ( + choice != UNRESOLVED and _HEX64.fullmatch(choice) is None + ): + return None, bad_request("choice must be a candidate hash or unresolved") + return {**data}, None + + +def _verify_and_read(store, capture, item, root, need_metadata: bool): + """Worker thread: ``(relative, facts, captured)``, or None off the root. + + Formats the item's target under the capture's root (resolving the root's + real path), re-hashes the current file, then reads the capture. + """ + found = format_review_target( + item["spelled_path"], + item["locator"], + [(capture.root_index, _realpath(str(root)))], + ) + if found is None: + return None + relative = found[1] + facts = store.verify(root, relative) + captured = captured_review(store, capture) if need_metadata else None + return relative, facts, captured + + +def _same_file(spelled: str, image_path: str, locator: str) -> bool: + """True when ``spelled`` reaches the physical file at ``locator``.""" + try: + if os.path.normcase(_realpath(spelled)) == locator: + return True + return os.path.samefile(spelled, image_path) + except (OSError, ValueError): + return False + + +def physical_spellings(db, image_path: str, locator: str): + """``(aliases, known_rows)`` for the legacy rows of one physical file. + + The roots are real paths, while the monitor links a file under the + directory it watches, so a symlinked or junctioned root, a ``subst`` or a + mapped drive spells the same file otherwise. Every row whose spelling + ends with the file's name (the link summaries' fallback lookup) is a + candidate; ``aliases`` are the candidates that resolve to ``locator`` + (``normcase(realpath)``, as the capture forms it) or are the same file, + and ``known_rows`` every candidate id, so a decision can refuse a + same-name row it did not classify. Blocking: call it on a worker. + """ + rows = db.review_same_name_rows(os.path.basename(image_path)) + aliases = tuple( + spelled + for _row_id, spelled in rows + if isinstance(spelled, str) + and spelled != image_path + and _same_file(spelled, image_path, locator) + ) + return tuple(dict.fromkeys(aliases)), frozenset(row_id for row_id, _ in rows) + + +def _candidate_still_offered(resolution: dict, choice: str) -> bool: + """True when the captured bytes still offer the chosen candidate. + + Only what the decision depends on is re-checked: the bytes are still + ``ambiguous`` and the chosen candidate (its hash covers its sampler, + sides and provenance) is still among their candidates. A resolver change + that rewords the top-level reason or reorders candidates leaves a + pending item applicable. + """ + if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous": + return False + evidence, _count, limited = review_evidence(resolution) + if limited: + return False + return any(c.get("hash") == choice for c in evidence.get("candidates", ())) diff --git a/py/api/review_recording.py b/py/api/review_recording.py new file mode 100644 index 0000000..26c2de2 --- /dev/null +++ b/py/api/review_recording.py @@ -0,0 +1,241 @@ +"""Needs review recording for the output scan and the live monitor (#249). + +The rule (maintainer test of #284/#288): a file is queued only when its +final outcome is ambiguous and unlinked. The strict resolver must find it +``ambiguous``, the scan's legacy fallback (the browser parser, +``_graph_prompt_text``) must not link it, and no saved row may link it +already. A file the fallback links, an exact pair the ingester stored and an +already linked file are never queued. Neither is an ambiguous file with +nothing to choose: fewer than two of its candidates carry a prompt +(``utils.review_capture.reviewable``); the scan reports it unresolved ("could +not be traced"). + +The queue heals itself: a pending item whose file is now linked (by the +fallback, a later resolution or the monitor) is removed by the next scan +that lists the file or by the monitor's next pass over it, and so is a +pending item without a choice whose file is still ambiguous without one +(``settle_untraced_review_items``). Only an undecided item goes +(``pending``, revision 0, no decision row); a decision is never deleted or +changed (``settle_linked_review_items``). + +The scan records through ``_record_scanned_review`` (best effort: a failure +is logged and counted); the monitor calls ``record_review_file`` from its +worker thread after its own link. Both go through +``utils.review_capture.record_ambiguous_file`` on the API's capture store. +Nothing here links a file. + +Recording changes records, so it follows the registry's ``records`` +admission (#289). The scan records under its own ``records`` job. The +monitor's pass lasts one file and is not a job: it is refused while a +``records`` or ``guarded`` job runs (``deferred``), and the next scan reads +the file again. +""" + +import os +from pathlib import Path + +from .admin import SCAN_LOG_PREFIXES, _plural, _scan_location, _scan_roots + +# The kind the monitor's recording checks admission under (never registered). +REVIEW_RECORDING_KIND = "review_recording" +# record_ambiguous_file outcomes that leave the file in the pending queue. +QUEUED_OUTCOMES = ("recorded", "limited", "duplicate") +NEW_OUTCOMES = ("recorded", "limited") + +try: + from ...utils.review_capture import record_ambiguous_file +except ImportError: + from utils.review_capture import record_ambiguous_file + +# The scan log's reason for items that leave the queue with nothing to choose. +NOTHING_TO_CHOOSE = "nothing to choose" + + +class ReviewRecordingMixin: + """Record the scan's and the monitor's ambiguous files for Needs review.""" + + def _record_scanned_review(self, media_file, roots, review=None): + """Record one ambiguous, unlinked scanned file (logged and counted).""" + try: + outcome = self._record_review_sync(media_file, roots, review) + except Exception: + self.logger.warning( + f"{media_file.name} could not be added to Needs review", + exc_info=True, + ) + if review is not None: + review["errors"] += 1 + return + if review is None: + return + if outcome in NEW_OUTCOMES: + review["added"] += 1 + if outcome in QUEUED_OUTCOMES: + review["queued"] += 1 + + def _record_review_sync(self, media_file, roots, review=None): + """``record_ambiguous_file`` for a file under one of ``roots`` (blocking). + + An ``untraced`` file (its captured bytes offer nothing to choose) also + loses its pending items that offer nothing either. + """ + located = self._review_location(media_file, roots) + if located is None: + return "skipped" + root, index, relative, _spelled = located + outcome = record_ambiguous_file( + self.db, self._capture_store(), Path(root), index, relative + ) + if outcome == "untraced": + self._settle_untraced_reviews([media_file], roots, review) + return outcome + + @staticmethod + def _review_location(media_file, roots, real_roots=None): + """``(root, root_index, relative, spelled)`` of a listed file, or None. + + ``spelled`` is the capture's spelling of the file (the real root joined + with the relative path); ``os.path.normcase`` of it is the locator + review items are stored under. ``real_roots`` caches each root's + ``realpath`` across a batch. + """ + relative, _label, index = _scan_location(media_file, roots) + root = next((r for r, _l, i in roots if i == index), None) + if index is None or root is None: + return None + if real_roots is None: + real_roots = {} + real = real_roots.get(index) + if real is None: + real = real_roots[index] = os.path.realpath(os.fspath(root)) + spelled = os.path.join(real, *relative.split("/")) + return root, index, relative, spelled + + def _review_file_linked(self, media_file, roots): + """True when a saved row links the scanned file (any spelling).""" + located = self._review_location(media_file, roots) + spellings = (str(media_file),) + ((located[3],) if located else ()) + return self.db.review_file_linked(spellings) + + def _settle_scanned_reviews(self, media_files, roots, review=None): + """Remove the pending items of ``media_files`` that are now linked. + + Blocking; best effort like recording (a failure is logged and + counted in ``review["errors"]``). Returns how many items left. + """ + real_roots = {} + files = [] + for media_file in media_files: + located = self._review_location(media_file, roots, real_roots) + if located is not None: + spelled = located[3] + files.append((os.path.normcase(spelled), (str(media_file), spelled))) + try: + settled = self.db.settle_linked_review_items(files) if files else 0 + except Exception: + self.logger.warning( + "Needs review items of linked files could not be removed", + exc_info=True, + ) + if review is not None: + review["errors"] += 1 + return 0 + if review is not None: + review["settled"] += settled + return settled + + def _settle_untraced_reviews(self, media_files, roots, review=None): + """Remove the pending items without a choice of untraced ``media_files``. + + ``media_files`` are files found ambiguous with fewer than two + candidates that carry a prompt. Blocking; best effort like the linked + settle (a failure is logged and counted in ``review["errors"]``). + Returns how many items left. + """ + locators = [] + for media_file in media_files: + located = self._review_location(media_file, roots) + if located is not None: + locators.append(os.path.normcase(located[3])) + try: + removed = self.db.settle_untraced_review_items(locators) if locators else 0 + except Exception: + self.logger.warning( + "Needs review items without a choice could not be removed", + exc_info=True, + ) + if review is not None: + review["errors"] += 1 + return 0 + if review is not None: + review["untraced"] += removed + return removed + + def _fallback_links(self, metadata): + """True when the scan's legacy fallback would link these records.""" + return bool(metadata.get("prompt")) and bool(self._graph_prompt_text(metadata)) + + def record_review_file(self, image_path): + """Apply the Needs review rule to one monitor output (its worker thread). + + Maps ``image_path`` to the gallery root that lists it; a file under no + current root is ``skipped``. A linked file is never queued: its + pending items are removed and the answer is ``linked``. An unlinked + file the scan's fallback would link is ``skipped`` (the next scan + links it); any other file is recorded when it is ambiguous with a + choice, and is ``untraced`` (its pending items without a choice + removed) when it is ambiguous without one. While a + job runs that a ``records`` job may not run beside, nothing is written + and the answer is ``deferred``. Errors propagate: the monitor logs + them and keeps its link. + """ + busy = self._jobs.blocker(REVIEW_RECORDING_KIND, admission="records") + if busy is not None: + self.logger.info( + f"Needs review was not updated for " + f"{os.path.basename(str(image_path))} while {busy.title} runs; " + "the next scan reads it again" + ) + return "deferred" + full = os.path.abspath(str(image_path)) + located = Path(os.path.realpath(os.path.dirname(full))) / os.path.basename(full) + roots = _scan_roots(self._get_all_output_dirs()) + if self._review_location(located, roots) is None: + return "skipped" + if self._review_file_linked(located, roots): + self._settle_scanned_reviews([located], roots) + return "linked" + if self._fallback_links(self._extract_comfyui_metadata(str(located))): + return "skipped" + return self._record_review_sync(located, roots) + + @staticmethod + def _log_review_items(job, review): + """The scan log's Needs review summary lines (none when nothing moved).""" + prefix = SCAN_LOG_PREFIXES["review"] + if review["added"]: + job.log_line( + f"{prefix} {_plural(review['added'], 'image')} added to Needs review" + ) + already = review["queued"] - review["added"] + if already: + job.log_line( + f"{prefix} {_plural(already, 'image')} already in Needs review" + ) + if review["settled"]: + job.log_line( + f"{SCAN_LOG_PREFIXES['known']} " + f"{_plural(review['settled'], 'image')} left Needs review · now linked" + ) + if review["untraced"]: + job.log_line( + f"{SCAN_LOG_PREFIXES['known']} " + f"{_plural(review['untraced'], 'image')} left Needs review · " + f"{NOTHING_TO_CHOOSE}" + ) + if review["errors"]: + job.log_line( + f"{SCAN_LOG_PREFIXES['errors']} " + f"{_plural(review['errors'], 'image')} could not be added to " + "Needs review" + ) diff --git a/py/api/system.py b/py/api/system.py index 16a62d7..092042e 100644 --- a/py/api/system.py +++ b/py/api/system.py @@ -9,9 +9,10 @@ kept. - ``GET /prompt_manager/system/overview`` is the System › Overview aggregate. Every value is real or null: there is no backup status until backups report - one, no health result until a check ran, and no Needs-review count until - the review queue exists. The missing-thumbnail count is not part of it: - the page reads ``/images/thumbnails/status``, which walks every source. + one and no health result until a check ran; ``needs_review.pending`` is the + Needs review queue's pending total. The missing-thumbnail count is not part + of it: the page reads ``/images/thumbnails/status``, which walks every + source. - ``GET /prompt_manager/system/statistics`` lists runs and ratings per category, with the legacy ``/stats`` totals and the Overview's count of rated images (images whose linked prompt is rated). @@ -187,7 +188,7 @@ class SystemRoutesMixin: }, "health": health_summary(self._current_health_report()), "backup": None, # no verified backup status exists yet (S8) - "needs_review": {"pending": None}, # no review queue yet (Q6) + "needs_review": {"pending": counts["review_pending"]}, "sources": { "configured": len(GalleryConfig.MONITORING_DIRECTORIES), "effective": len(output_dirs), diff --git a/tests/browser/server.py b/tests/browser/server.py index 9926c26..81d0e45 100644 --- a/tests/browser/server.py +++ b/tests/browser/server.py @@ -226,6 +226,8 @@ def create_app(root: Path) -> web.Application: for table in ( "saved_views", "image_favorites", + "media_review_decisions", + "media_review_items", "media_image_tags", "media_annotation_bytes", "media_annotation_files", @@ -591,6 +593,8 @@ def create_app(root: Path) -> web.Application: return web.json_response("dark") async def cleanup(application: web.Application) -> None: + # A scan of an ambiguous fixture file captures it for Needs review. + api.close_annotation_captures() api.db.close_all() seed() diff --git a/tests/browser/system-concept.spec.mjs b/tests/browser/system-concept.spec.mjs index 9317a19..6ff9145 100644 --- a/tests/browser/system-concept.spec.mjs +++ b/tests/browser/system-concept.spec.mjs @@ -169,8 +169,9 @@ function mapOverviewConcept(reference, overview) { const relabel = { 'Database integrity': 'Not checked', 'Positive prompts': number(overview.library.positive_prompts), 'Negative variants': number(overview.library.negative_variants), Images: number(overview.images.indexed) }; const rows = { - // D93 (Q6): no review queue yet, so no count and the action is unavailable (dimmed, as the Save row's D92). - '3 ambiguous image links': row => ({ ...row, title: 'Ambiguous image links', description: 'Not available yet.', + // D93: the real pending count; the action is unavailable (dimmed, as the Save row's D92) until the review page. + '3 ambiguous image links': row => ({ ...row, + title: `${number(overview.needs_review.pending)} ambiguous image link${overview.needs_review.pending === 1 ? '' : 's'}`, action: { ...row.action, opacity: '0.55', cursor: 'not-allowed' } }), // D93: no backup has been recorded (verified backups arrive with S8). 'Backup status': row => ({ ...row, description: 'No retained backup record.' }), @@ -230,12 +231,12 @@ for (const theme of THEMES) for (const [width, height] of SIZES) { }); } -// The recorded differences, by path: D93 the review row (title, Not available yet, dimmed action), the backup row, the +// The recorded differences, by path: D93 the review row (its real count, dimmed action), the backup row, the // preferences row and the sources row, and metric 4; D96 the statistics row; D92 the chip, the status and the dimmed // Save; D122 the nav label. (The fixture's first three metrics are the concept's own counts.) const UNMAPPED_PATHS = [ 'columns.surfaces.0.rows.0.action.cursor', 'columns.surfaces.0.rows.0.action.opacity', - 'columns.surfaces.0.rows.0.description', 'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description', + 'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description', 'columns.surfaces.1.rows.0.description', 'columns.surfaces.1.rows.1.title', 'columns.surfaces.1.rows.2.description', 'heading.actions.0', 'heading.actions.length', 'metrics.items.3.value', 'nav.label', 'save.button.cursor', 'save.button.opacity', 'save.status.text', diff --git a/tests/browser/workspace-system-sources.spec.mjs b/tests/browser/workspace-system-sources.spec.mjs index 6450344..5fca0dc 100644 --- a/tests/browser/workspace-system-sources.spec.mjs +++ b/tests/browser/workspace-system-sources.spec.mjs @@ -116,7 +116,7 @@ test('Scan changes opens it with that one source ("1 output folder") and starts const id = page.url().split('/').pop(); // The job walks only that source (the count is recorded once the walk has listed it). await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.scope) - .toEqual({ files: 'new', roots: [1], output_folders: 1 }); + .toMatchObject({ files: 'new', roots: [1], output_folders: 1 }); await expect(page.locator('#workspace-main h1')).toHaveText('Scan for metadata'); }); diff --git a/tests/browser/workspace-system.spec.mjs b/tests/browser/workspace-system.spec.mjs index a11cfdf..0a5f7bb 100644 --- a/tests/browser/workspace-system.spec.mjs +++ b/tests/browser/workspace-system.spec.mjs @@ -55,6 +55,8 @@ test('overview shows real counts and Not checked', async ({ page, request }) => expect([overview.library.positive_prompts, overview.images.indexed, thumbnails.pending]).toEqual([6, 24, 0]); // I2: the Overview never walks the sources for thumbnails; the row reads the status route itself. expect(overview).not.toHaveProperty('thumbnails'); + // The fixture records no ambiguous file: the Needs review queue is empty. + expect(overview.needs_review).toEqual({ pending: 0 }); const layout = await openOverview(page); expect(await layout.locator('.summary-metrics > .metric').evaluateAll(items => items.map(item => [item.querySelector('strong').textContent, item.querySelector('span').textContent]))).toEqual([ @@ -64,7 +66,7 @@ test('overview shows real counts and Not checked', async ({ page, request }) => ['Not checked', 'Database integrity'], ]); expect(await rowsOf(layout, 'Library health')).toEqual([ - ['Ambiguous image links', 'Not available yet.', 'Review metadata'], + ['0 ambiguous image links', 'Review metadata before assigning a positive / negative pair.', 'Review metadata'], ['Backup status', 'No retained backup record.', 'View backups'], ['No missing thumbnails', 'Generate only missing previews when necessary.', 'Review thumbnails'], ]); @@ -166,10 +168,11 @@ test('a failed thumbnail read is unavailable, never zero', async ({ page }) => { await expect(layout.locator('.metric').nth(2).locator('strong')).toHaveText('24'); }); -test('ambiguous row reads Not available yet (Q6)', async ({ page }) => { +test('ambiguous row reads the pending count; its action waits for the review page', async ({ page }) => { const layout = await openOverview(page); - const row = layout.locator('.operation').filter({ hasText: 'Ambiguous image links' }); - await expect(row.locator('p')).toHaveText('Not available yet.'); + const row = layout.locator('.operation').filter({ hasText: 'ambiguous image links' }); + await expect(row.locator('h3')).toHaveText('0 ambiguous image links'); + await expect(row.locator('p')).toHaveText('Review metadata before assigning a positive / negative pair.'); const review = row.getByRole('button', { name: 'Review metadata', exact: true }); await expect(review).toHaveAttribute('aria-disabled', 'true'); await review.click({ force: true }); diff --git a/tests/browser/workspace_concept_fixture.py b/tests/browser/workspace_concept_fixture.py index bcb306f..1cf0192 100644 --- a/tests/browser/workspace_concept_fixture.py +++ b/tests/browser/workspace_concept_fixture.py @@ -176,6 +176,8 @@ _TABLES = ( "execution_context_texts", "saved_views", "image_favorites", + "media_review_decisions", + "media_review_items", "media_image_tags", "media_annotation_bytes", "media_annotation_files", diff --git a/tests/browser/workspace_library_editor_fixture.py b/tests/browser/workspace_library_editor_fixture.py index 7b852a1..ab375c8 100644 --- a/tests/browser/workspace_library_editor_fixture.py +++ b/tests/browser/workspace_library_editor_fixture.py @@ -28,6 +28,8 @@ EDITOR_METADATA = { _TABLES = ( "saved_views", "image_favorites", + "media_review_decisions", + "media_review_items", "media_image_tags", "media_annotation_bytes", "media_annotation_files", diff --git a/tests/browser/workspace_tagging_fixture.py b/tests/browser/workspace_tagging_fixture.py index 31cbd2f..9bf0214 100644 --- a/tests/browser/workspace_tagging_fixture.py +++ b/tests/browser/workspace_tagging_fixture.py @@ -44,6 +44,8 @@ MAX_EXTRA_IMAGES = 160 _TABLES = ( "saved_views", "image_favorites", + "media_review_decisions", + "media_review_items", "media_image_tags", "media_annotation_bytes", "media_annotation_files", diff --git a/tests/browser/workspace_visual_fixture.py b/tests/browser/workspace_visual_fixture.py index 9a7cdad..3a98342 100644 --- a/tests/browser/workspace_visual_fixture.py +++ b/tests/browser/workspace_visual_fixture.py @@ -114,6 +114,8 @@ def seed_workspace_visual(api: PromptManagerAPI, root: Path) -> None: for table in ( "saved_views", "image_favorites", + "media_review_decisions", + "media_review_items", "media_image_tags", "media_annotation_bytes", "media_annotation_files", diff --git a/tests/browser/workspace_vocabulary_fixture.py b/tests/browser/workspace_vocabulary_fixture.py index 69e0fb9..10c5a8c 100644 --- a/tests/browser/workspace_vocabulary_fixture.py +++ b/tests/browser/workspace_vocabulary_fixture.py @@ -81,6 +81,8 @@ MAX_EXTRA_CATEGORIES = 200 _TABLES = ( "saved_views", "image_favorites", + "media_review_decisions", + "media_review_items", "media_image_tags", "media_annotation_bytes", "media_annotation_files", diff --git a/tests/test_annotation_schema.py b/tests/test_annotation_schema.py index cc37b9a..4961117 100644 --- a/tests/test_annotation_schema.py +++ b/tests/test_annotation_schema.py @@ -30,6 +30,11 @@ FILES = "media_annotation_files" BYTES = "media_annotation_bytes" IMAGE_TAGS = "media_image_tags" TABLES = (METADATA, FILES, BYTES, IMAGE_TAGS) +REVIEW_TABLES = ( + "media_review_decisions", + "media_review_items", + "media_review_metadata", +) # The exact DDL the plan pins; the module must produce precisely these objects. DDL = { METADATA: ( @@ -202,10 +207,14 @@ class AnnotationSchemaTests(unittest.TestCase): ) def drop_group(self, path): - """Remove the group from a closed database file (an old backup).""" + """Remove the group from a closed database file (an old backup). + + The review queue group cites annotation bytes and predates nothing, so + an old backup lacks it as well. + """ with closing(sqlite3.connect(path)) as raw, raw: raw.execute("PRAGMA foreign_keys=OFF") - for table in reversed(TABLES): + for table in REVIEW_TABLES + tuple(reversed(TABLES)): raw.execute(f"DROP TABLE {table}") def test_fresh_install_and_reopen_is_idempotent(self): diff --git a/tests/test_api_admin.py b/tests/test_api_admin.py index 8718199..a03164d 100644 --- a/tests/test_api_admin.py +++ b/tests/test_api_admin.py @@ -107,6 +107,9 @@ class AdminAPITestCase(AioHTTPTestCase): for task in tasks: task.cancel() await asyncio.gather(*tasks, return_exceptions=True) + if api is not None: + # A scan of an ambiguous file captures it for Needs review. + api.close_annotation_captures() PromptModel.close_all_instances() # restore may have swapped api.db assert_closed(self, self.tmpdir) shutil.rmtree(self.tmpdir, ignore_errors=True) @@ -2239,7 +2242,9 @@ class TestActivityJobs(AdminAPITestCase): for name in names: (self.output_dir / name).write_bytes(b"not an image") - def fake_ingest(media_file, metadata, detailed=False): + def fake_ingest( + media_file, metadata, detailed=False, ambiguous=None, untraced=None + ): key = Path(media_file).relative_to(self.output_dir).as_posix() outcome = names[key] if isinstance(outcome, Exception): @@ -2368,8 +2373,10 @@ class TestActivityJobs(AdminAPITestCase): first_done, release = threading.Event(), threading.Event() original = self.api._ingest_scan_batch_sync - def gated(batch_results, roots=(), outcomes=None): - counts = original(batch_results, roots=roots, outcomes=outcomes) + def gated(batch_results, roots=(), outcomes=None, review=None): + counts = original( + batch_results, roots=roots, outcomes=outcomes, review=review + ) first_done.set() release.wait(5) return counts diff --git a/tests/test_api_annotations.py b/tests/test_api_annotations.py index e59a65f..6476ea7 100644 --- a/tests/test_api_annotations.py +++ b/tests/test_api_annotations.py @@ -1045,7 +1045,15 @@ class TestFailures(AnnotationAPITestCase): await self.api.cleanup_annotation_captures(None) self.assertFalse(os.path.exists(directory)) self.assertNotIn(identity, observers._restore_observers) - # A later observation starts a fresh store. + # The shutdown cleanup is final: a later observation (or a monitor + # recording still in flight) creates no new store or directory. + status, body = await self.observe({"target": self.target("sub/a.png")}) + self.assertEqual((status, body["code"]), (500, "capture-error"), body) + self.assertIsNone(self.api._captures) + self.assertNotIn(identity, observers._restore_observers) + # An explicit, non-final close still lets a later observation start + # a fresh store (the tests' own store swaps rely on it). + self.api._captures_closed = False await self.observe_ok() self.assertNotEqual(self.api._captures.directory, directory) diff --git a/tests/test_api_core.py b/tests/test_api_core.py index 4af56a9..29f2e82 100644 --- a/tests/test_api_core.py +++ b/tests/test_api_core.py @@ -471,6 +471,19 @@ class TestUiRoutes(CoreRoutesTestCase): self.assertIn(("POST", "/prompt_manager/jobs/scan"), table) self.assertIn(("POST", "/prompt_manager/jobs/image-tags"), table) + async def test_review_routes_are_registered(self): + table = { + (route.method, route.resource.canonical) + for route in self.app.router.routes() + if route.resource is not None + } + for entry in ( + ("GET", "/prompt_manager/review/items"), + ("GET", "/prompt_manager/review/items/{item_id}"), + ("POST", "/prompt_manager/review/items/{item_id}/decision"), + ): + self.assertIn(entry, table) + class TestStaticRoutes(CoreRoutesTestCase): diff --git a/tests/test_api_review.py b/tests/test_api_review.py new file mode 100644 index 0000000..cb5e5a2 --- /dev/null +++ b/tests/test_api_review.py @@ -0,0 +1,1161 @@ +"""Needs review decisions and queue reads (#249 E3). + +The queue lists pending items oldest first behind an HMAC cursor bound to the +storage epoch and the review index revision; ``total`` is the exact pending +count. A decision is explicit: Apply links the chosen sampler's pair only +under the ingester's compatible rules and only for the current, re-hashed +bytes of the reviewed file, in one transaction with its decision row; Leave +unresolved records the decision and links nothing. Responses never carry a +server path. +""" + +import json +import ntpath +import os +import posixpath +import sqlite3 +import threading +from pathlib import Path +from unittest.mock import patch + +try: + from tests.test_review import ( + ReviewTestCase, + ambiguous_graph, + digest, + unlinkable_graph, + write_png, + ) +except ImportError: # discovered with tests/ as the top-level directory + from test_review import ( + ReviewTestCase, + ambiguous_graph, + digest, + unlinkable_graph, + write_png, + ) + +from database.review_schema import canonical_evidence, evidence_digest +from py.config import GalleryConfig +from utils import review_capture as review_capture_module +from utils.media_capture import ( + CaptureCapacity, + CaptureChanged, + CaptureStore, + FileFacts, +) +from utils.hashing import generate_prompt_hash + +try: # guarded so a missing module fails assertions, not test loading + from py.api import review as review_api +except ImportError: + review_api = None + +ITEMS_URL = "/prompt_manager/review/items" +# Every table a decision may touch; compared whole before and after. +WRITE_TABLES = ( + "prompts", + "generated_images", + "prompt_pairs", + "image_prompt_pairs", + "media_review_items", + "media_review_decisions", + "media_review_metadata", +) + + +class ReviewRoutesTestCase(ReviewTestCase): + def _require_routes(self): + self.assertIsNotNone(review_api, "py.api.review is not implemented") + + def _ambiguous(self, name="two.png", positives=("cat", "dog"), color=(255, 0, 0)): + path = self.output_dir / name + path.parent.mkdir(parents=True, exist_ok=True) + write_png(path, prompt=ambiguous_graph(*positives), color=color) + return path + + def _record(self, path): + self._roots() + outcome = self._record_raw(path) + self.assertIn(outcome, ("recorded", "limited"), outcome) + return outcome + + def _record_raw(self, path): + """Record ``path`` as the producers' recorder does, without their rule. + + These tests are about decisions on an item; whether the scan or the + monitor would queue the file (``tests/test_review_outcome.py``) is + not their subject, and an item can outlive the moment its file gets + linked (until the next scan or monitor pass settles it). + """ + from py.api.admin import _scan_roots + + roots = _scan_roots(self.api._get_all_output_dirs()) + return self.api._record_review_sync(Path(str(path)), roots) + + async def _json(self, method, url, **kwargs): + resp = await self.client.request(method, url, **kwargs) + data = await resp.json() + self.responses.append(data) + return resp.status, data + + async def asyncSetUp(self): + await super().asyncSetUp() + self.responses = [] + + async def _list(self, query=""): + status, data = await self._json("GET", ITEMS_URL + query) + self.assertEqual(status, 200, data) + return data + + async def _detail(self, item_id): + status, data = await self._json("GET", f"{ITEMS_URL}/{item_id}") + self.assertEqual(status, 200, data) + return data + + async def _observe(self, target): + status, data = await self._json( + "POST", "/prompt_manager/image-annotations/observe", json={"target": target} + ) + self.assertEqual(status, 200, data) + return data + + async def _open(self): + """The first pending item, its detail and a fresh capture of its file.""" + listed = await self._list() + item = listed["items"][0] + detail = await self._detail(item["id"]) + observed = await self._observe(item["target"]) + return item, detail, observed + + @staticmethod + def _candidate(detail, positive): + return next( + c for c in detail["candidates"] if c["positive"]["text"] == positive + ) + + def _body(self, detail, observed, choice): + return { + "expected_epoch": observed["library_epoch"], + "expected_revision": detail["revision"], + "capture_token": observed["capture"]["token"], + "sha256": observed["capture"]["sha256"], + "dependency_fingerprint": detail["dependency_fingerprint"], + "choice": choice, + } + + async def _decide(self, item_id, body): + return await self._json("POST", f"{ITEMS_URL}/{item_id}/decision", json=body) + + def _snapshot(self): + conn = self.api.db.model.get_connection() + return { + table: [tuple(row) for row in conn.execute(f"SELECT * FROM {table}")] + for table in WRITE_TABLES + } + + def _decisions(self): + conn = self.api.db.model.get_connection() + return [ + dict(row) + for row in conn.execute( + "SELECT * FROM media_review_decisions ORDER BY decision_revision" + ) + ] + + def _no_paths(self): + for data in self.responses: + text = canonical_evidence({"r": data}) + self.assertNotIn(self.tmpdir, text) + self.assertNotIn(self.tmpdir.replace(os.sep, "/"), text) + self.assertNotIn(os.path.normcase(self.tmpdir), text) + + +class TestQueueReads(ReviewRoutesTestCase): + async def test_total_is_exact_pending_count(self): + self._require_routes() + for index in range(3): + self._record(self._ambiguous(f"f{index}.png", color=(index, 0, 0))) + page = await self._list("?limit=1") + self.assertEqual(page["total"], 3) + self.assertEqual(len(page["items"]), 1) + seen = [page["items"][0]["id"]] + while page["next_cursor"]: + page = await self._list(f"?limit=1&cursor={page['next_cursor']}") + self.assertEqual(page["total"], 3) + seen.extend(item["id"] for item in page["items"]) + rows = self._items() + self.assertEqual(seen, [row["id"] for row in rows]) + item = (await self._list())["items"][0] + self.assertEqual( + set(item), + { + "id", + "state", + "revision", + "created_at", + "filename", + "target", + "unavailable", + "candidate_count", + "limited", + }, + ) + self.assertEqual( + (item["state"], item["revision"], item["filename"]), + ("pending", 0, "f0.png"), + ) + self.assertEqual(item["candidate_count"], 2) + self.assertIs(item["unavailable"], False) + self.assertEqual(item["target"]["path"], "f0.png") + self.assertEqual(item["target"]["root_index"], 0) + conn = self.api.db.model.get_connection() + with conn: + conn.execute( + "UPDATE media_review_items SET state='selected', revision=1 " + "WHERE id=?", + (seen[0],), + ) + conn.execute( + "UPDATE media_review_items SET state='unresolved', revision=1 " + "WHERE id=?", + (seen[1],), + ) + page = await self._list() + self.assertEqual(page["total"], 1) + self.assertEqual([i["id"] for i in page["items"]], [seen[2]]) + self.assertIsNone(page["next_cursor"]) + self._no_paths() + + async def test_empty_queue_and_limit_validation(self): + self._require_routes() + page = await self._list() + self.assertEqual( + (page["success"], page["total"], page["items"], page["next_cursor"]), + (True, 0, [], None), + ) + self.assertEqual(page["library_epoch"], self.api.db.get_execution_epoch()) + for query in ("?limit=0", "?limit=41", "?limit=abc", "?limit=-1", "?limit=1.5"): + status, data = await self._json("GET", ITEMS_URL + query) + self.assertEqual(status, 400, query) + self.assertFalse(data["success"]) + status, data = await self._json("GET", ITEMS_URL + "?cursor=" + "x" * 5000) + self.assertEqual(status, 400) + + async def test_cursor_invalidated_by_decision_and_restore(self): + self._require_routes() + for index in range(3): + self._record(self._ambiguous(f"f{index}.png", color=(index, 0, 0))) + first = await self._list("?limit=1") + cursor = first["next_cursor"] + self.assertTrue(cursor) + # Tampered: one character changed, or a cursor of another shape. + broken = ("A" if cursor[5] != "A" else "B").join((cursor[:5], cursor[6:])) + for token in ("!!!", "QUJD", cursor[:40]): + status, data = await self._json("GET", f"{ITEMS_URL}?cursor={token}") + self.assertEqual(status, 400, (token, data)) + # Well-formed but not this process's signature (tampered, or opened + # before a ComfyUI restart): a recoverable conflict (M3). + status, data = await self._json("GET", f"{ITEMS_URL}?cursor={broken}") + self.assertEqual( + (status, data["code"], data["total"]), (409, "review-conflict", 3) + ) + with patch.object(review_api, "_CURSOR_SECRET", b"\x01" * 32): + status, data = await self._json("GET", f"{ITEMS_URL}?cursor={cursor}") + self.assertEqual((status, data["code"]), (409, "review-conflict"), data) + self.assertEqual(data["library_epoch"], first["library_epoch"]) + # A decision moves the review index revision: the old cursor is stale. + item, detail, observed = await self._open() + status, data = await self._decide( + item["id"], {**self._body(detail, observed, "unresolved")} + ) + self.assertEqual(status, 200, data) + status, data = await self._json("GET", f"{ITEMS_URL}?limit=1&cursor={cursor}") + self.assertEqual(status, 409, data) + self.assertEqual(data["code"], "review-conflict") + self.assertEqual(data["total"], 2) + fresh = (await self._list("?limit=1"))["next_cursor"] + conn = self.api.db.model.get_connection() + with conn: # a restore replaces the storage epoch + conn.execute( + "UPDATE execution_history_metadata SET " + "storage_epoch='00000000-0000-4000-8000-000000000000'" + ) + status, data = await self._json("GET", f"{ITEMS_URL}?limit=1&cursor={fresh}") + self.assertEqual(status, 409, data) + self.assertEqual(data["code"], "storage-changed") + self.assertEqual(data["library_epoch"], "00000000-0000-4000-8000-000000000000") + self._no_paths() + + async def test_detail_size_cap_and_limited_item_has_no_candidates(self): + self._require_routes() + self._roots() + limited = self.output_dir / "limited.png" + # More than 16 candidates with a prompt: limited, still a choice. + many = [f"prompt {n}" for n in range(17)] + write_png(limited, prompt=unlinkable_graph(*many)) + self.assertEqual(self._record_raw(str(limited)), "limited") + self._record(self._ambiguous("two.png")) + items = (await self._list())["items"] + self.assertEqual([i["limited"] for i in items], [True, False]) + detail = await self._detail(items[0]["id"]) + self.assertEqual((detail["limited"], detail["candidates"]), (True, [])) + self.assertEqual(detail["candidate_count"], 0) + detail = await self._detail(items[1]["id"]) + self.assertEqual(len(detail["candidates"]), 2) + candidate = self._candidate(detail, "cat") + self.assertEqual( + set(candidate), + {"hash", "sampler_id", "state", "reason", "label", "positive", "negative"}, + ) + self.assertEqual(candidate["positive"], {"text": "cat"}) + self.assertEqual(candidate["negative"], {"text": "bad"}) + self.assertEqual(candidate["state"], "resolved") + self.assertEqual(candidate["label"], "cat / bad") + self.assertEqual(detail["sha256"], digest(self.output_dir / "two.png")) + self.assertRegex(detail["dependency_fingerprint"], r"^[0-9a-f]{64}$") + with patch.object(review_api, "MAX_DETAIL_BYTES", 200): + capped = await self._detail(items[1]["id"]) + self.assertEqual((capped["limited"], capped["candidates"]), (True, [])) + self.assertEqual(capped["id"], items[1]["id"]) + for missing in ("00000000-0000-4000-8000-000000000000", "not-a-uuid"): + status, data = await self._json("GET", f"{ITEMS_URL}/{missing}") + self.assertEqual(status, 404, data) + self.assertFalse(data["success"]) + self._no_paths() + + async def test_label_uses_library_title_of_a_saved_prompt(self): + self._require_routes() + prompt_id = self.api.db.save_prompt( + "cat", prompt_hash=generate_prompt_hash("cat") + ) + conn = self.api.db.model.get_connection() + with conn: + conn.execute( + "INSERT INTO prompt_library_state(prompt_id,title) VALUES(?,?)", + (prompt_id, "Dunes"), + ) + self._record(self._ambiguous("two.png")) + item = (await self._list())["items"][0] + detail = await self._detail(item["id"]) + self.assertEqual(self._candidate(detail, "cat")["label"], "Dunes / bad") + self.assertEqual(self._candidate(detail, "dog")["label"], "dog / bad") + + async def test_unavailable_when_no_current_root_lists_the_file(self): + self._require_routes() + self._record(self._ambiguous("two.png")) + other = self.output_dir / "elsewhere" + other.mkdir() + from py.config import GalleryConfig + + GalleryConfig.MONITORING_DIRECTORIES = [self._root(other)] + item = (await self._list())["items"][0] + self.assertEqual( + (item["unavailable"], item["target"], item["filename"]), + (True, None, "two.png"), + ) + detail = await self._detail(item["id"]) + self.assertIs(detail["unavailable"], True) + self._no_paths() + + def test_target_formatting_keeps_the_recorded_spelling_on_windows(self): + self._require_routes() + spelled = "C:\\Out\\Sub\\Image_0001.PNG" + locator = ntpath.normcase(spelled) + found = review_api.format_review_target( + spelled, locator, [(0, "D:\\Other"), (1, "C:\\Out")], pathmod=ntpath + ) + self.assertEqual(found, (1, "Sub/Image_0001.PNG")) + self.assertIsNone( + review_api.format_review_target( + spelled, locator, [(0, "C:\\Outer")], pathmod=ntpath + ) + ) + # A recorded spelling that no longer folds to the locator is refused. + self.assertIsNone( + review_api.format_review_target( + spelled, "c:\\out\\sub\\other.png", [(0, "C:\\Out")], pathmod=ntpath + ) + ) + self.assertEqual( + review_api.format_review_target( + "/out/a/b.png", "/out/a/b.png", [(0, "/out")], pathmod=posixpath + ), + (0, "a/b.png"), + ) + + +class TestDecisions(ReviewRoutesTestCase): + async def test_apply_candidate_links_pair_and_records_decision_atomically(self): + self._require_routes() + path = self._ambiguous("two.png") + self._record(path) + revision = self._review_revision() + item, detail, observed = await self._open() + cat = self._candidate(detail, "cat") + before = self._snapshot() + # A failure after the link rolls the link back with everything else. + with patch.object( + type(self.api.db), + "_insert_review_decision", + side_effect=sqlite3.OperationalError("disk I/O error at /secret/path"), + ): + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 500, data) + self.assertEqual(data["code"], "decision-failed") + self.assertNotIn("secret", data["error"]) + self.assertEqual(self._snapshot(), before) + + # A failed decision keeps its capture for the retry. + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 200, data) + self.assertTrue(data["success"]) + self.assertEqual(data["total"], 0) + self.assertEqual( + (data["item"]["id"], data["item"]["state"], data["item"]["revision"]), + (item["id"], "selected", 1), + ) + conn = self.api.db.model.get_connection() + image = conn.execute( + "SELECT gi.id, p.text FROM generated_images gi " + "JOIN prompts p ON p.id=gi.prompt_id WHERE gi.image_path=?", + (str(path),), + ).fetchone() + self.assertEqual(image[1], "cat") + pair = conn.execute( + "SELECT pp.id, p.text, n.text FROM prompt_pairs pp " + "JOIN prompts p ON p.id=pp.positive_prompt_id " + "JOIN prompts n ON n.id=pp.negative_prompt_id" + ).fetchall() + self.assertEqual([tuple(row)[1:] for row in pair], [("cat", "bad")]) + self.assertEqual(data["pair"]["id"], pair[0][0]) + self.assertEqual(data["pair"]["image_id"], image[0]) + links = self.api.db.get_image_prompt_pairs(image[0]) + self.assertEqual(len(links), 1) + provenance = links[0]["provenance"] + self.assertEqual(provenance["kind"], "user-selection") + self.assertEqual(provenance["candidate_hash"], cat["hash"]) + self.assertEqual(provenance["reviewed_digest"], digest(path)) + self.assertEqual(provenance["resolver_version"], 1) + self.assertEqual(provenance["positive"]["text"], "cat") + (decision,) = self._decisions() + self.assertEqual(provenance["decision_id"], decision["id"]) + self.assertEqual( + ( + decision["item_id"], + decision["decision_revision"], + decision["outcome"], + decision["candidate_hash"], + decision["pair_id"], + decision["image_id"], + ), + (item["id"], 1, "selected", cat["hash"], pair[0][0], image[0]), + ) + self.assertEqual(self._review_revision(), revision + 1) + self.assertEqual((await self._list())["total"], 0) + self.assertEqual(self._capture_files(), []) + self._no_paths() + + async def test_apply_conflicting_legacy_owner_returns_association_conflict_with_zero_writes( # noqa: E501 + self, + ): + self._require_routes() + path = self._ambiguous("two.png") + # The monitor's fallback link: the file is linked to "cat" already. + prompt_id = self.api.db.save_prompt( + "cat", prompt_hash=generate_prompt_hash("cat") + ) + image_id = self.api.db.link_image_to_prompt(prompt_id, str(path)) + self._record(path) + item, detail, observed = await self._open() + before = self._snapshot() + dog = self._candidate(detail, "dog") + status, data = await self._decide( + item["id"], self._body(detail, observed, dog["hash"]) + ) + self.assertEqual(status, 409, data) + self.assertEqual(data["code"], "association-conflict") + self.assertIn("linked", data["error"]) + self.assertEqual(self._snapshot(), before) + # The compatible choice reuses the existing image row. + cat = self._candidate(detail, "cat") + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 200, data) + self.assertEqual(data["pair"]["image_id"], image_id) + conn = self.api.db.model.get_connection() + self.assertEqual( + conn.execute("SELECT COUNT(*) FROM generated_images").fetchone()[0], 1 + ) + self._no_paths() + + async def test_changed_dependencies_are_an_association_conflict(self): + self._require_routes() + path = self._ambiguous("two.png") + self._record(path) + item, detail, observed = await self._open() + # Another writer links the file after the detail was read. + prompt_id = self.api.db.save_prompt( + "cat", prompt_hash=generate_prompt_hash("cat") + ) + self.api.db.link_image_to_prompt(prompt_id, str(path)) + before = self._snapshot() + cat = self._candidate(detail, "cat") + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 409, data) + self.assertEqual(data["code"], "association-conflict") + self.assertNotEqual( + data["dependency_fingerprint"], detail["dependency_fingerprint"] + ) + self.assertEqual(self._snapshot(), before) + + async def test_leave_unresolved_records_decision_and_leaves_pending_list(self): + self._require_routes() + path = self._ambiguous("two.png") + self._record(path) + self._record(self._ambiguous("other.png", color=(0, 0, 9))) + item, detail, _observed = await self._open() + body = { + "expected_epoch": self.api.db.get_execution_epoch(), + "expected_revision": 0, + "capture_token": None, + "sha256": detail["sha256"], + "dependency_fingerprint": detail["dependency_fingerprint"], + "choice": "unresolved", + } + status, data = await self._decide(item["id"], body) + self.assertEqual(status, 200, data) + self.assertEqual(data["item"]["state"], "unresolved") + self.assertEqual(data["total"], 1) + self.assertNotIn("pair", data) + (decision,) = self._decisions() + self.assertEqual( + (decision["outcome"], decision["candidate_hash"], decision["pair_id"]), + ("unresolved", None, None), + ) + conn = self.api.db.model.get_connection() + for table in ("prompt_pairs", "image_prompt_pairs", "generated_images"): + self.assertEqual( + conn.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0], 0, table + ) + listed = await self._list() + self.assertEqual(listed["total"], 1) + self.assertNotIn(item["id"], [i["id"] for i in listed["items"]]) + # The item still answers its detail, now decided. + self.assertEqual((await self._detail(item["id"]))["state"], "unresolved") + self._no_paths() + + async def test_system_overview_counts_pending_items(self): + self._require_routes() + + async def pending(): + status, data = await self._json("GET", "/prompt_manager/system/overview") + self.assertEqual(status, 200, data) + return data["needs_review"] + + self.assertEqual(await pending(), {"pending": 0}) + self._record(self._ambiguous("two.png")) + self._record(self._ambiguous("other.png", color=(0, 0, 9))) + self.assertEqual(await pending(), {"pending": 2}) + item, detail, _observed = await self._open() + body = { + "expected_epoch": self.api.db.get_execution_epoch(), + "expected_revision": 0, + "capture_token": None, + "sha256": detail["sha256"], + "dependency_fingerprint": detail["dependency_fingerprint"], + "choice": "unresolved", + } + status, data = await self._decide(item["id"], body) + self.assertEqual(status, 200, data) + # A decided item leaves the count: it is the queue's own total. + self.assertEqual(await pending(), {"pending": data["total"]}) + self.assertEqual(data["total"], 1) + + async def test_changed_bytes_or_evidence_reject_apply(self): + self._require_routes() + path = self._ambiguous("two.png") + self._record(path) + item, detail, observed = await self._open() + cat = self._candidate(detail, "cat") + before = self._snapshot() + + # The file is rewritten after the capture: the re-hash refuses it. + write_png(path, prompt=ambiguous_graph("cat", "dog"), color=(0, 255, 0)) + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual((status, data["code"]), (409, "file-changed"), data) + self.assertEqual(self._snapshot(), before) + + # A fresh capture of the new bytes is not the reviewed item's bytes. + observed = await self._observe(item["target"]) + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual((status, data["code"]), (409, "file-changed"), data) + self.assertEqual( + self._snapshot()["media_review_items"], before["media_review_items"] + ) + + # Back to the reviewed bytes, but the evidence they yield has changed. + write_png(path, prompt=ambiguous_graph("cat", "dog")) + self.assertEqual(digest(path), detail["sha256"]) + observed = await self._observe(item["target"]) + changed = { + "state": "ambiguous", + "reason": "multiple samplers", + "candidates": [], + } + with patch("utils.review_capture.resolve_prompt_pair", return_value=changed): + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual((status, data["code"]), (409, "review-conflict"), data) + + # A choice that is not one of the item's candidates is refused. + status, data = await self._decide( + item["id"], self._body(detail, observed, "f" * 64) + ) + self.assertEqual(status, 400, data) + # A restore since the capture. + status, data = await self._decide( + item["id"], + { + **self._body(detail, observed, cat["hash"]), + "expected_epoch": "00000000-0000-4000-8000-000000000000", + }, + ) + self.assertEqual((status, data["code"]), (409, "storage-changed"), data) + # A stale revision. + status, data = await self._decide( + item["id"], + {**self._body(detail, observed, cat["hash"]), "expected_revision": 3}, + ) + self.assertEqual((status, data["code"]), (409, "review-conflict"), data) + # Apply needs a capture; Leave unresolved does not. + status, data = await self._decide( + item["id"], + {**self._body(detail, observed, cat["hash"]), "capture_token": None}, + ) + self.assertEqual(status, 400, data) + # A capture of another file. + other = self._ambiguous("other.png", color=(1, 2, 3)) + self._record(other) + target = {**item["target"], "path": "other.png"} + foreign = await self._observe(target) + status, data = await self._decide( + item["id"], + { + **self._body(detail, observed, cat["hash"]), + "capture_token": foreign["capture"]["token"], + "sha256": foreign["capture"]["sha256"], + }, + ) + self.assertEqual((status, data["code"]), (409, "capture-expired"), data) + self.assertEqual(self._decisions(), []) + self.assertEqual(self._items()[0]["state"], "pending") + self._no_paths() + + async def test_decision_body_validation(self): + self._require_routes() + self._record(self._ambiguous("two.png")) + item, detail, observed = await self._open() + good = self._body(detail, observed, "unresolved") + cases = [ + {**good, "extra": 1}, + {k: v for k, v in good.items() if k != "choice"}, + {**good, "expected_epoch": "nope"}, + {**good, "expected_revision": -1}, + {**good, "expected_revision": True}, + {**good, "capture_token": "bad token!"}, + {**good, "sha256": "A" * 64}, + {**good, "dependency_fingerprint": "x"}, + {**good, "choice": "maybe"}, + {**good, "choice": 7}, + {**good, "sha256": "0" * 64}, + ] + for body in cases: + with self.subTest(body=body): + status, data = await self._decide(item["id"], body) + self.assertEqual(status, 400, data) + self.assertFalse(data["success"]) + status, data = await self._json( + "POST", f"{ITEMS_URL}/{item['id']}/decision", data=b"[1" + ) + self.assertEqual(status, 400, data) + status, data = await self._decide("00000000-0000-4000-8000-000000000000", good) + self.assertEqual(status, 404, data) + self.assertEqual(self._decisions(), []) + + async def test_duplicate_apply_after_commit_is_review_conflict_with_current_state( + self, + ): + self._require_routes() + self._record(self._ambiguous("two.png")) + item, detail, observed = await self._open() + cat = self._candidate(detail, "cat") + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 200, data) + after = self._snapshot() + observed = await self._observe(item["target"]) + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 409, data) + self.assertEqual(data["code"], "review-conflict") + self.assertEqual( + (data["item"]["state"], data["item"]["revision"]), ("selected", 1) + ) + self.assertEqual(data["total"], 0) + self.assertEqual(self._snapshot(), after) + self.assertEqual(len(self._decisions()), 1) + + async def test_tag_save_never_changes_review_item(self): + self._require_routes() + self._record(self._ambiguous("two.png")) + item, _detail, observed = await self._open() + before = self._snapshot() + status, data = await self._json( + "PUT", + "/prompt_manager/image-annotations/tags", + json={ + "target": item["target"], + "capture_token": observed["capture"]["token"], + "sha256": observed["capture"]["sha256"], + "expected_epoch": observed["library_epoch"], + "expected_tag_revision": observed["tag_revision"], + "tags": ["sunset"], + }, + ) + self.assertEqual(status, 200, data) + after = self._snapshot() + for table in ( + "media_review_items", + "media_review_decisions", + "media_review_metadata", + "generated_images", + "image_prompt_pairs", + ): + self.assertEqual(after[table], before[table], table) + self.assertEqual((await self._list())["total"], 1) + + async def test_run_and_usage_counts_unchanged(self): + self._require_routes() + path = self._ambiguous("two.png") + prompt_id = self.api.db.save_prompt( + "cat", prompt_hash=generate_prompt_hash("cat") + ) + conn = self.api.db.model.get_connection() + with conn: + conn.execute( + "UPDATE prompts SET run_count=5, last_used_at='2026-01-01T00:00:00Z' " + "WHERE id=?", + (prompt_id,), + ) + self._record(path) + executions = conn.execute( + "SELECT name FROM sqlite_master WHERE type='table' " + "AND name LIKE 'execution_%'" + ).fetchall() + counts = { + row[0]: conn.execute(f"SELECT COUNT(*) FROM {row[0]}").fetchone()[0] + for row in executions + } + item, detail, observed = await self._open() + status, data = await self._decide( + item["id"], + self._body(detail, observed, self._candidate(detail, "cat")["hash"]), + ) + self.assertEqual(status, 200, data) + rows = conn.execute( + "SELECT text, run_count, last_used_at FROM prompts" + ).fetchall() + self.assertIn(("cat", 5, "2026-01-01T00:00:00Z"), [tuple(r) for r in rows]) + self.assertEqual({tuple(r)[1] for r in rows if r[0] != "cat"}, {0}) + self.assertEqual( + { + name: conn.execute(f"SELECT COUNT(*) FROM {name}").fetchone()[0] + for name in counts + }, + counts, + ) + (pair,) = self.api.db.list_prompt_pairs()["pairs"] + self.assertEqual(pair["run_count"], 0) + + async def test_no_absolute_paths_in_responses(self): + self._require_routes() + self._record(self._ambiguous("two.png")) + item, detail, observed = await self._open() + await self._json("GET", f"{ITEMS_URL}?cursor=bad") + await self._decide(item["id"], {**self._body(detail, observed, "f" * 64)}) + with patch.object( + type(self.api.db), + "review_queue_page", + side_effect=sqlite3.OperationalError(f"no such table at {self.tmpdir}"), + ): + status, data = await self._json("GET", ITEMS_URL) + self.assertEqual((status, data["code"]), (500, "review-failed")) + with patch.object( + type(self.api.db), + "review_item", + side_effect=sqlite3.OperationalError(f"locked {self.tmpdir}"), + ): + status, data = await self._json("GET", f"{ITEMS_URL}/{item['id']}") + self.assertEqual((status, data["code"]), (500, "review-failed")) + status, data = await self._decide( + item["id"], self._body(detail, observed, "unresolved") + ) + self.assertEqual(status, 200, data) + self._no_paths() + + +class TestPhysicalIdentity(ReviewRoutesTestCase): + """Apply sees a legacy row of the same file under any spelling (I1). + + The roots are real paths, but the monitor links a file under the + directory it watches: a symlink, a Windows junction, ``subst`` or a + mapped drive reaches the same file under another spelling. + """ + + def _monitor_link(self, text, spelled): + prompt_id = self.api.db.save_prompt( + text, prompt_hash=generate_prompt_hash(text) + ) + return self.api.db.link_image_to_prompt(prompt_id, str(spelled)) + + def _rows(self): + conn = self.api.db.model.get_connection() + return [ + tuple(row) + for row in conn.execute( + "SELECT gi.image_path, p.text FROM generated_images gi " + "JOIN prompts p ON p.id=gi.prompt_id ORDER BY gi.id" + ) + ] + + async def _conflict_then_compatible(self, alias_path): + image_id = self._monitor_link("cat", alias_path) + item, detail, observed = await self._open() + before = self._snapshot() + dog = self._candidate(detail, "dog") + status, data = await self._decide( + item["id"], self._body(detail, observed, dog["hash"]) + ) + self.assertEqual((status, data.get("code")), (409, "association-conflict")) + self.assertEqual(self._snapshot(), before) + cat = self._candidate(detail, "cat") + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 200, data) + # The monitor's row is enriched; no second row for the same file. + self.assertEqual(data["pair"]["image_id"], image_id) + self.assertEqual(self._rows(), [(str(alias_path), "cat")]) + self._no_paths() + + def _symlinked_root(self): + alias = Path(self.tmpdir) / "alias-output" + try: + os.symlink(str(self.output_dir), str(alias), target_is_directory=True) + except (OSError, NotImplementedError, AttributeError) as exc: + self.skipTest(f"symlinks unavailable: {exc}") + return alias + + async def test_symlinked_root_legacy_row_is_an_association_conflict(self): + self._require_routes() + path = self._ambiguous("two.png") + alias = self._symlinked_root() + self._record(path) + await self._conflict_then_compatible(alias / "two.png") + + async def test_junction_or_mapped_drive_legacy_row_is_seen(self): + """A portable fake of a junction: its realpath is the real root.""" + self._require_routes() + path = self._ambiguous("two.png") + self._record(path) + junction = os.path.join(self.tmpdir, "Junction") + real = os.path.realpath + + def fake_realpath(value, *args, **kwargs): + text = os.fspath(value) + if os.path.normcase(text).startswith(os.path.normcase(junction)): + text = str(self.output_dir) + text[len(junction) :] + return real(text, *args, **kwargs) + + with patch.object(review_api, "_realpath", side_effect=fake_realpath): + await self._conflict_then_compatible(os.path.join(junction, "two.png")) + + async def test_two_rows_under_two_spellings_are_an_association_conflict(self): + self._require_routes() + path = self._ambiguous("two.png") + alias = self._symlinked_root() + self._monitor_link("cat", alias / "two.png") + self._monitor_link("cat", path) + self._record(path) + item, detail, observed = await self._open() + before = self._snapshot() + cat = self._candidate(detail, "cat") + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual((status, data.get("code")), (409, "association-conflict")) + self.assertEqual(self._snapshot(), before) + + async def test_same_name_in_another_folder_is_not_the_file(self): + self._require_routes() + path = self._ambiguous("two.png") + other = self.output_dir / "elsewhere" / "two.png" + other.parent.mkdir(parents=True) + write_png(other, prompt=ambiguous_graph("cat", "dog"), color=(9, 9, 9)) + self._monitor_link("dog", other) + self._record(path) + item, detail, observed = await self._open() + cat = self._candidate(detail, "cat") + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 200, data) + self.assertEqual( + sorted(self._rows()), sorted([(str(other), "dog"), (str(path), "cat")]) + ) + + async def test_a_new_alias_row_after_the_check_is_a_changed_conflict(self): + self._require_routes() + path = self._ambiguous("two.png") + alias = self._symlinked_root() + self._record(path) + item, detail, observed = await self._open() + cat = self._candidate(detail, "cat") + # The monitor links the file under its alias after the reviewer opened it. + self._monitor_link("cat", alias / "two.png") + before = self._snapshot() + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual((status, data.get("code")), (409, "association-conflict")) + self.assertNotEqual( + data["dependency_fingerprint"], detail["dependency_fingerprint"] + ) + self.assertEqual(self._snapshot(), before) + + +class TestApplyRefusals(ReviewRoutesTestCase): + """Every Apply refusal answers a code and writes nothing (I2, M1, M4).""" + + async def _opened(self): + path = self._ambiguous("two.png") + self._record(path) + item, detail, observed = await self._open() + return path, item, detail, observed, self._candidate(detail, "cat") + + async def _refused(self, item, body, status, code=None): + before = self._snapshot() + got, data = await self._decide(item["id"], body) + self.assertEqual(got, status, data) + if code is not None: + self.assertEqual(data["code"], code, data) + self.assertEqual(self._snapshot(), before) + self.assertEqual(self._decisions(), []) + return data + + async def test_resolver_change_keeps_the_chosen_candidate_applicable(self): + self._require_routes() + _path, item, detail, observed, cat = await self._opened() + real = review_capture_module.resolve_prompt_pair + + def reworded(metadata): + resolution = real(metadata) + return { + **resolution, + "reason": "a reworded reason from a newer resolver", + "candidates": list(reversed(resolution["candidates"])), + } + + with ( + patch.object( + review_capture_module, "resolve_prompt_pair", side_effect=reworded + ), + patch.object(review_capture_module, "RESOLVER_VERSION", 2), + ): + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 200, data) + self.assertEqual(data["item"]["state"], "selected") + + async def test_chosen_candidate_no_longer_offered_is_a_review_conflict(self): + self._require_routes() + _path, item, detail, observed, cat = await self._opened() + real = review_capture_module.resolve_prompt_pair + + def without_cat(metadata): + resolution = real(metadata) + kept = [ + c + for c in resolution["candidates"] + if (c.get("positive") or {}).get("text") != "cat" + ] + return {**resolution, "candidates": kept * 2} + + with patch.object( + review_capture_module, "resolve_prompt_pair", side_effect=without_cat + ): + await self._refused( + item, + self._body(detail, observed, cat["hash"]), + 409, + "review-conflict", + ) + + async def test_decision_passes_the_verified_digest(self): + self._require_routes() + path, item, detail, observed, cat = await self._opened() + real = type(self.api.db).decide_review_item + seen = [] + + def spy(db, **kwargs): + seen.append(kwargs["sha256"]) + return real(db, **kwargs) + + with patch.object(type(self.api.db), "decide_review_item", spy): + status, data = await self._decide( + item["id"], self._body(detail, observed, cat["hash"]) + ) + self.assertEqual(status, 200, data) + self.assertEqual(seen, [digest(path)]) + + async def test_expired_capture(self): + self._require_routes() + _path, item, detail, observed, cat = await self._opened() + self.api._captures.release(observed["capture"]["token"]) + await self._refused( + item, self._body(detail, observed, cat["hash"]), 409, "capture-expired" + ) + + async def test_a_candidate_that_cannot_be_applied(self): + self._require_routes() + _path, item, detail, observed, cat = await self._opened() + conn = self.api.db.model.get_connection() + (text,) = conn.execute( + "SELECT evidence FROM media_review_items WHERE id=?", (item["id"],) + ).fetchone() + evidence = json.loads(text) + for candidate in evidence["candidates"]: + if candidate["hash"] == cat["hash"]: + candidate["state"] = "unresolved" + text = canonical_evidence(evidence) + with conn: + conn.execute( + "UPDATE media_review_items SET evidence=?, evidence_hash=? WHERE id=?", + (text, evidence_digest(text), item["id"]), + ) + data = await self._refused(item, self._body(detail, observed, cat["hash"]), 400) + self.assertIn("cannot be applied", data["error"]) + + async def test_leave_unresolved_with_another_digest(self): + self._require_routes() + _path, item, detail, observed, _cat = await self._opened() + body = { + **self._body(detail, observed, "unresolved"), + "capture_token": None, + "sha256": "0" * 64, + } + data = await self._refused(item, body, 400) + self.assertIn("does not match the item", data["error"]) + + async def test_root_no_longer_configured_is_source_changed(self): + self._require_routes() + _path, item, detail, observed, cat = await self._opened() + # Another root takes index 0. + elsewhere = self.output_dir / "elsewhere" + elsewhere.mkdir() + GalleryConfig.MONITORING_DIRECTORIES = [self._root(elsewhere)] + await self._refused( + item, self._body(detail, observed, cat["hash"]), 409, "source-changed" + ) + + async def test_file_no_longer_under_its_root_is_source_changed(self): + self._require_routes() + _path, item, detail, observed, cat = await self._opened() + moved = os.path.join(self.tmpdir, "moved-root") + real = os.path.realpath + + def other_root(value, *args, **kwargs): + if os.path.normcase(os.fspath(value)) == os.path.normcase( + str(self.output_dir) + ): + return moved + return real(value, *args, **kwargs) + + with patch.object(review_api, "_realpath", side_effect=other_root): + await self._refused( + item, self._body(detail, observed, cat["hash"]), 409, "source-changed" + ) + + async def test_verify_failures(self): + self._require_routes() + _path, item, detail, observed, cat = await self._opened() + store = self.api._captures + body = self._body(detail, observed, cat["hash"]) + with patch.object(store, "verify", side_effect=CaptureChanged("gone")): + await self._refused(item, body, 409, "file-changed") + other = FileFacts(item_locator := self._items()[0]["locator"], "1" * 64, 1, 1) + with patch.object(store, "verify", return_value=other): + await self._refused(item, body, 409, "file-changed") + foreign = FileFacts(item_locator + "x", detail["sha256"], 1, 1) + with patch.object(store, "verify", return_value=foreign): + data = await self._refused(item, body, 409, "capture-expired") + self.assertIn("another file", data["error"]) + + +class TestInteractiveCapacity(ReviewRoutesTestCase): + async def test_background_recording_leaves_a_worker_for_the_tag_dialog(self): + """A scan or monitor recording in flight never starves an observe (I4).""" + self._require_routes() + self._record(self._ambiguous("two.png")) + busy = self._ambiguous("busy.png", color=(4, 5, 6)) + item = (await self._list())["items"][0] + store = self.api._captures + entered, proceed = threading.Event(), threading.Event() + original = CaptureStore._read + outcome = {} + + def blocking(store_, stream, size): + if threading.current_thread() is recorder and not entered.is_set(): + entered.set() + proceed.wait(30) + return original(store_, stream, size) + + def record(): + outcome["value"] = self._record_raw(str(busy)) + + recorder = threading.Thread(target=record) + with patch.object(CaptureStore, "_read", blocking): + recorder.start() + try: + self.assertTrue(entered.wait(30)) + self.assertEqual(store.stats()["workers"], 1) + # Another background recording finds no worker it may use... + with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01): + with self.assertRaises(CaptureCapacity): + self._record_raw(str(busy)) + # ...while the interactive observe still gets the last one. + observed = await self._observe(item["target"]) + self.assertEqual( + observed["capture"]["sha256"], digest(self.output_dir / "two.png") + ) + finally: + proceed.set() + recorder.join(30) + self.assertFalse(recorder.is_alive()) + self.assertEqual(outcome["value"], "recorded") diff --git a/tests/test_database_backup.py b/tests/test_database_backup.py index 9368697..7eddca7 100644 --- a/tests/test_database_backup.py +++ b/tests/test_database_backup.py @@ -466,6 +466,9 @@ class TestRestoreFromFile(BackupTestCase): def test_restore_old_backup_without_annotations(self): tables = ( + "media_review_decisions", + "media_review_items", + "media_review_metadata", "media_image_tags", "media_annotation_bytes", "media_annotation_files", @@ -502,6 +505,70 @@ class TestRestoreFromFile(BackupTestCase): 0, ) + def test_restore_annotations_era_backup_without_review_queue(self): + tables = ( + "media_review_decisions", + "media_review_items", + "media_review_metadata", + ) + src = self._make_source_db(["predates the review queue"]) + with closing(sqlite3.connect(src)) as conn: + for table in tables: + conn.execute(f"DROP TABLE {table}") + conn.commit() + self.assertTrue(PromptModel.verify_database_file(src)[0]) + ok, reason = PromptModel.verify_database_file(src, require_library_state=True) + self.assertFalse(ok) + self.assertEqual(reason, "Missing review queue schema") + + self.db.model.restore_from_file(src) + + self.assertEqual(_texts(self.path), ["predates the review queue"]) + with closing(sqlite3.connect(self.path)) as conn: + self.assertEqual( + conn.execute("SELECT * FROM media_review_metadata").fetchall(), + [(1, 1, 0)], + ) + self.assertEqual( + conn.execute("SELECT COUNT(*) FROM media_review_items").fetchone()[0], + 0, + ) + ok, reason = PromptModel.verify_database_file( + self.path, require_library_state=True + ) + self.assertTrue(ok, reason) + with closing(sqlite3.connect(src)) as conn: # the source is untouched + self.assertEqual( + conn.execute( + "SELECT COUNT(*) FROM sqlite_master WHERE name LIKE 'media_review%'" + ).fetchone()[0], + 0, + ) + + def test_restore_rejects_review_queue_without_annotations(self): + src = self._make_source_db(["review without annotations"]) + with closing(sqlite3.connect(src)) as conn: + conn.execute("PRAGMA foreign_keys=OFF") + for table in ( + "media_image_tags", + "media_annotation_bytes", + "media_annotation_files", + "media_annotation_metadata", + ): + conn.execute(f"DROP TABLE {table}") + conn.commit() + before = Path(src).read_bytes() + live_before = _texts(self.path) + ok, reason = PromptModel.verify_database_file(src) + self.assertFalse(ok) + self.assertIn("image annotation", reason) + + with self.assertRaises(ValueError): + self.db.model.restore_from_file(src) + + self.assertEqual(Path(src).read_bytes(), before) + self.assertEqual(_texts(self.path), live_before) + def test_restore_preserves_favorites_and_closes_handles(self): self.db.set_image_favorite("output", "live.png") with tempfile.TemporaryDirectory() as directory: diff --git a/tests/test_library.py b/tests/test_library.py index 242b76d..ef7566c 100644 --- a/tests/test_library.py +++ b/tests/test_library.py @@ -1744,6 +1744,7 @@ class LibraryCounts(unittest.TestCase): "negative_variants": 0, "images": 0, "rated_images": 0, + "review_pending": 0, }, ) diff --git a/tests/test_media_capture.py b/tests/test_media_capture.py index 53f3fdb..f4cc0b5 100644 --- a/tests/test_media_capture.py +++ b/tests/test_media_capture.py @@ -471,6 +471,94 @@ class CaptureStoreTestCase(unittest.TestCase): self.assertEqual(results["capture"].size, 60) self.capture(workers, "b.png") + def _hold_worker(self, store, relative, **options): + """Start a capture that holds a worker until ``proceed`` is set.""" + entered, proceed = threading.Event(), threading.Event() + original = CaptureStore._read + results = {} + + def blocking(store_, stream, size): + if threading.current_thread() is thread and not entered.is_set(): + entered.set() + self.assertTrue(proceed.wait(WAIT)) + return original(store_, stream, size) + + def worker(): + try: + results["capture"] = self.capture(store, relative, **options) + except Exception as error: # pragma: no cover - reported below + results["error"] = error + + patcher = patch.object(CaptureStore, "_read", blocking) + patcher.start() + self.addCleanup(patcher.stop) + thread = threading.Thread(target=worker) + thread.start() + self.assertTrue(entered.wait(WAIT)) + + def finish(): + proceed.set() + thread.join(WAIT) + self.assertFalse(thread.is_alive()) + self.assertNotIn("error", results) + return results["capture"] + + self.addCleanup(proceed.set) + return finish + + def test_background_capture_never_takes_the_last_worker(self): + for name in ("a", "b", "c"): + self.write(f"{name}.png", bytes(60)) + store = self.make_store(max_workers=2) + finish = self._hold_worker(store, "a.png", background=True) + # The second background capture waits, then fails: one worker is free + # but it is the interactive request's. + started = time.monotonic() + with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.05): + with self.assertRaises(CaptureCapacity): + self.capture(store, "b.png", background=True) + self.assertGreaterEqual(time.monotonic() - started, 0.04) + # An interactive capture (Tag dialog, Apply) still gets the last one. + interactive = self.capture(store, "c.png") + self.assertEqual(interactive.size, 60) + finish() + + def test_background_capture_waits_for_a_worker_to_free(self): + for name in ("a", "b"): + self.write(f"{name}.png", bytes(60)) + store = self.make_store(max_workers=2) + finish = self._hold_worker(store, "a.png", background=True) + waited = {} + + def background(): + waited["capture"] = self.capture(store, "b.png", background=True) + + thread = threading.Thread(target=background) + with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", WAIT): + thread.start() + finish() + thread.join(WAIT) + self.assertFalse(thread.is_alive()) + self.assertEqual(waited["capture"].size, 60) + + def test_single_worker_store_admits_no_background_capture(self): + self.write("a.png", bytes(60)) + store = self.make_store(max_workers=1) + with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01): + with self.assertRaises(CaptureCapacity): + self.capture(store, "a.png", background=True) + self.assertEqual(self.capture(store, "a.png").size, 60) + + def test_background_capture_leaves_the_last_capture_slot(self): + for name in ("a", "b", "c"): + self.write(f"{name}.png", bytes(60)) + store = self.make_store(max_captures=2) + self.capture(store, "a.png") + with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01): + with self.assertRaises(CaptureCapacity): + self.capture(store, "b.png", background=True) + self.assertEqual(self.capture(store, "c.png").size, 60) + # -- lifetime ------------------------------------------------------------ def test_idle_and_hard_expiry_and_in_use_retention(self): diff --git a/tests/test_pair_ingestion.py b/tests/test_pair_ingestion.py index 4cfdfa9..fab7ef9 100644 --- a/tests/test_pair_ingestion.py +++ b/tests/test_pair_ingestion.py @@ -160,3 +160,69 @@ class TestPairIngestion(unittest.TestCase): info = self.db.get_image_prompt_info(self.path) self.assertEqual(info["pair_context"]["positive_text"], "cat, lora_trigger") self.assertEqual(info["pair_context"]["text_source"], "execution") + + def test_public_ingest_resolved_pair_contract_unchanged(self): + """The public ingester keeps its result shape and its own transaction; + the extracted ``_ingest_resolved_pair_in`` runs in the caller's.""" + from database import pair_ingestion + + self.assertTrue( + hasattr(pair_ingestion, "_ingest_resolved_pair_in"), + "_ingest_resolved_pair_in is not extracted", + ) + for state in ("ambiguous", "unresolved", "conflict"): + self.assertEqual( + ingest_resolved_pair(self.db, self.path, {}, {"state": state}), + {"status": state, "pairs_added": 0, "links_added": 0}, + ) + first = ingest_resolved_pair(self.db, self.path, {}, resolved()) + self.assertEqual( + set(first), + {"status", "pairs_added", "links_added", "pair_id", "image_id"}, + ) + self.assertEqual(first["status"], "added") + (link,) = self.db.get_image_prompt_pairs(first["image_id"]) + self.assertEqual( + link["provenance"], + { + "resolver_version": 1, + "positive": resolved()["positive"], + "negative": resolved()["negative"], + }, + ) + conn = self.db.model.get_connection() + self.assertFalse(conn.in_transaction) + # Inside the caller's transaction, with the caller's provenance; an + # exception afterwards rolls the link back with the caller's writes. + provenance = {"kind": "user-selection", "resolver_version": 1} + with self.assertRaises(RuntimeError): + with self.db._pair_transaction() as held: + inner = pair_ingestion._ingest_resolved_pair_in( + held, + self.db, + self.path + "2", + {}, + resolved("dog"), + provenance=provenance, + ) + self.assertEqual(inner["status"], "added") + raise RuntimeError("caller failed") + self.assertIsNone(self.db.get_image_prompt_info(self.path + "2")) + with self.db._pair_transaction() as held: + inner = pair_ingestion._ingest_resolved_pair_in( + held, + self.db, + self.path + "2", + {}, + resolved("dog"), + provenance=provenance, + ) + (link,) = self.db.get_image_prompt_pairs(inner["image_id"]) + self.assertEqual(link["provenance"], provenance) + with self.db._pair_transaction() as held: + conflict = pair_ingestion._ingest_resolved_pair_in( + held, self.db, self.path, {}, resolved("bird") + ) + self.assertEqual( + conflict, {"status": "conflict", "pairs_added": 0, "links_added": 0} + ) diff --git a/tests/test_pair_scan.py b/tests/test_pair_scan.py index b8d914d..1597b1f 100644 --- a/tests/test_pair_scan.py +++ b/tests/test_pair_scan.py @@ -82,7 +82,8 @@ class TestPairScan(AdminAPITestCase): "class_type": "BasicGuider", "inputs": {"width": 2048, "height": 2048}, } - self.assertEqual(self._scanned("untraceable.png", g), "ambiguous") + # Ambiguous with nothing to choose: unresolved, never a review item. + self.assertEqual(self._scanned("untraceable.png", g), "unresolved") self.assertEqual(self.api.db.list_prompt_pairs()["total"], 0) self.assertEqual(self._texts(), []) @@ -159,9 +160,10 @@ class TestPairScan(AdminAPITestCase): "\nSteps: 20, Sampler: Euler" ) } + # No candidate to choose: reported unresolved, still not guessed. self.assertEqual( self.api._ingest_scanned_file(self.output_dir / "ambiguous.png", meta), - "ambiguous", + "unresolved", ) self.assertEqual(self.api.db.get_recent_prompts()["prompts"], []) diff --git a/tests/test_pair_schema_restore.py b/tests/test_pair_schema_restore.py index dff5420..24b1e87 100644 --- a/tests/test_pair_schema_restore.py +++ b/tests/test_pair_schema_restore.py @@ -258,6 +258,10 @@ class PairSchemaTests(unittest.TestCase): ], [(1, 1, 0)], ) + self.assertEqual( + [tuple(r) for r in conn.execute("SELECT * FROM media_review_metadata")], + [(1, 1, 0)], + ) self.assertTrue(PromptModel.verify_database_file(live.db_path)[0]) self.assertTrue( PromptModel.verify_database_file(live.db_path, require_library_state=True)[ diff --git a/tests/test_review.py b/tests/test_review.py new file mode 100644 index 0000000..70776c0 --- /dev/null +++ b/tests/test_review.py @@ -0,0 +1,1053 @@ +"""Needs review items (#249 E2): capture-bound ambiguous evidence. + +The scan and the live monitor persist one review item per ambiguous file: +the file is captured, its metadata is read again from the capture, the +default (ingestion) resolver runs on it, and only an ``ambiguous`` result is +stored, bound to the capture's byte record. Statuses, counters and links of +both producers stay what they were. +""" + +import asyncio +import hashlib +import json +import logging +import contextlib +import os +import sqlite3 +import unittest +from pathlib import Path +from unittest.mock import patch + +try: + from tests.open_handles import assert_closed + from tests.test_api_admin import AdminAPITestCase, _sse_payloads +except ImportError: # discovered with tests/ as the top-level directory + from open_handles import assert_closed + from test_api_admin import AdminAPITestCase, _sse_payloads + +from PIL import Image, PngImagePlugin + +from database.operations import PromptDatabase +from database.review_schema import canonical_evidence, evidence_digest +from py.api import PromptManagerAPI +from py.config import GalleryConfig +from utils.hashing import generate_prompt_hash +from utils.image_monitor import ImageGenerationHandler +from utils.media_capture import CaptureCapacity, CaptureError, CaptureStore +from utils.prompt_tracker import PromptTracker + +try: # guarded so a missing module fails assertions, not test loading + from utils import review_capture +except ImportError: + review_capture = None + +NOW = "2026-10-22T00:00:00+00:00" +# Two negative prompt delimiters: the A1111 record is ambiguous. +DELIMITERS = ( + "cat\nNegative prompt: blur\nNegative prompt: noise\nSteps: 20, Sampler: Euler" +) + + +def _sampler(positive, negative="bad"): + return {"positive": positive, "negative": negative} + + +def ambiguous_graph(*positives): + """One KSampler per positive text, each with its own encoder: ambiguous.""" + graph = { + "1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}}, + "2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}}, + } + for index, text in enumerate(positives or ("cat", "dog")): + encoder, sampler = str(10 + 2 * index), str(11 + 2 * index) + graph[encoder] = {"class_type": "CLIPTextEncode", "inputs": {"text": text}} + graph[sampler] = { + "class_type": "KSampler", + "inputs": { + "model": ["1", 0], + "positive": [encoder, 0], + "negative": ["2", 0], + }, + } + return graph + + +def unlinkable_graph(*positives): + """Ambiguous, and the scan's browser-parser fallback links nothing. + + Only such a file is queued (an ambiguous file the fallback links is not). + The parser reads the first sampler (node 3), whose positive input points + at a node missing from the graph; the resolver offers the other samplers + as candidates beside that unresolved one. + """ + graph = ambiguous_graph(*positives) + graph["3"] = { + "class_type": "KSampler", + "inputs": {"model": ["1", 0], "positive": ["99", 0], "negative": ["2", 0]}, + } + return graph + + +def resolved_texts(evidence): + """The positive texts of the candidates that resolved, sorted.""" + return sorted( + c["positive"]["text"] for c in evidence["candidates"] if c["positive"] + ) + + +def resolved_graph(text="a red square"): + return ambiguous_graph(text) + + +def write_png(path, *, prompt=None, parameters=None, color=(255, 0, 0)): + info = PngImagePlugin.PngInfo() + if prompt is not None: + info.add_text("prompt", json.dumps(prompt)) + if parameters is not None: + info.add_text("parameters", parameters) + Image.new("RGB", (2, 2), color).save(str(path), pnginfo=info) + + +def _reject_constant(name): + raise ValueError(f"non-strict JSON constant {name}") + + +def digest(path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +class ReviewTestCase(AdminAPITestCase): + """Scan helpers over a capture store kept inside the test directory.""" + + async def get_application(self): + app = await super().get_application() + self.capture_parent = Path(self.tmpdir) / "captures" + self.api._captures = CaptureStore(temp_parent=self.capture_parent) + return app + + async def tearDownAsync(self): + self.api.close_annotation_captures() + await super().tearDownAsync() + + def _roots(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + + async def _scan(self, scope="new"): + resp = await self.client.request( + "POST", "/prompt_manager/jobs/scan", json={"scope": scope} + ) + data = await resp.json() + self.assertEqual(resp.status, 202, data) + await asyncio.wait_for(self.api._scan_job.task, 20) + detail = await self.client.request( + "GET", f"/prompt_manager/jobs/{data['job']['id']}" + ) + return (await detail.json())["job"] + + def _items(self): + conn = self.api.db.model.get_connection() + return [ + dict(row) + for row in conn.execute( + "SELECT i.*, b.sha256, b.size, b.mtime_ns, f.locator " + "FROM media_review_items i " + "JOIN media_annotation_bytes b ON b.id=i.byte_id " + "JOIN media_annotation_files f ON f.id=b.file_id " + "ORDER BY i.created_at, i.id" + ) + ] + + def _review_revision(self): + conn = self.api.db.model.get_connection() + return conn.execute( + "SELECT review_index_revision FROM media_review_metadata" + ).fetchone()[0] + + def _capture_files(self): + if not self.capture_parent.exists(): + return [] + return [p for p in self.capture_parent.rglob("*") if p.is_file()] + + def _require_module(self): + self.assertIsNotNone(review_capture, "utils.review_capture is not implemented") + + +class TestScanRecording(ReviewTestCase): + async def test_scan_persists_only_ambiguous_files_with_capture_digest(self): + self._require_module() + self._roots() + ambiguous = self.output_dir / "sub" / "two.png" + ambiguous.parent.mkdir() + write_png(ambiguous, prompt=unlinkable_graph("cat", "dog")) + # Ambiguous too, but the browser-parser fallback links it: not queued. + write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b")) + write_png(self.output_dir / "resolved.png", prompt=resolved_graph()) + write_png(self.output_dir / "plain.png") + write_png( + self.output_dir / "a1111.png", + parameters="cat\nNegative prompt: blur\nSteps: 20, Sampler: Euler", + ) + + job = await self._scan() + + items = self._items() + self.assertEqual(len(items), 1, items) + item = items[0] + self.assertEqual(item["sha256"], digest(ambiguous)) + self.assertEqual(item["size"], ambiguous.stat().st_size) + self.assertEqual(item["mtime_ns"], ambiguous.stat().st_mtime_ns) + self.assertEqual( + item["locator"], os.path.normcase(os.path.realpath(str(ambiguous))) + ) + self.assertEqual( + (item["state"], item["revision"], item["limited"]), ("pending", 0, 0) + ) + self.assertEqual(item["resolver_version"], 1) + self.assertEqual(item["candidate_count"], 3) + self.assertEqual(item["evidence_hash"], evidence_digest(item["evidence"])) + evidence = json.loads(item["evidence"]) + self.assertEqual(canonical_evidence(evidence), item["evidence"]) + self.assertEqual(evidence["resolver_version"], 1) + self.assertEqual(resolved_texts(evidence), ["cat", "dog"]) + for candidate in evidence["candidates"]: + self.assertEqual( + set(candidate), + {"hash", "sampler_id", "state", "reason", "positive", "negative"}, + ) + if candidate["positive"] is None: # the parser's unresolved sampler + continue + self.assertEqual( + set(candidate["positive"]), {"text", "storage_text", "provenance"} + ) + self.assertEqual(candidate["negative"]["text"], "bad") + body = {k: v for k, v in candidate.items() if k != "hash"} + self.assertEqual( + candidate["hash"], + hashlib.sha256(canonical_evidence(body).encode()).hexdigest(), + ) + self.assertEqual(self._review_revision(), 1) + self.assertEqual(job["scope"]["review_items"], 1) + self.assertIn("? 1 image added to Needs review", "\n".join(job["log"])) + self.assertEqual(self._capture_files(), []) + self.assertEqual(self.api._captures.stats()["captures"], 0) + + async def test_scan_statuses_counters_and_outcome_rows_unchanged(self): + self._require_module() + self._roots() + write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog")) + write_png(self.output_dir / "one.png", prompt=resolved_graph()) + write_png( + self.output_dir / "delims.png", + parameters=DELIMITERS, + ) + write_png(self.output_dir / "none.png") + files = sorted(self.output_dir.iterdir()) + roots = self._scan_roots() + + def run(api, recorder=None): + batch = api._extract_batch_metadata_sync(files) + outcomes = [] + if recorder is None: + counts = api._ingest_scan_batch_sync(batch, roots, outcomes) + else: + from py.api import review_recording + + with patch.object(review_recording, "record_ambiguous_file", recorder): + counts = api._ingest_scan_batch_sync(batch, roots, outcomes) + prompts = api.db.get_recent_prompts()["prompts"] + return counts, outcomes, sorted(p["text"] for p in prompts) + + other_db = PromptDatabase(os.path.join(self.tmpdir, "other.db")) + other = PromptManagerAPI(db=other_db) + + def broken(*args, **kwargs): + raise RuntimeError("recording failed") + + recorded = run(self.api) + with self.assertLogs("prompt_manager.api", logging.WARNING) as logs: + without = run(other, broken) + self.assertEqual(recorded, without) + # delims.png offers nothing to choose: unresolved, not queued. + self.assertEqual(recorded[0]["ambiguous"], 1) + self.assertEqual(recorded[0]["unresolved"], 1) + self.assertEqual(recorded[0]["added"], 1) + self.assertTrue(any("Needs review" in line for line in logs.output)) + self.assertEqual(len(self._items()), 1) + other.close_annotation_captures() + other_db.close_all() + + def _scan_roots(self): + from py.api.admin import _scan_roots + + return _scan_roots(self.api._get_all_output_dirs()) + + async def test_new_scope_rereads_ambiguous_files_and_stays_idempotent(self): + self._require_module() + self._roots() + write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog")) + write_png(self.output_dir / "one.png", prompt=resolved_graph()) + first = await self._scan("new") + revision = self._review_revision() + reads = [] + original = self.api._extract_comfyui_metadata + + def spy(path): + reads.append(os.path.basename(path)) + return original(path) + + self.api._extract_comfyui_metadata = spy + second = await self._scan("new") + + self.assertEqual(reads, ["two.png"]) + self.assertEqual(first["scope"]["review_items"], 1) + # Still in the queue (review_items), but nothing new was added. + self.assertEqual(second["scope"]["review_items"], 1) + self.assertEqual(second["scope"]["review_added"], 0) + self.assertNotIn("added to Needs review", "\n".join(second["log"])) + self.assertIn("? 1 image already in Needs review", "\n".join(second["log"])) + self.assertEqual(len(self._items()), 1) + self.assertEqual(self._review_revision(), revision) + self.assertEqual(self._capture_files(), []) + + async def test_queued_fingerprint_skips_the_copy_and_hash(self): + """A rescan of a queued file whose size and mtime match its item's + verified fingerprint neither copies nor hashes it, and the file stays + in the default "new" scope (it is still read like any unknown file).""" + self._require_module() + self._roots() + path = self.output_dir / "two.png" + write_png(path, prompt=unlinkable_graph("cat", "dog")) + await self._scan("new") + captures = [] + original = CaptureStore.capture + + def spy(store, root, relative, **kwargs): + captures.append(relative) + return original(store, root, relative, **kwargs) + + reads = [] + extract = self.api._extract_comfyui_metadata + + def read_spy(file_path): + reads.append(os.path.basename(file_path)) + return extract(file_path) + + self.api._extract_comfyui_metadata = read_spy + with patch.object(CaptureStore, "capture", spy): + second = await self._scan("new") + self.assertEqual(captures, []) + self.assertEqual(reads, ["two.png"]) + self.assertEqual(second["scope"]["review_added"], 0) + self.assertEqual(second["scope"]["review_items"], 1) + # A touched file (same bytes, new mtime) is captured again: a + # duplicate of its item, whose fingerprint is refreshed. + stat = path.stat() + os.utime(path, ns=(stat.st_atime_ns, stat.st_mtime_ns + 10**9)) + await self._scan("new") + self.assertEqual(captures, ["two.png"]) + items = self._items() + self.assertEqual(len(items), 1) + self.assertEqual(items[0]["mtime_ns"], path.stat().st_mtime_ns) + self.assertEqual(self._capture_files(), []) + + async def test_repeat_scan_keeps_decisions(self): + self._require_module() + self._roots() + write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog")) + await self._scan("all") + item = self._items()[0] + candidate = json.loads(item["evidence"])["candidates"][0]["hash"] + conn = self.api.db.model.get_connection() + with conn: + conn.execute( + "UPDATE media_review_items SET state='selected', revision=1 WHERE id=?", + (item["id"],), + ) + conn.execute( + "INSERT INTO media_review_decisions(id,item_id,decision_revision," + "created_at," + "outcome,candidate_hash,evidence) VALUES(?,?,?,?,?,?,?)", + ( + "00000000-0000-4000-8000-0000000000d1", + item["id"], + 1, + NOW, + "selected", + candidate, + '{"kind":"user-selection"}', + ), + ) + + job = await self._scan("all") + + items = self._items() + self.assertEqual(len(items), 1) + self.assertEqual((items[0]["state"], items[0]["revision"]), ("selected", 1)) + self.assertEqual(items[0]["updated_at"], item["updated_at"]) + decisions = conn.execute("SELECT id FROM media_review_decisions").fetchall() + self.assertEqual(len(decisions), 1) + self.assertEqual(job["scope"]["review_items"], 0) + + async def test_changed_bytes_create_new_pending_item_and_keep_old(self): + self._require_module() + self._roots() + path = self.output_dir / "two.png" + write_png(path, prompt=unlinkable_graph("cat", "dog")) + await self._scan() + old = self._items()[0] + write_png(path, prompt=unlinkable_graph("cat", "dog"), color=(0, 0, 255)) + self.assertNotEqual(digest(path), old["sha256"]) + + job = await self._scan() + + items = self._items() + self.assertEqual(len(items), 2) + self.assertEqual(items[0]["id"], old["id"]) + self.assertEqual(items[0]["sha256"], old["sha256"]) + self.assertEqual(items[1]["sha256"], digest(path)) + self.assertEqual(items[1]["state"], "pending") + self.assertEqual(items[0]["locator"], items[1]["locator"]) + self.assertEqual(job["scope"]["review_items"], 1) + + async def test_many_candidates_are_limited_without_candidates(self): + """Only ``many.png`` is queued: the A1111 delimiter ambiguity offers + no candidate to choose (``tests/test_review_choosable.py``).""" + self._require_module() + self._roots() + write_png( + self.output_dir / "delims.png", + parameters=DELIMITERS, + ) + many = [f"prompt {n}" for n in range(17)] + write_png(self.output_dir / "many.png", prompt=unlinkable_graph(*many)) + + job = await self._scan() + + items = {Path(item["locator"]).name: item for item in self._items()} + self.assertEqual(set(items), {"many.png"}) + for item in items.values(): + self.assertEqual((item["limited"], item["candidate_count"]), (1, 0)) + evidence = json.loads(item["evidence"]) + self.assertEqual(evidence["candidates"], []) + self.assertIs(evidence["limited"], True) + self.assertEqual( + # 17 samplers and the parser's unresolved one. + json.loads(items["many.png"]["evidence"])["candidate_total"], + 18, + ) + self.assertEqual(job["scope"]["review_items"], 1) + + async def test_capacity_errors_are_logged_and_counted_not_fatal(self): + self._require_module() + self._roots() + write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog")) + + def full(*args, **kwargs): + raise CaptureCapacity("Too many captures are in progress") + + with patch.object(self.api._captures, "capture", side_effect=full): + job = await self._scan() + + self.assertEqual(job["state"], "finished") + # Nothing links this ambiguous file: Activity counts it under review. + self.assertEqual(job["counters"]["review"], 1) + self.assertEqual(job["scope"]["review_items"], 0) + self.assertEqual(job["scope"]["review_errors"], 1) + self.assertIn( + "! 1 image could not be added to Needs review", "\n".join(job["log"]) + ) + self.assertEqual(self._items(), []) + + async def test_legacy_scan_events_are_unchanged(self): + self._require_module() + self._roots() + write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog")) + resp = await self.client.request("POST", "/prompt_manager/scan") + payloads = _sse_payloads(await resp.text()) + final = payloads[-1] + self.assertEqual(final["type"], "complete") + self.assertNotIn("review_items", final) + self.assertEqual(len(self._items()), 1) + + +class TestNonFiniteGraphs(ReviewTestCase): + """ComfyUI stamps NaN/Infinity into graphs; review stores them as null.""" + + @staticmethod + def _nan_stamped(*positives): + graph = ambiguous_graph(*positives) + for node in graph.values(): + node["is_changed"] = [float("nan")] + if node["class_type"] == "KSampler": + node["inputs"].update({"cfg": float("inf"), "denoise": float("-inf")}) + return graph + + async def test_scan_records_a_nan_stamped_ambiguous_graph(self): + self._require_module() + self._roots() + path = self.output_dir / "nan.png" + # json.dumps' default writes the bare NaN/Infinity literals, as ComfyUI. + write_png(path, prompt=self._nan_stamped("cat NaN", "dog Infinity")) + self.assertIn(b"NaN", path.read_bytes()) + + job = await self._scan() + + self.assertEqual(job["scope"]["review_items"], 1, job) + (item,) = self._items() + self.assertEqual((item["limited"], item["candidate_count"]), (0, 2)) + evidence = json.loads(item["evidence"], parse_constant=_reject_constant) + self.assertEqual(canonical_evidence(evidence), item["evidence"]) + self.assertEqual( + sorted(c["positive"]["text"] for c in evidence["candidates"]), + ["cat NaN", "dog Infinity"], + ) + + async def test_record_review_file_records_a_nan_stamped_graph(self): + self._require_module() + self._roots() + path = self.output_dir / "nan.png" + write_png(path, prompt=self._nan_stamped("cat", "dog")) + + self.assertEqual(self.api.record_review_file(str(path)), "recorded") + (item,) = self._items() + self.assertEqual(item["candidate_count"], 2) + + def test_canonical_evidence_writes_non_finite_numbers_as_null(self): + value = {"b": [float("nan"), float("inf")], "a": -float("inf"), "t": "NaN"} + text = canonical_evidence(value) + self.assertEqual(text, '{"a":null,"b":[null,null],"t":"NaN"}') + self.assertEqual(canonical_evidence(json.loads(text)), text) + + +class TestRecordAmbiguousFile(ReviewTestCase): + def _record(self, relative): + return review_capture.record_ambiguous_file( + self.api.db, self.api._captures, self.output_dir, 0, relative + ) + + async def test_record_outcomes(self): + self._require_module() + write_png(self.output_dir / "two.png", prompt=ambiguous_graph("cat", "dog")) + write_png(self.output_dir / "one.png", prompt=resolved_graph()) + write_png( + self.output_dir / "delims.png", + parameters=DELIMITERS, + ) + self.assertEqual(self._record("two.png"), "recorded") + self.assertEqual(self._record("two.png"), "duplicate") + self.assertEqual(self._record("one.png"), "skipped") + self.assertEqual(self._record("missing.png"), "skipped") + self.assertEqual(self._record("delims.png"), "untraced") + many = [f"prompt {n}" for n in range(17)] + write_png(self.output_dir / "many.png", prompt=unlinkable_graph(*many)) + self.assertEqual(self._record("many.png"), "limited") + self.assertEqual(self._record("many.png"), "duplicate") + self.assertEqual(self._review_revision(), 2) + + async def test_restore_between_capture_and_write_skips(self): + self._require_module() + write_png(self.output_dir / "two.png", prompt=ambiguous_graph("cat", "dog")) + with patch.object( + self.api.db, "get_execution_epoch", return_value="stale-epoch" + ): + self.assertEqual(self._record("two.png"), "skipped") + self.assertEqual(self._items(), []) + + async def test_oversize_evidence_is_limited(self): + self._require_module() + big = "x" * (4 * 1024 * 1024 + 10) + side = {"text": big, "storage_text": big, "provenance": {}} + resolution = { + "state": "ambiguous", + "reason": "Multiple sampler branches", + "candidates": [ + { + "state": "resolved", + "sampler_id": "3", + "reason": "", + "positive": side, + "negative": {"text": "", "storage_text": "", "provenance": {}}, + "candidates": [], + } + ], + } + evidence, count, limited = review_capture.review_evidence(resolution) + self.assertEqual((count, limited), (0, True)) + self.assertEqual(evidence["candidates"], []) + self.assertLess(len(canonical_evidence(evidence)), 1024) + + async def test_nested_ambiguous_candidates_are_flattened_and_deduplicated(self): + self._require_module() + + def side(text): + return {"text": text, "storage_text": text, "provenance": {"n": text}} + + def leaf(sampler, text): + return { + "state": "resolved", + "sampler_id": sampler, + "reason": "", + "positive": side(text), + "negative": side("bad"), + "candidates": [], + "reason_code": None, + } + + resolution = { + "state": "ambiguous", + "reason": "Multiple sampler branches", + "candidates": [ + leaf("3", "cat"), + { + "state": "ambiguous", + "sampler_id": None, + "reason": "stages", + "positive": None, + "negative": None, + "candidates": [leaf("5", "dog"), leaf("3", "cat")], + }, + ], + } + evidence, count, limited = review_capture.review_evidence(resolution) + self.assertEqual((count, limited), (2, False)) + self.assertEqual( + [c["positive"]["text"] for c in evidence["candidates"]], ["cat", "dog"] + ) + + async def test_no_inference_or_model_access_during_recording(self): + self._require_module() + write_png(self.output_dir / "two.png", prompt=ambiguous_graph("cat", "dog")) + from py import autotag + + with ( + patch.object(autotag.AutoTagService, "load_model") as load, + patch.object(autotag.AutoTagService, "download_model") as download, + patch.object(autotag, "get_autotag_service") as service, + patch.object( + Image.Image, "load", autospec=True, side_effect=AssertionError("decode") + ) as decode, + ): + self.assertEqual(self._record("two.png"), "recorded") + load.assert_not_called() + download.assert_not_called() + service.assert_not_called() + decode.assert_not_called() + + async def test_capture_released_and_handles_closed(self): + self._require_module() + write_png(self.output_dir / "two.png", prompt=ambiguous_graph("cat", "dog")) + self.assertEqual(self._record("two.png"), "recorded") + self.assertEqual(self.api._captures.stats()["captures"], 0) + self.assertEqual(self._capture_files(), []) + write_png(self.output_dir / "three.png", prompt=ambiguous_graph("a", "b")) + with patch.object( + review_capture, + "resolve_prompt_pair", + side_effect=RuntimeError("resolver failed"), + ): + with self.assertRaises(RuntimeError): + self._record("three.png") + self.assertEqual(self.api._captures.stats()["captures"], 0) + self.assertEqual(self._capture_files(), []) + assert_closed(self, self.output_dir, self.capture_parent) + + +class TestReviewEdges(ReviewTestCase): + def _side(self, text): + return {"text": text, "storage_text": text, "provenance": {}} + + def _leaf(self, sampler, text): + return { + "state": "resolved", + "sampler_id": sampler, + "reason": "", + "positive": self._side(text), + "negative": self._side(""), + "candidates": [], + } + + async def test_evidence_over_the_bound_together_is_limited(self): + self._require_module() + half = "y" * (2 * 1024 * 1024 + 10) # twice per side, twice per item + resolution = { + "state": "ambiguous", + "reason": "Multiple sampler branches", + "candidates": [self._leaf("3", half), self._leaf("5", half + "z")], + } + evidence, count, limited = review_capture.review_evidence(resolution) + self.assertEqual((count, limited, evidence["candidate_total"]), (0, True, 2)) + + async def test_malformed_candidates_are_ignored_or_limited(self): + self._require_module() + self.assertEqual( + review_capture.review_evidence({"state": "ambiguous", "candidates": None}), + ( + { + "resolver_version": 1, + "reason": "", + "limited": True, + "candidate_total": 0, + "candidates": [], + }, + 0, + True, + ), + ) + resolution = { + "state": "ambiguous", + "reason": "x", + "candidates": ["junk", self._leaf("3", "cat"), self._leaf("5", "dog")], + } + self.assertEqual(review_capture.review_evidence(resolution)[1:], (2, False)) + broken = self._leaf("7", "eel") + broken["positive"] = {"storage_text": "no text"} + resolution["candidates"].append(broken) + self.assertEqual(review_capture.review_evidence(resolution)[1:], (0, True)) + + async def test_unreadable_or_promptless_captures_are_skipped(self): + self._require_module() + record = review_capture.record_ambiguous_file + (self.output_dir / "junk.png").write_bytes(b"not a png at all") + write_png(self.output_dir / "plain.png") + info = PngImagePlugin.PngInfo() + info.add_text("prompt", "{not json") + Image.new("RGB", (2, 2)).save(str(self.output_dir / "bad.png"), pnginfo=info) + for name in ("junk.png", "plain.png", "bad.png"): + self.assertEqual( + record(self.api.db, self.api._captures, self.output_dir, 0, name), + "skipped", + name, + ) + write_png(self.output_dir / "two.png", prompt=ambiguous_graph()) + with patch.object(self.api._captures, "acquire", return_value=None): + self.assertEqual( + record(self.api.db, self.api._captures, self.output_dir, 0, "two.png"), + "skipped", + ) + self.assertEqual(self._items(), []) + self.assertEqual(self._capture_files(), []) + + async def test_record_review_item_validates_its_facts(self): + self._require_module() + epoch = self.api.db.get_execution_epoch() + # The spelling keeps its case; the locator is folded the native way. + spelled = os.path.join(self.tmpdir, "Out", "A.png") + locator = os.path.normcase(spelled) + other = os.path.normcase(os.path.join(self.tmpdir, "Out", "B.png")) + good = { + "locator": locator, + "spelled_path": spelled, + "sha256": "a" * 64, + "size": 1, + "mtime_ns": 2, + "epoch": epoch, + "resolver_version": 1, + "evidence": {"candidates": []}, + "candidate_count": 0, + "limited": True, + } + for change in ( + {"resolver_version": 0}, + {"resolver_version": True}, + {"candidate_count": 17}, + {"candidate_count": -1}, + {"limited": 1}, + {"limited": True, "candidate_count": 2}, + {"sha256": "A" * 64}, + {"evidence": ["not", "an", "object"]}, + {"spelled_path": ""}, + {"spelled_path": None}, + {"spelled_path": os.path.join(self.tmpdir, "Out", "B.png")}, + {"locator": other}, + ): + with self.assertRaises(ValueError, msg=change): + self.api.db.record_review_item(**{**good, **change}) + self.assertTrue(self.api.db.record_review_item(**good)) + self.assertFalse(self.api.db.record_review_item(**good)) + items = self.api.db.review_items_for(locator) + self.assertEqual(len(items), 1) + self.assertEqual(items[0]["evidence"], {"candidates": []}) + self.assertIs(items[0]["limited"], True) + self.assertEqual(items[0]["sha256"], "a" * 64) + self.assertEqual(self.api.db.review_items_for(other), []) + self.assertEqual(self.api.db.review_index_revision(), 1) + with self.assertRaises(ValueError): + self.api.db.review_items_for("") + conn = self.api.db.model.get_connection() + with conn: + conn.execute("DELETE FROM media_review_metadata") + with self.assertRaises(sqlite3.DatabaseError): + self.api.db.review_index_revision() + + +class TestMonitorRecording(ReviewTestCase): + def _handler(self): + handler = ImageGenerationHandler(self.api.db, PromptTracker(self.api.db)) + return handler + + def _linked_prompt(self, path): + conn = self.api.db.model.get_connection() + row = conn.execute( + "SELECT prompt_id FROM generated_images WHERE image_path=?", (str(path),) + ).fetchone() + return row[0] if row else None + + async def test_monitor_keeps_its_link_and_never_queues_the_linked_file(self): + """The monitor's pass is best effort (a failure is logged, the link + stays); a file the monitor linked is never queued.""" + self._require_module() + self._roots() + path = self.output_dir / "two.png" + write_png(path, prompt=unlinkable_graph("cat", "dog")) + prompt_id = self.api.db.save_prompt( + text="latest", prompt_hash=generate_prompt_hash("latest") + ) + metadata = {"prompt": unlinkable_graph("cat", "dog")} + context = {"id": prompt_id, "text": "latest", "fallback": True} + + handler = self._handler() + + def broken(image_path): + raise sqlite3.OperationalError("database is locked") + + handler.review_recorder = broken + with self.assertLogs("prompt_manager.image_monitor", logging.WARNING) as logs: + handler.link_image_to_prompt(str(path), context, metadata) + self.assertEqual(self._linked_prompt(path), prompt_id) + self.assertTrue(any("Needs review" in line for line in logs.output)) + self.assertEqual(self._items(), []) + + other = self.output_dir / "three.png" + write_png(other, prompt=unlinkable_graph("cat", "dog")) + calls = [] + + def recorder(image_path): + calls.append(self.api.record_review_file(image_path)) + return calls[-1] + + handler.review_recorder = recorder + handler.link_image_to_prompt(str(other), context, metadata) + self.assertEqual(self._linked_prompt(other), prompt_id) + self.assertEqual(calls, ["linked"]) + self.assertEqual(self._items(), []) + self.assertEqual(self._capture_files(), []) + + async def test_monitor_skips_resolved_and_files_outside_the_roots(self): + self._require_module() + self._roots() + prompt_id = self.api.db.save_prompt( + text="latest", prompt_hash=generate_prompt_hash("latest") + ) + context = {"id": prompt_id, "text": "latest"} + calls = [] + handler = self._handler() + handler.review_recorder = lambda image_path: calls.append(image_path) + resolved = self.output_dir / "one.png" + write_png(resolved, prompt=resolved_graph()) + handler.link_image_to_prompt( + str(resolved), context, {"prompt": resolved_graph()} + ) + self.assertEqual(calls, []) + outside = self.outside_dir / "two.png" + write_png(outside, prompt=ambiguous_graph()) + self.assertEqual(self.api.record_review_file(str(outside)), "skipped") + self.assertEqual(self._items(), []) + + async def test_monitor_recording_is_refused_beside_record_changing_jobs(self): + """S1 admission: a ``records`` recording waits for no job; it is refused. + + While a ``records`` or ``guarded`` job runs, the monitor's recording + records nothing and answers ``deferred`` (its fallback link stands and + a later scan reads the ambiguous file again); a ``read`` job does not + refuse it, and it records once the blocking job has ended. + """ + self._require_module() + from py.api.jobs import RESTORE_JOB_KIND, THUMBNAILS_JOB_KIND + + self._roots() + path = self.output_dir / "held.png" + write_png(path, prompt=unlinkable_graph("cat", "dog")) + + for kind, admission in ( + (RESTORE_JOB_KIND, "records"), + (THUMBNAILS_JOB_KIND, "guarded"), + ): + with self.subTest(admission=admission): + async with self.api._jobs.hold( + kind, f"{kind} (admin page)", admission=admission + ): + with self.assertLogs("prompt_manager", logging.INFO) as logs: + outcome = await asyncio.to_thread( + self.api.record_review_file, str(path) + ) + self.assertEqual(outcome, "deferred") + self.assertEqual(self._items(), []) + self.assertTrue( + any(f"{kind} (admin page)" in line for line in logs.output), + logs.output, + ) + + async with self.api._jobs.hold("probe-read", "Read probe", admission="read"): + outcome = await asyncio.to_thread(self.api.record_review_file, str(path)) + self.assertEqual(outcome, "recorded") + self.assertEqual(len(self._items()), 1) + + async def test_monitor_wiring_reaches_new_and_existing_handlers(self): + self._require_module() + from utils.image_monitor import ImageMonitor + + monitor = ImageMonitor(self.api.db, PromptTracker(self.api.db)) + recorder = self.api.record_review_file + monitor.set_review_recorder(recorder) + self.assertIs(monitor.review_recorder, recorder) + monitor.start_monitoring([str(self.output_dir)]) + try: + self.assertIs(monitor.handler.review_recorder, recorder) + monitor.set_review_recorder(None) + self.assertIsNone(monitor.handler.review_recorder) + finally: + monitor.stop_monitoring() + + +if __name__ == "__main__": + unittest.main() + + +class _FoldingPath: + """``os.path`` of a case-folding filesystem (Windows, macOS APFS).""" + + def __getattr__(self, name): + return getattr(os.path, name) + + @staticmethod + def normcase(value): + return os.path.normcase(value).lower() + + +class _FoldingOs: + path = _FoldingPath() + + def __getattr__(self, name): + return getattr(os, name) + + +def _case_folding(): + """Fold case where locators are formed: recording, the store, the database.""" + import database.review as review_module + import utils.media_capture as capture_module + + stack = contextlib.ExitStack() + for module in (review_module, capture_module, review_capture): + stack.enter_context(patch.object(module, "os", _FoldingOs())) + return stack + + +class TestRecordedSpelling(ReviewTestCase): + """A case-only rename refreshes the recorded spelling (I5).""" + + def _spelled(self): + conn = self.api.db.model.get_connection() + return [ + row[0] + for row in conn.execute("SELECT spelled_path FROM media_review_items") + ] + + async def test_case_only_rename_refreshes_the_spelling_on_the_skip_path(self): + self._require_module() + self._roots() + upper = self.output_dir / "Two.PNG" + write_png(upper, prompt=unlinkable_graph("cat", "dog")) + with _case_folding(): + self.assertEqual(self.api.record_review_file(str(upper)), "recorded") + (before,) = self._items() + revision = self._review_revision() + lower = self.output_dir / "two.png" + os.rename(upper, lower) # keeps size and mtime_ns + self.assertEqual(self.api.record_review_file(str(lower)), "duplicate") + (after,) = self._items() + self.assertEqual(self._spelled(), [str(lower)]) + self.assertEqual(self._review_revision(), revision + 1) + # The item itself is unchanged: an open decision stays valid. + self.assertEqual( + {k: v for k, v in after.items() if k != "spelled_path"}, + {k: v for k, v in before.items() if k != "spelled_path"}, + ) + from py.api.review import format_review_target + + real_root = os.path.realpath(str(self.output_dir)) + self.assertEqual( + format_review_target( + after["spelled_path"], + after["locator"], + [(0, real_root)], + pathmod=_FoldingPath(), + ), + (0, "two.png"), + ) + # The new spelling opens; the recorded one would be refused. + capture = self.api._captures.capture( + self.output_dir, "two.png", epoch="e", scope="s" + ) + self.assertEqual(capture.locator, after["locator"]) + self.api._captures.release(capture.token) + # Seeing the same spelling again writes nothing. + self.assertEqual(self.api.record_review_file(str(lower)), "duplicate") + self.assertEqual(self._review_revision(), revision + 1) + + async def test_duplicate_insert_refreshes_the_spelling(self): + self._require_module() + db = self.api.db + root = os.path.realpath(str(self.output_dir)) + upper, lower = os.path.join(root, "Two.PNG"), os.path.join(root, "two.png") + facts = dict( + sha256="a" * 64, + size=10, + mtime_ns=1, + resolver_version=1, + evidence={"reason": "", "candidates": []}, + candidate_count=0, + limited=True, + ) + with _case_folding(): + locator = _FoldingPath.normcase(upper) + epoch = db.get_execution_epoch() + self.assertTrue( + db.record_review_item( + locator=locator, spelled_path=upper, epoch=epoch, **facts + ) + ) + revision = self._review_revision() + self.assertFalse( + db.record_review_item( + locator=locator, spelled_path=lower, epoch=epoch, **facts + ) + ) + self.assertEqual(self._spelled(), [lower]) + self.assertEqual(self._review_revision(), revision + 1) + self.assertFalse(db.refresh_review_spelling(locator, lower)) + self.assertEqual(self._review_revision(), revision + 1) + with self.assertRaises(ValueError): + db.refresh_review_spelling(locator, os.path.join(root, "other.png")) + + +class TestCaptureStoreShutdown(ReviewTestCase): + async def test_recording_after_close_creates_no_store(self): + """A monitor recording still in flight at shutdown leaks nothing (M2).""" + self._require_module() + self._roots() + path = self.output_dir / "two.png" + write_png(path, prompt=unlinkable_graph("cat", "dog")) + await self.api.cleanup_annotation_captures(None) # the shutdown hook + self.assertIsNone(self.api._captures) + with patch("utils.media_capture.tempfile.mkdtemp") as mkdtemp: + with self.assertRaises(CaptureError): + self.api.record_review_file(str(path)) + # The monitor logs the failure and keeps its link. + handler = ImageGenerationHandler(self.api.db, PromptTracker(self.api.db)) + handler.review_recorder = self.api.record_review_file + handler._record_review(str(path)) + mkdtemp.assert_not_called() + self.assertIsNone(self.api._captures) + self.assertIsNone(self.api._captures_unsubscribe) + self.assertEqual(self._items(), []) + self.api.close_annotation_captures() # idempotent diff --git a/tests/test_review_choosable.py b/tests/test_review_choosable.py new file mode 100644 index 0000000..de86540 --- /dev/null +++ b/tests/test_review_choosable.py @@ -0,0 +1,335 @@ +"""Needs review queues a file only when there is something to choose. + +The maintainer's live queue (#284 / #288) held 39 items whose 11 candidates +all read "unresolved", "No positive / No negative": the ingestion resolver +found several sampler branches but could trace no prompt on any of them. +An ambiguous result is reviewable only when at least two candidates carry a +resolved prompt text. With none, the file could not be traced; with exactly +one, the other branches may still have made the file, so the one text is +not certain either (the resolver never flattens unknown semantics). Both +stay unresolved and are never queued, and a rescan removes the pending, +undecided items such files already have. +""" + +import os +import unittest +from unittest.mock import patch + +try: + from tests.test_review import DELIMITERS, write_png + from tests.test_review_outcome import OutcomeTestCase +except ImportError: # discovered with tests/ as the top-level directory + from test_review import DELIMITERS, write_png + from test_review_outcome import OutcomeTestCase + +from database.review_schema import evidence_offers_choice +from utils import review_capture + + +def opaque_graph(*opaque, traced=()): + """Several KSamplers: ``opaque`` positives come from an encoder the + ingestion resolver does not know, ``traced`` ones from CLIPTextEncode. + + The first sampler is opaque, so the scan's browser-parser fallback links + nothing either. + """ + graph = { + "1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}}, + "2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}}, + } + sides = [("OpaqueTextEncode", "prompt", t) for t in opaque] + sides += [("CLIPTextEncode", "text", t) for t in traced] + for index, (encoder_class, key, text) in enumerate(sides): + encoder, sampler = str(10 + 2 * index), str(11 + 2 * index) + graph[encoder] = {"class_type": encoder_class, "inputs": {key: text}} + graph[sampler] = { + "class_type": "KSampler", + "inputs": { + "model": ["1", 0], + "positive": [encoder, 0], + "negative": ["2", 0], + }, + } + return graph + + +def _always_reviewable(_resolution): + return True + + +class ChoosableTestCase(OutcomeTestCase): + def _record_pre_fix(self, name): + """An item as the pre-fix recorder stored it (no choosable test).""" + with patch.object(review_capture, "reviewable", _always_reviewable): + self._record(name) + + def _write(self, name, graph): + write_png(self.output_dir / name, prompt=graph) + return self.output_dir / name + + +class TestChoosableCandidates(unittest.TestCase): + def _resolution(self, *states): + candidates = [] + for index, state in enumerate(states): + side = {"text": f"t{index}"} if state == "resolved" else None + candidates.append( + {"sampler_id": str(index), "state": state, "positive": side} + ) + return {"state": "ambiguous", "candidates": candidates} + + def test_two_resolved_candidates_are_reviewable(self): + resolution = self._resolution("resolved", "resolved", "unresolved") + self.assertTrue(review_capture.reviewable(resolution)) + + def test_none_or_one_resolved_candidate_is_not_reviewable(self): + for states in ((), ("unresolved",) * 11, ("resolved", "unresolved")): + with self.subTest(states=states): + resolution = self._resolution(*states) + self.assertFalse(review_capture.reviewable(resolution)) + + def test_blank_or_missing_text_does_not_count(self): + resolution = self._resolution("resolved", "resolved") + resolution["candidates"][0]["positive"] = {"text": " "} + self.assertFalse(review_capture.reviewable(resolution)) + resolution["candidates"][0]["positive"] = None + self.assertFalse(review_capture.reviewable(resolution)) + + def test_nested_ambiguous_stage_counts_its_own_candidates(self): + inner = self._resolution("resolved", "resolved")["candidates"] + resolution = { + "state": "ambiguous", + "candidates": [{"state": "ambiguous", "candidates": inner}], + } + self.assertTrue(review_capture.reviewable(resolution)) + + def test_a_result_that_is_not_ambiguous_is_not_reviewable(self): + resolution = self._resolution("resolved", "resolved") + resolution["state"] = "resolved" + self.assertFalse(review_capture.reviewable(resolution)) + + +class TestStoredEvidenceOffersChoice(unittest.TestCase): + def test_unlimited_evidence_needs_two_choosable_candidates(self): + side = {"text": "cat"} + two = [{"state": "resolved", "positive": side}] * 2 + self.assertTrue(evidence_offers_choice({"candidates": two})) + one = two[:1] + [{"state": "unresolved", "positive": None}] + self.assertFalse(evidence_offers_choice({"candidates": one})) + + def test_limited_evidence_offers_a_choice_only_with_two_candidates(self): + self.assertTrue( + evidence_offers_choice({"limited": True, "candidate_total": 18}) + ) + for total in (0, 1, None, "18", True): + with self.subTest(total=total): + evidence = {"limited": True, "candidate_total": total} + self.assertFalse(evidence_offers_choice(evidence)) + + def test_malformed_evidence_offers_nothing(self): + for evidence in (None, [], {"candidates": "x"}, {}): + with self.subTest(evidence=evidence): + self.assertFalse(evidence_offers_choice(evidence)) + + +class TestNothingToChooseIsNotQueued(ChoosableTestCase): + async def test_no_candidate_with_text_could_not_be_traced(self): + self._roots() + self._write("none.png", opaque_graph("cat", "dog")) + + job = await self._scan() + + self.assertEqual(self._items(), []) + self.assertEqual(self._pending(), 0) + self.assertEqual(job["scope"]["review_items"], 0) + (row,) = job["outcomes"]["review"] + self.assertEqual(row["path"], "none.png") + self.assertIn("could not be traced", row["detail"]) + self.assertEqual(job["counters"]["review"], 1, job["counters"]) + self.assertEqual(self._capture_files(), []) + + async def test_exactly_one_candidate_with_text_stays_unresolved(self): + """One traced branch beside untraced ones is not certain: no link.""" + self._roots() + self._write("one.png", opaque_graph("cat", traced=("dog",))) + + job = await self._scan() + + self.assertEqual(self._items(), []) + (row,) = job["outcomes"]["review"] + self.assertIn("could not be traced", row["detail"]) + self.assertEqual(job["counters"].get("added", 0), 0, job["counters"]) + self.assertEqual(job["counters"].get("known", 0), 0, job["counters"]) + + async def test_two_candidates_with_text_are_queued(self): + self._roots() + self._write("two.png", opaque_graph("x", traced=("cat", "dog"))) + + job = await self._scan() + + (item,) = self._items() + self.assertEqual(os.path.basename(item["locator"]), "two.png") + self.assertEqual(item["candidate_count"], 3) + self.assertEqual(job["scope"]["review_items"], 1) + (row,) = job["outcomes"]["review"] + self.assertIn("more than one prompt could match", row["detail"]) + + async def test_parameters_delimiter_ambiguity_is_not_queued(self): + """Two A1111 negative delimiters offer no candidate to choose.""" + self._roots() + write_png(self.output_dir / "delims.png", parameters=DELIMITERS) + + job = await self._scan() + + self.assertEqual(self._items(), []) + (row,) = job["outcomes"]["review"] + self.assertIn("could not be traced", row["detail"]) + + def test_recorder_reports_untraced_and_stores_nothing(self): + self._roots() + self._write("none.png", opaque_graph("cat", "dog")) + self._write("one.png", opaque_graph("cat", traced=("dog",))) + for name in ("none.png", "one.png"): + with self.subTest(name=name): + outcome = review_capture.record_ambiguous_file( + self.api.db, self.api._captures, self.output_dir, 0, name + ) + self.assertEqual(outcome, "untraced") + self.assertEqual(self._items(), []) + self.assertEqual(self._capture_files(), []) + + def test_monitor_does_not_queue_an_untraced_output(self): + self._roots() + path = self._write("none.png", opaque_graph("cat", "dog")) + + self.assertEqual(self.api.record_review_file(str(path)), "untraced") + self.assertEqual(self._items(), []) + + +class TestUntracedItemsLeaveTheQueue(ChoosableTestCase): + async def test_rescan_removes_pending_items_without_a_choice(self): + """The maintainer's 39 items: queued before the rule, gone on rescan.""" + self._roots() + self._write("none.png", opaque_graph("cat", "dog")) + self._write("one.png", opaque_graph("cat", traced=("dog",))) + self._write("two.png", opaque_graph("x", traced=("cat", "dog"))) + for name in ("none.png", "one.png", "two.png"): + self._record_pre_fix(name) + revision = self._review_revision() + self.assertEqual(self._pending(), 3) + + job = await self._scan("all") + + (item,) = self._items() + self.assertEqual(os.path.basename(item["locator"]), "two.png") + self.assertEqual(self._pending(), 1) + self.assertGreater(self._review_revision(), revision) + self.assertEqual(job["scope"]["review_untraced"], 2) + self.assertEqual(job["scope"]["review_items"], 1) + self.assertIn( + "= 2 images left Needs review · nothing to choose", "\n".join(job["log"]) + ) + + async def test_decided_items_of_untraced_files_are_never_touched(self): + self._roots() + for name in ("sel.png", "unres.png"): + self._write(name, opaque_graph("cat", traced=(name, name + "2"))) + self._record_pre_fix(name) + items = {os.path.basename(i["locator"]): i for i in self._items()} + self._decide(items["sel.png"], "selected") + self._decide(items["unres.png"], "unresolved") + before = (self._items(), self._decisions()) + # The files now carry graphs without a choice, queued pre-fix. + for name in ("sel.png", "unres.png", "pending.png"): + self._write(name, opaque_graph("a cow", "dog")) + self._record_pre_fix(name) + self.assertEqual(self._pending(), 3) + + job = await self._scan("all") + + self.assertEqual((self._items(), self._decisions()), before) + self.assertEqual(self._pending(), 0) + self.assertEqual(job["scope"]["review_untraced"], 3) + + async def test_an_item_with_a_choice_survives_beside_an_untraced_one(self): + """Removal judges each item by its own evidence: an item that offers + a choice is not removed although the file's new bytes offer none.""" + self._roots() + self._write("f.png", opaque_graph("x", traced=("cat", "dog"))) + self._record_pre_fix("f.png") + (choosable,) = self._items() + self._write("f.png", opaque_graph("a cow", "dog")) + self._record_pre_fix("f.png") + self.assertEqual(self._pending(), 2) + + job = await self._scan("all") + + self.assertEqual([i["id"] for i in self._items()], [choosable["id"]]) + self.assertEqual(job["scope"]["review_untraced"], 1) + + async def test_limited_items_go_only_without_enough_candidates(self): + """A pre-fix delimiter item (no candidate) goes; a limited item of 18 + candidates may hold a choice and stays, though the bytes now offer + none.""" + self._roots() + write_png(self.output_dir / "delims.png", parameters=DELIMITERS) + self._record_pre_fix("delims.png") + many = [f"prompt {n}" for n in range(17)] + self._write("many.png", opaque_graph("x", traced=many)) + self._record_pre_fix("many.png") + kept = [i["id"] for i in self._items() if "many" in i["locator"]] + self._write("many.png", opaque_graph("a cow", "dog")) + + job = await self._scan("all") + + self.assertEqual([i["id"] for i in self._items()], kept) + self.assertEqual(job["scope"]["review_untraced"], 1) + + async def test_choosable_pending_item_stays(self): + self._roots() + self._write("two.png", opaque_graph("x", traced=("cat", "dog"))) + await self._scan() + (item,) = self._items() + + job = await self._scan("all") + + self.assertEqual([i["id"] for i in self._items()], [item["id"]]) + self.assertEqual(job["scope"]["review_untraced"], 0) + + async def test_a_failed_removal_is_logged_and_counted_not_fatal(self): + import logging + import sqlite3 + + self._roots() + self._write("none.png", opaque_graph("cat", "dog")) + self._record_pre_fix("none.png") + with ( + patch.object( + self.api.db, + "settle_untraced_review_items", + side_effect=sqlite3.OperationalError("database is locked"), + ), + self.assertLogs("prompt_manager.api", logging.WARNING) as logs, + ): + job = await self._scan("all") + + self.assertEqual(job["state"], "finished") + self.assertEqual(job["scope"]["review_errors"], 1) + self.assertTrue(any("could not be removed" in line for line in logs.output)) + self.assertEqual(self._pending(), 1) + await self._scan("all") + self.assertEqual(self._pending(), 0) + + def test_monitor_removes_the_pending_item_of_an_untraced_output(self): + self._roots() + path = self._write("none.png", opaque_graph("cat", "dog")) + self._record_pre_fix("none.png") + # A pre-fix item of other bytes keeps the stat shortcut out of the way. + self._write("none.png", opaque_graph("a cow", "dog")) + + self.assertEqual(self.api.record_review_file(str(path)), "untraced") + self.assertEqual(self._items(), []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_review_outcome.py b/tests/test_review_outcome.py new file mode 100644 index 0000000..c4952c8 --- /dev/null +++ b/tests/test_review_outcome.py @@ -0,0 +1,374 @@ +"""Needs review takes only files whose final outcome is ambiguous and unlinked. + +The maintainer's live scan (#284 / #288) queued every file the strict +resolver found ambiguous, also those the legacy browser-parser fallback then +linked and those already linked, so the queue (10,232) was far larger than +the scan's Needs review group. The rule now: a file is queued only when the +resolver finds it ambiguous, the fallback does not link it and no saved row +links it. A pending, undecided item whose file is linked (by the fallback, a +later resolution or the monitor) is removed on the next scan or monitor +pass; an item that carries a decision is never touched. +""" + +import json +import logging +import os +import sqlite3 +import unittest +from unittest.mock import patch + +try: + from tests.test_review import ( + DELIMITERS, + NOW, + ReviewTestCase, + ambiguous_graph, + resolved_graph, + unlinkable_graph, + write_png, + ) +except ImportError: # discovered with tests/ as the top-level directory + from test_review import ( + DELIMITERS, + NOW, + ReviewTestCase, + ambiguous_graph, + resolved_graph, + unlinkable_graph, + write_png, + ) + +from utils.hashing import generate_prompt_hash +from utils.image_monitor import ImageGenerationHandler +from utils.prompt_tracker import PromptTracker +from utils import review_capture + + +def nan_stamped(graph): + """ComfyUI's ``is_changed: [NaN]`` stamp (written as a bare NaN literal).""" + for node in graph.values(): + node["is_changed"] = [float("nan")] + return graph + + +class OutcomeTestCase(ReviewTestCase): + def _pending(self): + return self.api.db.library_counts()["review_pending"] + + def _record(self, name): + """An item as the pre-fix producers recorded it (any ambiguous file).""" + outcome = review_capture.record_ambiguous_file( + self.api.db, self.api._captures, self.output_dir, 0, name + ) + self.assertIn(outcome, ("recorded", "limited")) + + def _decide(self, item, outcome): + conn = self.api.db.model.get_connection() + candidate = None + if outcome == "selected": + candidate = json.loads(item["evidence"])["candidates"][0]["hash"] + with conn: + conn.execute( + "UPDATE media_review_items SET state=?, revision=1 WHERE id=?", + (outcome, item["id"]), + ) + conn.execute( + "INSERT INTO media_review_decisions(id,item_id,decision_revision," + "created_at,outcome,candidate_hash,evidence) VALUES(?,?,?,?,?,?,?)", + ( + f"00000000-0000-4000-8000-{len(self._decisions()) + 1:012d}", + item["id"], + 1, + NOW, + outcome, + candidate, + '{"kind":"user-selection"}', + ), + ) + + def _decisions(self): + conn = self.api.db.model.get_connection() + return [ + tuple(row) + for row in conn.execute( + "SELECT id,item_id,outcome FROM media_review_decisions ORDER BY id" + ) + ] + + def _link(self, path, text="already saved"): + prompt_id = self.api.db.save_prompt( + text=text, prompt_hash=generate_prompt_hash(text) + ) + self.api.db.link_image_to_prompt(prompt_id, str(path)) + return prompt_id + + @staticmethod + def _rows(job, group): + return sorted(row["path"] for row in job["outcomes"].get(group, [])) + + +class TestOnlyUnlinkedAmbiguousFilesAreQueued(OutcomeTestCase): + async def test_nan_stamped_graph_the_fallback_links_is_not_queued(self): + self._roots() + path = self.output_dir / "nan.png" + write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog"))) + self.assertIn(b"NaN", path.read_bytes()) + + job = await self._scan() + + self.assertEqual(job["counters"]["added"], 1, job["counters"]) + self.assertEqual(self._items(), []) + self.assertEqual(job["scope"]["review_items"], 0) + self.assertEqual(self._pending(), 0) + + async def test_fallback_link_to_a_known_prompt_is_not_queued(self): + self._roots() + write_png(self.output_dir / "a.png", prompt=ambiguous_graph("cat", "dog")) + await self._scan() + write_png(self.output_dir / "b.png", prompt=ambiguous_graph("cat", "dog")) + + job = await self._scan() + + self.assertEqual(self._items(), []) + self.assertEqual(job["scope"]["review_items"], 0) + + async def test_already_linked_file_is_not_queued(self): + self._roots() + path = self.output_dir / "linked.png" + write_png(path, prompt=unlinkable_graph("cat", "dog")) + self._link(path) + + job = await self._scan() + + self.assertEqual(self._items(), []) + self.assertEqual(job["scope"]["review_items"], 0) + self.assertEqual(self._rows(job, "review"), []) + self.assertEqual(self._rows(job, "known"), ["linked.png"]) + + async def test_genuinely_ambiguous_file_is_queued(self): + self._roots() + path = self.output_dir / "two.png" + write_png(path, prompt=nan_stamped(unlinkable_graph("cat", "dog"))) + # Beside it, a file the fallback links: only the first is queued. + write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b")) + + job = await self._scan() + + (item,) = self._items() + self.assertEqual(os.path.basename(item["locator"]), "two.png") + self.assertEqual(item["state"], "pending") + self.assertEqual(item["candidate_count"], 3) + self.assertEqual(job["scope"]["review_items"], 1) + self.assertEqual(self._rows(job, "review"), ["two.png"]) + self.assertEqual(self._pending(), 1) + self.assertIn("? 1 image added to Needs review", "\n".join(job["log"])) + + async def test_counter_and_queue_agree_with_the_review_group(self): + """``review_items`` and the queue total equal the review group less + its unresolved files, on the first scan and on a rescan.""" + self._roots() + for name in ("u1.png", "u2.png"): + write_png(self.output_dir / name, prompt=unlinkable_graph("cat", name)) + write_png(self.output_dir / "delims.png", parameters=DELIMITERS) + write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b")) + unresolved = {"1": {"class_type": "KSampler", "inputs": {}}} + write_png(self.output_dir / "none.png", prompt=unresolved) + write_png(self.output_dir / "one.png", prompt=resolved_graph()) + + for _ in range(2): + job = await self._scan() + review = self._rows(job, "review") + unresolved_rows = sorted( + row["path"] + for row in job["outcomes"]["review"] + if "could not be traced" in row["detail"] + ) + queued = [name for name in review if name not in unresolved_rows] + # delims.png has nothing to choose: "could not be traced". + self.assertEqual(queued, ["u1.png", "u2.png"], job) + self.assertIn("delims.png", unresolved_rows) + self.assertEqual(job["scope"]["review_items"], 2) + self.assertEqual(self._pending(), 2) + self.assertIn("? 2 images already in Needs review", "\n".join(job["log"])) + + +class TestQueueSelfHeals(OutcomeTestCase): + async def test_pending_item_of_a_fallback_linked_file_is_removed(self): + """The maintainer's case: queued by the old rule, linked on rescan.""" + self._roots() + path = self.output_dir / "nan.png" + write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog"))) + self._record("nan.png") + revision = self._review_revision() + self.assertEqual(self._pending(), 1) + + job = await self._scan() + + self.assertEqual(self._items(), []) + self.assertEqual(self._pending(), 0) + self.assertGreater(self._review_revision(), revision) + self.assertEqual(job["scope"]["review_settled"], 1) + self.assertIn("= 1 image left Needs review · now linked", "\n".join(job["log"])) + + async def test_pending_item_of_an_already_linked_file_is_removed(self): + self._roots() + path = self.output_dir / "linked.png" + write_png(path, prompt=unlinkable_graph("cat", "dog")) + self._record("linked.png") + self._link(path) + + await self._scan() + + self.assertEqual(self._items(), []) + self.assertEqual(self._pending(), 0) + + async def test_pending_item_of_a_skipped_known_file_is_removed(self): + """A file the "new" scope skips as known still settles its old item.""" + self._roots() + path = self.output_dir / "one.png" + write_png(path, prompt=unlinkable_graph("cat", "dog")) + self._record("one.png") + write_png(path, prompt=resolved_graph()) + first = await self._scan() + self.assertEqual(self._rows(first, "added"), ["one.png"]) + # An item a pre-fix scan left for the file's old bytes. + self.assertEqual(self._items(), []) + write_png(path, prompt=unlinkable_graph("cat", "dog"), color=(0, 9, 0)) + self._record("one.png") + write_png(path, prompt=resolved_graph()) + second = await self._scan() + self.assertEqual(self._rows(second, "known"), ["one.png"]) + self.assertEqual(self._items(), []) + + async def test_a_failed_settle_is_logged_and_counted_not_fatal(self): + self._roots() + write_png(self.output_dir / "nan.png", prompt=ambiguous_graph("cat", "dog")) + self._record("nan.png") + + with ( + patch.object( + self.api.db, + "settle_linked_review_items", + side_effect=sqlite3.OperationalError("database is locked"), + ), + self.assertLogs("prompt_manager.api", logging.WARNING) as logs, + ): + job = await self._scan() + + self.assertEqual(job["state"], "finished") + self.assertEqual(job["scope"]["review_errors"], 1) + self.assertEqual(job["scope"]["review_settled"], 0) + self.assertTrue(any("could not be removed" in line for line in logs.output)) + self.assertEqual(self._pending(), 1) + # The next scan settles it. + await self._scan() + self.assertEqual(self._pending(), 0) + + async def test_still_unlinked_pending_item_stays(self): + self._roots() + write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog")) + await self._scan() + (item,) = self._items() + + job = await self._scan() + + self.assertEqual([i["id"] for i in self._items()], [item["id"]]) + self.assertEqual(job["scope"].get("review_settled", 0), 0) + + async def test_decided_items_are_never_touched(self): + self._roots() + for name in ("sel.png", "unres.png"): + write_png(self.output_dir / name, prompt=ambiguous_graph("cat", name)) + self._record(name) + items = {os.path.basename(i["locator"]): i for i in self._items()} + self._decide(items["sel.png"], "selected") + self._decide(items["unres.png"], "unresolved") + before = (self._items(), self._decisions()) + # A pending neighbour the fallback links: it alone leaves the queue. + write_png(self.output_dir / "pending.png", prompt=ambiguous_graph("x", "y")) + self._record("pending.png") + self.assertEqual(self._pending(), 1) + + job = await self._scan("all") + + self.assertEqual((self._items(), self._decisions()), before) + self.assertEqual(self._pending(), 0) + self.assertEqual(job["scope"]["review_settled"], 1) + + +class TestMonitorRule(OutcomeTestCase): + def _handler(self): + handler = ImageGenerationHandler(self.api.db, PromptTracker(self.api.db)) + handler.review_recorder = self.api.record_review_file + return handler + + def _context(self): + prompt_id = self.api.db.save_prompt( + text="latest", prompt_hash=generate_prompt_hash("latest") + ) + return {"id": prompt_id, "text": "latest", "fallback": True} + + async def test_monitor_linked_ambiguous_output_is_not_queued(self): + self._roots() + path = self.output_dir / "two.png" + graph = unlinkable_graph("cat", "dog") + write_png(path, prompt=graph) + + self._handler().link_image_to_prompt( + str(path), self._context(), {"prompt": graph} + ) + + self.assertEqual(self._items(), []) + + async def test_monitor_link_removes_the_files_pending_item(self): + self._roots() + path = self.output_dir / "two.png" + graph = unlinkable_graph("cat", "dog") + write_png(path, prompt=graph) + self._record("two.png") + self.assertEqual(self._pending(), 1) + + self._handler().link_image_to_prompt( + str(path), self._context(), {"prompt": graph} + ) + + self.assertEqual(self._items(), []) + self.assertEqual(self._pending(), 0) + + async def test_monitor_never_touches_a_decided_item(self): + self._roots() + path = self.output_dir / "two.png" + graph = unlinkable_graph("cat", "dog") + write_png(path, prompt=graph) + self._record("two.png") + (item,) = self._items() + self._decide(item, "unresolved") + before = (self._items(), self._decisions()) + other = self.output_dir / "other.png" + write_png(other, prompt=graph, color=(0, 0, 9)) + self._record("other.png") + handler, context = self._handler(), self._context() + + handler.link_image_to_prompt(str(path), context, {"prompt": graph}) + handler.link_image_to_prompt(str(other), context, {"prompt": graph}) + + self.assertEqual((self._items(), self._decisions()), before) + self.assertEqual(self._pending(), 0) + + async def test_record_review_file_applies_the_scan_rule(self): + self._roots() + write_png(self.output_dir / "fallback.png", prompt=ambiguous_graph("a", "b")) + write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog")) + linked = self.output_dir / "linked.png" + write_png(linked, prompt=unlinkable_graph("cat", "dog")) + self._link(linked) + + record = self.api.record_review_file + self.assertEqual(record(str(self.output_dir / "fallback.png")), "skipped") + self.assertEqual(record(str(linked)), "linked") + self.assertEqual(record(str(self.output_dir / "two.png")), "recorded") + self.assertEqual(len(self._items()), 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_review_schema.py b/tests/test_review_schema.py new file mode 100644 index 0000000..1ee935e --- /dev/null +++ b/tests/test_review_schema.py @@ -0,0 +1,918 @@ +"""Needs review schema group (#249): install, validation and restore safety.""" + +import hashlib +import json +import re +import sqlite3 +import tempfile +import unittest +from contextlib import closing +from pathlib import Path +from unittest.mock import patch + +try: + from tests.open_handles import assert_closed +except ImportError: # discovered with tests/ as the top-level directory + from open_handles import assert_closed + +from database.models import PromptModel +from database.operations import PromptDatabase + +try: # guarded so a missing module fails assertions, not test loading + from database.review_schema import ( + canonical_evidence, + evidence_digest, + migrate_review_state, + review_schema_error, + ) +except ImportError: + canonical_evidence = None + evidence_digest = None + migrate_review_state = None + review_schema_error = None + +METADATA = "media_review_metadata" +ITEMS = "media_review_items" +DECISIONS = "media_review_decisions" +TABLES = (METADATA, ITEMS, DECISIONS) +ANNOTATION_TABLES = ( + "media_annotation_metadata", + "media_annotation_files", + "media_annotation_bytes", + "media_image_tags", +) +MAX_EVIDENCE_BYTES = 8 * 1024 * 1024 +# The exact DDL the plan pins; the module must produce precisely these objects. +DDL = { + METADATA: ( + f"CREATE TABLE {METADATA}(singleton_id INTEGER PRIMARY KEY " + "CHECK(singleton_id=1), schema_version INTEGER NOT NULL " + "CHECK(schema_version=1), review_index_revision INTEGER NOT NULL " + "DEFAULT 0 CHECK(review_index_revision>=0))" + ), + ITEMS: ( + f"CREATE TABLE {ITEMS}(id TEXT PRIMARY KEY CHECK(length(id)=36), " + "byte_id TEXT NOT NULL REFERENCES media_annotation_bytes(id) " + "ON DELETE RESTRICT, spelled_path TEXT NOT NULL CHECK(length(CAST(" + "spelled_path AS BLOB)) BETWEEN 1 AND 16384), " + "resolver_version INTEGER NOT NULL, " + "evidence_hash TEXT NOT NULL CHECK(length(evidence_hash)=64 AND " + "evidence_hash NOT GLOB '*[^0-9a-f]*'), evidence TEXT NOT NULL " + f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), " + "candidate_count INTEGER NOT NULL CHECK(candidate_count BETWEEN 0 AND 16), " + "limited INTEGER NOT NULL CHECK(limited IN (0,1)), state TEXT NOT NULL " + "CHECK(state IN ('pending','selected','unresolved')), revision INTEGER " + "NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, updated_at TEXT " + "NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))" + ), + DECISIONS: ( + f"CREATE TABLE {DECISIONS}(id TEXT PRIMARY KEY CHECK(length(id)=36), " + f"item_id TEXT NOT NULL REFERENCES {ITEMS}(id) ON DELETE RESTRICT, " + "decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, " + "outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), " + "candidate_hash TEXT CHECK(candidate_hash IS NULL OR " + "(length(candidate_hash)=64 AND candidate_hash NOT GLOB '*[^0-9a-f]*')), " + "evidence TEXT NOT NULL " + f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), " + "pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, " + "image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, " + "UNIQUE(item_id, decision_revision), " + "CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))" + ), +} +INDEXES = { + "idx_media_review_items_queue": ( + f"CREATE INDEX idx_media_review_items_queue ON {ITEMS}" + "(state, created_at, id)" + ), + "idx_media_review_decisions_pair": ( + f"CREATE INDEX idx_media_review_decisions_pair ON {DECISIONS}(pair_id)" + ), + "idx_media_review_decisions_image": ( + f"CREATE INDEX idx_media_review_decisions_image ON {DECISIONS}(image_id)" + ), +} +AUTOINDEXES = {ITEMS: 2, DECISIONS: 2} +LEGACY_TABLES = ( + "prompts", + "generated_images", + "tags", + "prompt_tags", + "prompt_pairs", + "prompt_pair_tags", + "image_favorites", +) + ANNOTATION_TABLES +FILE_ID = "0f8e4b2a-1c3d-4e5f-8a9b-0c1d2e3f4a5b" +BYTE_ID = "6a1b2c3d-4e5f-4a6b-8c9d-0e1f2a3b4c5d" +ITEM_ID = "1d2e3f4a-5b6c-4d7e-8f9a-0b1c2d3e4f5a" +DECISION_ID = "2e3f4a5b-6c7d-4e8f-9a0b-1c2d3e4f5a6b" +OTHER_ID = "7b2c3d4e-5f6a-4b7c-9d0e-1f2a3b4c5d6e" +DIGEST = hashlib.sha256(b"img").hexdigest() +CANDIDATE = hashlib.sha256(b"candidate").hexdigest() +NOW = "2026-10-22T00:00:00.000000+00:00" +EVIDENCE = '{"candidates":[],"sha256":"' + DIGEST + '"}' +EVIDENCE_HASH = hashlib.sha256(EVIDENCE.encode("ascii")).hexdigest() + + +def _compact(sql): + return re.sub(r"\s+", "", sql or "").lower() + + +def expected_catalog(): + objects = {("table", name, _compact(sql)) for name, sql in DDL.items()} + objects |= {("index", name, _compact(sql)) for name, sql in INDEXES.items()} + objects |= { + ("index", f"sqlite_autoindex_{table}_{n}", "") + for table, count in AUTOINDEXES.items() + for n in range(1, count + 1) + } + return objects + + +def catalog(conn): + names = tuple(TABLES) + tuple(INDEXES) + marks = ",".join("?" * len(names)) + table_marks = ",".join("?" * len(TABLES)) + return { + (kind, name, _compact(sql)) + for kind, name, sql in conn.execute( + "SELECT type,name,sql FROM sqlite_master " + f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608 + names + tuple(TABLES), + ) + } + + +def rows(conn, sql, *params): + """Plain tuples whatever row factory *conn* uses.""" + return [tuple(row) for row in conn.execute(sql, params)] + + +def metadata(conn): + return rows(conn, f"SELECT schema_version,review_index_revision FROM {METADATA}") + + +def table_rows(conn, tables): + found = {} + for table in tables: + present = conn.execute( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,) + ).fetchone() + if present: + found[table] = rows(conn, f"SELECT * FROM {table}") # nosec B608 + return found + + +def insert_byte(conn, file_id=FILE_ID, byte_id=BYTE_ID, locator="/out/img.png"): + conn.execute( + "INSERT INTO media_annotation_files VALUES(?,?,?)", (file_id, locator, NOW) + ) + conn.execute( + "INSERT INTO media_annotation_bytes VALUES(?,?,?,?,?,?,?,0)", + (byte_id, file_id, DIGEST, 3, 1_700_000_000_000_000_000, NOW, NOW), + ) + + +def insert_item(conn, *, item_id=ITEM_ID, evidence=EVIDENCE, candidates=2, **extra): + values = { + "id": item_id, + "byte_id": BYTE_ID, + "spelled_path": "/out/img.png", + "resolver_version": 1, + "evidence_hash": hashlib.sha256(evidence.encode("utf-8")).hexdigest(), + "evidence": evidence, + "candidate_count": candidates, + "limited": 0, + "state": "pending", + "revision": 0, + "created_at": NOW, + "updated_at": NOW, + } + values.update(extra) + columns = ",".join(values) + marks = ",".join("?" * len(values)) + conn.execute( + f"INSERT INTO {ITEMS}({columns}) VALUES({marks})", # nosec B608 + tuple(values.values()), + ) + + +def insert_decision(conn, *, decision_id=DECISION_ID, **extra): + values = { + "id": decision_id, + "item_id": ITEM_ID, + "decision_revision": 1, + "created_at": NOW, + "outcome": "selected", + "candidate_hash": CANDIDATE, + "evidence": EVIDENCE, + "pair_id": None, + "image_id": None, + } + values.update(extra) + columns = ",".join(values) + marks = ",".join("?" * len(values)) + conn.execute( + f"INSERT INTO {DECISIONS}({columns}) VALUES({marks})", # nosec B608 + tuple(values.values()), + ) + + +def insert_group_rows(conn, **item): + insert_byte(conn) + insert_item(conn, **item) + insert_decision(conn) + + +def populate_legacy(conn): + conn.execute( + "INSERT INTO prompts(id,text,hash,category,notes,run_count) VALUES" + "(7,'positive','p','keep','notes',12)" + ) + conn.execute("INSERT INTO prompts(id,text,hash) VALUES(9,'negative','n')") + conn.execute( + "INSERT INTO generated_images(id,prompt_id,image_path,filename,file_path) " + "VALUES(4,7,'/tmp/image.png','image.png','/tmp/image.png')" + ) + conn.execute("INSERT INTO tags(id,name) VALUES(3,'keep-tag')") + conn.execute("INSERT INTO prompt_tags VALUES(7,3)") + conn.execute( + "INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) " + "VALUES(6,7,9)" + ) + conn.execute("INSERT INTO prompt_pair_tags VALUES(6,3)") + conn.execute("INSERT INTO image_favorites VALUES('root','a.png',?)", (NOW,)) + + +def full_capacity_evidence(): + """Sixteen candidates of control, lone-surrogate, astral and combining + text, exactly 8 MiB.""" + text = '\x00\x07\x1f
\ud800 é é "q" \\ \U0001f9ea' + candidates = [ + { + "hash": hashlib.sha256(str(n).encode()).hexdigest(), + "negative": {"text": text * 3}, + "positive": {"text": text * 5}, + "sampler_id": str(n), + "state": "resolved", + } + for n in range(16) + ] + value = {"candidates": candidates, "pad": ""} + base = len(canonical_evidence(value).encode("utf-8")) + value["pad"] = "x" * (MAX_EVIDENCE_BYTES - base) + return value + + +class ReviewSchemaTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.addCleanup(assert_closed, self, self.temp.name) + self.addCleanup(PromptModel.reset_schema_cache) + self.models = [] + self.addCleanup(self.close_models) + self.path = str(Path(self.temp.name) / "live.db") + + def close_models(self): + for model in self.models: + model.close_all() + + def model(self, path=None): + model = PromptModel(path or self.path) + self.models.append(model) + return model + + def require_module(self): + self.assertIsNotNone( + review_schema_error, "database.review_schema is not implemented" + ) + + def drop_tables(self, path, tables=TABLES): + """Remove tables from a closed database file (an old backup).""" + with closing(sqlite3.connect(path)) as raw, raw: + raw.execute("PRAGMA foreign_keys=OFF") + for table in reversed(tables): + raw.execute(f"DROP TABLE {table}") + + def test_fresh_install_and_reopen_is_idempotent(self): + self.require_module() + model = self.model() + with model.get_connection() as conn: + self.assertEqual(catalog(conn), expected_catalog()) + self.assertIsNone(review_schema_error(conn, allow_missing=False)) + self.assertEqual(metadata(conn), [(1, 0)]) + insert_group_rows(conn) + snapshot = table_rows(conn, TABLES) + self.assertEqual( + PromptModel.verify_database_file(self.path, require_library_state=True), + (True, "ok"), + ) + model.close_all() + for _ in range(2): + PromptModel.reset_schema_cache() + model = self.model() + with model.get_connection() as conn: + self.assertEqual(catalog(conn), expected_catalog()) + self.assertIsNone(review_schema_error(conn, allow_missing=False)) + self.assertEqual(table_rows(conn, TABLES), snapshot) + # Running the migration again on a current database changes nothing. + migrate_review_state(conn) + self.assertEqual(table_rows(conn, TABLES), snapshot) + model.close_all() + # Items and decisions keep the byte record and the item they cite. + PromptModel.reset_schema_cache() + model = self.model() + with model.get_connection() as conn: + with self.assertRaises(sqlite3.IntegrityError): + conn.execute("DELETE FROM media_annotation_files") + with self.assertRaises(sqlite3.IntegrityError): + conn.execute(f"DELETE FROM {ITEMS}") + self.assertEqual(table_rows(conn, TABLES), snapshot) + + def test_startup_validates_stored_evidence_once(self): + """A reopen streams the evidence once and the annotation group twice + (its own install), never again for the review group (I3).""" + self.require_module() + import database.annotation_schema as annotation_module + import database.review_schema as review_module + + model = self.model() + with model.get_connection() as conn: + insert_group_rows(conn) + model.close_all() + for _ in range(2): + PromptModel.reset_schema_cache() + with ( + patch.object( + review_module, + "_evidence_error", + wraps=review_module._evidence_error, + ) as evidence, + patch.object( + review_module, + "annotation_schema_error", + wraps=review_module.annotation_schema_error, + ) as nested, + patch.object( + annotation_module, + "_value_error", + wraps=annotation_module._value_error, + ) as annotation_rows, + ): + model = self.model() + model.get_connection() + self.assertEqual(evidence.call_count, 1) + self.assertEqual(nested.call_count, 0) + self.assertEqual(annotation_rows.call_count, 2) + model.close_all() + + def test_old_database_without_group_migrates_without_touching_legacy_rows(self): + self.require_module() + for label, dropped in ( + ("annotations only", TABLES), + ("predates annotations", ANNOTATION_TABLES + TABLES), + ): + with self.subTest(label): + path = str(Path(self.temp.name) / f"{label.replace(' ', '-')}.db") + model = self.model(path) + with model.get_connection() as conn: + populate_legacy(conn) + if dropped == TABLES: + insert_byte(conn) + model.close_all() + self.drop_tables(path, dropped) + with closing(sqlite3.connect(path)) as raw: + self.assertIsNone(review_schema_error(raw)) + self.assertEqual( + review_schema_error(raw, allow_missing=False), + "Missing review queue schema", + ) + legacy_rows_before = table_rows(raw, LEGACY_TABLES) + self.assertTrue(PromptModel.verify_database_file(path)[0]) + self.assertFalse( + PromptModel.verify_database_file(path, require_library_state=True)[ + 0 + ] + ) + PromptModel.reset_schema_cache() + + model = self.model(path) + + with model.get_connection() as conn: + self.assertEqual(catalog(conn), expected_catalog()) + self.assertIsNone(review_schema_error(conn, allow_missing=False)) + self.assertEqual(metadata(conn), [(1, 0)]) + counts = [ + conn.execute(f"SELECT COUNT(*) FROM {t}").fetchone()[0] + for t in (ITEMS, DECISIONS) + ] + legacy_rows_after = { + table: value + for table, value in table_rows(conn, LEGACY_TABLES).items() + if table in legacy_rows_before + } + self.assertEqual(counts, [0, 0]) + self.assertEqual(legacy_rows_after, legacy_rows_before) + self.assertEqual( + PromptModel.verify_database_file(path, require_library_state=True), + (True, "ok"), + ) + model.close_all() + + def test_annotations_only_installation_is_valid(self): + self.require_module() + model = self.model() + with model.get_connection() as conn: + insert_byte(conn) + model.close_all() + self.drop_tables(self.path) + with closing(sqlite3.connect(self.path)) as raw: + self.assertIsNone(review_schema_error(raw)) + self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok")) + self.assertEqual( + PromptModel.verify_database_file(self.path, require_library_state=True), + (False, "Missing review queue schema"), + ) + + def test_review_group_without_annotation_group_is_invalid(self): + self.require_module() + model = self.model() + model.close_all() + self.drop_tables(self.path, ANNOTATION_TABLES) + with closing(sqlite3.connect(self.path)) as raw: + for allow_missing in (True, False): + error = review_schema_error(raw, allow_missing=allow_missing) + self.assertIsNotNone(error) + self.assertIn("image annotation", error.lower()) + self.assertFalse(PromptModel.verify_database_file(self.path)[0]) + PromptModel.reset_schema_cache() + with self.assertRaises(sqlite3.DatabaseError): + self.model() + # An invalid annotation group invalidates the review group beside it. + path = self.tampered( + "bad annotations", + ("UPDATE media_annotation_bytes SET sha256=upper(sha256)",), + checks_off=True, + ) + with closing(sqlite3.connect(path)) as raw: + self.assertIn("image annotation", review_schema_error(raw).lower()) + + def tampered(self, name, statements, *, checks_off=False, rows_in=True): + """A closed current database whose group was altered after the fact.""" + path = str(Path(self.temp.name) / f"{name.replace(' ', '-')}.db") + model = self.model(path) + with model.get_connection() as conn: + populate_legacy(conn) + if rows_in: + insert_group_rows(conn) + model.close_all() + with closing(sqlite3.connect(path)) as raw, raw: + raw.execute("PRAGMA foreign_keys=OFF") + if checks_off: + raw.execute("PRAGMA ignore_check_constraints=ON") + for statement in statements: + if isinstance(statement, tuple): + raw.execute(*statement) + else: + raw.execute(statement) + return path + + def assert_rejected_unrepaired(self, path): + with closing(sqlite3.connect(path)) as raw: + schema_before = raw.execute( + "SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name" + ).fetchall() + rows_before = table_rows(raw, TABLES) + for allow_missing in (False, True): + error = review_schema_error(raw, allow_missing=allow_missing) + self.assertIsNotNone(error) + self.assertIn("review", error.lower()) + self.assertFalse(PromptModel.verify_database_file(path)[0]) + self.assertFalse( + PromptModel.verify_database_file(path, require_library_state=True)[0] + ) + PromptModel.reset_schema_cache() + with self.assertRaises(sqlite3.DatabaseError): + self.model(path) + PromptModel.reset_schema_cache() + with closing(sqlite3.connect(path)) as raw: + self.assertEqual( + raw.execute( + "SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name" + ).fetchall(), + schema_before, + ) + self.assertEqual(table_rows(raw, TABLES), rows_before) + + def test_partial_or_malformed_group_is_rejected_without_repair(self): + self.require_module() + cases = { + "missing metadata": (f"DROP TABLE {METADATA}",), + "missing items": (f"DROP TABLE {DECISIONS}", f"DROP TABLE {ITEMS}"), + "missing decisions": (f"DROP TABLE {DECISIONS}",), + "reserved view": ( + f"DROP TABLE {DECISIONS}", + f"CREATE VIEW {DECISIONS} AS SELECT 'x' AS id", + ), + "extra trigger": ( + f"CREATE TRIGGER media_review_trigger AFTER INSERT ON {ITEMS} " + "BEGIN SELECT 1; END", + ), + "extra index": ( + f"CREATE INDEX idx_media_review_extra ON {ITEMS}(updated_at)", + ), + "extra column": ( + f"DROP TABLE {DECISIONS}", + DDL[DECISIONS].replace("evidence TEXT", "note TEXT, evidence TEXT"), + ), + "wrong primary key": ( + f"DROP TABLE {DECISIONS}", + DDL[DECISIONS].replace("id TEXT PRIMARY KEY", "id TEXT", 1), + ), + "wrong foreign key": ( + f"DROP TABLE {DECISIONS}", + DDL[DECISIONS].replace( + "generated_images(id) ON DELETE SET NULL", + "generated_images(id) ON DELETE CASCADE", + ), + INDEXES["idx_media_review_decisions_pair"], + INDEXES["idx_media_review_decisions_image"], + ), + "wrong check": ( + f"DROP TABLE {METADATA}", + DDL[METADATA].replace("schema_version=1", "schema_version=2"), + f"INSERT INTO {METADATA} VALUES(1,2,0)", + ), + "missing unique": ( + "DROP INDEX idx_media_review_decisions_pair", + "DROP INDEX idx_media_review_decisions_image", + f"DROP TABLE {DECISIONS}", + DDL[DECISIONS].replace(" UNIQUE(item_id, decision_revision),", ""), + INDEXES["idx_media_review_decisions_pair"], + INDEXES["idx_media_review_decisions_image"], + ), + "missing queue index": ("DROP INDEX idx_media_review_items_queue",), + "wrong index columns": ( + "DROP INDEX idx_media_review_items_queue", + INDEXES["idx_media_review_items_queue"].replace(", id", ""), + ), + "orphan item": ((f"UPDATE {ITEMS} SET byte_id=?", (OTHER_ID,)),), + "orphan decision": ((f"UPDATE {DECISIONS} SET item_id=?", (OTHER_ID,)),), + "dangling pair": (f"UPDATE {DECISIONS} SET pair_id=404",), + "bad item uuid": ( + (f"UPDATE {ITEMS} SET id=?", ("x" * 36,)), + (f"UPDATE {DECISIONS} SET item_id=?", ("x" * 36,)), + ), + "uppercase decision uuid": (f"UPDATE {DECISIONS} SET id=upper(id)",), + "null decision id": (f"UPDATE {DECISIONS} SET id=NULL",), + "blob ids": (f"UPDATE {DECISIONS} SET id=CAST(id AS BLOB)",), + "no metadata row": (f"DELETE FROM {METADATA}",), + "long created_at": ((f"UPDATE {ITEMS} SET created_at=?", ("x" * 65,)),), + "blob updated_at": (f"UPDATE {ITEMS} SET updated_at=X'00'",), + "text resolver version": (f"UPDATE {ITEMS} SET resolver_version='one'",), + "blob spelled path": ( + f"UPDATE {ITEMS} SET spelled_path=CAST(spelled_path AS BLOB)", + ), + "evidence hash mismatch": ( + (f"UPDATE {ITEMS} SET evidence_hash=?", (CANDIDATE,)), + ), + } + for label, statements in cases.items(): + with self.subTest(label): + self.assert_rejected_unrepaired(self.tampered(label, statements)) + unchecked = { + "two metadata rows": (f"INSERT INTO {METADATA} VALUES(2,1,0)",), + "wrong schema version": (f"UPDATE {METADATA} SET schema_version=2",), + "negative index revision": ( + f"UPDATE {METADATA} SET review_index_revision=-1", + ), + "uppercase evidence hash": ( + f"UPDATE {ITEMS} SET evidence_hash=upper(evidence_hash)", + ), + "bad state": (f"UPDATE {ITEMS} SET state='done'",), + "limited two": (f"UPDATE {ITEMS} SET limited=2",), + "empty spelled path": (f"UPDATE {ITEMS} SET spelled_path=''",), + "negative revision": (f"UPDATE {ITEMS} SET revision=-1",), + "negative candidates": (f"UPDATE {ITEMS} SET candidate_count=-1",), + "bad outcome": (f"UPDATE {DECISIONS} SET outcome='maybe'",), + "short candidate hash": ( + f"UPDATE {DECISIONS} SET candidate_hash=substr(candidate_hash,1,63)", + ), + "negative decision revision": ( + f"UPDATE {DECISIONS} SET decision_revision=-1", + ), + } + for label, statements in unchecked.items(): + with self.subTest(label): + path = self.tampered(label, statements, checks_off=True) + self.assert_rejected_unrepaired(path) + + def test_selected_decision_requires_candidate_hash(self): + self.require_module() + model = self.model() + with model.get_connection() as conn: + insert_byte(conn) + insert_item(conn) + for outcome, candidate in (("selected", None), ("unresolved", CANDIDATE)): + with self.subTest(outcome): + with self.assertRaises(sqlite3.IntegrityError): + insert_decision(conn, outcome=outcome, candidate_hash=candidate) + insert_decision(conn, outcome="unresolved", candidate_hash=None) + insert_decision(conn, decision_id=OTHER_ID, decision_revision=2) + self.assertIsNone(review_schema_error(conn, allow_missing=False)) + self.assertEqual( + rows(conn, f"SELECT outcome FROM {DECISIONS} ORDER BY 1"), + [("selected",), ("unresolved",)], + ) + with self.assertRaises(sqlite3.IntegrityError): # one per revision + insert_decision(conn, decision_id=ITEM_ID, decision_revision=2) + model.close_all() + path = self.tampered( + "selected without hash", + (f"UPDATE {DECISIONS} SET candidate_hash=NULL",), + checks_off=True, + ) + self.assert_rejected_unrepaired(path) + + def test_oversize_or_noncanonical_evidence_rejected(self): + self.require_module() + self.assertEqual( + canonical_evidence({"b": [1, "é"], "a": None}), + '{"a":null,"b":[1,"\\u00e9"]}', + ) + self.assertEqual( + evidence_digest(EVIDENCE), hashlib.sha256(EVIDENCE.encode()).hexdigest() + ) + # The store's non-finite policy (utils.json_strict): written as null. + self.assertEqual( + canonical_evidence({"x": float("nan"), "y": [float("-inf")]}), + '{"x":null,"y":[null]}', + ) + for bad in ( + [], + "text", + {"x": object()}, + {"x": object(), "y": float("nan")}, + {1: 1, "a": 2}, + ): + with self.subTest(value=repr(bad)), self.assertRaises(ValueError): + canonical_evidence(bad) + with self.assertRaises(ValueError): + canonical_evidence({"pad": "x" * MAX_EVIDENCE_BYTES}) + oversize = '{"pad":"' + "x" * (MAX_EVIDENCE_BYTES - 9) + '"}' + self.assertEqual(len(oversize), MAX_EVIDENCE_BYTES + 1) + model = self.model() + with model.get_connection() as conn: + insert_byte(conn) + with self.assertRaises(sqlite3.IntegrityError): + insert_item(conn, evidence=oversize) + model.close_all() + noncanonical = { + "spaces": '{"candidates": [], "sha256": "x"}', + "unsorted": '{"sha256":"x","candidates":[]}', + "raw unicode": '{"text":"é"}', + "duplicate key": '{"a":1,"a":2}', + "not an object": "[]", + "not json": "{", + "nan": '{"a":NaN}', + "empty": "", + } + for label, evidence in noncanonical.items(): + for table in (ITEMS, DECISIONS): + with self.subTest(label, table=table): + digest = hashlib.sha256(evidence.encode("utf-8")).hexdigest() + statements = [ + (f"UPDATE {table} SET evidence=?", (evidence,)), + (f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)), + ] + if table == DECISIONS: + statements.pop() + self.assert_rejected_unrepaired( + self.tampered(f"{label} {table}", statements) + ) + for table in (ITEMS, DECISIONS): + with self.subTest("oversize", table=table): + digest = hashlib.sha256(oversize.encode("ascii")).hexdigest() + statements = [ + (f"UPDATE {table} SET evidence=?", (oversize,)), + (f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)), + ] + path = self.tampered(f"oversize {table}", statements, checks_off=True) + self.assert_rejected_unrepaired(path) + + def test_seventeen_candidates_rejected(self): + self.require_module() + model = self.model() + with model.get_connection() as conn: + insert_byte(conn) + with self.assertRaises(sqlite3.IntegrityError): + insert_item(conn, candidates=17) + insert_item(conn, candidates=16) + self.assertIsNone(review_schema_error(conn, allow_missing=False)) + model.close_all() + path = self.tampered( + "seventeen", (f"UPDATE {ITEMS} SET candidate_count=17",), checks_off=True + ) + self.assert_rejected_unrepaired(path) + + def test_prompt_or_image_deletion_nulls_navigation_and_keeps_decision(self): + self.require_module() + db = PromptDatabase(self.path) + self.models.append(db.model) + with db.model.get_connection() as conn: + populate_legacy(conn) + conn.execute( + "INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) " + "VALUES(8,9,NULL)" + ) + insert_byte(conn) + insert_item(conn) + insert_decision(conn, pair_id=8, image_id=4) + conn.commit() + self.assertTrue(db.delete_prompt_pair(8)) + with db.model.get_connection() as conn: + self.assertEqual( + rows(conn, f"SELECT pair_id, image_id FROM {DECISIONS}"), [(None, 4)] + ) + with db.model.get_connection() as conn: + conn.execute("DELETE FROM prompt_pairs WHERE id=6") + conn.commit() + self.assertTrue(db.delete_prompt(7)) + with db.model.get_connection() as conn: + self.assertEqual( + rows( + conn, + f"SELECT id, item_id, outcome, candidate_hash, pair_id, image_id " + f"FROM {DECISIONS}", + ), + [(DECISION_ID, ITEM_ID, "selected", CANDIDATE, None, None)], + ) + self.assertEqual(rows(conn, f"SELECT id FROM {ITEMS}"), [(ITEM_ID,)]) + self.assertIsNone(review_schema_error(conn, allow_missing=False)) + # The SET NULL actions look decisions up by index, not by a scan. + for column in ("pair_id", "image_id"): + plan = " ".join( + str(row[3]) + for row in conn.execute( + f"EXPLAIN QUERY PLAN UPDATE {DECISIONS} " # nosec B608 + f"SET {column}=NULL WHERE {column}=?", + (1,), + ) + ) + self.assertIn(f"idx_media_review_decisions_{column[:-3]}", plan) + + def test_full_capacity_unicode_control_pair_round_trips_within_8_mib(self): + self.require_module() + value = full_capacity_evidence() + evidence = canonical_evidence(value) + self.assertEqual(len(evidence.encode("utf-8")), MAX_EVIDENCE_BYTES) + self.assertTrue(evidence.isascii()) + source_path = str(Path(self.temp.name) / "source.db") + source = self.model(source_path) + with source.get_connection() as conn: + conn.execute("PRAGMA wal_autocheckpoint=0") + insert_byte(conn) + insert_item(conn, evidence=evidence, candidates=16) + insert_decision(conn, evidence=evidence) + conn.commit() + source_rows = table_rows(conn, TABLES) + self.assertGreater(Path(source_path + "-wal").stat().st_size, 0) + live = self.model() + backup_file = str(Path(self.temp.name) / "backup.db") + self.assertTrue(source.backup_database(backup_file)) + for restore_from in (source_path, backup_file): + with self.subTest(Path(restore_from).name): + live.restore_from_file(restore_from) + with live.get_connection() as conn: + self.assertEqual(table_rows(conn, TABLES), source_rows) + self.assertIsNone(review_schema_error(conn, allow_missing=False)) + stored = conn.execute( + f"SELECT i.evidence, d.evidence FROM {ITEMS} i " + f"JOIN {DECISIONS} d ON d.item_id = i.id" + ).fetchone() + self.assertEqual(json.loads(stored[0]), value) + self.assertEqual(json.loads(stored[1]), value) + PromptModel.close_all_instances() + assert_closed(self, self.temp.name) + + def test_install_failure_rolls_back_and_retries(self): + self.require_module() + with patch( + "database.review_schema.review_schema_error", + side_effect=[None, "injected final validation failure"], + ): + with self.assertRaisesRegex(sqlite3.DatabaseError, "injected"): + self.model() + with closing(sqlite3.connect(self.path)) as raw: + self.assertEqual(catalog(raw), set()) + self.assertIsNotNone( + raw.execute( + "SELECT 1 FROM sqlite_master WHERE name='media_annotation_bytes'" + ).fetchone() + ) + PromptModel.reset_schema_cache() + model = self.model() + self.assertEqual( + PromptModel.verify_database_file(model.db_path, require_library_state=True), + (True, "ok"), + ) + with model.get_connection() as conn: + self.assertEqual(metadata(conn), [(1, 0)]) + + def test_wal_backup_restore_round_trips_group(self): + self.require_module() + source_path = str(Path(self.temp.name) / "source.db") + source = self.model(source_path) + with source.get_connection() as conn: + conn.execute("PRAGMA wal_autocheckpoint=0") + insert_group_rows(conn) + conn.execute(f"UPDATE {METADATA} SET review_index_revision=3") + conn.commit() + source_rows = table_rows(conn, TABLES) + self.assertGreater(Path(source_path + "-wal").stat().st_size, 0) + live = self.model() + with live.get_connection() as conn: + insert_byte(conn) + insert_item(conn, item_id=OTHER_ID) + source_before = Path(source_path).read_bytes() + backup_file = str(Path(self.temp.name) / "backup.db") + self.assertTrue(source.backup_database(backup_file)) + with closing(sqlite3.connect(backup_file)) as raw: + self.assertEqual(table_rows(raw, TABLES), source_rows) + self.assertIsNone(review_schema_error(raw, allow_missing=False)) + + safety = live.restore_from_file(source_path) + + self.assertEqual(Path(source_path).read_bytes(), source_before) + with live.get_connection() as conn: + self.assertEqual(table_rows(conn, TABLES), source_rows) + self.assertEqual(metadata(conn), [(1, 3)]) + with closing(sqlite3.connect(safety)) as raw: + self.assertEqual(rows(raw, f"SELECT id FROM {ITEMS}"), [(OTHER_ID,)]) + PromptModel.reset_schema_cache() + live.restore_from_file(backup_file) + with live.get_connection() as conn: + self.assertEqual(table_rows(conn, TABLES), source_rows) + PromptModel.close_all_instances() + assert_closed(self, self.temp.name) + + def test_staged_failure_preserves_source_live_safety_and_handles(self): + self.require_module() + live = self.model() + with live.get_connection() as conn: + insert_byte(conn) + insert_item(conn, item_id=OTHER_ID) + live_rows = table_rows(conn, TABLES) + bad = self.tampered( + "restore bad", (f"UPDATE {ITEMS} SET state='done'",), checks_off=True + ) + old = str(Path(self.temp.name) / "old.db") + self.model(old).close_all() + self.drop_tables(old) + for source, patched in ((bad, False), (old, True)): + with self.subTest(source=Path(source).name): + before = Path(source).read_bytes() + safety = sorted(Path(self.temp.name).glob("live.db.backup_*")) + if patched: + context = patch( + "database.models.migrate_review_state", + side_effect=sqlite3.OperationalError("staging failed"), + ) + expected = sqlite3.OperationalError + else: + context = patch.object(PromptModel, "backup_database") + expected = ValueError + with context as hook, self.assertRaises(expected): + live.restore_from_file(source) + if not patched: + hook.assert_not_called() + self.assertEqual(Path(source).read_bytes(), before) + self.assertEqual( + sorted(Path(self.temp.name).glob("live.db.backup_*")), safety + ) + assert_closed(self, source) + with live.get_connection() as conn: + self.assertEqual(table_rows(conn, TABLES), live_rows) + self.assertFalse( + PromptModel.verify_database_file(old, require_library_state=True)[0] + ) + self.assertTrue(PromptModel.verify_database_file(old)[0]) + + def test_verify_requires_group_only_with_require_library_state(self): + self.require_module() + self.model().close_all() + self.assertEqual( + PromptModel.verify_database_file(self.path, require_library_state=True), + (True, "ok"), + ) + self.drop_tables(self.path) + self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok")) + self.assertEqual( + PromptModel.verify_database_file(self.path, require_library_state=True), + (False, "Missing review queue schema"), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_scan_scope.py b/tests/test_scan_scope.py index f5f085f..f10fb40 100644 --- a/tests/test_scan_scope.py +++ b/tests/test_scan_scope.py @@ -38,6 +38,11 @@ def _graph(text): } +def _scope_part(scope): + """The per-source scope keys this module pins (other features add counters).""" + return {key: scope[key] for key in ("files", "roots", "output_folders")} + + class ScanScopeTestCase(AdminAPITestCase): """Helpers: link files the way the scan and the monitor do, spy on reads.""" @@ -342,7 +347,8 @@ class TestPerSourceScope(ScanScopeTestCase): self.assertEqual(walked, [["b"]]) self.assertEqual(reads, ["in_b.png"]) self.assertEqual( - job["scope"], {"files": "all", "roots": [1], "output_folders": 1} + _scope_part(job["scope"]), + {"files": "all", "roots": [1], "output_folders": 1}, ) self.assertEqual( job["outcomes"]["added"], @@ -378,7 +384,8 @@ class TestPerSourceScope(ScanScopeTestCase): unscoped = await self._scan({"scope": "all"}) self.assertEqual( - unscoped["scope"], {"files": "all", "roots": None, "output_folders": 2} + _scope_part(unscoped["scope"]), + {"files": "all", "roots": None, "output_folders": 2}, ) self.assertIn("Scan started · 2 output folders · all files", unscoped["log"][0]) @@ -452,7 +459,7 @@ class TestPerSourceScope(ScanScopeTestCase): self.assertEqual(final["directories"], ["output/a", "output/b"]) self.assertEqual(final["added"], 2) self.assertEqual( - self.api._scan_job.scope, + _scope_part(self.api._scan_job.scope), {"files": "all", "roots": None, "output_folders": 2}, ) diff --git a/tests/test_system_api.py b/tests/test_system_api.py index 0a90208..99891bd 100644 --- a/tests/test_system_api.py +++ b/tests/test_system_api.py @@ -292,7 +292,7 @@ class TestOverviewRoute(SystemAPITestCase): body = await self.json("GET", "/prompt_manager/system/overview") self.assertIsNone(body["health"]) self.assertIsNone(body["backup"]) - self.assertEqual(body["needs_review"], {"pending": None}) + self.assertEqual(body["needs_review"], {"pending": 0}) self.save("Any") await self.run_health() report = (await self.json("GET", "/prompt_manager/system/health"))["report"] diff --git a/utils/image_monitor.py b/utils/image_monitor.py index a42504e..cfe1a6a 100644 --- a/utils/image_monitor.py +++ b/utils/image_monitor.py @@ -27,7 +27,7 @@ import os import queue import time import threading -from typing import Optional, Dict, Any, Tuple +from typing import Callable, Optional, Dict, Any, Tuple from watchdog.observers import Observer from watchdog.events import FileSystemEventHandler @@ -75,6 +75,10 @@ class ImageGenerationHandler(FileSystemEventHandler): self.prompt_tracker = prompt_tracker self.metadata_extractor = ComfyUIMetadataExtractor() self.logger = get_logger("prompt_manager.image_monitor") + # The Needs review pass for ambiguous outputs (the API's + # ``record_review_file``, wired at start-up): a linked output is + # never queued and its pending items leave the queue. None: no pass. + self.review_recorder: Optional[Callable[[str], str]] = None # Read from GalleryConfig if available, otherwise use defaults try: @@ -579,6 +583,29 @@ class ImageGenerationHandler(FileSystemEventHandler): ) except Exception as e: self.logger.error(f"Failed to link image to prompt: {e}") + return + if resolution["state"] == "ambiguous": + self._record_review(image_path) + + def _record_review(self, image_path: str) -> None: + """The Needs review pass for an ambiguous output, best effort. + + The output is linked by now, so it is never queued; a pending item + an earlier scan left for its path leaves the queue. The link above + stays whatever happens here: a failure is logged, never raised. + """ + recorder = getattr(self, "review_recorder", None) + if recorder is None: + return + try: + outcome = recorder(image_path) + except Exception: + self.logger.warning( + f"{os.path.basename(image_path)} could not be added to Needs review", + exc_info=True, + ) + return + self.logger.debug(f"Needs review {outcome}: {os.path.basename(image_path)}") class ImageMonitor: @@ -608,6 +635,16 @@ class ImageMonitor: self.handler = None self.monitored_directories = [] self.logger = get_logger("prompt_manager.image_monitor") + self.review_recorder: Optional[Callable[[str], str]] = None + + def set_review_recorder(self, recorder: Optional[Callable[[str], str]]) -> None: + """Record ambiguous outputs through ``recorder`` (None stops recording). + + Applies to the running handler and to every handler started later. + """ + self.review_recorder = recorder + if self.handler is not None: + self.handler.review_recorder = recorder def start_monitoring(self, output_directories: Optional[list] = None): """ @@ -665,6 +702,7 @@ class ImageMonitor: # Only a started observer becomes self.observer, so a failed start never # leaves the monitor stuck "already running" with nothing to stop. handler = ImageGenerationHandler(self.db_manager, self.prompt_tracker) + handler.review_recorder = self.review_recorder observer = Observer() scheduled = [] for output_dir in valid_directories: diff --git a/utils/media_capture.py b/utils/media_capture.py index 1932f0b..b826bc8 100644 --- a/utils/media_capture.py +++ b/utils/media_capture.py @@ -46,6 +46,10 @@ DEFAULT_MAX_BYTES = 512 * 2**20 DEFAULT_MAX_WORKERS = 2 DEFAULT_IDLE_SECONDS = 900 DEFAULT_HARD_SECONDS = 1800 +# Background recording (Needs review from the scan and the monitor) may use +# every worker and capture slot but the last one, which stays free for an +# interactive Tag dialog or Apply; it waits this long for room before failing. +BACKGROUND_WAIT_SECONDS = 2.0 MAX_INFERENCE_SIDE = 4096 MAX_INFERENCE_PIXELS = 16_777_216 MAX_RELATIVE_LENGTH = 1024 @@ -80,7 +84,7 @@ class CaptureError(Exception): class CaptureCapacity(CaptureError): - """Count, byte or worker admission failed (nonblocking).""" + """Count, byte or worker admission failed (interactive: nonblocking).""" code = "review-capacity" @@ -220,6 +224,8 @@ class CaptureStore: # mkdtemp creates the directory with owner-only permissions. self.directory = tempfile.mkdtemp(prefix=_DIRECTORY_PREFIX, dir=parent) self._lock = threading.Lock() + # Signalled whenever a worker or a capture slot frees up. + self._room = threading.Condition(self._lock) self._entries: dict[str, _Entry] = {} self._completed_bytes = 0 self._pending = 0 @@ -238,8 +244,14 @@ class CaptureStore: epoch: str, scope: str, root_index: int = 0, + background: bool = False, ) -> Capture: - """Copy and hash one rooted media file into a private capture.""" + """Copy and hash one rooted media file into a private capture. + + A ``background`` capture never takes the last worker or capture slot + and waits up to ``BACKGROUND_WAIT_SECONDS`` for room; an interactive + one is admitted or refused at once. + """ if isinstance(root_index, bool) or not isinstance(root_index, int): raise CaptureDenied("Invalid gallery root") if root_index < 0: @@ -250,7 +262,7 @@ class CaptureStore: parts, extension = _relative_parts(relative) candidate, locator = _resolve(root, parts) before = self._pre_stat(candidate) - self._admit(before.st_size, reserve=True) + self._admit(before.st_size, reserve=True, background=background) try: capture = self._copy( candidate, locator, extension, before, root_index, epoch, scope @@ -386,35 +398,52 @@ class CaptureStore: # -- admission ------------------------------------------------------------- - def _admit(self, size: int, *, reserve: bool) -> None: + def _admit(self, size: int, *, reserve: bool, background: bool = False) -> None: try: - self._admit_locked(size, reserve=reserve) + self._admit_locked(size, reserve=reserve, background=background) finally: self._flush() - def _admit_locked(self, size: int, *, reserve: bool) -> None: + def _admit_locked(self, size: int, *, reserve: bool, background: bool) -> None: + deadline = time.monotonic() + BACKGROUND_WAIT_SECONDS with self._lock: - if self._closed: - raise RuntimeError("The capture store is closed") - self._sweep_locked(self._clock()) - if self._workers >= self.max_workers: - raise CaptureCapacity("Too many captures are in progress") + while True: + if self._closed: + raise RuntimeError("The capture store is closed") + self._sweep_locked(self._clock()) + try: + self._check_room_locked(size, reserve, background) + break + except CaptureCapacity: + remaining = deadline - time.monotonic() + if not background or remaining <= 0: + raise + self._room.wait(remaining) if reserve: - if len(self._entries) + self._pending >= self.max_captures: - raise CaptureCapacity("Too many captures are held") - reserved = self._completed_bytes + self._pending_bytes - if reserved + size > self.max_bytes: - raise CaptureCapacity("Capture byte budget exhausted") self._pending += 1 self._pending_bytes += size self._workers += 1 + def _check_room_locked(self, size: int, reserve: bool, background: bool) -> None: + """Raise ``CaptureCapacity`` unless one more capture or verify fits.""" + spare = 1 if background else 0 + if self._workers + spare >= self.max_workers: + raise CaptureCapacity("Too many captures are in progress") + if reserve: + held = len(self._entries) + self._pending + if held + spare >= self.max_captures: + raise CaptureCapacity("Too many captures are held") + reserved = self._completed_bytes + self._pending_bytes + if reserved + size > self.max_bytes: + raise CaptureCapacity("Capture byte budget exhausted") + def _release_admission(self, size: int, *, reserve: bool) -> None: with self._lock: self._workers -= 1 if reserve: self._pending -= 1 self._pending_bytes -= size + self._room.notify_all() last_out = self._closed and self._pending == 0 and self._workers == 0 if last_out: self._remove_directory() @@ -457,6 +486,7 @@ class CaptureStore: if self._entries.pop(entry.capture.token, None) is not None: self._completed_bytes -= entry.capture.size self._stale.append(entry.capture.path) + self._room.notify_all() def _flush(self) -> None: """Unlink files released under the lock, without holding it (slow I/O).""" diff --git a/utils/prompt_pair_resolver.py b/utils/prompt_pair_resolver.py index 3f42b0d..4ad10a5 100644 --- a/utils/prompt_pair_resolver.py +++ b/utils/prompt_pair_resolver.py @@ -39,6 +39,12 @@ from .pair_execution import ExecutionEvidenceError, read_pair_execution _LOGGER = logging.getLogger("prompt_manager.prompt_pair_resolver") +# The ingestion resolver's version, recorded in pair-link provenance and on +# Needs review items. Bump it when a change would resolve the same metadata +# otherwise; a pending item stays applicable while its chosen candidate is +# still offered by the captured bytes. +RESOLVER_VERSION = 1 + @dataclass(frozen=True) class OutputAdapter: diff --git a/utils/review_capture.py b/utils/review_capture.py new file mode 100644 index 0000000..f26a46b --- /dev/null +++ b/utils/review_capture.py @@ -0,0 +1,299 @@ +"""Capture-bound Needs review items for ambiguous files (#249 E2). + +The output scan and the live monitor hand every file the ingestion resolver +finds ``ambiguous`` to :func:`record_ambiguous_file`. It captures the file +(the bounded private copy of ``utils.media_capture``, hashed while copied), +reads the metadata again from the capture, resolves it with the default +(ingestion) profile and stores only an ``ambiguous`` result, bound to the +capture's byte record. The evidence a reviewer chooses from is therefore the +evidence of exactly those bytes. An ambiguous result is stored only when it +is ``reviewable``: at least two candidates carry a resolved prompt text. With +none the file could not be traced; with exactly one, the untraced branches +may still have made the file, so that text is not certain and the file stays +unresolved (the resolver never flattens unknown semantics into a guess). +Nothing here links a file, loads a model or decodes pixels; the capture is +released before the database write. + +Evidence is small on purpose (startup validates every row): the resolver's +reason and, per candidate sampler, ``{sampler_id, state, reason, positive, +negative}`` where each side is ``{text, storage_text, provenance}``. A +candidate's ``hash`` is the SHA-256 of its canonical JSON without the hash. +More than ``MAX_CANDIDATES`` candidates or evidence over the 8 MiB bound make +a ``limited`` item without candidates (a result without any candidate is not +``reviewable`` and is never stored). +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import os +from pathlib import Path +from typing import Any + +try: + from ..database.library_commands import LibraryCommandError + from ..database.review_schema import ( + MAX_CANDIDATES, + MIN_CHOOSABLE, + canonical_evidence, + choosable, + ) + from .media_capture import CaptureCapacity, CaptureError + from .prompt_inspector import MetadataReadError, read_inspection_file + from .prompt_pair_resolver import RESOLVER_VERSION, resolve_prompt_pair +except ImportError: # imported with the repo root on sys.path + from database.library_commands import LibraryCommandError + from database.review_schema import ( + MAX_CANDIDATES, + MIN_CHOOSABLE, + canonical_evidence, + choosable, + ) + from utils.media_capture import CaptureCapacity, CaptureError + from utils.prompt_inspector import MetadataReadError, read_inspection_file + from utils.prompt_pair_resolver import RESOLVER_VERSION, resolve_prompt_pair + +logger = logging.getLogger("prompt_manager.review_capture") + +# Capture scope label: review captures are never handed to a client. +REVIEW_SCOPE = "needs-review" +# Nested ambiguous stages are flattened at most this deep. +MAX_CANDIDATE_DEPTH = 8 + + +def record_ambiguous_file(db, store, root, root_index: int, relative: str) -> str: + """Persist the ambiguous evidence of one rooted file. + + Returns ``recorded`` (new item with candidates), ``limited`` (new item + without candidates), ``duplicate`` (these bytes already carry this + evidence, or the file's current stat is the verified fingerprint of + bytes an item cites, so it is not even copied, and that item is still + pending), ``decided`` (as ``duplicate``, but the item carries a decision, + so the file is not in the queue), ``untraced`` (ambiguous in its captured + bytes, but not ``reviewable``: nothing to choose, nothing stored) or + ``skipped`` (the file + changed, vanished, is not ambiguous in its captured bytes, or storage was + restored meanwhile). + ``CaptureCapacity`` and database errors propagate to the caller, which + logs and counts them. + """ + spelled_path, queued = _queued_spelling(db, root, relative) + if queued is not None: + return _already_queued(db, *queued) + epoch = db.get_execution_epoch() + try: + # Background: the last worker stays free for an interactive request. + capture = store.capture( + root, + relative, + epoch=epoch, + scope=REVIEW_SCOPE, + root_index=root_index, + background=True, + ) + except CaptureCapacity: + raise + except CaptureError as error: + logger.debug("Review capture skipped (%s): %s", error.code, error) + return "skipped" + try: + captured = captured_review(store, capture) + finally: + store.release(capture.token) + resolution = captured[1] if captured is not None else None + if resolution is None or resolution.get("state") != "ambiguous": + return "skipped" + if not reviewable(resolution): + return "untraced" + evidence, count, limited = review_evidence(resolution) + try: + inserted = db.record_review_item( + locator=capture.locator, + spelled_path=spelled_path, + sha256=capture.sha256, + size=capture.size, + mtime_ns=capture.mtime_ns, + epoch=capture.epoch, + resolver_version=RESOLVER_VERSION, + evidence=evidence, + candidate_count=count, + limited=limited, + ) + except LibraryCommandError: + return "skipped" + if not inserted: + return _already_queued(db, capture.locator, capture.size, capture.mtime_ns) + return "limited" if limited else "recorded" + + +def _already_queued(db, locator: str, size: int, mtime_ns: int) -> str: + """``duplicate`` while the existing item is pending, else ``decided``.""" + if db.review_pending_at(locator, size, mtime_ns, RESOLVER_VERSION): + return "duplicate" + return "decided" + + +def _queued_spelling(db, root, relative: str) -> tuple[str, tuple | None]: + """The file's spelling, and ``(locator, size, mtime_ns)`` when queued. + + The second value is None unless the file's current stat is already + queued. + + The spelling is the capture's own candidate path (the real root joined + with the listed relative path), so it folds to the capture's locator. A + file whose size and ``mtime_ns`` equal the verified fingerprint of bytes + an item already cites is not copied or hashed again (maintainer ruling: + the rescan stays cheap and the file stays in the default scope); its + items take the current spelling. A stat failure leaves the decision to + the capture. + """ + spelled = os.path.join(os.path.realpath(os.fspath(root)), *relative.split("/")) + try: + facts = os.lstat(spelled) + except OSError: + return spelled, None + locator = os.path.normcase(spelled) + if db.review_fingerprint_queued( + locator, facts.st_size, facts.st_mtime_ns, RESOLVER_VERSION + ): + # Not copied again, but a case-only rename keeps the stat: record + # the current spelling so the item's target still opens. + db.refresh_review_spelling(locator, spelled) + return spelled, (locator, facts.st_size, facts.st_mtime_ns) + return spelled, None + + +def captured_review(store, capture) -> tuple[dict, dict] | None: + """``(metadata, resolution)`` of the capture's own bytes, else None. + + ``metadata`` is what the scan hands the ingester: prompt and workflow + JSON-decoded and ``file_info`` (size, dimensions, format) from the + capture. None when the capture is gone or carries no prompt metadata. + """ + if store.acquire(capture.token) is None: + return None + try: + try: + read = read_inspection_file(Path(capture.path)) + except MetadataReadError: + return None + finally: + store.release_use(capture.token) + canonical = dict(read.metadata) + for key in ("prompt", "workflow"): + if isinstance(canonical.get(key), str): + try: + canonical[key] = json.loads(canonical[key]) + except (ValueError, TypeError, RecursionError): + # Malformed metadata stays as text for the resolver to reject. + pass + if not canonical.get("prompt") and not canonical.get("parameters"): + return None + info = {"size": capture.size} + if read.width is not None and read.height is not None: + info["dimensions"] = [read.width, read.height] + if read.format is not None: + info["format"] = read.format + canonical["file_info"] = info + return canonical, resolve_prompt_pair(canonical) + + +def reviewable(resolution: Any) -> bool: + """True when an ambiguous result offers a choice: ``MIN_CHOOSABLE`` of its + selectable candidates (nested ambiguous stages flattened) are choosable. + + Counted before the evidence is bounded, so a ``limited`` item (more than + ``MAX_CANDIDATES`` candidates) with prompts to choose is still queued. + """ + if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous": + return False + count = 0 + for candidate in _flattened(resolution.get("candidates"), 0): + if choosable(candidate): + count += 1 + if count >= MIN_CHOOSABLE: + return True + return False + + +def review_evidence(resolution: dict) -> tuple[dict, int, bool]: + """``(evidence, candidate_count, limited)`` for one ambiguous result.""" + base = { + "resolver_version": RESOLVER_VERSION, + "reason": _text(resolution.get("reason")), + } + candidates: list[dict] = [] + seen: set[str] = set() + total = 0 + for candidate in _flattened(resolution.get("candidates"), 0): + total += 1 + try: + projected = _projection(candidate) + digest = hashlib.sha256( + canonical_evidence(projected).encode("utf-8") + ).hexdigest() + except (ValueError, TypeError, KeyError): + return _limited(base, total), 0, True + if digest in seen: + total -= 1 + continue + seen.add(digest) + candidates.append({**projected, "hash": digest}) + if not candidates or len(candidates) > MAX_CANDIDATES: + return _limited(base, total), 0, True + evidence = {**base, "candidates": candidates} + try: + canonical_evidence(evidence) + except ValueError: + return _limited(base, total), 0, True + return evidence, len(candidates), False + + +def _limited(base: dict, total: int) -> dict: + return {**base, "limited": True, "candidate_total": total, "candidates": []} + + +def _flattened(candidates: Any, depth: int): + """Selectable candidates; a nested ambiguous stage yields its own.""" + if not isinstance(candidates, list): + return + for candidate in candidates: + if not isinstance(candidate, dict): + continue + nested = candidate.get("candidates") + if ( + candidate.get("state") == "ambiguous" + and isinstance(nested, list) + and nested + and depth < MAX_CANDIDATE_DEPTH + ): + yield from _flattened(nested, depth + 1) + else: + yield candidate + + +def _projection(candidate: dict) -> dict: + return { + "sampler_id": candidate.get("sampler_id"), + "state": _text(candidate.get("state")), + "reason": _text(candidate.get("reason")), + "positive": _side(candidate.get("positive")), + "negative": _side(candidate.get("negative")), + } + + +def _side(side: Any) -> dict | None: + if not isinstance(side, dict): + return None + text = side["text"] + return { + "text": text, + "storage_text": side.get("storage_text", text), + "provenance": side.get("provenance"), + } + + +def _text(value: Any) -> str: + return value if isinstance(value, str) else "" diff --git a/web/js/workspace/system-overview.mjs b/web/js/workspace/system-overview.mjs index ac90206..7d770ca 100644 --- a/web/js/workspace/system-overview.mjs +++ b/web/js/workspace/system-overview.mjs @@ -323,7 +323,7 @@ function overviewNodes(overview, { scope, navigate, openStatistics }) { description: Number.isSafeInteger(overview.needs_review?.pending) ? 'Review metadata before assigning a positive / negative pair.' : NOT_AVAILABLE, label: 'Review metadata', onClick: () => {} }); - // D93 (Q6): the review queue (Needs review) is not part of the workspace yet. + // D93: the row reads the queue's pending count; its action waits for the Needs review page (#288). review.querySelector('button').setAttribute('aria-disabled', 'true'); const [sourcesTitle, sourcesDescription] = sourcesCopy(overview.sources); const thumbnails = operationRow({ scope, title: THUMBNAILS_LOADING, description: 'Generate only missing previews when necessary.',