From c86739178d89b26d578b2a31502ba306d7a49ecf Mon Sep 17 00:00:00 2001 From: Vito Date: Fri, 23 Oct 2026 04:59:18 -0700 Subject: [PATCH 1/4] feat(system): System page shell, review dialog and Overview (#298) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(system): read-only health check and truthful overview counts POST /prompt_manager/system/health runs the health check off-loop as a read registry job of kind "health" (one at a time, beside a scan, not cancellable, no file outcomes) and keeps the last report; GET returns it or null. The check reads one snapshot on its own mode=ro connection, closed in finally: integrity_check(100), foreign_key_check, stored hashes against utils.hashing, duplicate pair rows, the legacy consistency queries (now shared with check_consistency), and image links whose file is gone, counted only under readable sources. An unavailable source, or the absent Needs-review queue, makes its check unavailable and the report incomplete instead of producing a count. GET /prompt_manager/system/overview returns Library-eligible positive prompts and active negative variants, indexed images, the last health summary or null, backup null, the Needs-review count null, the effective sources with whether they were auto-detected, and the thumbnail plan's missing count. GET /prompt_manager/system/statistics lists runs and average rating per category of Library parents, with the /stats totals. * feat(system): native System route, drafts and shared review dialog - #/system/
renders the concept's System shell: section nav, settings layout and Save row with truthful status (D92); each section not delivered yet hands off to its legacy tools (D94). Overview's System card opens #/system/overview (Q12). - A module-level draft store saves browser, then host (expected_revision), then runtime; failed domains stay dirty, a stale revision is a conflict. - System row builders (setting, operation, section nav) on the scope. - openReviewDialog and backupNotice extracted from the scan review, which is now a thin caller with identical markup. - Overview images gain a "rated" count (images whose linked prompt is rated); the missing-files health check reports per source, so partial availability is visible. * feat(system): native Overview with health report and statistics System > Overview renders the concept's metrics and its Library health and Workspace preferences surfaces from GET /system/overview: real counts, "Not checked" until a health check ran, no invented review count, backup or thumbnail status (D93). Explore opens "Usage statistics" with the real runs and ratings per category and the Library totals (D96); the statistics route now also returns rated_images. Maintenance shows the concept's notice with Run health checks above its hand-off: the read-only check runs as a registry read job and opens the "Health report" with a chip per real result, missing files per source (D95). A load failure shows Retry and no zeros; late answers never repaint a newer section. Adds the System concept comparison (shell, Overview, both dialogs, the notice, mutation checks), the System workflow spec and its fixture hooks, and the TESTING.md "Native System" section. * test(system): coverage floors for the native Overview Adds the system-overview.mjs baseline and raises the system.mjs floors to their measured coverage; no other floor changes. * fix(system): keep health reports per database and off the event loop - A restore that replaces the database forgets the stored health report, and a check that started before the swap is not kept (database generation stamped at start). - The Overview no longer walks every source for the missing-thumbnail count; the page reads /images/thumbnails/status on its own. - Image sources for the health check and the Overview resolve in the executor; a failure to start the check answers the coded envelope. - Any exception fails only its own check; missing files reads its rows in the snapshot and stats the files after it ended. - The read-only URI keeps Windows UNC paths (empty authority). - An integrity count at SQLite's limit is flagged as capped. * fix(system): thumbnail row read, tracked health job and guarded host save - The Overview's thumbnail row is its own read of the thumbnail status: "Checking thumbnails…" until it answers, unavailable when it fails. - The health job is tracked in the Activity store (new store.track) and followed through its event stream instead of polling every 150 ms. - A host save before a read returned a revision fails without a request; System reads the host values on mount and after a conflict. - "100+ problems found" when the integrity count is capped. - D-NEW-S2-5 is D123 everywhere; the unused Overview hand-off is gone. - The concept spec builds the health report from the concept's own rows and asserts the full unmapped path list; surface tops pin the 18px offset. - TESTING.md: D96's reason, the read-only (WAL/SHM) standard, the UNC URI and the new behavior. Coverage floors raised for the touched files. * fix(system): make two health tests portable to Windows - Windows forbids "?" in file names, so the URI-character open test uses "#", "%" and a space there. - ntpath.realpath on Python 3.11+ returns a NUL-byte path instead of raising, so the bad-path test raises as POSIX does to reach the missing-files failure path on every platform. --- TESTING.md | 136 ++++ database/health.py | 354 +++++++++++ database/library.py | 51 ++ database/operations.py | 34 +- py/api/__init__.py | 27 +- py/api/admin.py | 1 + py/api/jobs.py | 3 + py/api/system.py | 207 +++++++ tests/browser/server.py | 2 + tests/browser/system-concept.spec.mjs | 410 +++++++++++++ tests/browser/workspace-system.spec.mjs | 252 ++++++++ tests/browser/workspace.spec.mjs | 97 ++- tests/browser/workspace_system_fixture.py | 116 ++++ tests/js/workspace-activity-store.test.js | 18 + tests/js/workspace-state.test.js | 46 +- tests/js/workspace-system-controls.test.js | 214 +++++++ tests/js/workspace-system-overview.test.js | 255 ++++++++ tests/js/workspace-system-state.test.js | 261 ++++++++ tests/js/workspace-toolbar.test.js | 143 +++++ tests/test_database_health.py | 614 +++++++++++++++++++ tests/test_library.py | 97 +++ tests/test_system_api.py | 422 +++++++++++++ tools/vendor/frontend-coverage-baseline.json | 24 + web/js/workspace/activity-store.mjs | 10 + web/js/workspace/index.mjs | 29 +- web/js/workspace/state.mjs | 34 +- web/js/workspace/system-controls.mjs | 132 ++++ web/js/workspace/system-overview.mjs | 432 +++++++++++++ web/js/workspace/system-state.mjs | 201 ++++++ web/js/workspace/system.mjs | 148 +++++ web/js/workspace/workspace-dialogs.mjs | 143 +++-- web/js/workspace/workspace.css | 9 + 32 files changed, 4831 insertions(+), 91 deletions(-) create mode 100644 database/health.py create mode 100644 py/api/system.py create mode 100644 tests/browser/system-concept.spec.mjs create mode 100644 tests/browser/workspace-system.spec.mjs create mode 100644 tests/browser/workspace_system_fixture.py create mode 100644 tests/js/workspace-system-controls.test.js create mode 100644 tests/js/workspace-system-overview.test.js create mode 100644 tests/js/workspace-system-state.test.js create mode 100644 tests/test_database_health.py create mode 100644 tests/test_system_api.py create mode 100644 web/js/workspace/system-controls.mjs create mode 100644 web/js/workspace/system-overview.mjs create mode 100644 web/js/workspace/system-state.mjs create mode 100644 web/js/workspace/system.mjs diff --git a/TESTING.md b/TESTING.md index cab865a..208e190 100644 --- a/TESTING.md +++ b/TESTING.md @@ -1816,6 +1816,142 @@ npm run --prefix tools/vendor test:workflows -- inspect-concept.spec.mjs workspa npm run --prefix tools/vendor test:comfyui -- inspect-sender.spec.mjs ``` +### Native System (#249 System) + +`#/system/
` is a native workspace route rendered by `web/js/workspace/system.mjs` +on the concept's `renderSystem` markup (app.js a698–706; `concept.css` 709–895). `#/system` +is Overview under the sidebar's own hash; `#/system/overview|appearance|sources| +integrations|tagging|maintenance|backups|diagnostics` select a section, and an unknown +child (the concept's `#/system/general`) is replaced by `#/system/overview`. The +production hub `#/overview` is unchanged (D28); its System card opens +`#/system/overview` (Q12). Switching sections keeps the shell, the drafts and focus on +the pressed section button and disposes the previous section's scope, so its reads, +listeners and dialogs never outlive it. + +Drafts (`system-state.mjs`) live for the page session (Q10). Save writes the dirty +browser fields, then the host fields (`POST /prompt_manager/settings` with +`expected_revision`, #220), then the runtime fields. A failed domain keeps its drafts and +highlights their controls; `settings-changed` is a conflict and keeps the draft. System +reads the host values on mount and again after a conflict; until a read returned a +revision, a host save fails without a request, so it never sends a blind +`expected_revision: null` (which S1 merges into the file on disk). No delivered section +edits a setting yet, so Save stays disabled until Appearance (S3). + +Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per +visit through `latestOnly()` on the section scope; a late answer after the section left +paints nothing. Until it answers, the section shows "Loading the overview…"; a failure +shows "The overview could not be loaded. Your saved data has not changed." and Retry, +never zeros. The thumbnail row is a separate read of `/images/thumbnails/status` (a +walk of every source), so the other values never wait for it: "Checking thumbnails…" +until it answers. The health check (`POST /prompt_manager/system/health`) is a `read` +registry job of kind `health`: one at a time, beside a scan, not cancellable and with +no file outcomes. The page tracks the started job in the Activity store (its event +stream; the store marks it lost if the server stops listing it), never by polling. A press while one runs (another tab) waits +for that one. The last report lives for this server session (`GET +/prompt_manager/system/health`) and describes the database it was taken from: a +restore that replaces the database forgets it, and a check that started before the +swap is not kept. The check's own connection opens a `mode=ro` URI that keeps a Windows +UNC path (`file:////server/share/prompts.db`; pathlib's `file://server/share/...` is a +remote authority SQLite refuses). A check that raises fails only itself, and the +missing-files check reads its rows inside the snapshot but stats the files after it +ended, so a slow source never pins the WAL. Statistics +read `GET /prompt_manager/system/statistics` (`by_category`, the legacy `/stats` totals +and `rated_images`). + +Read-only standard for a check or preview (S2's health check; S6 and S8 previews use +the same): the database and `-wal` bytes are unchanged, the `-shm` size is unchanged, +the connection's `total_changes` is 0 at close and the data reads back unchanged. The +`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no +reader can keep it byte-identical (`test_health_check_never_writes`). + +Recorded deviations (plan D60–D64 renumbered D92–D96; D121, D122 and D123 assigned): + +- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No + unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server + configuration." / "Applied until the server restarts.", "Some changes were not saved. + Review the highlighted settings.", "Saved for this tab only; browser storage is + unavailable." and "Settings changed elsewhere. Reload this section to see them." + "Save preferences" is `aria-disabled` (opacity .55, `not-allowed`) while clean or + saving; a setting the last save could not save gets a red outline. CSS under D92. +- **D93** (Overview): real values. The metrics are the Library's eligible positives + and active saved negatives and the indexed `generated_images` rows (Q2). "Database + integrity" is "Not checked" until a check ran this session, then "N issues" (checks + that found issues, even beside a check that could not run), "Passed" or "Incomplete". + "Ambiguous image links" reads "Not available yet." with its Review metadata action + `aria-disabled` (CSS under D93) until the Needs review queue reports a count (Q6). + "Backup status" reads "No retained backup record." until verified backups exist (S8). + The thumbnail row is "Checking thumbnails…" until its own read answers, then "No + missing thumbnails", "N missing thumbnails" (the pending count of + `/images/thumbnails/status`) or "Thumbnail status unavailable"; Review thumbnails + opens Sources until the thumbnail dialog (S5). "One display configuration" says "Grid + and density settings are shared across views." (no paging). The sources row is + "N output sources" with "Monitoring and enrichment share one source list.", prefixed + by "M of N configured sources available." or "None configured is available; ComfyUI's + output folder is used." when that is the case. +- **D94** (all sections): a section not delivered yet shows the hand-off panel "Your + current tools are ready" with a link to each legacy tool that still owns it (the + plan's legacy coverage table); retired section by section, gone after S12. +- **D95** (health report): the concept's `health-check` dialog titled "Health report", + with no sample note. It lists every check in the server's order: Integrity check, + Foreign keys & pair references, Canonical hashes, Pair uniqueness (added), Missing + files, Ambiguous metadata, Consistency (added). Missing files has one row per image + source when there are several ("Missing files · output"). Each chip is the real + result: "Passed", a count with what was found ("2 broken references", "1 link needs + review", the concept's "N links need review" and "N images need review"; "100+ + problems found" when SQLite's integrity check stopped at its limit), "Not + available" (an unavailable source), "Not available yet" (Ambiguous metadata, Q6) or + "Could not check". A `p.small` after the list says when it was checked and whether + every check passed, some could not run, or issues need review. Close / Review missing + links (to Maintenance, its hand-off until S7). Nothing is changed by a check. +- **D96** (Statistics): "Usage statistics" without the sample note: the concept's + "Runs by category" table (Category / Runs / Avg. rating) with the real runs and + average rating of eligible parents per category ("Uncategorized", "Not rated"), + then a second table "Library totals" (Total / Count: Prompts, Tags, Images from the + legacy statistics, and Rated images: indexed images whose linked prompt is rated). + The Overview row says "Runs and ratings by category, with library totals." because + that is what the dialog shows (the concept's own dialog shows only "Runs by + category"); per-prompt run counts, prompt tags and dated runs are recorded, but no + table of them exists yet. A failed read toasts "Statistics could not + be loaded. Your saved data has not changed." +- **D121** (production hub): the Overview hub's System card reads the concept's System + heading and "Open System"; the concept has no hub card. +- **D122** (shell): the section nav's `aria-label` is "System sections"; the concept's + literal label is its prototype action name "system-tab sections". +- **D123** (Maintenance): the concept's Maintenance notice + ("Check first. Review changes. Then apply." + Run health checks) is delivered ahead of + the rest of Maintenance, above its D94 hand-off, as the native place that runs the + health check. A press shows "Checking…" (`aria-disabled`) and opens the report; a + failed check toasts "The health check could not finish. Nothing was changed." + +`tests/browser/system-concept.spec.mjs` compares the shell and Overview with the concept +(dark and light, 1600×1100 and 390×844), mapping only the deviations above; unmapped, +the difference is exactly the D92, D93, D96 and D122 paths. Each surface's `top` is +measured, which pins the concept's `.surface + .surface` 18px offset that concept.css +also applies inside `.two-col` (copied as is). `statistics and health dialogs match +the concept` maps D95 from the concept's own five rows (labels verbatim): it inserts +Pair uniqueness after Canonical hashes and Consistency last, splits Missing files per +source when there are several, replaces only the chips with the fixture's expected +results, and asserts the full unmapped path list; it maps D96 and compares the +Maintenance notice unchanged. `system comparison detects mutations` removes a row, +renames a metric label, swaps the `--line` token, recolours a chip, renames a check, +removes the totals table and renames a caption. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`, +`{concept,production}-{statistics,health}-{dark,light}.png`) are test artifacts; there +are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts, +the health check and metric 4, a failed check, the thumbnail row against the status +endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the +route, a load failure with Retry and late answers after leaving. Its hooks are +`tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected +failures, an absent source, a deleted linked file, unthumbnailed files) on the +concept-shaped fixture, with delays through the shared request holds. + +```sh +python3 -m unittest tests.test_database_health tests.test_system_api tests.test_library -v +node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-controls.test.js \ + tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js +npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \ + --project chromium --project touch --project firefox --project webkit +``` + ## Workspace data fields for the concept pages (#249) These fields are additive. The legacy `/prompt_manager/images/output` listing (no diff --git a/database/health.py b/database/health.py new file mode 100644 index 0000000..7feedda --- /dev/null +++ b/database/health.py @@ -0,0 +1,354 @@ +"""The read-only health check behind System › Overview and Maintenance. + +``run_health_checks`` opens its own ``mode=ro`` connection, reads one +snapshot and closes the connection in ``finally``; it never writes. Each +check reports ``passed``, ``issues`` (with an exact ``count``; the +integrity check stops at ``INTEGRITY_LIMIT`` problems and then says +``capped``), +``unavailable`` (it could not be checked here, so ``count`` is None, or +``missing_files`` checked only some sources and found nothing) or ``failed`` +(it raised; the other checks still run). ``missing_files`` also reports each +source and whether the check was ``partial``; it reads the image rows inside +the snapshot and checks their files after the snapshot ended, so a slow source +never holds the WAL. At most ``MAX_DETAIL`` detail strings are returned per +check, and none of them carries a server path. +""" + +import datetime +import logging +import ntpath +import os +import sqlite3 +from pathlib import PurePosixPath, PureWindowsPath +from urllib.parse import quote + +try: + from ..utils.hashing import generate_prompt_hash + from .models import normalize_image_path +except ImportError: + from utils.hashing import generate_prompt_hash + from database.models import normalize_image_path + +logger = logging.getLogger("prompt_manager.database.health") + +MAX_DETAIL = 10 # detail strings returned per check +INTEGRITY_LIMIT = 100 # PRAGMA integrity_check(N): problems SQLite reports +BATCH_ROWS = 100 # image rows read per fetch +REVIEW_UNAVAILABLE = "Needs review is not available yet." +CHECK_FAILED = "The check could not finish." +# Check ids and the labels of the concept's health report, in its order; +# pair uniqueness and consistency are the added rows (D63, now D95). +CHECKS = ( + ("integrity", "Integrity check"), + ("foreign_keys", "Foreign keys & pair references"), + ("canonical_hashes", "Canonical hashes"), + ("pair_uniqueness", "Pair uniqueness"), + ("missing_files", "Missing files"), + ("ambiguous_metadata", "Ambiguous metadata"), + ("consistency", "Consistency"), +) +_DATABASE_CHECKS = ( + "integrity", + "foreign_keys", + "canonical_hashes", + "pair_uniqueness", + "missing_files", + "consistency", +) + + +def iter_consistency_issues(conn): + """The legacy consistency issues: tag and image rows of a missing prompt.""" + for ref in conn.execute( + "SELECT pt.prompt_id FROM prompt_tags pt " + "LEFT JOIN prompts p ON pt.prompt_id = p.id WHERE p.id IS NULL" + ): + yield f"prompt_tags entry references non-existent prompt {ref[0]}" + for ref in conn.execute( + "SELECT gi.id, gi.prompt_id FROM generated_images gi " + "LEFT JOIN prompts p ON gi.prompt_id = p.id WHERE p.id IS NULL" + ): + yield f"Image {ref[0]} references non-existent prompt {ref[1]}" + + +def _result(count, detail, *, status=None): + return { + "status": status or ("issues" if count else "passed"), + "count": count, + "detail": detail[:MAX_DETAIL], + } + + +def _counted(lines): + """(count, first MAX_DETAIL lines) of an iterable, consumed once.""" + count, detail = 0, [] + for line in lines: + count += 1 + if len(detail) < MAX_DETAIL: + detail.append(line) + return _result(count, detail) + + +def readonly_uri(db_path, *, windows=None): + """The ``mode=ro`` SQLite URI of *db_path*, on this platform or the given one. + + pathlib spells a UNC path ``\\\\server\\share\\x.db`` as + ``file://server/share/x.db``, a remote authority SQLite refuses to open. + With an empty authority (``file:////server/share/x.db``) SQLite hands + Windows the ``//server/share`` path, which it opens as the UNC path. + """ + if windows is None: + windows = os.name == "nt" + db_path = os.path.abspath(db_path) + if not windows: + return PurePosixPath(db_path).as_uri() + "?mode=ro" + path = PureWindowsPath(db_path) + if ntpath.splitdrive(str(path))[0].startswith("\\\\"): + return "file:" + quote("//" + path.as_posix()) + "?mode=ro" + return path.as_uri() + "?mode=ro" + + +def _integrity(conn, _roots): + rows = [ + row[0] for row in conn.execute(f"PRAGMA integrity_check({INTEGRITY_LIMIT})") + ] + if rows == ["ok"]: + return {**_result(0, []), "capped": False} + # SQLite stops listing at the limit: there may be more problems than that. + return {**_result(len(rows), rows), "capped": len(rows) >= INTEGRITY_LIMIT} + + +def _foreign_keys(conn, _roots): + return _counted( + f"{table} row {rowid} references a missing {parent} row" + for table, rowid, parent, _fkid in conn.execute("PRAGMA foreign_key_check") + ) + + +def _prompt_hash(text): + return generate_prompt_hash(text) if isinstance(text, str) else None + + +def _canonical_hashes(conn, _roots): + conn.create_function("pm_prompt_hash", 1, _prompt_hash, deterministic=True) + return _counted( + f"Prompt {row[0]}: the stored hash does not match its text" + for row in conn.execute( + "SELECT id FROM prompts WHERE hash IS NOT pm_prompt_hash(text) ORDER BY id" + ) + ) + + +def _pair_uniqueness(conn, _roots): + groups = conn.execute( + "SELECT positive_prompt_id,negative_prompt_id,COUNT(*) FROM prompt_pairs " + "GROUP BY positive_prompt_id,negative_prompt_id HAVING COUNT(*)>1 " + "ORDER BY positive_prompt_id,negative_prompt_id" + ).fetchall() + detail = [ + f"Positive prompt {positive} with " + + ("no negative" if negative is None else f"negative {negative}") + + f": {rows} pair rows" + for positive, negative, rows in groups + ] + # The surplus rows: what keeping one row per pair would leave out. + return _result(sum(rows - 1 for _p, _n, rows in groups), detail) + + +def _consistency(conn, _roots): + return _counted(iter_consistency_issues(conn)) + + +def _readable(root): + return os.path.isdir(root) and os.access(root, os.R_OK | os.X_OK) + + +def _root_prefix(root): + return normalize_image_path(root).rstrip("\\/") + os.sep + + +def _source_name(root, index): + """A source's folder name for the report (never its server path).""" + return os.path.basename(str(root).rstrip("\\/")) or f"Source {index + 1}" + + +def _owner(path, prefixes): + """Index of the longest source prefix *path* lies under, or None.""" + owner, longest = None, -1 + for index, prefix in enumerate(prefixes): + if path.startswith(prefix) and len(prefix) > longest: + owner, longest = index, len(prefix) + return owner + + +def _image_owner(image_path, file_path, prefixes): + spelled = file_path or normalize_image_path(image_path) + owner = _owner(spelled, prefixes) + if owner is None: + owner = _owner(normalize_image_path(os.path.realpath(image_path)), prefixes) + return owner + + +def _unavailable_line(unavailable, *, partial): + noun = "source is" if unavailable == 1 else "sources are" + effect = "its files were" if unavailable == 1 else "their files were" + if not partial: + effect = "missing files were" + return f"{unavailable} image {noun} unavailable, so {effect} not counted." + + +def _missing_files(conn, roots): + """Links whose file is gone, counted per source (only readable ones). + + Each source reports ``checked`` with its own count, or ``unavailable`` + (an unmounted drive) with no count: its links are never turned into a + count, while the readable sources still are. The check's ``count`` is the + readable sources' total (None when none is readable) and ``partial`` says + some sources were not checked. Links outside every source are named in the + detail, not counted. + + Inside the snapshot this only reads the image rows; it returns the step + that checks their files, which runs after the snapshot ended. + """ + if not roots: + result = _result(None, ["No image source is available."], status="unavailable") + return {**result, "partial": False, "sources": []} + rows = [] + cursor = conn.execute( + "SELECT id,image_path,filename,file_path FROM generated_images ORDER BY id" + ) + while batch := cursor.fetchmany(BATCH_ROWS): + rows.extend(batch) + return lambda: _check_files(rows, roots) + + +def _check_files(rows, roots): + """The ``missing_files`` result of the image *rows*, from the file system.""" + readable = [_readable(root) for root in roots] + sources = [ + { + "name": _source_name(root, index), + "status": "checked" if ok else "unavailable", + "count": 0 if ok else None, + } + for index, (root, ok) in enumerate(zip(roots, readable)) + ] + unavailable = readable.count(False) + if not any(readable): + result = _result( + None, [_unavailable_line(unavailable, partial=False)], status="unavailable" + ) + return {**result, "partial": False, "sources": sources} + prefixes = [_root_prefix(root) for root in roots] + outside, missing = 0, [] + for image_id, image_path, filename, file_path in rows: + owner = _image_owner(image_path, file_path, prefixes) + if owner is None: + outside += 1 + elif readable[owner] and not os.path.exists(image_path): + sources[owner]["count"] += 1 + if len(missing) < MAX_DETAIL: + name = filename or os.path.basename(image_path) + missing.append(f"Image {image_id}: {name} is missing") + count = sum(source["count"] or 0 for source in sources) + detail = [_unavailable_line(unavailable, partial=True)] if unavailable else [] + detail += missing + if outside: + links = "link" if outside == 1 else "links" + verb = "was" if outside == 1 else "were" + detail.append( + f"{outside} {links} outside the image sources {verb} not checked." + ) + status = None if count or not unavailable else "unavailable" + result = _result(count, detail, status=status) + return {**result, "partial": bool(unavailable), "sources": sources} + + +def _ambiguous(review_count): + if review_count is None: + return _result(None, [REVIEW_UNAVAILABLE], status="unavailable") + return _result(review_count, []) + + +def overall_status(checks) -> str: + """``passed`` only when every check passed; any check not run is incomplete.""" + statuses = {check["status"] for check in checks} + if statuses & {"unavailable", "failed"} or any( + check.get("partial") for check in checks + ): + return "incomplete" + return "issues" if "issues" in statuses else "passed" + + +def _failed(): + return _result(None, [CHECK_FAILED], status="failed") + + +def _run_check(check_id, step, *args): + """One check's result, or None (reported as failed) when it raised.""" + try: + return step(*args) + except Exception: # any error fails this check only; the others still run + logger.exception(f"Health check {check_id} failed") + return None + + +def _run_database_checks(db_path, roots): + """Each database check's result (None when it raised), from one snapshot. + + A check may return a step instead of its result: the step runs after the + snapshot ended and the connection closed (``missing_files``' file checks). + """ + results = dict.fromkeys(_DATABASE_CHECKS) + try: + conn = sqlite3.connect(readonly_uri(db_path), uri=True) + except sqlite3.Error: + logger.exception("Health check could not open the database") + return results + try: + conn.execute("BEGIN") # one read snapshot for every check + for check_id in _DATABASE_CHECKS: + runner = globals()["_" + check_id] # looked up per run (tests patch it) + results[check_id] = _run_check(check_id, runner, conn, roots) + except sqlite3.Error: + logger.exception("Health check could not read the database") + finally: + conn.close() + for check_id, result in results.items(): + if callable(result): + results[check_id] = _run_check(check_id, result) + return results + + +class HealthOperations: + """Mixin for PromptDatabase: the read-only System health check.""" + + def run_health_checks(self, roots, *, review_count=None) -> dict: + """Run every check and return a ``HealthReport``. + + Args: + roots: The image source directories to judge missing files under: + the effective sources plus any configured source that is not + available (reported as an unavailable source of ``missing_files``). + review_count: The Needs-review pending count, or None while no + review queue exists (``ambiguous_metadata`` is then unavailable). + """ + if review_count is not None and ( + type(review_count) is not int or review_count < 0 + ): + raise ValueError("review_count must be a non-negative integer or None") + roots = [str(root) for root in roots] + database = _run_database_checks(self.model.db_path, roots) + checks = [] + for check_id, label in CHECKS: + if check_id == "ambiguous_metadata": + result = _ambiguous(review_count) + else: + result = database[check_id] or _failed() + checks.append({"id": check_id, "label": label, **result}) + return { + "success": True, + "checked_at": datetime.datetime.now(datetime.timezone.utc).isoformat(), + "status": overall_status(checks), + "checks": checks, + } diff --git a/database/library.py b/database/library.py index 4e532b9..7393e6d 100644 --- a/database/library.py +++ b/database/library.py @@ -628,3 +628,54 @@ class LibraryOperations: "kind": "prompt-manager-library", "positives": [owners[key] for key in order], } + + def library_counts(self) -> dict: + """System › Overview counts from one snapshot, writing nothing. + + ``positive_prompts`` and ``negative_variants`` are the unfiltered + Library's ``total`` and ``variant_total``: eligible parents and active + pairs (every pair's positive is eligible). ``images`` counts indexed + ``generated_images`` rows, not files on disk. An image has no rating + of its own (it shows its linked prompt's), so ``rated_images`` counts + the indexed images whose linked prompt is rated. + """ + with self.library_snapshot() as (conn, _epoch, _ceiling): + row = conn.execute( + "SELECT (SELECT COUNT(*) FROM prompts p WHERE " + _ELIGIBLE + ")," + "(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0)," + "(SELECT COUNT(*) FROM generated_images)," + "(SELECT COUNT(*) FROM generated_images gi JOIN prompts r " + "ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL)" + ).fetchone() + return { + "positive_prompts": row[0], + "negative_variants": row[1], + "images": row[2], + "rated_images": row[3], + } + + def usage_by_category(self) -> list[dict]: + """Runs and average rating of the Library's parents, per category. + + Only eligible parents count, so a negative-only prompt's runs (which + repeat its pairs' runs) are never added twice. A blank category groups + with no category as ``None``. Unrated parents are left out of the + average, which is ``None`` when no parent is rated. Most runs first. + """ + with self.library_snapshot() as (conn, _epoch, _ceiling): + rows = conn.execute( + "SELECT * FROM (SELECT NULLIF(TRIM(p.category),'') AS category," + "COALESCE(SUM(p.run_count),0) AS runs,AVG(p.rating) AS rating " + "FROM prompts p WHERE " + _ELIGIBLE + " GROUP BY 1) " + "ORDER BY runs DESC,category IS NULL,category" + ).fetchall() + return [ + { + "category": row["category"], + "runs": row["runs"], + "average_rating": ( + None if row["rating"] is None else round(row["rating"], 2) + ), + } + for row in rows + ] diff --git a/database/operations.py b/database/operations.py index 28bf35d..8424e88 100644 --- a/database/operations.py +++ b/database/operations.py @@ -20,6 +20,7 @@ from .execution_history import ExecutionHistoryOperations from .execution_contexts import ExecutionContextOperations, LIVE_HINT_CTES from .execution_outputs import ExecutionOutputOperations from .history_reads import HistoryReadOperations +from .health import HealthOperations, iter_consistency_issues # Import logging system try: @@ -342,6 +343,7 @@ class PromptDatabase( ImageFavoriteOperations, AnnotationOperations, SavedViewOperations, + HealthOperations, ): """Database operations class for managing prompts.""" @@ -2570,33 +2572,13 @@ class PromptDatabase( return cursor.rowcount def check_consistency(self) -> List[str]: - """Run consistency checks and return a list of issue descriptions.""" - issues: List[str] = [] - with self.model.get_connection() as conn: - # Check for orphaned prompt_tags entries - cursor = conn.execute(""" - SELECT pt.prompt_id, pt.tag_id FROM prompt_tags pt - LEFT JOIN prompts p ON pt.prompt_id = p.id - WHERE p.id IS NULL - """) - for ref in cursor.fetchall(): - issues.append( - "prompt_tags entry references non-existent prompt " - f"{ref['prompt_id']}" - ) + """Run consistency checks and return a list of issue descriptions. - # Check for orphaned image entries - cursor = conn.execute(""" - SELECT gi.id, gi.prompt_id FROM generated_images gi - LEFT JOIN prompts p ON gi.prompt_id = p.id - WHERE p.id IS NULL - """) - for ref in cursor.fetchall(): - issues.append( - f"Image {ref['id']} references non-existent prompt " - f"{ref['prompt_id']}" - ) - return issues + The same queries back the System health check's Consistency row + (database/health.py). + """ + with self.model.get_connection() as conn: + return list(iter_consistency_issues(conn)) def _image_row_to_dict(self, row: sqlite3.Row) -> Dict[str, Any]: """ diff --git a/py/api/__init__.py b/py/api/__init__.py index d97b359..fbff121 100644 --- a/py/api/__init__.py +++ b/py/api/__init__.py @@ -8,6 +8,7 @@ Split into domain-specific mixins for maintainability: - AutotagRoutesMixin: auto-tagging model management and tagging - JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py) - ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs +- SystemRoutesMixin: System health check, Overview and usage statistics """ import asyncio @@ -35,6 +36,7 @@ from .admin import AdminRoutesMixin from .settings_routes import SettingsRoutesMixin from .job_routes import JobRoutesMixin from .image_tag_jobs import ImageTagJobsMixin +from .system import SystemRoutesMixin from .jobs import JobRegistry from .logging_routes import LoggingRoutesMixin from .autotag_routes import AutotagRoutesMixin @@ -269,6 +271,7 @@ class PromptManagerAPI( SettingsRoutesMixin, JobRoutesMixin, ImageTagJobsMixin, + SystemRoutesMixin, LoggingRoutesMixin, AutotagRoutesMixin, LoraIntegrationMixin, @@ -325,6 +328,11 @@ class PromptManagerAPI( self._scan_job = None # The current duplicate scan (a py/api/jobs.Job outside the registry). self._duplicate_job = None + # The last System health report this session and the database + # generation it describes, bumped when a restore replaces self.db + # (py/api/system.py). + self._health_report = None + self._db_generation = 0 # Run cleanup on initialization to remove any existing duplicates try: @@ -482,6 +490,7 @@ class PromptManagerAPI( self._register_admin_routes(routes) self._register_settings_routes(routes) self._register_job_routes(routes) + self._register_system_routes(routes) self._register_logging_routes(routes) self._register_autotag_routes(routes) self._register_lora_routes(routes) @@ -728,6 +737,13 @@ class PromptManagerAPI( Returns: List[Path]: Validated directory paths, possibly empty. """ + return self._output_dirs_with_origin()[0] + + def _output_dirs_with_origin(self): + """``(dirs, auto_detected)``: the ``_get_all_output_dirs`` list, and + whether it comes from detection or ComfyUI's output rather than from + a valid configured root (False when there is no directory at all). + """ from ..config import GalleryConfig configured = list(GalleryConfig.MONITORING_DIRECTORIES) @@ -746,15 +762,16 @@ class PromptManagerAPI( output_dirs.append(Path(real)) if output_dirs: - return output_dirs + return output_dirs, False if configured: # Every configured root was rejected: never let auto-detection # resurrect it, serve ComfyUI's own output directory instead. - return self._comfyui_output_dir_list() - - fallback = self._find_comfyui_output_dir() - return [Path(fallback)] if fallback else [] + fallback_dirs = self._comfyui_output_dir_list() + else: + fallback = self._find_comfyui_output_dir() + fallback_dirs = [Path(fallback)] if fallback else [] + return fallback_dirs, bool(fallback_dirs) def _comfyui_output_dir_list(self): """ComfyUI's output directory as a one-element list, or [] if unknown.""" diff --git a/py/api/admin.py b/py/api/admin.py index 7ad05da..6410489 100644 --- a/py/api/admin.py +++ b/py/api/admin.py @@ -1271,6 +1271,7 @@ class AdminRoutesMixin: self.logger.info(f"Removed {len(removed)} old safety backups") previous = self.db self.db = PromptDatabase(model.db_path) + self._database_replaced() # a health report described the old one # Its connections would otherwise linger (a file lock on Windows) previous.close_all() prompt_count = self.db.model.get_database_info().get("total_prompts", 0) diff --git a/py/api/jobs.py b/py/api/jobs.py index 568a99e..09ab15a 100644 --- a/py/api/jobs.py +++ b/py/api/jobs.py @@ -88,6 +88,9 @@ DUPLICATE_CLEANUP_JOB_KIND = "duplicate_cleanup" THUMBNAILS_JOB_KIND = "thumbnails" LORA_IMPORT_JOB_KIND = "lora_import" AUTOTAG_BATCH_JOB_KIND = "autotag" +# The System health check (py/api/system.py): a read-only ``read`` job with +# no file outcomes, so it may run beside a scan. +HEALTH_JOB_KIND = "health" def resolve_admission(admission=None, record_changing=None) -> str: diff --git a/py/api/system.py b/py/api/system.py new file mode 100644 index 0000000..9539ae9 --- /dev/null +++ b/py/api/system.py @@ -0,0 +1,207 @@ +"""System routes: the read-only health check, the Overview and usage statistics. + +- ``POST /prompt_manager/system/health`` starts the health check + (``database/health.py``) off-loop as a ``read`` job of kind ``health``: one + at a time, beside a scan, not cancellable and with no file outcomes. The + API keeps the last report for this session; ``GET`` returns it or null. + A report describes the database it was taken from: replacing the database + (a restore) forgets it, and a check that started before the swap is not + kept. +- ``GET /prompt_manager/system/overview`` is the System › Overview aggregate. + Every value is real or null: there is no backup status until backups report + one, no health result until a check ran, and no Needs-review count until + the review queue exists. The missing-thumbnail count is not part of it: + the page reads ``/images/thumbnails/status``, which walks every source. +- ``GET /prompt_manager/system/statistics`` lists runs and ratings per + category, with the legacy ``/stats`` totals and the Overview's count of + rated images (images whose linked prompt is rated). + +New routes answer errors with ``{success:false, code, error}``. +""" + +import asyncio +import os +from collections import namedtuple + +from aiohttp import web + +from .job_routes import busy_response +from .jobs import HEALTH_JOB_KIND, JobBusy + +HEALTH_JOB_TITLE = "Health check" +_HEALTH_STATUS_LINES = { + "passed": "Health check passed", + "issues": "Health check found issues", + "incomplete": "Health check incomplete", +} + + +# The last health report and the database generation it was taken from. +_StoredReport = namedtuple("_StoredReport", "generation report") + + +def _failure(message, status=500, code=None): + return web.json_response( + {"success": False, "code": code, "error": message}, status=status + ) + + +def health_summary(report): + """What the Overview shows of a report: when, the outcome, checks with issues.""" + if report is None: + return None + return { + "checked_at": report["checked_at"], + "status": report["status"], + "issues": sum(1 for check in report["checks"] if check["status"] == "issues"), + } + + +class SystemRoutesMixin: + """Mixin providing the /prompt_manager/system endpoints.""" + + def _register_system_routes(self, routes): + @routes.post("/prompt_manager/system/health") + async def start_health_check_route(request): + return await self.start_health_check(request) + + @routes.get("/prompt_manager/system/health") + async def health_report_route(request): + return await self.get_health_report(request) + + @routes.get("/prompt_manager/system/overview") + async def system_overview_route(request): + return await self.get_system_overview(request) + + @routes.get("/prompt_manager/system/statistics") + async def system_statistics_route(request): + return await self.get_system_statistics(request) + + # ── health ──────────────────────────────────────────────────────── + + def _health_roots(self): + """The sources missing files are judged under. + + The effective sources, plus every configured source that is not an + existing directory (an unmounted drive): the check then reports + missing files as unavailable instead of counting that source's links. + """ + from ..config import GalleryConfig + + roots = [str(path) for path in self._get_all_output_dirs()] + for configured in GalleryConfig.MONITORING_DIRECTORIES: + try: + resolved = GalleryConfig.resolve_gallery_root(configured) + except (OSError, TypeError, ValueError): + continue + if not os.path.isdir(resolved): + roots.append(resolved) + return roots + + def _database_replaced(self): + """Forget what described the previous database (call after swapping ``db``).""" + self._db_generation += 1 + self._health_report = None + + def _current_health_report(self): + """The last report, or None when none was kept for the current database.""" + stored = self._health_report + generation = self._db_generation + if isinstance(stored, _StoredReport) and stored.generation == generation: + return stored.report + return None + + async def start_health_check(self, request): + """Start the health check job (202), or 409 naming the running check.""" + generation = self._db_generation + db = self.db + try: + roots = await self._run_in_executor(self._health_roots) + except Exception: + self.logger.exception("Health check could not resolve the image sources") + return _failure("Failed to start the health check") + + async def run(job): + job.update(status_line="Checking the database and image links") + report = await self._run_in_executor(db.run_health_checks, roots) + if generation == self._db_generation: + self._health_report = _StoredReport(generation, report) + else: + self.logger.info("Health report dropped: the database was replaced") + job.update(progress=100, status_line=_HEALTH_STATUS_LINES[report["status"]]) + + try: + job = self._jobs.start( + HEALTH_JOB_KIND, + HEALTH_JOB_TITLE, + admission="read", + cancellable=False, + scope={}, + runner=run, + ) + except JobBusy as busy: + return busy_response(busy) + return web.json_response({"success": True, "job": job.summary()}, status=202) + + async def get_health_report(self, request): + return web.json_response( + {"success": True, "report": self._current_health_report()} + ) + + # ── overview ────────────────────────────────────────────────────── + + async def get_system_overview(self, request): + from ..config import GalleryConfig + + try: + (output_dirs, auto_detected), counts = await asyncio.gather( + self._run_in_executor(self._output_dirs_with_origin), + self._run_in_executor(self.db.library_counts), + ) + except Exception: + self.logger.exception("System overview failed") + return _failure("Failed to read the overview") + return web.json_response( + { + "success": True, + "library": { + "positive_prompts": counts["positive_prompts"], + "negative_variants": counts["negative_variants"], + }, + "images": { + "indexed": counts["images"], + "rated": counts["rated_images"], + }, + "health": health_summary(self._current_health_report()), + "backup": None, # no verified backup status exists yet (S8) + "needs_review": {"pending": None}, # no review queue yet (Q6) + "sources": { + "configured": len(GalleryConfig.MONITORING_DIRECTORIES), + "effective": len(output_dirs), + "auto_detected": auto_detected, + }, + }, + headers={"Cache-Control": "no-store"}, + ) + + # ── statistics ──────────────────────────────────────────────────── + + async def get_system_statistics(self, request): + try: + by_category, totals, counts = await asyncio.gather( + self._run_in_executor(self.db.usage_by_category), + self._run_in_executor(self.db.get_statistics), + self._run_in_executor(self.db.library_counts), + ) + except Exception: + self.logger.exception("System statistics failed") + return _failure("Failed to read statistics") + return web.json_response( + { + "success": True, + "by_category": by_category, + "totals": totals, + "rated_images": counts["rated_images"], + }, + headers={"Cache-Control": "no-store"}, + ) diff --git a/tests/browser/server.py b/tests/browser/server.py index 9b60d7b..9926c26 100644 --- a/tests/browser/server.py +++ b/tests/browser/server.py @@ -21,6 +21,7 @@ from PIL.PngImagePlugin import PngInfo from tests.browser.request_hold import register_request_holds from tests.browser import slow_reset_watch from tests.browser.workspace_activity_fixture import register_activity_commands +from tests.browser.workspace_system_fixture import register_system_commands from tests.browser.workspace_concept_fixture import ( MAX_EXTRA_IMAGES as CONCEPT_MAX_EXTRA_IMAGES, MAX_SEEDLESS_IMAGES as CONCEPT_MAX_SEEDLESS_IMAGES, @@ -198,6 +199,7 @@ def create_app(root: Path) -> web.Application: register_tagging_commands(routes, api, root, model) reveal_stub = register_reveal_stub(routes, api, root) register_activity_commands(app, routes, api) + register_system_commands(app, routes, api, root) request_holds = register_request_holds(app, routes) slow_reset_watch.install(app, api) diff --git a/tests/browser/system-concept.spec.mjs b/tests/browser/system-concept.spec.mjs new file mode 100644 index 0000000..c32a62e --- /dev/null +++ b/tests/browser/system-concept.spec.mjs @@ -0,0 +1,410 @@ +import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs'; +import { test, expect } from './fixtures.mjs'; +import { SIZES, THEMES, seedConcept, openConcept, openProduction, diffInventory, mutate } from './concept-inventory.mjs'; + +// #249 System S2 Task 6: System's shell and its Overview section (app.js renderSystem 658–706), the Statistics dialog +// ("Usage statistics", a1600–1604) and the health report (a1439–1456) against the immutable concept, on the +// concept-shaped real fixture. Each concept measurement is mapped only by the recorded deviations (D92–D96, D121, +// D122, D123), each commented where it applies; after that the difference must be empty, and a mutation of a +// row, a label or a token must be noticed. +test.use({ locale: 'en-US', timezoneId: 'UTC' }); + +test.beforeEach(async ({ page, request, browserName }) => { + await startCoverage(page, browserName); + await seedConcept(request); + await expect(await request.post('/__test__/system/reset')).toBeOK(); + await expect(await request.post('/__test__/jobs/restart')).toBeOK(); + page.conceptErrors = []; + page.on('pageerror', error => page.conceptErrors.push(error.message)); +}); + +test.afterEach(async ({ page, request, browserName }) => { + await saveCoverage(page, browserName); + await request.post('/__test__/system/reset'); + expect(page.conceptErrors).toEqual([]); +}); + +/** Runs in the page; self-contained. System's heading, section nav, Overview body and Save row under `mainSelector`. */ +function measureSystem(mainSelector) { + const main = document.querySelector(mainSelector); + const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null; + const name = node => [node.localName, ...[...node.classList].sort()].join('.'); + const box = (node, origin = main) => { + const rect = node.getBoundingClientRect(), base = origin.getBoundingClientRect(); + return { left: Math.round(rect.left - base.left), width: Math.round(rect.width) }; + }; + // A surface's top: from `.two-col` beside its neighbour, else from the bottom of the surface above it. This pins the + // concept's `.surface + .surface` 18px offset, which concept.css applies inside `.two-col` too (copied, not ruled). + const top = node => { + const rect = node.getBoundingClientRect(), previous = node.previousElementSibling?.getBoundingClientRect(); + if (previous && rect.top >= previous.bottom) return Math.round(rect.top - previous.bottom); + return Math.round(rect.top - node.parentElement.getBoundingClientRect().top); + }; + const text = node => node?.textContent.trim() ?? null; + const button = node => ({ text: text(node), className: node.className, + ...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopWidth', 'borderTopColor', + 'borderTopLeftRadius', 'opacity', 'cursor') }); + const heading = main.querySelector(':scope > .heading'); + const nav = main.querySelector(':scope > .section-nav'); + const layout = main.querySelector(':scope > .settings-layout'); + const metrics = layout.querySelector(':scope > .summary-metrics'); + const columns = layout.querySelector(':scope > .two-col'); + const save = layout.querySelector(':scope > .save-row'); + const firstMetric = metrics.querySelector('.metric'); + const firstSurface = columns.querySelector('.surface'); + const firstOperation = firstSurface.querySelector('.operation'); + return { + tree: [...main.children].map(name), + heading: { + title: text(heading.querySelector('h1')), description: text(heading.querySelector('p.muted')), + actions: [...(heading.querySelector('.heading-actions')?.children ?? [])].map(name), + }, + nav: { + role: nav.getAttribute('role'), label: nav.getAttribute('aria-label'), + ...css(nav, 'display', 'flexWrap', 'columnGap', 'rowGap', 'paddingBottom', 'marginBottom', 'borderBottomWidth', 'borderBottomColor'), + box: box(nav), + buttons: [...nav.children].map(item => ({ pressed: item.getAttribute('aria-pressed'), ...button(item) })), + }, + layout: { tree: [...layout.children].map(name), ...css(layout, 'maxWidth'), box: box(layout) }, + metrics: { + ...css(metrics, 'display', 'gridTemplateColumns', 'columnGap', 'rowGap', 'marginBottom'), box: box(metrics), + items: [...metrics.children].map(item => ({ tree: [...item.children].map(name), value: text(item.querySelector('strong')), + label: text(item.querySelector('span')), box: box(item) })), + metric: css(firstMetric, 'paddingTop', 'paddingLeft', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius', 'backgroundColor'), + strong: css(firstMetric.querySelector('strong'), 'display', 'fontSize', 'fontWeight', 'color'), + span: css(firstMetric.querySelector('span'), 'fontSize', 'color', 'marginTop'), + }, + columns: { + ...css(columns, 'display', 'gridTemplateColumns', 'columnGap', 'rowGap'), box: box(columns), + surfaces: [...columns.children].map(surface => ({ tree: [...surface.children].map(name), box: box(surface), top: top(surface), + heading: text(surface.querySelector(':scope > h2')), + rows: [...surface.querySelectorAll(':scope > .operation')].map(row => ({ tree: [...row.children].map(name), + title: text(row.querySelector('h3')), description: text(row.querySelector('p')), + action: button(row.querySelector('.row-actions > button')) })) })), + surface: css(firstSurface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius'), + h2: css(firstSurface.querySelector('h2'), 'fontSize', 'fontWeight', 'color', 'marginBottom'), + operation: css(firstOperation, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', + 'borderTopWidth', 'borderTopColor'), + h3: css(firstOperation.querySelector('h3'), 'fontSize', 'fontWeight', 'color'), + p: css(firstOperation.querySelector('p'), 'fontSize', 'color', 'marginTop'), + actions: css(firstOperation.querySelector('.row-actions'), 'display', 'columnGap', 'flexShrink'), + }, + save: { + tree: [...save.children].map(name), ...css(save, 'display', 'alignItems', 'justifyContent', 'columnGap', 'marginTop', 'paddingTop'), + status: { text: text(save.querySelector('#settings-status')), ...css(save.querySelector('#settings-status'), 'fontSize', 'color') }, + button: button(save.querySelector('button')), + }, + pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth, + }; +} + +/** Runs in the page: the open dialog's title, body sequence, tables, operation rows and actions. */ +function measureDialog() { + const dialog = document.querySelector('dialog#modal[open]'); + const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null; + const name = node => [node.localName, ...[...node.classList].sort()].join('.'); + const body = dialog.querySelector('#modal-body'); + const firstCell = selector => body.querySelector(selector); + const list = body.querySelector(':scope > .operation-list'); + const actions = body.querySelector(':scope > .dialog-actions'); + return { + title: dialog.querySelector('#modal-title').textContent, + body: [...body.children].map(name), + tables: [...body.querySelectorAll(':scope > table')].map(table => ({ + caption: table.caption?.textContent ?? null, + head: [...table.tHead.rows[0].cells].map(cell => [cell.localName, cell.scope, cell.textContent]), + rows: [...table.tBodies[0].rows].map(row => [...row.cells].map(cell => cell.textContent)), + })), + tableStyle: firstCell(':scope > table') && { + table: css(firstCell(':scope > table'), 'width', 'borderCollapse', 'fontSize', 'marginTop'), + caption: css(firstCell(':scope > table caption'), 'textAlign', 'fontSize', 'fontWeight', 'color', 'paddingBottom'), + th: css(firstCell(':scope > table thead th'), 'textAlign', 'fontSize', 'fontWeight', 'color', 'paddingTop', 'paddingLeft', + 'borderBottomWidth', 'borderBottomColor'), + td: css(firstCell(':scope > table tbody td'), 'textAlign', 'fontSize', 'color', 'paddingTop', 'paddingLeft', + 'borderBottomWidth', 'borderBottomColor'), + }, + operations: list && { + ...css(list, 'display', 'rowGap', 'marginTop'), + rows: [...list.children].map(row => ({ tree: [...row.children].map(name), title: row.querySelector('h3').textContent, + chip: row.querySelector('.chip').textContent })), + operation: css(list.querySelector('.operation'), 'display', 'alignItems', 'justifyContent', 'paddingTop', 'borderTopWidth'), + h3: css(list.querySelector('.operation h3'), 'fontSize', 'fontWeight', 'color'), + chip: css(list.querySelector('.operation .chip'), 'fontSize', 'color', 'backgroundColor', 'borderTopLeftRadius', 'paddingLeft'), + }, + actions: actions && { + ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'), + buttons: [...actions.children].map(item => ({ text: item.textContent, className: item.className, + ...css(item, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft') })), + }, + }; +} + +/** Runs in the page: a `.notice` (Maintenance's "Check first…") with its copy. */ +function measureNotice(mainSelector) { + const notice = document.querySelector(`${mainSelector} .settings-layout > .notice`); + const css = (node, ...keys) => Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])); + const name = node => [node.localName, ...[...node.classList].sort()].join('.'); + const run = notice.querySelector(':scope > button'); + return { + tree: [...notice.children].map(name), text: [...notice.querySelectorAll(':scope > div > *')].map(node => [node.localName, node.textContent]), + frame: css(notice, 'display', 'alignItems', 'justifyContent', 'columnGap', 'borderTopWidth', 'borderTopColor', 'backgroundColor', + 'borderTopLeftRadius', 'paddingTop', 'paddingLeft', 'marginBottom'), + h2: css(notice.querySelector('h2'), 'fontSize', 'fontWeight', 'color'), + p: css(notice.querySelector('p'), 'fontSize', 'color', 'marginTop'), + run: { text: run.textContent, className: run.className, ...css(run, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft') }, + }; +} + +async function serverJson(request, url) { + const response = await request.get(url); + await expect(response).toBeOK(); + return response.json(); +} + +const number = value => value.toLocaleString('en-US'); +const ratingText = value => value === null ? 'Not rated' : `${number(value)} / 5`; + +/** The concept's shell and Overview with the recorded deviations applied, given the server's real overview. */ +function mapOverviewConcept(reference, overview) { + const relabel = { 'Database integrity': 'Not checked', 'Positive prompts': number(overview.library.positive_prompts), + 'Negative variants': number(overview.library.negative_variants), Images: number(overview.images.indexed) }; + const rows = { + // D93 (Q6): no review queue yet, so no count and the action is unavailable (dimmed, as the Save row's D92). + '3 ambiguous image links': row => ({ ...row, title: 'Ambiguous image links', description: 'Not available yet.', + action: { ...row.action, opacity: '0.55', cursor: 'not-allowed' } }), + // D93: no backup has been recorded (verified backups arrive with S8). + 'Backup status': row => ({ ...row, description: 'No retained backup record.' }), + // D93: there is no paging, so the shared settings are grid and density. + 'One display configuration': row => ({ ...row, description: 'Grid and density settings are shared across views.' }), + // D93: the real sources, not samples. + '2 sample output sources': row => ({ ...row, title: `${number(overview.sources.effective)} output sources` }), + // D96: the dialog shows runs and ratings by category and library totals, not prompt, tag or date usage. + Statistics: row => ({ ...row, description: 'Runs and ratings by category, with library totals.' }), + }; + return { + ...reference, + // D92: no "Sample configuration" chip. + heading: { ...reference.heading, actions: [] }, + // D122: the concept's literal label is its prototype action name ("system-tab sections"). + nav: { ...reference.nav, label: 'System sections' }, + metrics: { ...reference.metrics, items: reference.metrics.items.map(item => ({ ...item, value: relabel[item.label] })) }, + columns: { ...reference.columns, surfaces: reference.columns.surfaces.map(surface => ({ ...surface, + rows: surface.rows.map(row => (rows[row.title] ?? (value => value))(row)) })) }, + save: { ...reference.save, + // D92: truthful status copy, and "Save preferences" is aria-disabled (dimmed) while nothing is unsaved. + status: { ...reference.save.status, text: 'No unsaved changes.' }, + button: { ...reference.save.button, opacity: '0.55', cursor: 'not-allowed' } }, + }; +} + +async function conceptOverview(page, theme) { + await openConcept(page, 'system', theme); + await expect(page.locator('#main .summary-metrics')).toBeVisible(); + await page.mouse.move(0, 0); + return page.evaluate(measureSystem, '#main'); +} + +async function productionOverview(page, theme) { + await openProduction(page, 'system/overview', theme); + await expect(page.locator('#workspace-main .summary-metrics')).toBeVisible(); + // The thumbnail row is its own read (I2): compare once it answered. + await expect(page.locator('#workspace-main .operation h3').filter({ hasText: 'thumbnails' })).toHaveText('No missing thumbnails'); + await page.mouse.move(0, 0); + return page.evaluate(measureSystem, '#workspace-main'); +} + +for (const theme of THEMES) for (const [width, height] of SIZES) { + test(`system shell and overview match the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => { + await page.setViewportSize({ width, height }); + const overview = await serverJson(request, '/prompt_manager/system/overview'); + // The fixture's thumbnails all exist, so the concept's "No missing thumbnails" is also the real title. + expect((await serverJson(request, '/prompt_manager/images/thumbnails/status')).pending).toBe(0); + const concept = await conceptOverview(page, theme); + expect(concept.heading.actions).toEqual(['span.chip']); + // Paired captures are artifacts only; there are no PNG baselines. + await page.screenshot({ path: testInfo.outputPath(`concept-system-${theme}-${width}.png`), fullPage: true }); + const production = await productionOverview(page, theme); + await page.screenshot({ path: testInfo.outputPath(`production-system-${theme}-${width}.png`), fullPage: true }); + expect(diffInventory(mapOverviewConcept(concept, overview), production)).toEqual([]); + expect(production.pageScroll).toBe(false); + }); +} + +// The recorded differences, by path: D93 the review row (title, Not available yet, dimmed action), the backup row, the +// preferences row and the sources row, and metric 4; D96 the statistics row; D92 the chip, the status and the dimmed +// Save; D122 the nav label. (The fixture's first three metrics are the concept's own counts.) +const UNMAPPED_PATHS = [ + 'columns.surfaces.0.rows.0.action.cursor', 'columns.surfaces.0.rows.0.action.opacity', + 'columns.surfaces.0.rows.0.description', 'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description', + 'columns.surfaces.1.rows.0.description', 'columns.surfaces.1.rows.1.title', 'columns.surfaces.1.rows.2.description', + 'heading.actions.0', 'heading.actions.length', 'metrics.items.3.value', 'nav.label', + 'save.button.cursor', 'save.button.opacity', 'save.status.text', +]; + +test('system comparison detects mutations', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const overview = await serverJson(request, '/prompt_manager/system/overview'); + const concept = await conceptOverview(page, 'dark'); + const expected = mapOverviewConcept(concept, overview); + const production = await productionOverview(page, 'dark'); + expect(diffInventory(expected, production)).toEqual([]); + // Unmapped, the concept differs exactly where a deviation is recorded, and nowhere else. + expect(diffInventory(concept, production).map(item => item.path).sort()).toEqual(UNMAPPED_PATHS); + const measure = () => page.evaluate(measureSystem, '#workspace-main'); + const paths = async () => diffInventory(expected, await measure()).map(item => item.path); + + // A removed row. + const undoRow = await mutate(page, { remove: ['#workspace-main .two-col > .surface:first-child > .operation:nth-of-type(2)'] }); + expect(await paths()).toEqual(expect.arrayContaining(['columns.surfaces.0.rows.length', 'columns.surfaces.0.tree.length'])); + await undoRow(); + // A changed label. + await page.evaluate(() => { document.querySelector('#workspace-main .metric span').textContent = 'Positive prompt'; }); + expect(await paths()).toEqual(['metrics.items.0.label']); + await page.evaluate(() => { document.querySelector('#workspace-main .metric span').textContent = 'Positive prompts'; }); + // A swapped token. + const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' }); + expect(await paths()).toEqual(expect.arrayContaining(['metrics.metric.borderTopColor', 'columns.surface.borderTopColor'])); + await undoToken(); + expect(diffInventory(expected, await measure())).toEqual([]); +}); + +/** D96: the concept's table, its rows the real ones; then "Library totals" in the same table markup. */ +function mapStatisticsConcept(reference, statistics) { + const [runs] = reference.tables; + return { + ...reference, + // No sample note (prototype copy); a second table follows the first. + body: ['table', 'table'], + tables: [ + { ...runs, rows: statistics.by_category.map(row => [row.category ?? 'Uncategorized', number(row.runs), ratingText(row.average_rating)]) }, + { caption: 'Library totals', head: [['th', 'col', 'Total'], ['th', 'col', 'Count']], + rows: [['Prompts', number(statistics.totals.total_prompts)], ['Tags', number(statistics.totals.total_tags)], + ['Images', number(statistics.totals.total_images)], ['Rated images', number(statistics.rated_images)]] }, + ], + }; +} + +// The fixture's image sources and each check's real chip on it: nothing is damaged, Needs review does not exist yet (Q6). +const FIXTURE_SOURCES = ['output', 'input']; +const FIXTURE_CHIPS = Object.freeze({ 'Integrity check': 'Passed', 'Foreign keys & pair references': 'Passed', + 'Canonical hashes': 'Passed', 'Pair uniqueness': 'Passed', 'Missing files': 'Passed', 'Ambiguous metadata': 'Not available yet', + Consistency: 'Passed' }); +const chipOf = title => { + if (!(title in FIXTURE_CHIPS)) throw new Error(`No expected chip for ${title}`); + return FIXTURE_CHIPS[title]; +}; + +/** + * D95: the concept's health report from its own five rows, labels verbatim, titled "Health report". Only the D95 rows + * are inserted where they are recorded (Pair uniqueness after Canonical hashes, Consistency last); Missing files has a + * row per image source when there are several; each chip is the fixture's expected result. + */ +function mapHealthConcept(reference, sources = FIXTURE_SOURCES) { + const [first] = reference.operations.rows; + const added = title => ({ ...first, title, chip: chipOf(title) }); + const rows = reference.operations.rows.flatMap(row => { + const mapped = row.title === 'Missing files' && sources.length > 1 + ? sources.map(source => ({ ...row, title: `Missing files · ${source}`, chip: chipOf(row.title) })) + : [{ ...row, chip: chipOf(row.title) }]; + return row.title === 'Canonical hashes' ? [...mapped, added('Pair uniqueness')] : mapped; + }); + return { + ...reference, + title: 'Health report', + // No sample note (prototype copy); D95 adds the checked line after the list. + body: ['div.operation-list', 'p.small', 'div.dialog-actions'], + operations: { ...reference.operations, rows: [...rows, added('Consistency')] }, + }; +} + +// Unmapped, the health report differs exactly where D95 is recorded: the title, the body sequence (no sample note, the +// checked line), the rows after Canonical hashes (Pair uniqueness inserted, Missing files per source) and the chips of +// the concept's sample findings. +const UNMAPPED_HEALTH_PATHS = [ + 'body.0', 'body.1', + ...[3, 4].flatMap(row => [`operations.rows.${row}.chip`, `operations.rows.${row}.title`]), + // The concept's five rows end at row 4; rows 5–7 exist only with the D95 rows and the second source. + ...[5, 6, 7].flatMap(row => ['chip', 'title', 'tree.0', 'tree.1', 'tree.length'].map(key => `operations.rows.${row}.${key}`)), + 'operations.rows.length', 'title', +]; + +async function conceptDialogs(page, theme, testInfo) { + await openConcept(page, 'system', theme); + await page.locator('#main').getByRole('button', { name: 'Explore', exact: true }).click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics'); + await page.mouse.move(0, 0); + const statistics = await page.evaluate(measureDialog); + await page.screenshot({ path: testInfo.outputPath(`concept-statistics-${theme}.png`) }); + await page.keyboard.press('Escape'); + await page.locator('#main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click(); + const notice = await page.evaluate(measureNotice, '#main'); + await page.locator('#main .notice').getByRole('button', { name: 'Run health checks', exact: true }).click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Sample health report'); + await page.mouse.move(0, 0); + await page.screenshot({ path: testInfo.outputPath(`concept-health-${theme}.png`) }); + return { statistics, notice, health: await page.evaluate(measureDialog) }; +} + +async function productionDialogs(page, theme, testInfo) { + await openProduction(page, 'system/overview', theme); + await page.locator('#workspace-main').getByRole('button', { name: 'Explore', exact: true }).click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics'); + await page.mouse.move(0, 0); + const statistics = await page.evaluate(measureDialog); + await page.screenshot({ path: testInfo.outputPath(`production-statistics-${theme}.png`) }); + await page.keyboard.press('Escape'); + await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click(); + await expect(page.locator('#workspace-main .notice')).toBeVisible(); + const notice = await page.evaluate(measureNotice, '#workspace-main'); + await page.locator('#workspace-main .notice').getByRole('button', { name: 'Run health checks', exact: true }).click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Health report'); + await page.mouse.move(0, 0); + await page.screenshot({ path: testInfo.outputPath(`production-health-${theme}.png`) }); + return { statistics, notice, health: await page.evaluate(measureDialog) }; +} + +for (const theme of THEMES) test(`statistics and health dialogs match the concept ${theme}`, async ({ page, request }, testInfo) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const concept = await conceptDialogs(page, theme, testInfo); + expect(concept.statistics.body).toEqual(['p.demo-note', 'table']); + expect(concept.health.body).toEqual(['p.demo-note', 'div.operation-list', 'div.dialog-actions']); + const production = await productionDialogs(page, theme, testInfo); + const statistics = await serverJson(request, '/prompt_manager/system/statistics'); + const { report } = await serverJson(request, '/prompt_manager/system/health'); + // The fixture the expected chips describe: its two sources, and the concept's checks plus the D95 ones. + const missing = report.checks.find(check => check.id === 'missing_files'); + expect(missing.sources.map(source => source.name)).toEqual(FIXTURE_SOURCES); + expect(diffInventory(mapStatisticsConcept(concept.statistics, statistics), production.statistics)).toEqual([]); + expect(diffInventory(mapHealthConcept(concept.health), production.health)).toEqual([]); + // D123: Maintenance's notice is the concept's own, delivered ahead of the rest of the section. + expect(diffInventory(concept.notice, production.notice)).toEqual([]); + + // Unmapped, the dialogs differ only where D95 and D96 are recorded. + expect(diffInventory(concept.health, production.health).map(item => item.path).sort()).toEqual(UNMAPPED_HEALTH_PATHS); + expect(diffInventory(concept.statistics, production.statistics).map(item => item.path.split('.')[0]) + .filter((part, index, all) => all.indexOf(part) === index).sort()).toEqual(['body', 'tables']); + + // Mutation check: a changed chip token in the health report is noticed, then undone to an exact match again. + const undo = await mutate(page, { css: '#modal .operation .chip { background: rgb(255, 0, 0); }' }); + expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog)).map(item => item.path)) + .toEqual(['operations.chip.backgroundColor']); + await undo(); + expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog))).toEqual([]); + // A renamed check (database/health.py's label) is noticed against the concept's own label. + await page.evaluate(() => { document.querySelector('#modal .operation h3').textContent = 'Integrity checks'; }); + expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog)).map(item => item.path)) + .toEqual(['operations.rows.0.title']); + await page.evaluate(() => { document.querySelector('#modal .operation h3').textContent = 'Integrity check'; }); + // And a removed "Library totals" table and a changed caption in the statistics dialog. + await page.keyboard.press('Escape'); + await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click(); + await page.locator('#workspace-main').getByRole('button', { name: 'Explore', exact: true }).click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics'); + const expectedStatistics = mapStatisticsConcept(concept.statistics, statistics); + const undoTable = await mutate(page, { remove: ['#modal-body > table:nth-of-type(2)'] }); + expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path)) + .toEqual(expect.arrayContaining(['body.length', 'tables.length'])); + await undoTable(); + await page.evaluate(() => { document.querySelector('#modal-body caption').textContent = 'Runs per category'; }); + expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path)).toEqual(['tables.0.caption']); +}); diff --git a/tests/browser/workspace-system.spec.mjs b/tests/browser/workspace-system.spec.mjs new file mode 100644 index 0000000..75d2069 --- /dev/null +++ b/tests/browser/workspace-system.spec.mjs @@ -0,0 +1,252 @@ +import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs'; +import { test, expect } from './fixtures.mjs'; +import { seedConcept } from './concept-inventory.mjs'; + +// System › Overview (#249 System S2 Task 6) on the concept-shaped real fixture: the metrics, rows, Statistics dialog and +// Health report read the server, never sample values (D93, D95, D96). The hooks are workspace_system_fixture.py's. +const workspace = '/prompt_manager/workspace'; +const OVERVIEW_URL = '/prompt_manager/system/overview'; +const THUMBNAILS_URL = '/prompt_manager/images/thumbnails/status'; +const number = value => value.toLocaleString('en-US'); + +test.beforeEach(async ({ page, request, browserName }) => { + await startCoverage(page, browserName); + await seedConcept(request); + await expect(await request.post('/__test__/system/reset')).toBeOK(); + await expect(await request.post('/__test__/jobs/restart')).toBeOK(); + page.systemErrors = []; + page.on('pageerror', error => page.systemErrors.push(error.message)); +}); + +test.afterEach(async ({ page, request, browserName }) => { + await saveCoverage(page, browserName); + await request.post('/__test__/release'); + await request.post('/__test__/system/reset'); + expect(page.systemErrors).toEqual([]); +}); + +async function serverJson(request, url) { + const response = await request.get(url); + await expect(response).toBeOK(); + return response.json(); +} + +const layoutOf = page => page.locator('#workspace-main .settings-layout'); + +async function openOverview(page) { + await page.goto(`${workspace}#/system/overview`); + await expect(layoutOf(page).locator(':scope > .summary-metrics')).toBeVisible(); + return layoutOf(page); +} + +/** The rows of one Overview surface: `[h3, p, button]`. */ +function rowsOf(layout, heading) { + return layout.locator('.two-col > section.surface').filter({ has: layout.page().getByRole('heading', { name: heading, exact: true }) }) + .locator('.operation').evaluateAll(rows => rows.map(row => [row.querySelector('h3').textContent, + row.querySelector('p').textContent, row.querySelector('.row-actions button').textContent])); +} + +test('overview shows real counts and Not checked', async ({ page, request }) => { + const writes = []; + page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) writes.push(item.url()); }); + const overview = await serverJson(request, OVERVIEW_URL); + const thumbnails = await serverJson(request, THUMBNAILS_URL); + // The concept fixture: six positives, their saved negatives and 24 linked images, all with thumbnails. + expect([overview.library.positive_prompts, overview.images.indexed, thumbnails.pending]).toEqual([6, 24, 0]); + // I2: the Overview never walks the sources for thumbnails; the row reads the status route itself. + expect(overview).not.toHaveProperty('thumbnails'); + const layout = await openOverview(page); + expect(await layout.locator('.summary-metrics > .metric').evaluateAll(items => items.map(item => + [item.querySelector('strong').textContent, item.querySelector('span').textContent]))).toEqual([ + [number(overview.library.positive_prompts), 'Positive prompts'], + [number(overview.library.negative_variants), 'Negative variants'], + [number(overview.images.indexed), 'Images'], + ['Not checked', 'Database integrity'], + ]); + expect(await rowsOf(layout, 'Library health')).toEqual([ + ['Ambiguous image links', 'Not available yet.', 'Review metadata'], + ['Backup status', 'No retained backup record.', 'View backups'], + ['No missing thumbnails', 'Generate only missing previews when necessary.', 'Review thumbnails'], + ]); + expect(await rowsOf(layout, 'Workspace preferences')).toEqual([ + ['One display configuration', 'Grid and density settings are shared across views.', 'Appearance'], + ['2 output sources', 'Monitoring and enrichment share one source list.', 'Manage sources'], + ['Statistics', 'Runs and ratings by category, with library totals.', 'Explore'], + ]); + expect(writes).toEqual([]); +}); + +test('Run health checks opens Health report and updates metric 4', async ({ page, request }) => { + await page.goto(`${workspace}#/system/maintenance`); + const layout = layoutOf(page); + await expect(layout.locator(':scope > .notice h2')).toHaveText('Check first. Review changes. Then apply.'); + // The rest of Maintenance is still the existing tools' (D94). + await expect(layout.locator(':scope > section.handoff h2')).toHaveText('Your current tools are ready'); + const run = layout.getByRole('button', { name: 'Run health checks', exact: true }); + await run.click(); + const dialog = page.locator('dialog#modal[open]'); + await expect(dialog.locator('#modal-title')).toHaveText('Health report'); + const { report } = await serverJson(request, '/prompt_manager/system/health'); + expect(report.status).toBe('incomplete'); + expect(await dialog.locator('.operation-list > .operation').evaluateAll(rows => rows.map(row => + [row.querySelector('h3').textContent, row.querySelector('span.chip').textContent]))).toEqual([ + ['Integrity check', 'Passed'], ['Foreign keys & pair references', 'Passed'], ['Canonical hashes', 'Passed'], + ['Pair uniqueness', 'Passed'], ['Missing files · output', 'Passed'], ['Missing files · input', 'Passed'], + ['Ambiguous metadata', 'Not available yet'], ['Consistency', 'Passed'], + ]); + await expect(dialog.locator('#modal-body > p.small')).toHaveText(/^Checked .+\. Some checks could not run, so this report is incomplete\.$/); + await expect(dialog.locator('.dialog-actions button')).toHaveText(['Close', 'Review missing links']); + await dialog.getByRole('button', { name: 'Close', exact: true }).click(); + await expect(page.locator('dialog#modal[open]')).toHaveCount(0); + await expect(run).toBeFocused(); + await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click(); + await expect(layout.locator('.metric').nth(3).locator('strong')).toHaveText('Incomplete'); + + // A missing file and an unmounted source: the report counts the readable sources and names the other one. + await expect(await request.post('/__test__/system/missing-file')).toBeOK(); + await expect(await request.post('/__test__/system/missing-source')).toBeOK(); + await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click(); + await layout.getByRole('button', { name: 'Run health checks', exact: true }).click(); + await expect(dialog.locator('.operation').filter({ hasText: 'Missing files' }).locator('.chip')) + .toHaveText(['1 link needs review', 'Passed', 'Not available']); + await expect(dialog.locator('.operation h3').filter({ hasText: 'Missing files' })) + .toHaveText(['Missing files · output', 'Missing files · input', 'Missing files · unmounted-drive']); + // Review missing links goes to Maintenance, where the cleanup's existing tool is linked. + await dialog.getByRole('button', { name: 'Review missing links', exact: true }).click(); + await expect(page.locator('dialog#modal[open]')).toHaveCount(0); + await expect(page).toHaveURL(`${workspace}#/system/maintenance`); + await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click(); + await expect(layout.locator('.metric').nth(3).locator('strong')).toHaveText('1 issue'); +}); + +test('a health check that fails says so and keeps the last result', async ({ page, request }) => { + await expect(await request.post('/__test__/system/fail', { data: { path: '/prompt_manager/system/health' } })).toBeOK(); + await page.goto(`${workspace}#/system/maintenance`); + const run = layoutOf(page).getByRole('button', { name: 'Run health checks', exact: true }); + await run.click(); + await expect(page.locator('#workspace-toast')).toHaveText('The health check could not finish. Nothing was changed.'); + await expect(page.locator('dialog#modal[open]')).toHaveCount(0); + await expect(run).not.toHaveAttribute('aria-disabled', 'true'); + await openOverview(page); + await expect(layoutOf(page).locator('.metric').nth(3).locator('strong')).toHaveText('Not checked'); +}); + +test('thumbnail row reflects the status endpoint', async ({ page, request }) => { + await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 3 } })).toBeOK(); + const status = await serverJson(request, '/prompt_manager/images/thumbnails/status'); + expect(status.pending).toBe(3); + const layout = await openOverview(page); + const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) }); + await expect(row.locator('h3')).toHaveText('3 missing thumbnails'); + // Until the thumbnail review dialog (S5), Review thumbnails opens Sources. + await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click(); + await expect(page).toHaveURL(`${workspace}#/system/sources`); +}); + +test('the thumbnail row paints when its own status read answers', async ({ page, request }) => { + const held = async () => (await serverJson(request, '/__test__/hold')).held[THUMBNAILS_URL] ?? 0; + await expect(await request.post('/__test__/hold', { data: { path: THUMBNAILS_URL } })).toBeOK(); + const layout = await openOverview(page); + const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) }); + // The other values do not wait for the walk of every source. + await expect(row.locator('h3')).toHaveText('Checking thumbnails…'); + await expect(layout.locator('.summary-metrics > .metric')).toHaveCount(4); + await expect.poll(held).toBe(1); + await expect(await request.post('/__test__/release')).toBeOK(); + await expect(row.locator('h3')).toHaveText('No missing thumbnails'); +}); + +test('a failed thumbnail read is unavailable, never zero', async ({ page }) => { + await page.route(`**${THUMBNAILS_URL}`, route => route.fulfill({ status: 500, contentType: 'application/json', + body: JSON.stringify({ success: false, error: 'Injected failure' }) })); + const layout = await openOverview(page); + const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) }); + await expect(row.locator('h3')).toHaveText('Thumbnail status unavailable'); + await expect(layout.getByRole('alert')).toHaveCount(0); + await expect(layout.locator('.metric').nth(2).locator('strong')).toHaveText('24'); +}); + +test('ambiguous row reads Not available yet (Q6)', async ({ page }) => { + const layout = await openOverview(page); + const row = layout.locator('.operation').filter({ hasText: 'Ambiguous image links' }); + await expect(row.locator('p')).toHaveText('Not available yet.'); + const review = row.getByRole('button', { name: 'Review metadata', exact: true }); + await expect(review).toHaveAttribute('aria-disabled', 'true'); + await review.click({ force: true }); + await expect(page).toHaveURL(`${workspace}#/system/overview`); +}); + +test('View backups / Appearance / Manage sources switch sections', async ({ page }) => { + for (const [label, section, pressed] of [['View backups', 'backups', 'Backup & export'], ['Appearance', 'appearance', 'Appearance'], + ['Manage sources', 'sources', 'Sources & scans']]) { + const layout = await openOverview(page); + await layout.getByRole('button', { name: label, exact: true }).click(); + await expect(page).toHaveURL(`${workspace}#/system/${section}`); + await expect(page.locator('#workspace-main .section-nav [aria-pressed="true"]')).toHaveText(pressed); + await expect(layout.locator(':scope > .summary-metrics')).toHaveCount(0); + } +}); + +test('statistics dialog lists runs by category and library totals', async ({ page, request }) => { + const statistics = await serverJson(request, '/prompt_manager/system/statistics'); + expect(statistics.by_category.length).toBeGreaterThan(1); + const layout = await openOverview(page); + const explore = layout.getByRole('button', { name: 'Explore', exact: true }); + await explore.click(); + const dialog = page.locator('dialog#modal[open]'); + await expect(dialog.locator('#modal-title')).toHaveText('Usage statistics'); + const tables = dialog.locator('#modal-body > table'); + await expect(tables.locator('caption')).toHaveText(['Runs by category', 'Library totals']); + await expect(tables.first().locator('thead th')).toHaveText(['Category', 'Runs', 'Avg. rating']); + const cells = table => table.locator('tbody tr').evaluateAll(rows => rows.map(row => [...row.cells].map(cell => cell.textContent))); + expect(await cells(tables.first())).toEqual(statistics.by_category.map(row => [row.category ?? 'Uncategorized', + number(row.runs), row.average_rating === null ? 'Not rated' : `${number(row.average_rating)} / 5`])); + expect(await cells(tables.nth(1))).toEqual([['Prompts', number(statistics.totals.total_prompts)], + ['Tags', number(statistics.totals.total_tags)], ['Images', number(statistics.totals.total_images)], + ['Rated images', number(statistics.rated_images)]]); + await page.keyboard.press('Escape'); + await expect(explore).toBeFocused(); + // A statistics read that fails says so; the Overview stays. + await expect(await request.post('/__test__/system/fail', { data: { path: '/prompt_manager/system/statistics' } })).toBeOK(); + await explore.click(); + await expect(page.locator('#workspace-toast')).toHaveText('Statistics could not be loaded. Your saved data has not changed.'); + await expect(page.locator('dialog#modal[open]')).toHaveCount(0); +}); + +test('load failure shows Retry and no zeros', async ({ page, request }) => { + await expect(await request.post('/__test__/system/fail', { data: { path: OVERVIEW_URL } })).toBeOK(); + await page.goto(`${workspace}#/system/overview`); + const layout = layoutOf(page); + await expect(layout.getByRole('alert')).toHaveText('The overview could not be loaded. Your saved data has not changed.'); + await expect(layout.locator('.metric')).toHaveCount(0); + await expect(layout).not.toContainText('0'); + await layout.getByRole('button', { name: 'Retry', exact: true }).click(); + await expect(layout.locator('.summary-metrics > .metric')).toHaveCount(4); + await expect(layout.getByRole('alert')).toHaveCount(0); +}); + +test('out-of-order overview responses never repaint a newer section', async ({ page, request }) => { + const held = async () => (await serverJson(request, '/__test__/hold')).held[OVERVIEW_URL] ?? 0; + await expect(await request.post('/__test__/hold', { data: { path: OVERVIEW_URL } })).toBeOK(); + await page.goto(`${workspace}#/system/overview`); + const layout = layoutOf(page); + await expect(layout.getByRole('status')).toHaveText('Loading the overview…'); + await expect.poll(held).toBe(1); + // Leave for Appearance, come back (a second read), then leave again before either answers. + const nav = page.locator('#workspace-main .section-nav'); + await nav.getByRole('button', { name: 'Appearance', exact: true }).click(); + await nav.getByRole('button', { name: 'Overview', exact: true }).click(); + await expect.poll(held).toBe(2); + await nav.getByRole('button', { name: 'Sources & scans', exact: true }).click(); + await expect(await request.post('/__test__/release')).toBeOK(); + await expect.poll(held).toBe(0); + // Both late answers reached a section that had left: Sources keeps its own body. + await expect(layout.locator(':scope > section.handoff')).toBeVisible(); + await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))); + await expect(layout.locator('.summary-metrics')).toHaveCount(0); + expect(await layout.evaluate(node => [...node.children].map(child => child.className))).toEqual(['surface handoff', 'save-row']); + // Back on Overview, one fresh read paints one set of metrics. + await nav.getByRole('button', { name: 'Overview', exact: true }).click(); + await expect(layout.locator('.summary-metrics')).toHaveCount(1); + await expect(layout.locator('.two-col')).toHaveCount(1); +}); diff --git a/tests/browser/workspace.spec.mjs b/tests/browser/workspace.spec.mjs index fff42a6..f065b4c 100644 --- a/tests/browser/workspace.spec.mjs +++ b/tests/browser/workspace.spec.mjs @@ -18,7 +18,8 @@ const routes = [ ['tags', 'Tags & categories', null], // #249 local Inspect: the native Inspect a file page; Overview's tool card still links the legacy metadata page. ['inspect', 'Inspect a file', null], - ['system', 'System', '/prompt_manager/admin'], + // #249 System S2: native System; its undelivered sections hand off per section (D94, asserted below). + ['system', 'System', null], // #249 D51: SAVED VIEWS' Favorites, Images filtered to favorites. ['favorites', 'Favorites', null], // #249 Activity-1: the concept's Activity page in the tools-and-system group. @@ -409,6 +410,8 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true); await sidebarLink(page, 'System').click(); + // System's Overview is native (S2 Task 6); a section not delivered yet still hands off (D94). + await main.locator('.section-nav').getByRole('button', { name: 'Integrations', exact: true }).click(); const handoff = main.locator('section.handoff'); await expect(handoff).toHaveClass(/\bsurface\b/); expect(await handoff.evaluate(surfaceStyle)).toMatchObject(SURFACE); @@ -417,6 +420,98 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true); }); +// #249 System S2 Task 5: the concept's renderSystem shell (app.js 698–706) at `#/system/
`, every section a D94 +// hand-off until its PR, and the Save row's truthful status (D92). +const SYSTEM_SECTION_LABELS = ['Overview', 'Appearance', 'Sources & scans', 'Integrations', 'Auto-tagging', 'Maintenance', + 'Backup & export', 'Diagnostics & logs']; +const SYSTEM_HASHES = ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics']; +const SYSTEM_HANDOFFS = { appearance: ['/prompt_manager/gallery', '/prompt_manager/admin'], sources: ['/prompt_manager/admin', '/prompt_manager/gallery'], + maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] }; + +test('System is native: the concept shell, section navigation, sections, hand-offs and the Save row', async ({ page }) => { + const mutations = []; + page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) mutations.push(item.url()); }); + await page.goto(`${workspace}#/system`); + const main = page.locator('#workspace-main'); + await expect(main.locator('h1')).toHaveText('System'); + await expect(main.locator('.heading p.muted')).toHaveText('Configuration, maintenance and recovery—with browsing kept in the workspace.'); + // D92: no "Sample configuration" chip. + await expect(main.locator('.heading .chip')).toHaveCount(0); + expect(await main.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`))) + .toEqual(['div.heading', 'div.section-nav', 'div.settings-layout']); + const nav = main.getByRole('group', { name: 'System sections', exact: true }); + await expect(nav.getByRole('button')).toHaveText(SYSTEM_SECTION_LABELS); + await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true'); + await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1); + await expect(page.locator('aside.sidebar a[href="#/system"]')).toHaveAttribute('aria-current', 'page'); + const layout = main.locator('.settings-layout'); + await expect(layout.locator(':scope > .save-row > span.small#settings-status')).toHaveText('No unsaved changes.'); + const save = layout.locator(':scope > .save-row > button.ca-primary'); + await expect(save).toHaveText('Save preferences'); + await expect(save).toHaveAttribute('aria-disabled', 'true'); + // Playwright waits on an aria-disabled control, so the ignored press is forced. + await save.click({ force: true }); + await expect(layout.locator('#settings-status')).toHaveText('No unsaved changes.'); + + for (const [index, label] of SYSTEM_SECTION_LABELS.entries()) { + const hash = SYSTEM_HASHES[index]; + const button = nav.getByRole('button', { name: label, exact: true }); + await button.click(); + await expect(page).toHaveURL(`${workspace}#/system/${hash}`); + await expect(button).toHaveAttribute('aria-pressed', 'true'); + await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1); + // The section changes in place: focus stays on the pressed section, not the page title. + await expect(button).toBeFocused(); + // S2 Task 6: Overview is native (workspace-system.spec.mjs); Maintenance shows its health-check notice above + // the hand-off (D123); every other section hands off. + if (hash === 'overview') await expect(layout.locator(':scope > .summary-metrics')).toBeVisible(); + expect(await layout.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`))) + .toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'], + maintenance: ['div.notice', 'section.surface handoff', 'div.save-row'] }[hash] + ?? ['section.surface handoff', 'div.save-row']); + if (hash === 'overview') continue; + const handoff = layout.locator('section.handoff'); + await expect(handoff.locator('h2')).toHaveText('Your current tools are ready'); + expect(await handoff.locator('a.workspace-link').evaluateAll(links => links.map(link => link.getAttribute('href')))) + .toEqual(SYSTEM_HANDOFFS[hash] ?? ['/prompt_manager/admin']); + await expect(layout.locator('#settings-status')).toHaveText('No unsaved changes.'); + await expect(page.locator('aside.sidebar a[href="#/system"]')).toHaveAttribute('aria-current', 'page'); + } + // Deep links, reload and history keep the section; an unknown section is Overview under its own hash. + await page.reload(); + await expect(nav.getByRole('button', { name: 'Diagnostics & logs', exact: true })).toHaveAttribute('aria-pressed', 'true'); + await page.goBack(); + await expect(page).toHaveURL(`${workspace}#/system/backups`); + await expect(nav.getByRole('button', { name: 'Backup & export', exact: true })).toHaveAttribute('aria-pressed', 'true'); + await page.goto(`${workspace}#/system/general`); + await expect(page).toHaveURL(`${workspace}#/system/overview`); + await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true'); + // The sidebar's System returns to the Overview section. + await nav.getByRole('button', { name: 'Integrations', exact: true }).click(); + await sidebarLink(page, 'System').click(); + await expect(page).toHaveURL(`${workspace}#/system`); + await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true'); + // Leaving System and coming back renders it once, with a fresh section. + await sidebarLink(page, 'Images').click(); + await expect(main.locator('h1')).toHaveText('Images'); + await expect(main.locator('.section-nav')).toHaveCount(0); + await page.goBack(); + await expect(main.locator('.section-nav')).toHaveCount(1); + await expect(main.locator('.settings-layout')).toHaveCount(1); + expect(mutations).toEqual([]); +}); + +test('Overview opens native System from its System card (Q12)', async ({ page }) => { + await page.goto(`${workspace}#/overview`); + const card = page.locator('#workspace-main .tool-card').filter({ has: page.getByRole('heading', { name: 'System', exact: true }) }); + await expect(card.locator('p.small')).toHaveText('Configuration, maintenance and recovery—with browsing kept in the workspace.'); + const link = card.getByRole('link', { name: 'Open System', exact: true }); + await expect(link).toHaveAttribute('href', '#/system/overview'); + await link.click(); + await expect(page.locator('#workspace-main h1')).toHaveText('System'); + await expect(page.locator('#workspace-main .section-nav [aria-pressed="true"]')).toHaveText('Overview'); +}); + // Task A3: the concept sidebar (index.html h16-41, app.js nav()) rendered from real data (D27, D28). const NAV_TOTALS = [ ['images', 'Images', '/prompt_manager/images/output?mode=workspace&limit=1'], diff --git a/tests/browser/workspace_system_fixture.py b/tests/browser/workspace_system_fixture.py new file mode 100644 index 0000000..6027c64 --- /dev/null +++ b/tests/browser/workspace_system_fixture.py @@ -0,0 +1,116 @@ +"""System test hooks for the browser tests (#249 System S2 Task 6). + +The data is the concept-shaped real fixture (``POST /__test__/concept/reset``, +``workspace_concept_fixture.py``): its parents, saved pairs, images, ratings +and runs are what the Overview, the Statistics dialog and the health check +read. These per-server hooks only change what a System test needs on top: + +- ``POST /__test__/system/reset`` restores the configured sources, forgets the + session's health report and ends every injected failure; +- ``POST /__test__/system/fail`` ``{path, times}`` answers the next ``times`` + requests for ``path`` (a ``/prompt_manager/system/`` route) with the routes' + 500 error envelope; +- ``POST /__test__/system/missing-source`` configures one more source that + does not exist (an unmounted drive); +- ``POST /__test__/system/missing-file`` deletes the newest linked image's + file from disk, leaving its database row (a missing-file link); +- ``POST /__test__/system/unthumbnailed`` ``{count}`` writes ``count`` plain + PNGs with no thumbnail into the output folder. + +Delays use the shared request holds (``request_hold.py``). +""" + +import os +from pathlib import Path + +from aiohttp import web +from PIL import Image + +FAILABLE_PREFIX = "/prompt_manager/system/" +MAX_UNTHUMBNAILED = 50 + + +def register_system_commands(app, routes, api, root: Path): + from py.config import GalleryConfig + + configured = list(GalleryConfig.MONITORING_DIRECTORIES) + failures: dict[str, int] = {} + + @web.middleware + async def inject_failures(request, handler): + left = failures.get(request.path, 0) + if left <= 0: + return await handler(request) + failures[request.path] = left - 1 + return web.json_response( + {"success": False, "code": None, "error": "Injected failure"}, status=500 + ) + + app.middlewares.append(inject_failures) + + def reset(): + failures.clear() + GalleryConfig.MONITORING_DIRECTORIES = list(configured) + api._health_report = None + + @routes.post("/__test__/system/reset") + async def system_reset(request): + reset() + return web.json_response({"success": True}) + + @routes.post("/__test__/system/fail") + async def system_fail(request): + data = await request.json() + path, times = data.get("path"), data.get("times", 1) + if ( + not isinstance(path, str) + or not path.startswith(FAILABLE_PREFIX) + or isinstance(times, bool) + or not isinstance(times, int) + or not 1 <= times <= 10 + ): + raise web.HTTPBadRequest() + failures[path] = times + return web.json_response({"success": True}) + + @routes.post("/__test__/system/missing-source") + async def missing_source(request): + absent = str(root / "unmounted-drive") + if os.path.exists(absent): + raise web.HTTPConflict() + GalleryConfig.MONITORING_DIRECTORIES = [*configured, absent] + return web.json_response( + {"success": True, "configured": len(GalleryConfig.MONITORING_DIRECTORIES)} + ) + + @routes.post("/__test__/system/missing-file") + async def missing_file(request): + with api.db.model.get_connection() as conn: + row = conn.execute( + "SELECT file_path FROM generated_images ORDER BY id DESC LIMIT 1" + ).fetchone() + if row is None: + raise web.HTTPNotFound() + Path(row[0]).unlink() + api.invalidate_gallery_cache() + return web.json_response({"success": True, "filename": Path(row[0]).name}) + + @routes.post("/__test__/system/unthumbnailed") + async def unthumbnailed(request): + data = await request.json() + count = data.get("count") + if ( + isinstance(count, bool) + or not isinstance(count, int) + or not 1 <= count <= MAX_UNTHUMBNAILED + ): + raise web.HTTPBadRequest() + folder = root / "output" / "unthumbnailed" + folder.mkdir(exist_ok=True) + for number in range(count): + with Image.new("RGB", (64, 64), "#2b3442") as image: + image.save(folder / f"plain_{number + 1:03}.png") + api.invalidate_gallery_cache() + return web.json_response({"success": True}) + + return reset diff --git a/tests/js/workspace-activity-store.test.js b/tests/js/workspace-activity-store.test.js index 5fce2cd..82b69da 100644 --- a/tests/js/workspace-activity-store.test.js +++ b/tests/js/workspace-activity-store.test.js @@ -182,6 +182,24 @@ test('start returns busy with the running job', async () => { world.dispose(); }); +test('track follows a job another route started, with no read of its own', async () => { + const { createActivityStore } = await source; + const { world, ...deps } = harness(); + const store = createActivityStore(deps); + const seen = []; + store.subscribe(() => seen.push(store.find(ID_A)?.state)); + const job = store.track(summary({ kind: 'health', title: 'Health check' })); + assert.equal(job.id, ID_A); + assert.equal(world.gets, 0, 'the started job is known without a list read'); + assert.deepEqual(world.open().map(stream => stream.url), [`/prompt_manager/jobs/${ID_A}/events`]); + assert.deepEqual(world.toasts, [], 'the route that started it says so itself'); + send(world.open()[0], summary({ kind: 'health', title: 'Health check', state: 'finished', progress: 100 })); + assert.deepEqual(seen, ['running', 'finished']); + assert.equal(world.open().length, 0, 'an ended job is no longer followed'); + assert.throws(() => store.track({ id: 'nope' }), /Invalid job/); + world.dispose(); +}); + test('a list that is not a job list counts as a failed read', async () => { const { createActivityStore } = await source; const { world, api, ...deps } = harness(); diff --git a/tests/js/workspace-state.test.js b/tests/js/workspace-state.test.js index a02129a..c05714b 100644 --- a/tests/js/workspace-state.test.js +++ b/tests/js/workspace-state.test.js @@ -23,8 +23,50 @@ test('tags is a native route titled Tags; only unbuilt routes hand off to legacy assert.equal(routeTitle(route), route.label); } assert.equal(routeTitle(resolveRoute('#/library/7')), 'Prompt library'); - for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id); - for (const id of ['system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), true, id); + // System is native from S2 (#249 System): its undelivered sections hand off per section (D94), not the route. + for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity', 'system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id); +}); + +test('system resolves its sections; #/overview stays the production hub', async () => { + const { resolveRoute, routeTitle, isHandoff, routes } = await source; + // `#/system` is the Overview section and keeps the sidebar's hash. + const system = resolveRoute('#/system'); + assert.deepEqual([system.id, system.section, system.hash, routeTitle(system), isHandoff(system)], + ['system', 'overview', '#/system', 'System', false]); + for (const section of ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics']) { + const route = resolveRoute(`#/system/${section}`); + assert.deepEqual([route.id, route.section, route.hash], ['system', section, `#/system/${section}`]); + } + // An unknown child is the Overview section with a replaced hash. + for (const hash of ['#/system/general', '#/system/', '#/system/__proto__', '#/system/backups/1', '#/system/']) { + const route = resolveRoute(hash); + assert.deepEqual([route.id, route.section, route.hash], ['system', 'overview', '#/system/overview'], hash); + } + // D28: the production hub is untouched. + const hub = resolveRoute('#/overview'); + assert.deepEqual([hub.id, hub.section], ['overview', undefined]); + // Q12: Overview's System card opens the native System Overview, with no legacy href or action. + const card = routes.find(route => route.id === 'system'); + assert.equal(card.href, '#/system/overview'); + assert.equal(card.native, true); + assert.equal(card.action, 'Open System'); +}); + +test('the router renders again when only the System section changes', async () => { + const { mountRouter } = await source; + const win = new EventTarget(); + win.location = { hash: '#/system' }; + const replacements = []; + win.history = { state: null, replaceState(state, unused, hash) { replacements.push(hash); win.location.hash = hash; } }; + const seen = []; + const dispose = mountRouter(win, route => seen.push([route.id, route.section])); + for (const hash of ['#/system/appearance', '#/system/appearance', '#/system/bogus', '#/system/overview', '#/images']) { + win.location.hash = hash; + win.dispatchEvent(new Event('hashchange')); + } + assert.deepEqual(seen, [['system', 'overview'], ['system', 'appearance'], ['system', 'overview'], ['images', undefined]]); + assert.deepEqual(replacements, ['#/system/overview']); + dispose(); }); test('inspect is the native Inspect a file page with the concept description', async () => { diff --git a/tests/js/workspace-system-controls.test.js b/tests/js/workspace-system-controls.test.js new file mode 100644 index 0000000..19a99b8 --- /dev/null +++ b/tests/js/workspace-system-controls.test.js @@ -0,0 +1,214 @@ +// System controls (#249 System S2 Task 5): the concept's `setting()` (app.js 624–634, its toggles hydrated into Toolkit +// switches by `hydrateSwitches` 708–717), `operation()` (645–654) and `sectionNav()` (306–308) as DOM builders whose +// listeners live on the caller's scope. +const test = require('node:test'); +const assert = require('node:assert/strict'); + +const controlsModule = import('../../web/js/workspace/system-controls.mjs'); +const coreModule = import('../../web/js/shared/toolkit/core.js'); + +/** A minimal DOM whose elements count their live listeners (a listener's signal aborts it, as in a browser). */ +function fakeDOM() { + const previous = global.document; + const elements = []; + class Element { + constructor(tag) { + this.tagName = tag.toUpperCase(); + this.children = []; + this.attributes = {}; + this.listeners = []; + this.dataset = {}; + this.className = ''; + this.id = ''; + this.value = ''; + this._text = ''; + elements.push(this); + } + get textContent() { return this._text + this.children.map(child => child.textContent).join(''); } + set textContent(text) { this.children = []; this._text = String(text); } + append(...nodes) { + for (const node of nodes) { + const child = typeof node === 'string' ? Object.assign(new Element('#text'), { _text: node }) : node; + child.parentNode = this; + this.children.push(child); + } + } + replaceChildren(...nodes) { this.children = []; this._text = ''; this.append(...nodes); } + setAttribute(name, value) { this.attributes[name] = String(value); } + getAttribute(name) { return Object.hasOwn(this.attributes, name) ? this.attributes[name] : null; } + removeAttribute(name) { delete this.attributes[name]; } + addEventListener(type, callback, options = {}) { + if (options.signal?.aborted) return; + const entry = { type, callback }; + this.listeners.push(entry); + options.signal?.addEventListener('abort', () => { this.listeners = this.listeners.filter(item => item !== entry); }); + } + removeEventListener(type, callback) { this.listeners = this.listeners.filter(item => item.type !== type || item.callback !== callback); } + dispatch(type) { for (const { callback } of this.listeners.filter(item => item.type === type)) callback({ type, target: this }); } + click() { this.dispatch('click'); } + } + global.document = { + createElement: tag => new Element(tag), + createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }), + }; + const walk = (node, found = []) => { found.push(node); node.children.forEach(child => walk(child, found)); return found; }; + return { + listenerCount: () => elements.reduce((sum, element) => sum + element.listeners.length, 0), + find: (root, predicate) => walk(root).find(predicate), + tags: node => node.children.map(child => child.tagName.toLowerCase()), + restore() { global.document = previous; }, + }; +} + +test('settingRow is the concept row: div > label[for] + p, then the control', async () => { + const { settingRow } = await controlsModule; + const { createScope } = await coreModule; + const dom = fakeDOM(); + const scope = createScope(); + try { + const changes = []; + const row = settingRow({ id: 'timeout', label: 'Node search auto-hide (seconds)', + description: 'Node search behavior only. Use 0 to keep results visible.', + control: { type: 'number', value: 30, min: 0, max: 300 }, scope, onChange: value => changes.push(value) }); + assert.equal(row.className, 'setting-row'); + assert.deepEqual(dom.tags(row), ['div', 'input']); + const [text, input] = row.children; + assert.deepEqual(dom.tags(text), ['label', 'p']); + assert.equal(text.children[0].textContent, 'Node search auto-hide (seconds)'); + assert.equal(text.children[0].htmlFor, 'timeout'); + assert.equal(text.children[1].textContent, 'Node search behavior only. Use 0 to keep results visible.'); + assert.deepEqual([input.id, input.type, input.value, input.min, input.max], ['timeout', 'number', '30', '0', '300']); + assert.equal(input.dataset.setting, 'timeout'); + input.value = '45'; + input.dispatch('change'); + assert.deepEqual(changes, [45]); + + const select = settingRow({ id: 'open-mode', label: 'Open from ComfyUI', description: 'Keep the node pack’s popup / new-tab preference.', + control: { type: 'select', options: [['newtab', 'New tab'], ['popup', 'Popup window']], value: 'popup' }, scope, + onChange: value => changes.push(value) }).children[1]; + assert.equal(select.tagName, 'SELECT'); + assert.deepEqual(select.children.map(option => [option.value, option.textContent]), [['newtab', 'New tab'], ['popup', 'Popup window']]); + assert.equal(select.value, 'popup'); + select.value = 'newtab'; + select.dispatch('change'); + assert.deepEqual(changes, [45, 'newtab']); + + // The concept's range is 1–16 unless the row says otherwise (D113: the CPU count). + const range = settingRow({ id: 'workers', label: 'Worker threads', description: 'Balance background scans.', + control: { type: 'range', value: 4 } }).children[1]; + assert.deepEqual([range.type, range.min, range.max, range.value], ['range', '1', '16', '4']); + const password = settingRow({ id: 'civitai-key', label: 'CivitAI API key', description: 'Credentials stay on the server.', + control: { type: 'password', value: '' } }).children[1]; + assert.equal(password.type, 'password'); + + // A ready-made control is placed as given and takes the row id. + const custom = document.createElement('textarea'); + const customRow = settingRow({ id: 'custom', label: 'Custom', description: 'Any control.', control: custom }); + assert.equal(customRow.children[1], custom); + assert.equal(custom.id, 'custom'); + assert.throws(() => settingRow({ id: '', label: 'x', description: 'y', control: { type: 'number' } }), TypeError); + assert.throws(() => settingRow({ id: 'x', label: 'x', description: 'y', control: { type: 'colour' } }), TypeError); + } finally { + scope.dispose(); + dom.restore(); + } +}); + +test('a toggle row holds the Toolkit switch, labelled by the row label', async () => { + const { settingRow } = await controlsModule; + const { createScope } = await coreModule; + const dom = fakeDOM(); + const scope = createScope(); + try { + const changes = []; + const row = settingRow({ id: 'video-auto', label: 'Autoplay videos', description: 'Videos stay within the same gallery.', + control: { type: 'toggle', checked: false }, scope, onChange: value => changes.push(value) }); + const field = row.children[1]; + // createSwitch('', '', id): `label.ca-switch` > input[type=checkbox][role=switch] + an empty label text. + assert.equal(field.tagName, 'LABEL'); + assert.equal(field.className, 'ca-switch'); + const input = field.children[0]; + assert.deepEqual([input.type, input.role, input.id, input.checked], ['checkbox', 'switch', 'video-auto', false]); + assert.equal(row.children[0].children[0].htmlFor, input.id); + assert.equal(input.dataset.setting, 'video-auto'); + input.checked = true; + input.dispatch('change'); + assert.deepEqual(changes, [true]); + // The concept's toggles are on unless configured off (`val !== false`). + const on = settingRow({ id: 'video-loop', label: 'Loop videos', description: 'Repeat a clip.', control: { type: 'toggle' } }); + assert.equal(on.children[1].children[0].checked, true); + } finally { + scope.dispose(); + dom.restore(); + } +}); + +test('operationRow is the concept operation with a danger class only when flagged', async () => { + const { operationRow } = await controlsModule; + const { createScope } = await coreModule; + const dom = fakeDOM(); + const scope = createScope(); + try { + const clicks = []; + const row = operationRow({ title: 'Missing-file links', + description: 'Remove stale database links only. Files on disk are never deleted by this operation.', + label: 'Review plan', danger: true, scope, onClick: button => clicks.push(button) }); + assert.equal(row.className, 'operation'); + assert.deepEqual(dom.tags(row), ['div', 'div']); + assert.deepEqual(dom.tags(row.children[0]), ['h3', 'p']); + assert.equal(row.children[0].children[0].textContent, 'Missing-file links'); + assert.equal(row.children[1].className, 'row-actions'); + const button = row.children[1].children[0]; + assert.deepEqual([button.tagName, button.type, button.textContent, button.className], ['BUTTON', 'button', 'Review plan', 'danger']); + button.click(); + assert.deepEqual(clicks, [button]); + const plain = operationRow({ title: 'Statistics', description: 'Usage by prompt, category, tag and date.', label: 'Explore', + scope, onClick() {} }); + assert.equal(plain.children[1].children[0].className, ''); + assert.throws(() => operationRow({ title: 'x', description: 'y', label: 'z', onClick() {} }), TypeError); + } finally { + scope.dispose(); + dom.restore(); + } +}); + +test('sectionNav marks the active section and reports the chosen one', async () => { + const { sectionNav } = await controlsModule; + const { createScope } = await coreModule; + const dom = fakeDOM(); + const scope = createScope(); + try { + const chosen = []; + const nav = sectionNav({ label: 'System sections', sections: [['overview', 'Overview'], ['appearance', 'Appearance']], + active: 'appearance', scope, onSelect: id => chosen.push(id) }); + assert.equal(nav.className, 'section-nav'); + assert.equal(nav.getAttribute('role'), 'group'); + assert.equal(nav.getAttribute('aria-label'), 'System sections'); + assert.deepEqual(nav.children.map(button => [button.textContent, button.getAttribute('aria-pressed'), button.dataset.value]), + [['Overview', 'false', 'overview'], ['Appearance', 'true', 'appearance']]); + nav.children[0].click(); + assert.deepEqual(chosen, ['overview']); + } finally { + scope.dispose(); + dom.restore(); + } +}); + +test('disposing the scope leaves no listener behind', async () => { + const { settingRow, operationRow, sectionNav } = await controlsModule; + const { createScope } = await coreModule; + const dom = fakeDOM(); + const scope = createScope(); + try { + settingRow({ id: 'a', label: 'A', description: 'a', control: { type: 'toggle' }, scope, onChange() {} }); + settingRow({ id: 'b', label: 'B', description: 'b', control: { type: 'select', options: ['Grid', 'List'] }, scope, onChange() {} }); + operationRow({ title: 'C', description: 'c', label: 'Go', scope, onClick() {} }); + sectionNav({ label: 'System sections', sections: [['overview', 'Overview']], active: 'overview', scope, onSelect() {} }); + assert.equal(dom.listenerCount() > 0, true); + scope.dispose(); + assert.equal(dom.listenerCount(), 0); + } finally { + scope.dispose(); + dom.restore(); + } +}); diff --git a/tests/js/workspace-system-overview.test.js b/tests/js/workspace-system-overview.test.js new file mode 100644 index 0000000..bafcd0f --- /dev/null +++ b/tests/js/workspace-system-overview.test.js @@ -0,0 +1,255 @@ +// System › Overview (#249 System S2 Task 6): the concept's Overview copy on real values (D93), the Health report rows +// (D95), the Statistics tables (D96) and the health check run that waits for its read job. +const test = require('node:test'); +const assert = require('node:assert/strict'); + +const overviewModule = import('../../web/js/workspace/system-overview.mjs'); + +const OVERVIEW = Object.freeze({ + library: { positive_prompts: 1234, negative_variants: 7 }, + images: { indexed: 24, rated: 20 }, + health: null, + backup: null, + needs_review: { pending: null }, + sources: { configured: 2, effective: 2, auto_detected: false }, +}); + +test('metrics are the server counts and Not checked before any health check', async () => { + const { metricValues } = await overviewModule; + assert.deepEqual(metricValues(OVERVIEW), [ + [(1234).toLocaleString(), 'Positive prompts'], ['7', 'Negative variants'], ['24', 'Images'], + ['Not checked', 'Database integrity'], + ]); +}); + +test('metric 4 is the last health result', async () => { + const { integrityText } = await overviewModule; + assert.equal(integrityText(null), 'Not checked'); + assert.equal(integrityText({ status: 'passed', issues: 0 }), 'Passed'); + assert.equal(integrityText({ status: 'issues', issues: 1 }), '1 issue'); + assert.equal(integrityText({ status: 'issues', issues: 3 }), '3 issues'); + assert.equal(integrityText({ status: 'incomplete', issues: 0 }), 'Incomplete'); + // Issues found beside a check that could not run (Needs review, Q6) are still named. + assert.equal(integrityText({ status: 'incomplete', issues: 2 }), '2 issues'); +}); + +test('library health rows never invent a review count, a backup or thumbnails', async () => { + const { ambiguousTitle, backupText } = await overviewModule; + assert.equal(ambiguousTitle({ pending: null }), 'Ambiguous image links'); + assert.equal(ambiguousTitle(undefined), 'Ambiguous image links'); + assert.equal(ambiguousTitle({ pending: 1 }), '1 ambiguous image link'); + assert.equal(ambiguousTitle({ pending: 3 }), '3 ambiguous image links'); + assert.equal(backupText(null), 'No retained backup record.'); + assert.equal(backupText({ verified_at: 'not a time' }), 'No retained backup record.'); + assert.match(backupText({ verified_at: '2026-10-22T09:30:00Z' }), /^Last verified backup .+\.$/); +}); + +test('the thumbnail row is its own read of the thumbnail status', async () => { + const { thumbnailTitle, THUMBNAILS_LOADING } = await overviewModule; + // Until `/images/thumbnails/status` answers the row says it is checking; a failed read is unavailable, never 0. + assert.equal(THUMBNAILS_LOADING, 'Checking thumbnails…'); + assert.equal(thumbnailTitle(undefined), THUMBNAILS_LOADING); + assert.equal(thumbnailTitle({ pending: 0 }), 'No missing thumbnails'); + assert.equal(thumbnailTitle({ pending: 1 }), '1 missing thumbnail'); + assert.equal(thumbnailTitle({ pending: 12 }), '12 missing thumbnails'); + assert.equal(thumbnailTitle(null), 'Thumbnail status unavailable'); + assert.equal(thumbnailTitle({ pending: 'x' }), 'Thumbnail status unavailable'); + // With no image source there is nothing to count. + assert.equal(thumbnailTitle({ pending: 0 }, { effective: 0 }), 'Thumbnail status unavailable'); + assert.equal(thumbnailTitle({ pending: 2 }, { effective: 1 }), '2 missing thumbnails'); +}); + +test('an integrity count SQLite stopped at reads as a lower bound', async () => { + const { checkChip } = await overviewModule; + assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 100, capped: true }), '100+ problems found'); + assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 37, capped: false }), '37 problems found'); + assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 1 }), '1 problem found'); +}); + +test('the sources row tells configured, available and detected sources apart', async () => { + const { sourcesCopy } = await overviewModule; + const shared = 'Monitoring and enrichment share one source list.'; + assert.deepEqual(sourcesCopy({ configured: 2, effective: 2, auto_detected: false }), ['2 output sources', shared]); + assert.deepEqual(sourcesCopy({ configured: 2, effective: 1, auto_detected: false }), + ['1 output source', `1 of 2 configured sources available. ${shared}`]); + assert.deepEqual(sourcesCopy({ configured: 0, effective: 1, auto_detected: true }), + ['1 output source', `None configured is available; ComfyUI's output folder is used. ${shared}`]); +}); + +const REPORT = Object.freeze({ + checked_at: '2026-10-23T08:00:00Z', + status: 'issues', + checks: [ + { id: 'integrity', label: 'Integrity check', status: 'passed', count: 0 }, + { id: 'foreign_keys', label: 'Foreign keys & pair references', status: 'issues', count: 2 }, + { id: 'canonical_hashes', label: 'Canonical hashes', status: 'issues', count: 1 }, + { id: 'pair_uniqueness', label: 'Pair uniqueness', status: 'failed', count: null }, + { id: 'missing_files', label: 'Missing files', status: 'issues', count: 2, + sources: [{ name: 'output', status: 'checked', count: 2 }] }, + { id: 'ambiguous_metadata', label: 'Ambiguous metadata', status: 'unavailable', count: null }, + { id: 'consistency', label: 'Consistency', status: 'issues', count: null }, + ], +}); + +test('health rows give each check a chip from its real status, in the server order', async () => { + const { healthRows } = await overviewModule; + assert.deepEqual(healthRows(REPORT), [ + ['Integrity check', 'Passed'], + ['Foreign keys & pair references', '2 broken references'], + ['Canonical hashes', '1 hash does not match its text'], + ['Pair uniqueness', 'Could not check'], + ['Missing files', '2 links need review'], + ['Ambiguous metadata', 'Not available yet'], + ['Consistency', 'Issues found'], + ]); +}); + +test('missing files is reported per image source when there are several', async () => { + const { healthRows } = await overviewModule; + const report = { ...REPORT, checks: [{ id: 'missing_files', label: 'Missing files', status: 'issues', count: 1, partial: true, + sources: [{ name: 'output', status: 'checked', count: 1 }, { name: 'input', status: 'checked', count: 0 }, + { name: 'archive', status: 'unavailable', count: null }] }] }; + assert.deepEqual(healthRows(report), [ + ['Missing files · output', '1 link needs review'], + ['Missing files · input', 'Passed'], + ['Missing files · archive', 'Not available'], + ]); + const unavailable = { ...REPORT, checks: [{ id: 'missing_files', label: 'Missing files', status: 'unavailable', count: null, sources: [] }] }; + assert.deepEqual(healthRows(unavailable), [['Missing files', 'Not available']]); +}); + +test('the report summary says when it was checked and whether it is complete', async () => { + const { healthSummary } = await overviewModule; + const when = new Date(REPORT.checked_at).toLocaleString(); + assert.equal(healthSummary({ ...REPORT, status: 'passed' }), `Checked ${when}. Every check passed.`); + assert.equal(healthSummary({ ...REPORT, status: 'incomplete' }), `Checked ${when}. Some checks could not run, so this report is incomplete.`); + assert.equal(healthSummary({ ...REPORT, checked_at: 'x' }), 'Nothing was changed; review the checks with issues.'); +}); + +test('statistics rows are the real runs and ratings, then the library totals', async () => { + const { categoryRows, totalsRows } = await overviewModule; + const statistics = { + by_category: [{ category: 'Landscapes', runs: 1200, average_rating: 4.5 }, + { category: null, runs: 3, average_rating: null }], + totals: { total_prompts: 12, total_tags: 15, total_images: 24 }, + rated_images: 20, + }; + assert.deepEqual(categoryRows(statistics), [['Landscapes', (1200).toLocaleString(), '4.5 / 5'], ['Uncategorized', '3', 'Not rated']]); + assert.deepEqual(totalsRows(statistics), [['Prompts', '12'], ['Tags', '15'], ['Images', '24'], ['Rated images', '20']]); +}); + +/** A fake api answering each `get`/`post` from a queue per URL, recording the calls. */ +function fakeApi(answers) { + const calls = []; + const answer = (method, url) => { + calls.push(`${method} ${url}`); + const queue = answers[`${method} ${url}`]; + if (!queue?.length) throw new Error(`unexpected ${method} ${url}`); + const next = queue.shift(); + return next instanceof Error ? Promise.reject(next) : Promise.resolve(next); + }; + return { calls, get: url => answer('GET', url), post: url => answer('POST', url) }; +} + +const JOB = '0123456789abcdef0123456789abcdef'; +const job = (state, id = JOB, kind = 'health') => ({ id, kind, state }); + +/** + * A fake Activity store holding `jobs`: `track(job)` lists a started job, `load()` applies the next of `reads` (the + * server's job lists) and `push(job)` is a stream update; each notifies as the real store does. It never polls. + */ +function fakeActivity({ jobs = [], reads = [] } = {}) { + const listeners = new Set(); + const emit = () => { for (const listener of [...listeners]) listener(); }; + const store = { + loads: 0, + find: id => jobs.find(item => item.id === id), + running: () => jobs.filter(item => item.state === 'running'), + subscribe(listener) { listeners.add(listener); return () => listeners.delete(listener); }, + listeners: () => listeners.size, + async load() { + store.loads += 1; + if (reads.length) jobs = reads.shift(); + emit(); + }, + track(item) { + if (!item?.id) throw new TypeError('Invalid job'); + store.push(item); + return item; + }, + push(item) { + jobs = [...jobs.filter(other => other.id !== item.id), item]; + emit(); + }, + }; + return store; +} + +test('a health check tracks its job in the Activity store, then reads the report', async () => { + const { runHealthCheck } = await overviewModule; + const api = fakeApi({ + 'POST /prompt_manager/system/health': [{ job: job('running') }], + 'GET /prompt_manager/system/health': [{ report: REPORT }], + }); + const activity = fakeActivity(); + const checking = runHealthCheck(api, { activity }); + await new Promise(resolve => setImmediate(resolve)); + // The store follows the started job's event stream; nothing reads or polls the job here. + assert.equal(activity.loads, 0); + assert.equal(activity.find(JOB).state, 'running'); + assert.deepEqual(api.calls, ['POST /prompt_manager/system/health']); + activity.push(job('finished')); + assert.equal(await checking, REPORT); + assert.deepEqual(api.calls, ['POST /prompt_manager/system/health', 'GET /prompt_manager/system/health']); + assert.equal(activity.listeners(), 0); +}); + +test('a check already running elsewhere is waited for, not refused', async () => { + const { runHealthCheck } = await overviewModule; + const busy = Object.assign(new Error('busy'), { status: 409, code: 'busy' }); + const api = fakeApi({ + 'POST /prompt_manager/system/health': [busy], + 'GET /prompt_manager/system/health': [{ report: REPORT }], + }); + const activity = fakeActivity({ reads: [[job('running', 'f'.repeat(32), 'scan'), job('running')]] }); + const checking = runHealthCheck(api, { activity }); + await new Promise(resolve => setImmediate(resolve)); + assert.equal(activity.loads, 1); + activity.push(job('finished')); + assert.equal(await checking, REPORT); + // The other check ended between the refusal and the read: its report is the latest. + const ended = fakeApi({ 'POST /prompt_manager/system/health': [busy], 'GET /prompt_manager/system/health': [{ report: REPORT }] }); + assert.equal(await runHealthCheck(ended, { activity: fakeActivity({ reads: [[]] }) }), REPORT); +}); + +test('a failed, lost or reportless check rejects; other errors pass through', async () => { + const { runHealthCheck, waitForJob, HEALTH_FAILED_TEXT } = await overviewModule; + const failed = fakeApi({ 'POST /prompt_manager/system/health': [{ job: job('failed') }] }); + await assert.rejects(runHealthCheck(failed, { activity: fakeActivity() }), { message: HEALTH_FAILED_TEXT }); + // The store marks a job lost when the server no longer lists it (ComfyUI restarted). + const restarted = fakeActivity({ jobs: [job('running')] }); + const waiting = waitForJob(restarted, JOB); + restarted.push(job('lost')); + assert.equal(await waiting, 'lost'); + const empty = fakeApi({ 'POST /prompt_manager/system/health': [{ job: job('finished') }], + 'GET /prompt_manager/system/health': [{ report: null }] }); + await assert.rejects(runHealthCheck(empty, { activity: fakeActivity() }), { message: HEALTH_FAILED_TEXT }); + const broken = fakeApi({ 'POST /prompt_manager/system/health': [Object.assign(new Error('server'), { status: 500 })] }); + await assert.rejects(runHealthCheck(broken, { activity: fakeActivity() }), { message: 'server' }); + const invalid = fakeApi({ 'POST /prompt_manager/system/health': [{ job: null }] }); + await assert.rejects(runHealthCheck(invalid, { activity: fakeActivity() }), { message: 'Invalid job' }); +}); + +test('waiting stops when its signal aborts', async () => { + const { waitForJob } = await overviewModule; + const controller = new AbortController(); + const activity = fakeActivity({ jobs: [job('running')] }); + const waiting = waitForJob(activity, JOB, { signal: controller.signal }); + await new Promise(resolve => setImmediate(resolve)); + controller.abort(new Error('left')); + await assert.rejects(waiting, { message: 'left' }); + assert.equal(activity.listeners(), 0); + const gone = new AbortController(); + gone.abort(new Error('already')); + await assert.rejects(waitForJob(fakeActivity(), JOB, { signal: gone.signal }), { message: 'already' }); +}); diff --git a/tests/js/workspace-system-state.test.js b/tests/js/workspace-system-state.test.js new file mode 100644 index 0000000..a9e0911 --- /dev/null +++ b/tests/js/workspace-system-state.test.js @@ -0,0 +1,261 @@ +// System draft store (#249 System S2 Task 5): one Save row for browser, host and runtime fields (concept app.js +// renderSystem `.save-row` 705, `save-settings` 1320). It saves browser, then host with the GET revision, then runtime; +// a domain that fails keeps its fields dirty, and the status copy (D92) says what really happened. +const test = require('node:test'); +const assert = require('node:assert/strict'); + +const source = import('../../web/js/workspace/system-state.mjs'); + +/** Adapters that record every call; `host` answers like POST /prompt_manager/settings. */ +function adapters({ browser = {}, host = {}, runtime = {}, revision = 'sha256:one' } = {}) { + const calls = []; + const outcome = { browser: null, host: null, runtime: null }; + return { + calls, + outcome, + readBrowser: () => { calls.push(['readBrowser']); return { ...browser }; }, + readHost: async () => { calls.push(['readHost']); return { values: { ...host }, revision }; }, + readRuntime: async () => { calls.push(['readRuntime']); return { ...runtime }; }, + writeBrowser: async changes => { + calls.push(['writeBrowser', changes]); + if (outcome.browser) return outcome.browser(changes); + return { persisted: true }; + }, + writeHost: async (changes, options) => { + calls.push(['writeHost', changes, options]); + if (outcome.host) return outcome.host(changes, options); + return { success: true, persisted: true, revision: 'sha256:two', runtime_applied: true }; + }, + writeRuntime: async changes => { + calls.push(['writeRuntime', changes]); + if (outcome.runtime) return outcome.runtime(changes); + return { success: true }; + }, + }; +} + +async function loaded(domains) { + const { createSystemDrafts } = await source; + const drafts = createSystemDrafts(domains); + await Promise.all(['browser', 'host', 'runtime'].map(domain => drafts.load(domain))); + domains.calls.length = 0; + return drafts; +} + +test('sections follow the concept order and the production hashes', async () => { + const { SYSTEM_SECTIONS, isSystemSection } = await source; + assert.deepEqual(SYSTEM_SECTIONS.map(([id, label]) => [id, label]), [ + ['overview', 'Overview'], ['appearance', 'Appearance'], ['sources', 'Sources & scans'], + ['integrations', 'Integrations'], ['tagging', 'Auto-tagging'], ['maintenance', 'Maintenance'], + ['backups', 'Backup & export'], ['diagnostics', 'Diagnostics & logs'], + ]); + assert.equal(isSystemSection('backups'), true); + for (const value of ['general', '', 'constructor', '__proto__', undefined]) assert.equal(isSystemSection(value), false, value); +}); + +test('drafts read baselines and report dirty only for changed values', async () => { + const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 } }); + const drafts = await loaded(domains); + assert.equal(drafts.get('browser', 'layout'), 'grid'); + assert.equal(drafts.get('host', 'result_timeout'), 30); + assert.equal(drafts.dirty(), false); + drafts.set('host', 'result_timeout', 45); + assert.equal(drafts.get('host', 'result_timeout'), 45); + assert.equal(drafts.dirty(), true); + assert.equal(drafts.dirty('host'), true); + assert.equal(drafts.dirty('browser'), false); + // Setting the saved value back is clean again. + drafts.set('host', 'result_timeout', 30); + assert.equal(drafts.dirty(), false); + assert.throws(() => drafts.set('elsewhere', 'x', 1), TypeError); + assert.throws(() => drafts.get('elsewhere', 'x'), TypeError); +}); + +test('save commits browser then host then runtime', async () => { + const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 }, runtime: { level: 'INFO' } }); + const drafts = await loaded(domains); + drafts.set('runtime', 'level', 'DEBUG'); + drafts.set('host', 'result_timeout', 45); + drafts.set('browser', 'layout', 'list'); + const outcome = await drafts.save(); + assert.deepEqual(outcome, { browser: 'saved', host: 'saved', runtime: 'applied' }); + assert.deepEqual(domains.calls, [ + ['writeBrowser', { layout: 'list' }], + ['writeHost', { result_timeout: 45 }, { expected_revision: 'sha256:one' }], + ['writeRuntime', { level: 'DEBUG' }], + ]); + assert.equal(drafts.dirty(), false); + // The next host save sends the revision the last one returned. + drafts.set('host', 'result_timeout', 60); + await drafts.save(); + assert.deepEqual(domains.calls.at(-1), ['writeHost', { result_timeout: 60 }, { expected_revision: 'sha256:two' }]); +}); + +test('host failure keeps host fields dirty and browser saved', async () => { + const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 } }); + const drafts = await loaded(domains); + domains.outcome.host = () => { throw Object.assign(new Error('Settings could not be written to config.json.'), { code: 'config-write-failed' }); }; + drafts.set('browser', 'layout', 'list'); + drafts.set('host', 'result_timeout', 45); + const outcome = await drafts.save(); + assert.deepEqual(outcome, { browser: 'saved', host: 'failed', runtime: 'clean' }); + assert.equal(drafts.dirty('browser'), false); + assert.equal(drafts.dirty('host'), true); + assert.equal(drafts.get('host', 'result_timeout'), 45); + assert.deepEqual([...drafts.failed('host')], ['result_timeout']); + assert.deepEqual([...drafts.failed('browser')], []); + assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.'); + // A save answered `persisted: false` without an error is not saved either. + domains.outcome.host = () => ({ success: false, persisted: false }); + assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'failed', runtime: 'clean' }); + // Editing the failed field clears its highlight and reads as unsaved again. + drafts.set('host', 'result_timeout', 50); + assert.deepEqual([...drafts.failed('host')], []); + assert.equal(drafts.statusText(), 'Unsaved changes.'); +}); + +test('stale revision reports conflict and keeps the draft', async () => { + const domains = adapters({ host: { result_timeout: 30 } }); + const drafts = await loaded(domains); + domains.outcome.host = () => { throw Object.assign(new Error('Settings changed since they were loaded.'), { code: 'settings-changed' }); }; + drafts.set('host', 'result_timeout', 45); + assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'conflict', runtime: 'clean' }); + assert.equal(drafts.dirty('host'), true); + assert.equal(drafts.get('host', 'result_timeout'), 45); + assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.'); + // The conflict read the host values again (their new revision), keeping the draft. + assert.deepEqual(domains.calls.slice(-2).map(([name]) => name), ['writeHost', 'readHost']); + // Reloading the section reads the new revision and keeps the draft (Q10). + domains.outcome.host = null; + await drafts.load('host'); + assert.equal(drafts.get('host', 'result_timeout'), 45); + await drafts.save(); + assert.deepEqual(domains.calls.at(-1)[2], { expected_revision: 'sha256:one' }); +}); + +test('host save without a read revision never posts', async () => { + const { createSystemDrafts } = await source; + // Never read: an edit made before the host values (and their revision) arrived must not merge blindly. + const unread = adapters({ host: { result_timeout: 30 } }); + const drafts = createSystemDrafts(unread); + drafts.set('host', 'result_timeout', 45); + assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'failed', runtime: 'clean' }); + assert.deepEqual(unread.calls, []); + assert.equal(drafts.dirty('host'), true); + assert.deepEqual([...drafts.failed('host')], ['result_timeout']); + // A read of an unreadable configuration has no revision either. + const unreadable = adapters({ host: { result_timeout: 30 }, revision: null }); + const blind = createSystemDrafts(unreadable); + await blind.load('host'); + blind.set('host', 'result_timeout', 45); + assert.equal((await blind.save()).host, 'failed'); + assert.equal(unreadable.calls.filter(([name]) => name === 'writeHost').length, 0); + // Once read, the same draft saves with the revision. + await drafts.load('host'); + assert.equal((await drafts.save()).host, 'saved'); + assert.deepEqual(unread.calls.at(-1), ['writeHost', { result_timeout: 45 }, { expected_revision: 'sha256:one' }]); +}); + +test('reloadHost reads the host values and says whether it could', async () => { + const { createSystemDrafts } = await source; + const domains = adapters({ host: { result_timeout: 30 } }); + const drafts = createSystemDrafts(domains); + assert.equal(await drafts.reloadHost(), true); + assert.equal(drafts.get('host', 'result_timeout'), 30); + const broken = createSystemDrafts({ ...domains, readHost: async () => { throw new Error('offline'); } }); + assert.equal(await broken.reloadHost(), false); + broken.set('host', 'result_timeout', 45); + assert.equal((await broken.save()).host, 'failed'); +}); + +test('no domain called when clean', async () => { + const domains = adapters({ browser: { layout: 'grid' } }); + const drafts = await loaded(domains); + assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'clean', runtime: 'clean' }); + assert.deepEqual(domains.calls, []); + drafts.set('browser', 'layout', 'list'); + drafts.discard(); + assert.equal(drafts.dirty(), false); + assert.equal(drafts.get('browser', 'layout'), 'grid'); + assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'clean', runtime: 'clean' }); + assert.deepEqual(domains.calls, []); +}); + +test('status text per outcome', async () => { + const { statusText, SETTINGS_STATUS } = await source; + assert.deepEqual(SETTINGS_STATUS, { + clean: 'No unsaved changes.', + dirty: 'Unsaved changes.', + browser: 'Saved in this browser.', + host: 'Saved to the server configuration.', + runtime: 'Applied until the server restarts.', + partial: 'Some changes were not saved. Review the highlighted settings.', + tabOnly: 'Saved for this tab only; browser storage is unavailable.', + conflict: 'Settings changed elsewhere. Reload this section to see them.', + }); + const clean = { browser: 'clean', host: 'clean', runtime: 'clean' }; + assert.equal(statusText(clean), 'No unsaved changes.'); + assert.equal(statusText({ ...clean, browser: 'saved' }), 'Saved in this browser.'); + assert.equal(statusText({ ...clean, host: 'saved' }), 'Saved to the server configuration.'); + assert.equal(statusText({ ...clean, runtime: 'applied' }), 'Applied until the server restarts.'); + assert.equal(statusText({ ...clean, browser: 'tab-only' }), 'Saved for this tab only; browser storage is unavailable.'); + assert.equal(statusText({ browser: 'saved', host: 'saved', runtime: 'applied' }), + 'Saved in this browser. Saved to the server configuration. Applied until the server restarts.'); + assert.equal(statusText({ ...clean, browser: 'saved', runtime: 'failed' }), + 'Some changes were not saved. Review the highlighted settings.'); + assert.equal(statusText({ browser: 'failed', host: 'conflict', runtime: 'clean' }), + 'Settings changed elsewhere. Reload this section to see them.'); +}); + +test('the store says unsaved, then the save outcome, and notifies subscribers', async () => { + const domains = adapters({ browser: { layout: 'grid' } }); + const drafts = await loaded(domains); + const seen = []; + const unsubscribe = drafts.subscribe(() => seen.push(drafts.statusText())); + assert.equal(drafts.statusText(), 'No unsaved changes.'); + drafts.set('browser', 'layout', 'list'); + domains.outcome.browser = () => ({ persisted: false }); + await drafts.save(); + assert.deepEqual(seen, ['Unsaved changes.', 'Saved for this tab only; browser storage is unavailable.']); + unsubscribe(); + drafts.set('browser', 'layout', 'masonry'); + assert.equal(seen.length, 2); + // The tab-only value is this tab's baseline now. + drafts.set('browser', 'layout', 'list'); + assert.equal(drafts.dirty(), false); +}); + +test('an edit made while a save runs stays dirty', async () => { + const domains = adapters({ browser: { layout: 'grid' } }); + const drafts = await loaded(domains); + let release; + domains.outcome.browser = () => new Promise(resolve => { release = () => resolve({ persisted: true }); }); + drafts.set('browser', 'layout', 'list'); + const saving = drafts.save(); + // A second Save while one runs joins it instead of writing twice. + assert.equal(drafts.save(), saving); + assert.equal(drafts.saving(), true); + await new Promise(resolve => setImmediate(resolve)); + drafts.set('browser', 'layout', 'masonry'); + release(); + assert.deepEqual(await saving, { browser: 'saved', host: 'clean', runtime: 'clean' }); + assert.equal(drafts.saving(), false); + assert.equal(drafts.dirty('browser'), true); + assert.equal(drafts.get('browser', 'layout'), 'masonry'); + assert.equal(domains.calls.filter(([name]) => name === 'writeBrowser').length, 1); +}); + +test('a failed read keeps the store usable and drafts survive a reload (Q10)', async () => { + const { createSystemDrafts } = await source; + const domains = adapters({ browser: { layout: 'grid' } }); + const drafts = createSystemDrafts(domains); + await drafts.load('browser'); + drafts.set('browser', 'layout', 'list'); + await drafts.load('browser'); + assert.equal(drafts.get('browser', 'layout'), 'list'); + assert.equal(drafts.dirty(), true); + const failing = createSystemDrafts({ ...domains, readHost: async () => { throw new Error('offline'); } }); + await assert.rejects(failing.load('host'), /offline/); + assert.equal(failing.dirty(), false); + assert.throws(() => createSystemDrafts({ ...domains, writeHost: null }), TypeError); +}); diff --git a/tests/js/workspace-toolbar.test.js b/tests/js/workspace-toolbar.test.js index 91e1c3b..052bc0b 100644 --- a/tests/js/workspace-toolbar.test.js +++ b/tests/js/workspace-toolbar.test.js @@ -76,6 +76,7 @@ function fakeDOM() { get activeElement() { return active ?? body; }, createElement: tag => new Element(tag), createElementNS: (namespace, tag) => new Element(tag), + createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }), }; const all = (root, predicate, found = []) => { if (predicate(root)) found.push(root); @@ -790,3 +791,145 @@ test('the Filters button shows the active refinement count', async () => { dom.restore(); } }); + +// #249 System S2 Task 5: the scan review's dialog and backup box (D55) generalised for every record-changing review. +const scanModule = import('../../web/js/workspace/activity-scan.mjs'); +const bodyShape = dom => byId(dom, 'modal-body').children.map(node => [node.tagName.toLowerCase(), node.className].filter(Boolean).join('.')); + +test('backupNotice is the concept .notice with its copy and one Back up now action', async () => { + const { backupNotice, SCAN_BACKUP_COPY } = await dialogsModule; + const dom = fakeDOM(); + try { + let calls = 0; + const notice = backupNotice({ copy: SCAN_BACKUP_COPY, onBackup: () => { calls++; } }); + assert.equal(notice.className, 'notice'); + assert.deepEqual(tags(notice), ['div', 'div']); + assert.deepEqual(tags(notice.children[0]), ['h3', 'p']); + assert.equal(notice.children[0].children[0].textContent, 'Back up the database first (recommended)'); + assert.equal(notice.children[0].children[1].textContent, + 'Downloads a copy of the prompts database before the scan changes any records.'); + assert.equal(notice.children[1].className, 'row-actions'); + const action = button(notice, 'Back up now'); + action.click(); + assert.equal(calls, 1); + assert.throws(() => backupNotice({ copy: { heading: '', text: 'x' }, onBackup() {} }), TypeError); + assert.throws(() => backupNotice({ copy: SCAN_BACKUP_COPY }), TypeError); + } finally { + dom.restore(); + } +}); + +test('openReviewDialog lays out heading, grid, options, sentence, backup box and actions', async () => { + const { openReviewDialog, closeWorkspaceDialog } = await dialogsModule; + const { createScope } = await coreModule; + const dom = fakeDOM(); + const scope = createScope(); + try { + const trigger = new dom.Element('button'); + dom.body.append(trigger); + const option = new dom.Element('div'); + option.className = 'setting-row'; + const started = []; + const review = openReviewDialog({ scope, title: 'Review maintenance plan', heading: 'Missing-file links', + grid: [['Scope', '2 stale image links'], ['Media on disk', 'No source files removed']], options: [option], + sentence: 'Cancelling leaves all data untouched.', backup: { copy: { heading: 'Back up first', text: 'A copy.' }, onBackup() {} }, + cancelLabel: 'Cancel', primary: { label: 'Apply reviewed operation' }, onStart: () => started.push('start'), trigger }); + assert.equal(byId(dom, 'modal-title').textContent, 'Review maintenance plan'); + assert.deepEqual(bodyShape(dom), ['h3', 'dl.meta-grid', 'div.setting-row', 'p.small', 'div.notice', 'div.dialog-actions']); + const grid = byId(dom, 'modal-body').children[1]; + assert.deepEqual(grid.children.map(entry => [tags(entry), entry.children[0].textContent, entry.children[1].textContent]), [ + [['dt', 'dd'], 'Scope', '2 stale image links'], [['dt', 'dd'], 'Media on disk', 'No source files removed']]); + review.setValue('Scope', '3 stale image links'); + assert.equal(grid.children[0].children[1].textContent, '3 stale image links'); + const actions = byId(dom, 'modal-body').children.at(-1); + assert.deepEqual(actions.children.map(node => [node.textContent, node.className]), + [['Cancel', ''], ['Apply reviewed operation', 'ca-primary']]); + actions.children[1].click(); + assert.deepEqual(started, ['start']); + assert.equal(modal(dom).open, false); + assert.equal(document.activeElement, trigger); + + // A danger primary (the concept's Restore) has no sentence, heading, options or backup box when none is given. + openReviewDialog({ scope, title: 'Restore preview', grid: [['Integrity', 'Verified']], backup: null, + cancelLabel: 'Cancel', primary: { label: 'Restore', danger: true }, onStart() {} }); + assert.deepEqual(bodyShape(dom), ['dl.meta-grid', 'div.dialog-actions']); + assert.equal(button(byId(dom, 'modal-body'), 'Restore').className, 'danger'); + button(byId(dom, 'modal-body'), 'Cancel').click(); + assert.equal(modal(dom).open, false); + assert.throws(() => openReviewDialog({ scope, title: 'x', grid: [['a']], cancelLabel: 'Cancel', primary: { label: 'Go' }, + onStart() {} }), TypeError); + assert.throws(() => openReviewDialog({ scope, title: 'x', grid: [], cancelLabel: 'Cancel', primary: {}, onStart() {} }), TypeError); + } finally { + scope.dispose(); + closeWorkspaceDialog(); + dom.restore(); + } +}); + +test('review dialog backup box keeps the dialog open and toasts the outcome', async () => { + const { openReviewDialog, closeWorkspaceDialog, SCAN_BACKUP_COPY } = await dialogsModule; + const { downloadDatabaseBackup, BACKUP_DONE, BACKUP_FAILED } = await scanModule; + const { createScope } = await coreModule; + const dom = fakeDOM(); + const scope = createScope(); + const created = URL.createObjectURL; + const revoked = URL.revokeObjectURL; + URL.createObjectURL = () => 'blob:backup'; + URL.revokeObjectURL = () => {}; + try { + const toasts = []; + let answer; + const fetchUrl = () => new Promise(resolve => { answer = resolve; }); + openReviewDialog({ scope, title: 'Review metadata scan', grid: [['Sources', '2 output folders']], + backup: { copy: SCAN_BACKUP_COPY, onBackup: () => downloadDatabaseBackup({ fetchUrl, notify: text => toasts.push(text) }) }, + cancelLabel: 'Cancel', primary: { label: 'Start scan' }, onStart() { throw new Error('not started'); } }); + const action = button(byId(dom, 'modal-body'), 'Back up now'); + action.click(); + assert.equal(action.textContent, 'Backing up…'); + assert.equal(action.getAttribute('aria-disabled'), 'true'); + // A second press while the copy downloads is ignored. + action.click(); + answer({ ok: true, blob: async () => new Blob(['db']), headers: { get: () => 'attachment; filename="prompts_backup_1.db"' } }); + await tick(); + assert.deepEqual(toasts, [BACKUP_DONE]); + assert.equal(modal(dom).open, true); + assert.equal(action.textContent, 'Back up now'); + assert.equal(action.getAttribute('aria-disabled'), null); + action.click(); + answer({ ok: false, status: 500 }); + await tick(); + assert.deepEqual(toasts, [BACKUP_DONE, BACKUP_FAILED]); + assert.equal(modal(dom).open, true); + } finally { + URL.createObjectURL = created; + URL.revokeObjectURL = revoked; + scope.dispose(); + closeWorkspaceDialog(); + dom.restore(); + } +}); + +test('the scan review is a thin openReviewDialog caller with its scope switch', async () => { + const { openScanReviewDialog, closeWorkspaceDialog } = await dialogsModule; + const { createScope } = await coreModule; + const dom = fakeDOM(); + const scope = createScope(); + try { + const starts = []; + openScanReviewDialog({ scope, sources: 2, onBackup() {}, onStart: value => starts.push(value) }); + assert.equal(byId(dom, 'modal-title').textContent, 'Review metadata scan'); + assert.deepEqual(bodyShape(dom), ['dl.meta-grid', 'div.setting-row', 'p.small', 'div.notice', 'div.dialog-actions']); + const grid = byId(dom, 'modal-body').children[0]; + assert.deepEqual(grid.children.map(entry => entry.textContent), ['Sources2 output folders', 'ScopeNew and unresolved files']); + const allFiles = byId(dom, 'scan-all-files'); + allFiles.checked = true; + allFiles.dispatchEvent(new Event('change')); + assert.equal(grid.children[1].children[1].textContent, 'All files'); + button(byId(dom, 'modal-body'), 'Start scan').click(); + assert.deepEqual(starts, [{ scope: 'all' }]); + } finally { + scope.dispose(); + closeWorkspaceDialog(); + dom.restore(); + } +}); diff --git a/tests/test_database_health.py b/tests/test_database_health.py new file mode 100644 index 0000000..2c0b40f --- /dev/null +++ b/tests/test_database_health.py @@ -0,0 +1,614 @@ +"""The read-only health check (System › Overview and Maintenance). + +``run_health_checks`` reads one snapshot on its own ``mode=ro`` connection: +SQLite's integrity and foreign-key checks, stored prompt hashes against +``utils.hashing``, duplicate pair rows, the legacy consistency queries, image +links whose file is gone (only under readable sources) and the Needs-review +count when one is supplied. It never writes, and it never turns a check it +could not run into a count. +""" + +import hashlib +import ntpath +import os +import sqlite3 +import sys +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from database import health # noqa: E402 +from database.operations import PromptDatabase # noqa: E402 +from utils.hashing import generate_prompt_hash # noqa: E402 + +try: + from tests.open_handles import assert_closed # noqa: E402 +except ImportError: # discovered with tests/ as the top-level directory + from open_handles import assert_closed # noqa: E402 + +CHECK_IDS = [ + "integrity", + "foreign_keys", + "canonical_hashes", + "pair_uniqueness", + "missing_files", + "ambiguous_metadata", + "consistency", +] + + +def _digest(path): + if not os.path.exists(path): + return None + with open(path, "rb") as handle: + return hashlib.sha256(handle.read()).hexdigest() + + +class HealthTestCase(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.root = Path(os.path.realpath(self.temp.name)) + self.output = self.root / "output" + self.output.mkdir() + self.db_path = str(self.root / "prompts.db") + self.db = PromptDatabase(self.db_path) + + def tearDown(self): + self.db.close_all() + assert_closed(self, self.root) + self.temp.cleanup() + + def save(self, text, **metadata): + return self.db.save_prompt( + text, prompt_hash=generate_prompt_hash(text), **metadata + ) + + def image(self, prompt_id, name, *, exists=True, folder=None): + folder = Path(folder or self.output) + path = folder / name + if exists: + path.write_bytes(b"png") + return self.db.link_image_to_prompt(prompt_id, str(path)) + + def run_checks(self, roots=None, **kwargs): + roots = [str(self.output)] if roots is None else roots + return self.db.run_health_checks(roots, **kwargs) + + @staticmethod + def check(report, check_id): + return next(c for c in report["checks"] if c["id"] == check_id) + + def raw(self): + """A plain connection for planting damage the app itself never writes.""" + conn = sqlite3.connect(self.db_path) + conn.execute("PRAGMA foreign_keys=OFF") + return conn + + +class TestHealthReport(HealthTestCase): + def test_clean_library_passes_except_unavailable_review(self): + prompt = self.save("A clean prompt") + self.db.save_prompt_pair("Cat", "Blur") + self.image(prompt, "a.png") + report = self.run_checks(review_count=0) + self.assertTrue(report["success"]) + self.assertEqual([c["id"] for c in report["checks"]], CHECK_IDS) + self.assertEqual( + [c["label"] for c in report["checks"]], + [ + "Integrity check", + "Foreign keys & pair references", + "Canonical hashes", + "Pair uniqueness", + "Missing files", + "Ambiguous metadata", + "Consistency", + ], + ) + for check in report["checks"]: + self.assertEqual(check["status"], "passed", check) + self.assertEqual(check["count"], 0, check) + self.assertEqual(report["status"], "passed") + self.assertRegex(report["checked_at"], r"^\d{4}-\d\d-\d\dT.*\+00:00$") + + def test_health_check_never_writes(self): + prompt = self.save("Untouched") + self.image(prompt, "kept.png") + self.image(prompt, "gone.png", exists=False) + with self.raw() as conn: + conn.execute("UPDATE prompts SET hash='stale' WHERE id=?", (prompt,)) + conn.close() + # The app's own connection stays open, as in the running server. + self.db.get_statistics() + files = [self.db_path, self.db_path + "-wal"] + before = [_digest(path) for path in files] + shm_size = os.path.getsize(self.db_path + "-shm") + media = sorted(os.listdir(self.output)) + opened = [] + real_connect = sqlite3.connect + + class Recording(sqlite3.Connection): + changes_at_close = None + + def close(self): + Recording.changes_at_close = self.total_changes + super().close() + + def connect(*args, **kwargs): + conn = real_connect(*args, factory=Recording, **kwargs) + opened.append((args, kwargs, conn)) + return conn + + with patch.object(health.sqlite3, "connect", side_effect=connect): + report = self.run_checks() + self.assertEqual(report["status"], "incomplete") + self.assertEqual(len(opened), 1) + args, kwargs, conn = opened[0] + self.assertTrue(str(args[0]).endswith("?mode=ro"), args) + self.assertTrue(kwargs.get("uri")) + self.assertEqual(Recording.changes_at_close, 0) + with self.assertRaises(sqlite3.ProgrammingError): # closed in finally + conn.execute("SELECT 1") + self.assertEqual([_digest(path) for path in files], before) + # The -shm holds WAL read marks every reader updates; it never grows. + self.assertEqual(os.path.getsize(self.db_path + "-shm"), shm_size) + self.assertEqual(sorted(os.listdir(self.output)), media) + with self.db.model.get_connection() as live: + stored = live.execute( + "SELECT hash FROM prompts WHERE id=?", (prompt,) + ).fetchone()[0] + self.assertEqual(stored, "stale") + + def test_connection_closed_when_a_check_raises(self): + self.save("Any") + with ( + patch.object( + health, + "_canonical_hashes", + side_effect=sqlite3.OperationalError("boom"), + ), + self.assertLogs("prompt_manager.database.health", "ERROR"), + ): + report = self.run_checks(review_count=0) + canonical = self.check(report, "canonical_hashes") + self.assertEqual(canonical["status"], "failed") + self.assertIsNone(canonical["count"]) + self.assertEqual(canonical["detail"], ["The check could not finish."]) + self.assertEqual(self.check(report, "integrity")["status"], "passed") + self.assertEqual(report["status"], "incomplete") + self.db.close_all() + assert_closed(self, self.root) + + def test_unopenable_database_fails_every_database_check(self): + missing = PromptDatabase.__new__(PromptDatabase) + missing.model = type("M", (), {"db_path": str(self.root / "nope.db")})() + with self.assertLogs("prompt_manager.database.health", "ERROR"): + report = missing.run_health_checks([str(self.output)], review_count=0) + statuses = {c["id"]: c["status"] for c in report["checks"]} + self.assertEqual(statuses["integrity"], "failed") + self.assertEqual(statuses["missing_files"], "failed") + self.assertEqual(statuses["ambiguous_metadata"], "passed") + self.assertEqual(report["status"], "incomplete") + self.assertFalse(os.path.exists(self.root / "nope.db")) + + def test_integrity_and_foreign_key_issues_reported(self): + prompt = self.save("Linked") + with self.raw() as conn: + conn.execute("CREATE TABLE probe(a)") + conn.execute("CREATE INDEX probe_a ON probe(a)") + conn.executemany("INSERT INTO probe VALUES(?)", [(i,) for i in range(16)]) + conn.execute( + "INSERT INTO generated_images(prompt_id,image_path,filename,file_path)" + " VALUES(?,?,?,?)", + (9999, str(self.output / "orphan.png"), "orphan.png", "orphan"), + ) + conn.execute( + "INSERT INTO prompt_tags(prompt_id,tag_id) VALUES(?,?)", (prompt, 777) + ) + conn.close() + with self.raw() as conn: + conn.execute("PRAGMA writable_schema=ON") + conn.execute( + "UPDATE sqlite_master SET sql='CREATE INDEX probe_a ON probe(a DESC)'" + " WHERE name='probe_a'" + ) + conn.close() + self.db.close_all() # the next read sees the edited schema + (self.output / "orphan.png").write_bytes(b"png") + with self.raw() as conn: + expected = [r[0] for r in conn.execute("PRAGMA integrity_check(100)")] + conn.close() + self.assertGreater(len(expected), 10) + report = self.run_checks(review_count=0) + integrity = self.check(report, "integrity") + self.assertEqual(integrity["status"], "issues") + self.assertEqual(integrity["count"], len(expected)) + self.assertEqual(integrity["detail"], expected[:10]) + self.assertIn("missing from index probe_a", integrity["detail"][0]) + foreign = self.check(report, "foreign_keys") + self.assertEqual(foreign["status"], "issues") + self.assertEqual(foreign["count"], 2) + self.assertEqual( + sorted(foreign["detail"]), + [ + "generated_images row 1 references a missing prompts row", + "prompt_tags row 1 references a missing tags row", + ], + ) + consistency = self.check(report, "consistency") + self.assertEqual(consistency["status"], "issues") + self.assertEqual(consistency["count"], 1) + self.assertEqual( + consistency["detail"], ["Image 1 references non-existent prompt 9999"] + ) + self.assertEqual(report["status"], "issues") + + def test_canonical_hash_mismatch_counts_rows(self): + ids = [self.save(f"Prompt {n}") for n in range(14)] + with self.raw() as conn: + for n, prompt_id in enumerate(ids[:12]): + conn.execute( + "UPDATE prompts SET hash=? WHERE id=?", (f"bad{n}", prompt_id) + ) + conn.execute("UPDATE prompts SET hash=NULL WHERE id=?", (ids[12],)) + conn.close() + # Normalised spellings hash alike: this one is not a mismatch. + self.save(" MIXED Case ") + report = self.run_checks(review_count=0) + canonical = self.check(report, "canonical_hashes") + self.assertEqual(canonical["status"], "issues") + self.assertEqual(canonical["count"], 13) + self.assertEqual(len(canonical["detail"]), 10) + self.assertEqual( + canonical["detail"][0], + f"Prompt {ids[0]}: the stored hash does not match its text", + ) + + def test_duplicate_pair_rows_reported(self): + pair = self.db.save_prompt_pair("Cat", "") + positive = self.db.get_prompt_pair(pair)["positive_prompt_id"] + self.db.save_prompt_pair("Dog", "Blur") + with self.raw() as conn: + # Only the partial index keeps positive-only pairs unique; a restored + # or hand-edited database may have lost it. + conn.execute("DROP INDEX idx_prompt_pairs_positive_only") + conn.executemany( + "INSERT INTO prompt_pairs(positive_prompt_id,negative_prompt_id)" + " VALUES(?,NULL)", + [(positive,), (positive,)], + ) + conn.close() + report = self.run_checks(review_count=0) + pairs = self.check(report, "pair_uniqueness") + self.assertEqual(pairs["status"], "issues") + self.assertEqual(pairs["count"], 2) + self.assertEqual( + pairs["detail"], + [f"Positive prompt {positive} with no negative: 3 pair rows"], + ) + + def test_integrity_count_at_the_limit_is_flagged_capped(self): + class Rows: + def __init__(self, rows): + self.rows = rows + + def execute(self, _sql): + return iter([(row,) for row in self.rows]) + + capped = health._integrity(Rows(["bad"] * health.INTEGRITY_LIMIT), []) + self.assertEqual(capped["status"], "issues") + self.assertEqual(capped["count"], health.INTEGRITY_LIMIT) + self.assertTrue(capped["capped"]) + exact = health._integrity(Rows(["bad"] * 3), []) + self.assertEqual(exact["count"], 3) + self.assertFalse(exact["capped"]) + self.assertFalse(health._integrity(Rows(["ok"]), [])["capped"]) + + def test_an_unexpected_error_fails_only_its_own_check(self): + self.save("Any") + with ( + patch.object(health, "_pair_uniqueness", side_effect=KeyError("boom")), + self.assertLogs("prompt_manager.database.health", "ERROR"), + ): + report = self.run_checks(review_count=0) + statuses = {c["id"]: c["status"] for c in report["checks"]} + self.assertEqual(statuses.pop("pair_uniqueness"), "failed") + self.assertEqual(set(statuses.values()), {"passed"}) + + +class TestDatabaseUri(unittest.TestCase): + """The ``mode=ro`` URI of the database file, for every path form SQLite opens.""" + + def test_posix_path_is_a_local_file_uri(self): + self.assertEqual( + health.readonly_uri("/data/my db#1?.db", windows=False), + "file:///data/my%20db%231%3F.db?mode=ro", + ) + + def test_windows_drive_path(self): + self.assertEqual( + health.readonly_uri(r"C:\Users\Me\prompts.db", windows=True), + "file:///C:/Users/Me/prompts.db?mode=ro", + ) + + def test_windows_unc_path_has_an_empty_authority(self): + # pathlib spells \\server\share as file://server/share, which SQLite + # rejects (a non-local authority); an empty one keeps the UNC path. + uri = health.readonly_uri(r"\\server\share\Comfy UI\prompts.db", windows=True) + self.assertEqual(uri, "file:////server/share/Comfy%20UI/prompts.db?mode=ro") + self.assertTrue(ntpath.isabs(r"\\server\share\Comfy UI\prompts.db")) + + def test_uri_opens_a_path_with_uri_characters(self): + with tempfile.TemporaryDirectory() as temp: + # "?" is not a valid Windows file name character; "#", "%" and spaces are. + name = "a #b%20 c.db" if os.name == "nt" else "a #b?c%20.db" + path = os.path.join(os.path.realpath(temp), name) + conn = sqlite3.connect(path) + try: + conn.execute("CREATE TABLE t(a)") + conn.commit() + finally: + conn.close() + ro = sqlite3.connect(health.readonly_uri(path), uri=True) + try: + self.assertEqual(ro.execute("SELECT COUNT(*) FROM t").fetchone(), (0,)) + with self.assertRaises(sqlite3.OperationalError): + ro.execute("INSERT INTO t VALUES(1)") + finally: + ro.close() + + @unittest.skipUnless(os.name == "nt", "a native UNC path exists on Windows only") + def test_native_unc_path_opens(self): + with tempfile.TemporaryDirectory() as temp: + path = os.path.join(os.path.realpath(temp), "prompts.db") + conn = sqlite3.connect(path) + try: + conn.execute("CREATE TABLE t(a)") + conn.commit() + finally: + conn.close() + drive, rest = ntpath.splitdrive(path) + unc = "\\\\localhost\\" + drive[0] + "$" + rest + if not os.path.exists(unc): + self.skipTest("the administrative share is not reachable here") + ro = sqlite3.connect(health.readonly_uri(unc), uri=True) + try: + self.assertEqual(ro.execute("SELECT COUNT(*) FROM t").fetchone(), (0,)) + finally: + ro.close() + + +class TestMissingFiles(HealthTestCase): + def test_missing_files_counted_under_readable_sources_only(self): + prompt = self.save("Images") + self.image(prompt, "here.png") + for n in range(12): + self.image(prompt, f"gone-{n}.png", exists=False) + elsewhere = self.root / "elsewhere" + elsewhere.mkdir() + self.image(prompt, "outside.png", exists=False, folder=elsewhere) + report = self.run_checks(review_count=0) + missing = self.check(report, "missing_files") + self.assertEqual(missing["status"], "issues") + self.assertEqual(missing["count"], 12) + self.assertEqual(len(missing["detail"]), 10) + self.assertRegex(missing["detail"][0], r"^Image \d+: gone-0\.png is missing$") + self.assertNotIn(str(self.root), " ".join(missing["detail"])) + with self.db.model.get_connection() as conn: + self.assertEqual( + conn.execute("SELECT COUNT(*) FROM generated_images").fetchone()[0], + 14, + ) + + def test_links_outside_every_source_are_named_not_counted(self): + prompt = self.save("Images") + elsewhere = self.root / "elsewhere" + elsewhere.mkdir() + self.image(prompt, "outside.png", exists=False, folder=elsewhere) + missing = self.check(self.run_checks(review_count=0), "missing_files") + self.assertEqual(missing["status"], "passed") + self.assertEqual(missing["count"], 0) + self.assertEqual( + missing["detail"], + ["1 link outside the image sources was not checked."], + ) + + def test_missing_file_vs_unavailable_root(self): + # Per source (S2 ruling): the readable source is counted, the absent + # one reports "unavailable" with no count, and the report is incomplete. + prompt = self.save("Images") + absent = self.root / "unmounted" + absent.mkdir() + self.image(prompt, "on-nas.png", folder=absent) + self.image(prompt, "gone.png", exists=False) + os.unlink(absent / "on-nas.png") + absent.rmdir() + report = self.run_checks([str(self.output), str(absent)], review_count=0) + missing = self.check(report, "missing_files") + self.assertEqual(missing["status"], "issues") + self.assertEqual(missing["count"], 1) + self.assertTrue(missing["partial"]) + self.assertEqual( + missing["sources"], + [ + {"name": "output", "status": "checked", "count": 1}, + {"name": "unmounted", "status": "unavailable", "count": None}, + ], + ) + self.assertEqual( + missing["detail"][0], + "1 image source is unavailable, so its files were not counted.", + ) + self.assertRegex(missing["detail"][1], r"^Image \d+: gone\.png is missing$") + # The link under the unavailable source is neither counted nor "outside". + self.assertEqual(len(missing["detail"]), 2) + self.assertNotIn(str(self.root), " ".join(missing["detail"])) + self.assertEqual(report["status"], "incomplete") + + def test_partial_sources_with_no_missing_file_are_not_passed(self): + prompt = self.save("Images") + self.image(prompt, "here.png") + absent = self.root / "unmounted" + missing = self.check( + self.run_checks([str(self.output), str(absent)], review_count=0), + "missing_files", + ) + self.assertEqual(missing["status"], "unavailable") + self.assertEqual(missing["count"], 0) + self.assertTrue(missing["partial"]) + self.assertEqual( + [s["status"] for s in missing["sources"]], ["checked", "unavailable"] + ) + + def test_every_source_unavailable_counts_nothing(self): + prompt = self.save("Images") + absent = self.root / "unmounted" + self.image(prompt, "gone.png", exists=False, folder=self.output) + report = self.run_checks([str(absent)], review_count=0) + missing = self.check(report, "missing_files") + self.assertEqual(missing["status"], "unavailable") + self.assertIsNone(missing["count"]) + self.assertFalse(missing["partial"]) + self.assertEqual( + missing["sources"], + [{"name": "unmounted", "status": "unavailable", "count": None}], + ) + self.assertEqual( + missing["detail"], + ["1 image source is unavailable, so missing files were not counted."], + ) + + def test_each_source_counts_its_own_missing_files(self): + prompt = self.save("Images") + archive = self.root / "archive" + archive.mkdir() + self.image(prompt, "a.png", exists=False) + self.image(prompt, "b.png", exists=False, folder=archive) + self.image(prompt, "c.png", exists=False, folder=archive) + missing = self.check( + self.run_checks([str(self.output), str(archive)], review_count=0), + "missing_files", + ) + self.assertEqual(missing["count"], 3) + self.assertFalse(missing["partial"]) + self.assertEqual( + [(s["name"], s["count"]) for s in missing["sources"]], + [("output", 1), ("archive", 2)], + ) + + def test_no_sources_is_unavailable(self): + self.save("Images") + missing = self.check(self.run_checks([], review_count=0), "missing_files") + self.assertEqual(missing["status"], "unavailable") + self.assertIsNone(missing["count"]) + self.assertEqual(missing["detail"], ["No image source is available."]) + self.assertEqual(missing["sources"], []) + self.assertFalse(missing["partial"]) + + def test_a_bad_stored_path_fails_only_missing_files(self): + prompt = self.save("Images") + self.image(prompt, "here.png") + with self.raw() as conn: + conn.execute( + "INSERT INTO generated_images(prompt_id,image_path,filename)" + " VALUES(?,?,?)", + (prompt, str(self.root / "else\x00where.png"), "bad.png"), + ) + conn.close() + real_realpath = os.path.realpath + + def realpath(path, *args, **kwargs): + # POSIX (and Windows before Python 3.11) raise for the NUL byte; + # newer ntpath returns it unchanged. Raise everywhere, as POSIX does. + if "\x00" in str(path): + raise ValueError("embedded null byte") + return real_realpath(path, *args, **kwargs) + + with ( + patch.object(health.os.path, "realpath", side_effect=realpath), + self.assertLogs("prompt_manager.database.health", "ERROR"), + ): + report = self.run_checks(review_count=0) + statuses = {c["id"]: c["status"] for c in report["checks"]} + self.assertEqual(statuses.pop("missing_files"), "failed") + self.assertEqual(set(statuses.values()), {"passed"}) + + def test_files_are_checked_after_the_read_snapshot_ends(self): + prompt = self.save("Images") + self.image(prompt, "here.png") + self.image(prompt, "gone.png", exists=False) + opened = [] + real_connect = sqlite3.connect + + def connect(*args, **kwargs): + conn = real_connect(*args, **kwargs) + opened.append(conn) + return conn + + real_exists = os.path.exists + in_snapshot = [] + + def exists(path): + for conn in opened: + try: + in_snapshot.append(conn.in_transaction) + except sqlite3.ProgrammingError: # closed: no snapshot + in_snapshot.append(False) + return real_exists(path) + + with ( + patch.object(health.sqlite3, "connect", side_effect=connect), + patch.object(health.os.path, "exists", side_effect=exists), + ): + missing = self.check(self.run_checks(review_count=0), "missing_files") + self.assertEqual((missing["status"], missing["count"]), ("issues", 1)) + self.assertEqual(len(opened), 1) + self.assertTrue(in_snapshot) + self.assertNotIn(True, in_snapshot) + + +class TestAmbiguousMetadata(HealthTestCase): + def test_ambiguous_metadata_unavailable_without_review_queue(self): + self.save("Any") + report = self.run_checks() + ambiguous = self.check(report, "ambiguous_metadata") + self.assertEqual(ambiguous["status"], "unavailable") + self.assertIsNone(ambiguous["count"]) + self.assertEqual(ambiguous["detail"], ["Needs review is not available yet."]) + self.assertEqual(report["status"], "incomplete") + + def test_review_count_reports_images_needing_review(self): + report = self.run_checks(review_count=3) + ambiguous = self.check(report, "ambiguous_metadata") + self.assertEqual(ambiguous["status"], "issues") + self.assertEqual(ambiguous["count"], 3) + self.assertEqual(ambiguous["detail"], []) + self.assertEqual(report["status"], "issues") + + def test_invalid_review_count_is_rejected(self): + for value in (-1, True, "3", 1.5): + with self.assertRaises(ValueError): + self.run_checks(review_count=value) + + +class TestOverallStatus(unittest.TestCase): + def test_overall_status_precedence(self): + def checks(*statuses): + return [{"status": status} for status in statuses] + + self.assertEqual(health.overall_status(checks("passed", "passed")), "passed") + self.assertEqual(health.overall_status(checks("passed", "issues")), "issues") + self.assertEqual( + health.overall_status(checks("issues", "unavailable")), "incomplete" + ) + self.assertEqual(health.overall_status(checks("failed")), "incomplete") + # A check that found issues under only some sources is incomplete too. + partial = [{"status": "issues", "partial": True}, {"status": "passed"}] + self.assertEqual(health.overall_status(partial), "incomplete") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_library.py b/tests/test_library.py index fd83cb7..242b76d 100644 --- a/tests/test_library.py +++ b/tests/test_library.py @@ -1706,6 +1706,103 @@ class TestLibrary(unittest.TestCase): self.db.list_library_images(positive, pair_id=foreign, archive="all") +class LibraryCounts(unittest.TestCase): + """System › Overview counts and usage statistics follow Library eligibility.""" + + setUp = TestLibrary.setUp + tearDown = TestLibrary.tearDown + save = TestLibrary.save + state = TestLibrary.state + + def test_counts_use_library_eligibility(self): + self.db.save_prompt_pair("Cat", "Blur") + archived = self.db.save_prompt_pair("Cat", "Noise") + self.db.archive_prompt_pair(archived) + titled_pair = self.db.save_prompt_pair("Dog", "Titled negative") + titled = self.db.get_prompt_pair(titled_pair)["negative_prompt_id"] + self.state(titled, "A titled parent") # zero pairs of its own: counted + legacy = self.save("Legacy") + self.db.link_image_to_prompt(legacy, str(Path(self.temp.name) / "a.png")) + self.db.link_image_to_prompt(legacy, str(Path(self.temp.name) / "b.png")) + counts = self.db.library_counts() + # Cat, Dog, the titled negative and Legacy; "Blur" and "Noise" are + # negative-only. + self.assertEqual(counts["positive_prompts"], 4) + # Cat+Blur and Dog+Titled negative; the archived pair is excluded. + self.assertEqual(counts["negative_variants"], 2) + self.assertEqual(counts["images"], 2) + self.assertEqual(counts["rated_images"], 0) + listed = self.db.list_library() + self.assertEqual(counts["positive_prompts"], listed["total"]) + self.assertEqual(counts["negative_variants"], listed["variant_total"]) + + def test_counts_of_an_empty_library_are_zero(self): + self.assertEqual( + self.db.library_counts(), + { + "positive_prompts": 0, + "negative_variants": 0, + "images": 0, + "rated_images": 0, + }, + ) + + def test_rated_images_follow_their_linked_prompt_rating(self): + # An image carries no rating of its own: it shows its linked prompt's + # (the Images "Highest rated" sort), so "rated" counts those images. + folder = Path(self.temp.name) + rated = self.save("Rated", rating=4) + unrated = self.save("Unrated") + for name in ("a.png", "b.png", "c.png"): + self.db.link_image_to_prompt(rated, str(folder / name)) + self.db.link_image_to_prompt(unrated, str(folder / "d.png")) + counts = self.db.library_counts() + self.assertEqual(counts["images"], 4) + self.assertEqual(counts["rated_images"], 3) + self.db.update_prompt_rating(unrated, 2) + self.assertEqual(self.db.library_counts()["rated_images"], 4) + + def test_statistics_sum_real_run_counts(self): + a = self.save("Alpha", category="Landscapes", rating=5) + b = self.save("Beta", category="Landscapes", rating=4) + self.save("Gamma", category="Landscapes") # unrated: not in the average + c = self.save("Delta", category="Portraits", rating=3) + blank = self.save("Epsilon", category=" ") + uncategorised = self.save("Zeta") + self.db.record_prompt_use(a, times=3) + self.db.record_prompt_use(b, times=2) + self.db.record_prompt_use(c) + self.db.record_prompt_use(blank) + self.db.record_prompt_use(uncategorised, times=4) + # A negative-only prompt's runs are the pair's runs: never counted twice. + pair = self.db.save_prompt_pair("Alpha", "Only negative") + negative = self.db.get_prompt_pair(pair)["negative_prompt_id"] + with self.db.model.get_connection() as connection: + connection.execute( + "UPDATE prompts SET category='Landscapes',run_count=50 WHERE id=?", + (negative,), + ) + rows = self.db.usage_by_category() + self.assertEqual( + rows, + [ + {"category": "Landscapes", "runs": 5, "average_rating": 4.5}, + {"category": None, "runs": 5, "average_rating": None}, + {"category": "Portraits", "runs": 1, "average_rating": 3.0}, + ], + ) + + def test_counts_and_usage_write_nothing(self): + self.save("Alpha", category="A") + with self.db.model.get_connection() as connection: + before = connection.total_changes + self.db.library_counts() + self.db.usage_by_category() + with self.db.model.get_connection() as connection: + self.assertEqual(connection.total_changes, before) + self.assertFalse(connection.in_transaction) + + class DisplayTitleUnits(unittest.TestCase): """The first-line title is cut like the Library's JavaScript title().""" diff --git a/tests/test_system_api.py b/tests/test_system_api.py new file mode 100644 index 0000000..f72e8d4 --- /dev/null +++ b/tests/test_system_api.py @@ -0,0 +1,422 @@ +"""System routes (py/api/system.py): health check job, Overview and statistics. + +The health check runs off-loop as a ``read`` registry job of kind ``health`` +(one at a time, beside a scan), and the API keeps its last report. The +Overview reads only real values: Library-eligible counts, indexed images, the +last health result or null, no backup until one exists, and the effective +sources and whether they were auto-detected. The missing-thumbnail count is not +part of it: the Overview reads ``/images/thumbnails/status`` on its own (its +walk of every source must not hold the other values back). +""" + +import asyncio +import os +import sqlite3 +import sys +import threading +from pathlib import Path +from unittest.mock import MagicMock, patch + +_mock_server = MagicMock() +_mock_server.PromptServer.instance.routes = MagicMock() +sys.modules.setdefault("server", _mock_server) + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from py.api.jobs import HEALTH_JOB_KIND # noqa: E402 +from py.config import GalleryConfig # noqa: E402 +from utils.hashing import generate_prompt_hash # noqa: E402 + +try: + from tests.test_api_admin import AdminAPITestCase # noqa: E402 +except ImportError: # discovered with tests/ as the top-level directory + from test_api_admin import AdminAPITestCase # noqa: E402 + +WAIT = 5 + + +class SystemAPITestCase(AdminAPITestCase): + async def get_application(self): + app = await super().get_application() + # Auto-detection searches the disk; here it finds the fake ComfyUI only. + self.api._find_comfyui_output_dir = lambda: ( + str(self.output_dir) if self.output_dir.is_dir() else None + ) + return app + + def save(self, text, **metadata): + return self.api.db.save_prompt( + text, prompt_hash=generate_prompt_hash(text), **metadata + ) + + def image(self, prompt_id, name, *, exists=True, folder=None): + path = Path(folder or self.output_dir) / name + if exists: + path.write_bytes(b"png") + return self.api.db.link_image_to_prompt(prompt_id, str(path)) + + async def json(self, method, url, status=200): + response = await self.client.request(method, url) + self.assertEqual(response.status, status, await response.text()) + return await response.json() + + async def restore_snapshot(self): + """Restore the live database from a copy of itself through the legacy route.""" + from contextlib import closing + + from aiohttp import FormData + + snapshot = os.path.join(self.tmpdir, "snapshot.db") + with closing(sqlite3.connect(self.api.db.model.db_path)) as src: + with closing(sqlite3.connect(snapshot)) as dst: + src.backup(dst) + with open(snapshot, "rb") as handle: + upload = handle.read() + form = FormData() + form.add_field("database_file", upload, filename="upload.db") + response = await self.client.request( + "POST", "/prompt_manager/restore", data=form + ) + self.assertEqual(response.status, 200, await response.text()) + + async def run_health(self): + body = await self.json("POST", "/prompt_manager/system/health", 202) + job = self.api._jobs.get(body["job"]["id"]) + await asyncio.wait_for(asyncio.shield(job.task), WAIT) + return body["job"], job + + +class TestHealthRoute(SystemAPITestCase): + async def test_no_report_before_a_check(self): + body = await self.json("GET", "/prompt_manager/system/health") + self.assertEqual(body, {"success": True, "report": None}) + + async def test_health_check_runs_as_a_read_job_and_keeps_its_report(self): + prompt = self.save("Images") + self.image(prompt, "kept.png") + self.image(prompt, "gone.png", exists=False) + summary, job = await self.run_health() + self.assertEqual(summary["kind"], HEALTH_JOB_KIND) + self.assertEqual(summary["title"], "Health check") + self.assertEqual(summary["admission"], "read") + self.assertFalse(summary["cancellable"]) + self.assertEqual(summary["counters"], {}) + self.assertEqual(job.state, "finished") + self.assertEqual(job.progress, 100) + self.assertEqual(job.status_line, "Health check incomplete") + report = (await self.json("GET", "/prompt_manager/system/health"))["report"] + self.assertEqual(report["status"], "incomplete") + missing = next(c for c in report["checks"] if c["id"] == "missing_files") + self.assertEqual((missing["status"], missing["count"]), ("issues", 1)) + ambiguous = next(c for c in report["checks"] if c["id"] == "ambiguous_metadata") + self.assertEqual(ambiguous["status"], "unavailable") + self.assertNotIn(self.tmpdir, str(report)) + + async def test_health_check_is_single_flight_and_runs_beside_a_scan(self): + release = threading.Event() + entered = threading.Event() + real = self.api.db.run_health_checks + + def gated(*args, **kwargs): + entered.set() + release.wait(WAIT) + return real(*args, **kwargs) + + scan_gate = asyncio.Event() + + async def scanning(job): + await scan_gate.wait() + + scan = self.api._jobs.start( + "scan", "Metadata scan", admission="records", runner=scanning + ) + with patch.object(self.api.db, "run_health_checks", side_effect=gated): + first = await self.json("POST", "/prompt_manager/system/health", 202) + loop = asyncio.get_running_loop() + await loop.run_in_executor(None, entered.wait, WAIT) + busy = await self.json("POST", "/prompt_manager/system/health", 409) + self.assertEqual(busy["code"], "busy") + self.assertEqual( + busy["running"], {"id": first["job"]["id"], "title": "Health check"} + ) + release.set() + await asyncio.wait_for(self.api._jobs.get(first["job"]["id"]).task, WAIT) + scan_gate.set() + await asyncio.wait_for(scan.task, WAIT) + + async def test_failed_check_keeps_the_previous_report(self): + await self.run_health() + before = (await self.json("GET", "/prompt_manager/system/health"))["report"] + with ( + patch.object( + self.api.db, "run_health_checks", side_effect=RuntimeError("boom") + ), + self.assertLogs("prompt_manager.api", "ERROR"), + ): + body = await self.json("POST", "/prompt_manager/system/health", 202) + job = self.api._jobs.get(body["job"]["id"]) + await asyncio.wait_for(asyncio.shield(job.task), WAIT) + self.assertEqual(job.state, "failed") + after = (await self.json("GET", "/prompt_manager/system/health"))["report"] + self.assertEqual(after, before) + + async def test_absent_configured_source_is_reported_per_source(self): + prompt = self.save("Images") + self.image(prompt, "gone.png", exists=False) + absent = self.comfy_dir / "output" / "unmounted" + GalleryConfig.MONITORING_DIRECTORIES = [ + self._root(self.output_dir), + str(absent), + ] + await self.run_health() + report = (await self.json("GET", "/prompt_manager/system/health"))["report"] + missing = next(c for c in report["checks"] if c["id"] == "missing_files") + # Per source: the readable output is counted, the absent one is not. + self.assertEqual(missing["status"], "issues") + self.assertEqual(missing["count"], 1) + self.assertTrue(missing["partial"]) + self.assertEqual( + [(s["status"], s["count"]) for s in missing["sources"]], + [("checked", 1), ("unavailable", None)], + ) + self.assertEqual(report["status"], "incomplete") + + async def test_restore_clears_health_report(self): + self.save("Any") + await self.run_health() + self.assertIsNotNone( + (await self.json("GET", "/prompt_manager/system/health"))["report"] + ) + await self.restore_snapshot() + body = await self.json("GET", "/prompt_manager/system/health") + self.assertEqual(body, {"success": True, "report": None}) + overview = await self.json("GET", "/prompt_manager/system/overview") + self.assertIsNone(overview["health"]) + + async def test_check_finishing_after_a_restore_is_not_kept(self): + # A check that started on the old database describes it, not the new one. + self.save("Any") + release = threading.Event() + entered = threading.Event() + real = self.api.db.run_health_checks + + def gated(*args, **kwargs): + entered.set() + release.wait(WAIT) + return real(*args, **kwargs) + + with patch.object(self.api.db, "run_health_checks", side_effect=gated): + body = await self.json("POST", "/prompt_manager/system/health", 202) + loop = asyncio.get_running_loop() + await loop.run_in_executor(None, entered.wait, WAIT) + await self.restore_snapshot() + release.set() + job = self.api._jobs.get(body["job"]["id"]) + await asyncio.wait_for(asyncio.shield(job.task), WAIT) + self.assertEqual(job.state, "finished") + body = await self.json("GET", "/prompt_manager/system/health") + self.assertEqual(body, {"success": True, "report": None}) + # A check started on the restored database is kept again. + await self.run_health() + self.assertIsNotNone( + (await self.json("GET", "/prompt_manager/system/health"))["report"] + ) + + async def test_health_sources_resolve_off_the_event_loop(self): + loop_thread = threading.get_ident() + threads = [] + real = self.api._health_roots + + def recording(): + threads.append(threading.get_ident()) + return real() + + with patch.object(self.api, "_health_roots", side_effect=recording): + await self.run_health() + self.assertEqual(len(threads), 1) + self.assertNotEqual(threads[0], loop_thread) + + async def test_health_start_failure_is_an_error_envelope(self): + with ( + patch.object(self.api, "_health_roots", side_effect=RuntimeError("secret")), + self.assertLogs("prompt_manager.api", "ERROR"), + ): + body = await self.json("POST", "/prompt_manager/system/health", 500) + self.assertEqual( + body, + { + "success": False, + "code": None, + "error": "Failed to start the health check", + }, + ) + self.assertEqual(self.api._jobs.list(), []) + + +class TestOverviewRoute(SystemAPITestCase): + async def test_overview_reports_real_counts(self): + self.api.db.save_prompt_pair("Cat", "Blur") + archived = self.api.db.save_prompt_pair("Cat", "Noise") + self.api.db.archive_prompt_pair(archived) + legacy = self.save("Legacy") + self.image(legacy, "a.png") + self.image(legacy, "b.png") + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertTrue(body["success"]) + self.assertEqual( + body["library"], {"positive_prompts": 2, "negative_variants": 1} + ) + self.assertEqual(body["images"], {"indexed": 2, "rated": 0}) + self.assertEqual( + set(body), + { + "success", + "library", + "images", + "health", + "backup", + "needs_review", + "sources", + }, + ) + + async def test_overview_rated_counts_images_of_rated_prompts(self): + rated = self.save("Rated", rating=5) + self.image(rated, "a.png") + self.image(rated, "b.png") + self.image(self.save("Unrated"), "c.png") + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertEqual(body["images"], {"indexed": 3, "rated": 2}) + + async def test_overview_never_fabricates_backup_or_health(self): + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertIsNone(body["health"]) + self.assertIsNone(body["backup"]) + self.assertEqual(body["needs_review"], {"pending": None}) + self.save("Any") + await self.run_health() + report = (await self.json("GET", "/prompt_manager/system/health"))["report"] + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertEqual( + body["health"], + {"checked_at": report["checked_at"], "status": "incomplete", "issues": 0}, + ) + self.assertIsNone(body["backup"]) + + async def test_overview_health_counts_checks_with_issues(self): + prompt = self.save("Images") + self.image(prompt, "gone.png", exists=False) + self.image(prompt, "gone-too.png", exists=False) + await self.run_health() + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertEqual(body["health"]["issues"], 1) + + async def test_overview_sources_distinguish_auto_detect(self): + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertEqual( + body["sources"], {"configured": 0, "effective": 1, "auto_detected": True} + ) + second = self.output_dir / "archive" + second.mkdir() + GalleryConfig.MONITORING_DIRECTORIES = [ + self._root(self.output_dir), + self._root(second), + ] + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertEqual( + body["sources"], {"configured": 2, "effective": 2, "auto_detected": False} + ) + # Every configured source unavailable: ComfyUI's output is used instead. + GalleryConfig.MONITORING_DIRECTORIES = [str(self.comfy_dir / "absent")] + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertEqual( + body["sources"], {"configured": 1, "effective": 1, "auto_detected": True} + ) + + async def test_overview_never_walks_the_sources_for_thumbnails(self): + (self.output_dir / "one.png").write_bytes(b"png") + with patch.object( + self.api, + "_plan_gallery_thumbnails", + side_effect=AssertionError("walked"), + ): + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertTrue(body["success"]) + self.assertNotIn("thumbnails", body) + + async def test_overview_sources_resolve_off_the_event_loop(self): + loop_thread = threading.get_ident() + threads = [] + real = self.api._output_dirs_with_origin + + def recording(): + threads.append(threading.get_ident()) + return real() + + with patch.object(self.api, "_output_dirs_with_origin", side_effect=recording): + body = await self.json("GET", "/prompt_manager/system/overview") + self.assertEqual(body["sources"]["effective"], 1) + self.assertEqual(len(threads), 1) + self.assertNotEqual(threads[0], loop_thread) + + async def test_overview_failure_is_an_error_envelope(self): + with ( + patch.object( + self.api.db, "library_counts", side_effect=RuntimeError("secret") + ), + self.assertLogs("prompt_manager.api", "ERROR"), + ): + body = await self.json("GET", "/prompt_manager/system/overview", 500) + self.assertEqual( + body, + {"success": False, "code": None, "error": "Failed to read the overview"}, + ) + + +class TestStatisticsRoute(SystemAPITestCase): + async def test_statistics_sum_real_run_counts(self): + a = self.save("Alpha", category="Landscapes", rating=5) + b = self.save("Beta", category="Portraits", rating=2) + self.api.db.record_prompt_use(a, times=3) + self.api.db.record_prompt_use(b) + body = await self.json("GET", "/prompt_manager/system/statistics") + self.assertEqual( + body["by_category"], + [ + {"category": "Landscapes", "runs": 3, "average_rating": 5.0}, + {"category": "Portraits", "runs": 1, "average_rating": 2.0}, + ], + ) + self.assertEqual(body["totals"], self.api.db.get_statistics()) + self.assertTrue(body["success"]) + + async def test_statistics_lists_rated_images_for_library_totals(self): + # D96: the Statistics dialog's "Library totals" reads the rated count + # with the runs (one read), from the same images the Overview counts. + rated = self.save("Rated", rating=4) + self.image(rated, "a.png") + self.image(self.save("Unrated"), "b.png") + body = await self.json("GET", "/prompt_manager/system/statistics") + self.assertEqual(body["rated_images"], 1) + self.assertEqual( + set(body), {"success", "by_category", "totals", "rated_images"} + ) + + async def test_statistics_failure_is_an_error_envelope(self): + with ( + patch.object( + self.api.db, "usage_by_category", side_effect=RuntimeError("secret") + ), + self.assertLogs("prompt_manager.api", "ERROR"), + ): + body = await self.json("GET", "/prompt_manager/system/statistics", 500) + self.assertEqual( + body, + {"success": False, "code": None, "error": "Failed to read statistics"}, + ) + + +if __name__ == "__main__": + import unittest + + unittest.main() diff --git a/tools/vendor/frontend-coverage-baseline.json b/tools/vendor/frontend-coverage-baseline.json index 7d82941..88dfa16 100644 --- a/tools/vendor/frontend-coverage-baseline.json +++ b/tools/vendor/frontend-coverage-baseline.json @@ -443,6 +443,30 @@ "functions": 98, "statements": 98 }, + "workspace/system-controls.mjs": { + "lines": 98, + "branches": 87, + "functions": 89, + "statements": 98 + }, + "workspace/system-overview.mjs": { + "lines": 96, + "branches": 92, + "functions": 98, + "statements": 96 + }, + "workspace/system-state.mjs": { + "lines": 98, + "branches": 92, + "functions": 92, + "statements": 98 + }, + "workspace/system.mjs": { + "lines": 93, + "branches": 85, + "functions": 69, + "statements": 93 + }, "workspace/theme.mjs": { "lines": 98, "branches": 93, diff --git a/web/js/workspace/activity-store.mjs b/web/js/workspace/activity-store.mjs index 72eb6ab..0729d44 100644 --- a/web/js/workspace/activity-store.mjs +++ b/web/js/workspace/activity-store.mjs @@ -336,6 +336,16 @@ export function createActivityStore({ api, scope, connect = defaultConnect, noti return { busy: store.running()[0] ?? null, message: error.message }; } }, + /** + * Follow a job another route started (its summary from that route's answer, e.g. the health check's + * `POST /system/health`): it is listed and its stream opened at once, with no read and no start toast. + */ + track(raw) { + const job = parseJob(raw); + if (!job) throw new TypeError('Invalid job'); + upsert(job); + return job; + }, /** Ask a running job to stop at its next file boundary; false once it has ended. */ async cancel(id) { const result = await api.post(`${JOBS_URL}/${encodeURIComponent(id)}/cancel`, {}, { signal: scope.signal }); diff --git a/web/js/workspace/index.mjs b/web/js/workspace/index.mjs index 70adf1a..138a6ef 100644 --- a/web/js/workspace/index.mjs +++ b/web/js/workspace/index.mjs @@ -1,6 +1,6 @@ import '../api-client.js'; import { createButton, createScope } from '../shared/toolkit/core.js'; -import { routes, mountRouter, summarizeStats, routeTitle, isHandoff, loadNavCounts, +import { routes, mountRouter, summarizeStats, routeTitle, loadNavCounts, LIBRARY_WRITE_COUNTS, NAV_COUNT_ROUTES, FAVORITE_WRITE_COUNTS, FAVORITE_COUNT_DELAY_MS } from './state.mjs'; import { mountWorkspaceTheme } from './theme.mjs'; import { mountImages, onFavoriteChange } from './images.mjs'; @@ -16,6 +16,7 @@ import { mountActivityPage, mountActivityTopbar } from './activity.mjs'; import { mountActivityJobView } from './activity-job-view.mjs'; import { mountInspect } from './inspect.mjs'; import { createPromptSender } from './prompt-sender.mjs'; +import { mountSystem } from './system.mjs'; const { createApiClient, resolveUrl } = window.ApiClient; @@ -28,7 +29,8 @@ function element(tag, text, className) { function legacyLink(route) { const link = element('a', route.action, 'workspace-link'); - link.href = resolveUrl(route.href); + // A native card (System, Q12) links its workspace hash; a legacy tool's page goes through the mount prefix. + link.href = route.native ? route.href : resolveUrl(route.href); return link; } @@ -137,6 +139,9 @@ function mountWorkspace() { let leaveGuard = null, renderToken = 0; scope.cleanup(() => viewScope?.dispose()); + // The mounted System page: another of its sections is shown in place, keeping the shell and the drafts. + let system = null; + function render(route) { const token = ++renderToken; const hold = leaveGuard?.(); @@ -151,6 +156,11 @@ function mountWorkspace() { leaveGuard = null; // A shared page dialog never stays open over the next route. closeWorkspaceDialog(); + if (route.id === 'system' && system && currentKey === 'system') { + system.show(route.section); + return; + } + system = null; viewScope?.dispose(); images = null; viewScope = createScope(); @@ -251,21 +261,18 @@ function mountWorkspace() { else mountActivityPage(main, { store: activity, scope: view }); return; } + if (route.id === 'system') { + // The concept's System (app.js renderSystem): its heading description, section nav and Save row. + description.textContent = route.description; + system = mountSystem(main, api, view, { section: route.section, activity }); + return; + } if (route.id === 'inspect') { // The concept's Inspect a file (app.js renderMetadata): a local file, read only in this browser. description.textContent = route.description; ({ leaveGuard } = mountInspect(main, { parentScope: view, api, sender: createPromptSender({ scope: view }) })); return; } - if (isHandoff(route)) { - description.textContent = route.description; - const panel = element('section', undefined, 'surface handoff'); - panel.append(element('h2', 'Your current tools are ready'), - element('p', 'This workspace is being introduced in stages. Continue in the existing tool with your saved data.'), - legacyLink(route)); - main.append(panel); - return; - } description.textContent = 'A home for your prompts, images and creative history.'; const stats = element('section', undefined, 'overview-stats'); stats.setAttribute('aria-label', 'Library statistics'); diff --git a/web/js/workspace/state.mjs b/web/js/workspace/state.mjs index 7957f4d..3255381 100644 --- a/web/js/workspace/state.mjs +++ b/web/js/workspace/state.mjs @@ -14,11 +14,29 @@ export const routes = [ { id: 'favorites', label: 'Favorites' }, // The concept's Activity (app.js renderActivity 732–738): this server session's jobs. No href: no Overview card. { id: 'activity', label: 'Activity' }, - { id: 'system', label: 'System', href: '/prompt_manager/admin', - description: 'Open the current dashboard, then choose Settings or Maintenance for configuration, scans, backups and diagnostics.', action: 'Open existing administration' }, + // Native System (#249 System S2): `#/system/
`. Q12: Overview's card opens its Overview section (`native`: + // a workspace hash, not a legacy page); each undelivered section hands off to its legacy tool (D94). The + // description is the concept's System heading (app.js renderSystem 700). + { id: 'system', label: 'System', href: '#/system/overview', native: true, + description: 'Configuration, maintenance and recovery—with browsing kept in the workspace.', action: 'Open System' }, ]; -const NATIVE_ROUTES = new Set(['images', 'library', 'tags', 'inspect', 'overview', 'saved', 'favorites', 'activity']); +const NATIVE_ROUTES = new Set(['images', 'library', 'tags', 'inspect', 'overview', 'saved', 'favorites', 'activity', 'system']); + +/** + * System's sections in the concept's order (app.js `systemSections` 635–644); the concept's `general` is production's + * `overview`, so `#/system/overview` sits beside the production hub `#/overview` (D28). + */ +export const SYSTEM_SECTIONS = Object.freeze([ + ['overview', 'Overview'], ['appearance', 'Appearance'], ['sources', 'Sources & scans'], ['integrations', 'Integrations'], + ['tagging', 'Auto-tagging'], ['maintenance', 'Maintenance'], ['backups', 'Backup & export'], ['diagnostics', 'Diagnostics & logs'], +].map(section => Object.freeze(section))); +const SYSTEM_SECTION_IDS = new Set(SYSTEM_SECTIONS.map(([id]) => id)); + +/** Whether `id` names a System section. */ +export function isSystemSection(id) { + return typeof id === 'string' && SYSTEM_SECTION_IDS.has(id); +} /** A saved view (#249 D16): the shell resolves its id to the view's page, name and state. */ const SAVED_ROUTE = Object.freeze({ id: 'saved', label: 'Saved view' }); @@ -39,6 +57,14 @@ export function resolveRoute(hash) { // One job of the Activity page (its id is the registry's uuid4 hex). const job = /^#\/activity\/([0-9a-f]{32})$/.exec(hash); if (job) return {...routes.find(route => route.id === 'activity'), jobId:job[1], hash}; + // `#/system` is the Overview section under the sidebar's own hash; an unknown child is Overview, its hash replaced. + const system = /^#\/system(?:\/(.*))?$/.exec(hash); + if (system) { + const route = routes.find(item => item.id === 'system'); + if (system[1] === undefined) return {...route, section:'overview', hash}; + const section = isSystemSection(system[1]) ? system[1] : 'overview'; + return {...route, section, hash:`#/system/${section}`}; + } const match = /^#\/library\/([1-9]\d*)$/.exec(hash); if (match && Number.isSafeInteger(Number(match[1]))) { return {...routes.find(route => route.id === 'library'), positiveId:Number(match[1]), hash}; @@ -117,7 +143,7 @@ export function mountRouter(win, onRoute) { const hash = route.hash || `#/${route.id}`; if (win.location.hash !== hash) win.history.replaceState(win.history.state, '', hash); if (current?.id === route.id && current?.positiveId === route.positiveId && current?.savedId === route.savedId && - current?.jobId === route.jobId) return; + current?.jobId === route.jobId && current?.section === route.section) return; current = route; onRoute(route); }; diff --git a/web/js/workspace/system-controls.mjs b/web/js/workspace/system-controls.mjs new file mode 100644 index 0000000..79bf618 --- /dev/null +++ b/web/js/workspace/system-controls.mjs @@ -0,0 +1,132 @@ +/** + * System's row builders on the concept's markup (#249 System S2 Task 5), built with the DOM, never HTML strings: + * - `settingRow` is the concept's `setting(id, label, description, type, options)` (app.js 624–634): `.setting-row` > + * `div` > `label[for]` + `p`, then the control. A toggle is the Toolkit switch, as the concept's `hydrateSwitches` + * (708–717) makes it; a range is the concept's `min=1 max=16` unless the row gives its own bounds. + * - `operationRow` is `operation(title, desc, action, label, extra, danger)` (645–654): `.operation` > `div` > `h3` + `p`, + * then `.row-actions` with one button, `danger` when flagged. + * - `sectionNav` is `sectionNav(items, active, action)` (306–308): `.section-nav[role=group]` buttons with `aria-pressed`. + * Every listener is registered on the caller's `scope`, so disposing it leaves none behind. + */ +import { createButton, createSwitch } from '../shared/toolkit/core.js'; + +const INPUT_TYPES = new Set(['number', 'password', 'text', 'range']); +const RANGE_DEFAULTS = Object.freeze({ min: 1, max: 16 }); + +function el(tag, text, className) { + const node = document.createElement(tag); + if (text !== undefined) node.textContent = text; + if (className) node.className = className; + return node; +} + +function requireText(values) { + for (const [name, value] of Object.entries(values)) { + if (typeof value !== 'string' || !value) throw new TypeError(`The row needs a ${name}`); + } +} + +function requireScope(scope, callback, name) { + if (callback === undefined) return; + if (typeof callback !== 'function') throw new TypeError(`The row ${name} must be a function`); + if (typeof scope?.listen !== 'function') throw new TypeError('The row needs the section scope for its listener'); +} + +/** `[value, label]` pairs from strings or pairs. */ +function optionPairs(options) { + if (!Array.isArray(options) || options.length === 0) throw new TypeError('A select row needs its options'); + return options.map(option => (Array.isArray(option) ? option.map(String) : [String(option), String(option)])); +} + +/** The control a spec describes, and how to read its value. */ +function buildControl(id, spec) { + if (spec?.type === 'toggle') { + const field = createSwitch('', '', id); + const input = field.children[0]; + // The concept's toggles are on unless configured off (`val !== false`). + input.checked = spec.checked !== false; + return { node: field, input, read: () => input.checked }; + } + if (spec?.type === 'select') { + const select = el('select'); + for (const [value, label] of optionPairs(spec.options)) { + const option = el('option', label); + option.value = value; + select.append(option); + } + if (spec.value !== undefined) select.value = String(spec.value); + return { node: select, input: select, read: () => select.value }; + } + if (INPUT_TYPES.has(spec?.type)) { + const input = el('input'); + input.type = spec.type; + const bounds = spec.type === 'range' ? { ...RANGE_DEFAULTS, ...spec } : spec; + for (const key of ['min', 'max', 'step']) if (bounds[key] !== undefined) input[key] = String(bounds[key]); + input.value = spec.value === undefined || spec.value === null ? '' : String(spec.value); + const numeric = spec.type === 'number' || spec.type === 'range'; + return { node: input, input, read: () => (numeric ? Number(input.value) : input.value) }; + } + throw new TypeError(`Unknown setting control: ${spec?.type}`); +} + +/** + * One `.setting-row`. `control` is a spec (`{type:'toggle', checked}`, `{type:'select', options, value}` with string or + * `[value, label]` options, `{type:'number'|'password'|'text'|'range', value, min, max, step}`) or a ready-made node, + * which takes `id`. `onChange(value)` (on `scope`) reports the checked state, a number for number and range, else the + * text. + */ +export function settingRow({ id, label, description, control, scope, onChange }) { + requireText({ id, label, description }); + requireScope(scope, onChange, 'onChange'); + let built; + if (control && typeof control === 'object' && typeof control.type !== 'string' && 'tagName' in control) { + control.id = id; + built = { node: control, input: control, read: () => control.value }; + } else { + built = buildControl(id, control); + built.input.id = id; + } + built.input.dataset.setting = id; + const row = el('div', undefined, 'setting-row'); + const text = el('div'); + const caption = el('label', label); + caption.htmlFor = id; + text.append(caption, el('p', description)); + row.append(text, built.node); + if (onChange) scope.listen(built.input, 'change', () => onChange(built.read())); + return row; +} + +/** One `.operation` row; `onClick(button)` runs on `scope`. */ +export function operationRow({ title, description, label, danger = false, onClick, scope }) { + requireText({ title, description, label }); + if (typeof onClick !== 'function') throw new TypeError('The operation needs onClick'); + requireScope(scope, onClick, 'onClick'); + const row = el('div', undefined, 'operation'); + const text = el('div'); + text.append(el('h3', title), el('p', description)); + const actions = el('div', undefined, 'row-actions'); + const action = createButton(label, danger ? 'danger' : ''); + scope.listen(action, 'click', () => onClick(action)); + actions.append(action); + row.append(text, actions); + return row; +} + +/** The `.section-nav` group: one button per `[id, title]`, the `active` one pressed; `onSelect(id)` runs on `scope`. */ +export function sectionNav({ label, sections, active, onSelect, scope }) { + requireText({ label }); + if (typeof onSelect !== 'function') throw new TypeError('The section nav needs onSelect'); + requireScope(scope, onSelect, 'onSelect'); + const nav = el('div', undefined, 'section-nav'); + nav.setAttribute('role', 'group'); + nav.setAttribute('aria-label', label); + for (const [id, title] of sections) { + const item = createButton(title); + item.dataset.value = id; + item.setAttribute('aria-pressed', String(id === active)); + scope.listen(item, 'click', () => onSelect(id)); + nav.append(item); + } + return nav; +} diff --git a/web/js/workspace/system-overview.mjs b/web/js/workspace/system-overview.mjs new file mode 100644 index 0000000..ac90206 --- /dev/null +++ b/web/js/workspace/system-overview.mjs @@ -0,0 +1,432 @@ +/** + * System › Overview (#249 System S2 Task 6): the concept's `renderSystem` "general" body (app.js 658–675) on real + * values, its Statistics dialog ("Usage statistics", a1600–1604) and its health report (`health-check`, a1439–1456). + * + * - The four `.summary-metrics` and the two `.two-col` surfaces ("Library health", "Workspace preferences") come from + * one `GET /prompt_manager/system/overview`. D93: metric 4 is the last health result (Not checked / Passed / + * N issues / Incomplete); the review, backup, thumbnail and sources rows say what the server reports, and nothing + * reads 0 before the read answers: a load shows a status line, a failure an alert and Retry. The thumbnail row is + * its own read of `/images/thumbnails/status` (a walk of every source): "Checking thumbnails…" until it answers. + * - D96: the Statistics dialog lists the real runs and average rating per category, then "Library totals". + * - D95: the "Health report" lists every check the server ran, Missing files per image source, with a chip per real + * result and when it was checked. The check runs from Maintenance's notice ("Run health checks", D123) as a + * read job of the registry, tracked in the Activity store (its event stream); the report opens when it ends. + * + * Reads run through `latestOnly()` on the section's scope, so a late answer never repaints a newer section. + */ +import { createButton } from '../shared/toolkit/core.js'; +import { navigateTo } from './state.mjs'; +import { closeWorkspaceDialog, openContentDialog } from './workspace-dialogs.mjs'; +import { showWorkspaceToast } from './toast.mjs'; +import { noun } from './plural.mjs'; +import { operationRow } from './system-controls.mjs'; + +export const OVERVIEW_URL = '/prompt_manager/system/overview'; +export const STATISTICS_URL = '/prompt_manager/system/statistics'; +export const HEALTH_URL = '/prompt_manager/system/health'; +export const THUMBNAILS_URL = '/prompt_manager/images/thumbnails/status'; +export const HEALTH_KIND = 'health'; + +export const LOADING_TEXT = 'Loading the overview…'; +export const THUMBNAILS_LOADING = 'Checking thumbnails…'; +export const LOAD_FAILED_TEXT = 'The overview could not be loaded. Your saved data has not changed.'; +export const STATISTICS_FAILED_TEXT = 'Statistics could not be loaded. Your saved data has not changed.'; +export const HEALTH_FAILED_TEXT = 'The health check could not finish. Nothing was changed.'; +export const NOT_AVAILABLE = 'Not available yet.'; + +const number = value => Number(value).toLocaleString(); + +function el(tag, text, className) { + const node = document.createElement(tag); + if (text !== undefined) node.textContent = text; + if (className) node.className = className; + return node; +} + +/** + * D93: metric 4, the last health result of this server session: the checks that found issues, even when another + * check could not run; otherwise Passed, or Incomplete while a check was unavailable or failed. + */ +export function integrityText(health) { + if (!health) return 'Not checked'; + if (health.issues > 0) return `${number(health.issues)} ${noun(health.issues, 'issue')}`; + return health.status === 'passed' ? 'Passed' : 'Incomplete'; +} + +/** The concept's four metrics `[value, label]`, in its order. */ +export function metricValues(overview) { + return [ + [number(overview.library.positive_prompts), 'Positive prompts'], + [number(overview.library.negative_variants), 'Negative variants'], + [number(overview.images.indexed), 'Images'], + [integrityText(overview.health), 'Database integrity'], + ]; +} + +/** D93 (Q6): the review row's title; a count only once the Needs review queue reports one. */ +export function ambiguousTitle(needsReview) { + const pending = needsReview?.pending; + if (!Number.isSafeInteger(pending)) return 'Ambiguous image links'; + return `${number(pending)} ambiguous image ${noun(pending, 'link')}`; +} + +/** D93: the backup row; only a recorded, verified backup is ever named. */ +export function backupText(backup) { + if (!backup) return 'No retained backup record.'; + const when = new Date(backup.verified_at ?? backup.created_at); + return Number.isNaN(when.getTime()) ? 'No retained backup record.' : `Last verified backup ${when.toLocaleString()}.`; +} + +/** + * D93: the thumbnail row's title from the `/images/thumbnails/status` answer: undefined while it is read, null when + * the read failed; with no image source (`sources.effective` 0) there is nothing to count. + */ +export function thumbnailTitle(status, sources) { + if (status === undefined) return THUMBNAILS_LOADING; + const missing = status?.pending; + if (!Number.isSafeInteger(missing) || sources?.effective === 0) return 'Thumbnail status unavailable'; + if (missing === 0) return 'No missing thumbnails'; + return `${number(missing)} missing ${noun(missing, 'thumbnail')}`; +} + +const SOURCES_SHARED = 'Monitoring and enrichment share one source list.'; + +/** D93: the sources row, `[title, description]`: the sources in use and how they were chosen. */ +export function sourcesCopy(sources) { + const { configured, effective, auto_detected: auto } = sources; + const title = `${number(effective)} output ${noun(effective, 'source')}`; + if (auto) return [title, `None configured is available; ComfyUI's output folder is used. ${SOURCES_SHARED}`]; + if (configured !== effective) return [title, `${number(effective)} of ${number(configured)} configured ${noun(configured, 'source')} available. ${SOURCES_SHARED}`]; + return [title, SOURCES_SHARED]; +} + +// D95: an issue chip names what was found; the concept's own wording for missing files and ambiguous metadata. +const ISSUE_NOUNS = Object.freeze({ + integrity: ['problem found', 'problems found'], + foreign_keys: ['broken reference', 'broken references'], + canonical_hashes: ['hash does not match its text', 'hashes do not match their text'], + pair_uniqueness: ['duplicate pair row', 'duplicate pair rows'], + missing_files: ['link needs review', 'links need review'], + ambiguous_metadata: ['image needs review', 'images need review'], + consistency: ['inconsistency', 'inconsistencies'], +}); + +/** D95: a check's chip from its real status. */ +export function checkChip(check) { + if (check.status === 'passed') return 'Passed'; + if (check.status === 'failed') return 'Could not check'; + if (check.status === 'unavailable') return check.id === 'ambiguous_metadata' ? NOT_AVAILABLE.slice(0, -1) : 'Not available'; + const [one, many] = ISSUE_NOUNS[check.id] ?? ['issue', 'issues']; + if (!Number.isSafeInteger(check.count)) return 'Issues found'; + // SQLite stops listing integrity problems at its limit, so a capped count is a lower bound ("100+"). + if (check.capped === true) return `${number(check.count)}+ ${many}`; + return `${number(check.count)} ${check.count === 1 ? one : many}`; +} + +function sourceChip(source) { + if (source.status !== 'checked') return 'Not available'; + return source.count === 0 ? 'Passed' : checkChip({ id: 'missing_files', status: 'issues', count: source.count }); +} + +/** D95: the report's rows `[label, chip]` in the server's order; Missing files once per image source when several. */ +export function healthRows(report) { + return report.checks.flatMap(check => { + const sources = check.id === 'missing_files' && Array.isArray(check.sources) ? check.sources : []; + if (sources.length < 2) return [[check.label, checkChip(check)]]; + return sources.map(source => [`${check.label} · ${source.name}`, sourceChip(source)]); + }); +} + +/** D95: when the report was taken, and whether it is complete. */ +export function healthSummary(report) { + const when = new Date(report.checked_at); + const time = Number.isNaN(when.getTime()) ? '' : `Checked ${when.toLocaleString()}. `; + if (report.status === 'passed') return `${time}Every check passed.`; + if (report.status === 'incomplete') return `${time}Some checks could not run, so this report is incomplete.`; + return `${time}Nothing was changed; review the checks with issues.`; +} + +/** D96: the concept's "Runs by category" rows `[category, runs, average rating]`. */ +export function categoryRows(statistics) { + return statistics.by_category.map(row => [ + row.category ?? 'Uncategorized', + number(row.runs), + row.average_rating === null || row.average_rating === undefined ? 'Not rated' + : `${Number(row.average_rating).toLocaleString(undefined, { maximumFractionDigits: 2 })} / 5`, + ]); +} + +/** D96: the added "Library totals" rows `[label, count]` (the legacy statistics and the rated images). */ +export function totalsRows(statistics) { + const { totals } = statistics; + return [['Prompts', totals.total_prompts], ['Tags', totals.total_tags], ['Images', totals.total_images], + ['Rated images', statistics.rated_images]].map(([label, value]) => [label, number(value)]); +} + +function table(caption, head, rows, empty) { + const node = el('table'); + const header = el('tr'); + for (const text of head) { + const cell = el('th', text); + cell.scope = 'col'; + header.append(cell); + } + const thead = el('thead'); + thead.append(header); + const tbody = el('tbody'); + for (const values of rows) { + const row = el('tr'); + row.append(...values.map(text => el('td', text))); + tbody.append(row); + } + if (rows.length === 0) { + const row = el('tr'); + const cell = el('td', empty); + cell.colSpan = head.length; + row.append(cell); + tbody.append(row); + } + node.append(el('caption', caption), thead, tbody); + return node; +} + +/** The Statistics dialog: "Runs by category" (the concept's table), then "Library totals" (D96). */ +export function statisticsNodes(statistics) { + return [ + table('Runs by category', ['Category', 'Runs', 'Avg. rating'], categoryRows(statistics), 'No prompts in the library yet.'), + table('Library totals', ['Total', 'Count'], totalsRows(statistics), ''), + ]; +} + +/** The concept's health report body: `.operation-list` of `h3` + chip, the D95 summary, then Close / Review missing links. */ +export function healthNodes(report, { scope, onReviewMissing }) { + const list = el('div', undefined, 'operation-list'); + for (const [label, chip] of healthRows(report)) { + const row = el('div', undefined, 'operation'); + row.append(el('h3', label), el('span', chip, 'chip')); + list.append(row); + } + const close = createButton('Close'); + const review = createButton('Review missing links', 'ca-primary'); + scope.listen(close, 'click', () => closeWorkspaceDialog()); + scope.listen(review, 'click', () => onReviewMissing()); + const actions = el('div', undefined, 'dialog-actions'); + actions.append(close, review); + return [list, el('p', healthSummary(report), 'small'), actions]; +} + +/** + * Wait until the job `id` has ended, following it in the Activity store (`activity`), which streams the job's events + * and marks it lost when the server no longer lists it; nothing here polls. Resolves its final state ('finished', + * 'failed', 'cancelled' or 'lost'). + */ +export function waitForJob(activity, id, { signal } = {}) { + if (signal?.aborted) return Promise.reject(signal.reason); + return new Promise((resolve, reject) => { + const check = () => { + const state = activity.find(id)?.state; + if (state && state !== 'running') finish(resolve, state); + }; + const onAbort = () => finish(reject, signal.reason); + const unsubscribe = activity.subscribe(check); + function finish(settle, value) { + unsubscribe(); + signal?.removeEventListener('abort', onAbort); + settle(value); + } + signal?.addEventListener('abort', onAbort, { once: true }); + check(); + }); +} + +/** + * Run the health check and resolve its report. The started job is tracked in the Activity store; a check already + * running (409 busy, another tab) is found in the store's job list and waited for instead of refused. Rejects when the + * check fails or ends without a report. + */ +export async function runHealthCheck(api, { activity, signal } = {}) { + let id; + try { + const { job } = await api.post(HEALTH_URL, {}, { signal }); + id = activity.track(job).id; + } catch (error) { + if (error?.status !== 409 || error?.code !== 'busy') throw error; + await activity.load(); + id = activity.running().find(job => job.kind === HEALTH_KIND)?.id; + // The other check ended between the refusal and this read: its report is the latest. + } + const state = id ? await waitForJob(activity, id, { signal }) : 'finished'; + if (state !== 'finished') throw new Error(HEALTH_FAILED_TEXT); + const { report } = await api.get(HEALTH_URL, { signal, cache: 'no-store' }); + if (!report) throw new Error(HEALTH_FAILED_TEXT); + return report; +} + +/** Open the Health report for `report` (the Maintenance notice's and, from S7, its Check action's dialog). */ +export function openHealthReport(report, { scope, trigger = null, navigate = navigateTo }) { + openContentDialog({ scope, title: 'Health report', trigger, nodes: healthNodes(report, { scope, + // Missing-link cleanup lives in Maintenance (its D94 hand-off until S7). + onReviewMissing: () => { closeWorkspaceDialog(); navigate('#/system/maintenance'); } }) }); +} + +/** + * The concept's Maintenance notice ("Check first. Review changes. Then apply." + "Run health checks", a689), which + * S2 delivers ahead of the rest of Maintenance (D123): the press runs the read-only check, then opens the report. + */ +export function healthNotice({ api, activity, scope, navigate = navigateTo, notify = showWorkspaceToast }) { + const notice = el('div', undefined, 'notice'); + const text = el('div'); + text.append(el('h2', 'Check first. Review changes. Then apply.'), + el('p', 'One home for database and media maintenance. Scanning is separate from changing or removing records.')); + const run = createButton('Run health checks', 'ca-primary'); + notice.append(text, run); + scope.listen(run, 'click', async () => { + if (run.getAttribute('aria-disabled') === 'true') return; + run.setAttribute('aria-disabled', 'true'); + run.textContent = 'Checking…'; + try { + const report = await runHealthCheck(api, { activity, signal: scope.signal }); + if (scope.signal.aborted) return; + openHealthReport(report, { scope, trigger: run, navigate }); + } catch (error) { + if (!scope.signal.aborted) notify(HEALTH_FAILED_TEXT); + } finally { + if (!scope.signal.aborted) { + run.removeAttribute('aria-disabled'); + run.textContent = 'Run health checks'; + } + } + }); + return notice; +} + +function metricsNode(overview) { + const metrics = el('div', undefined, 'summary-metrics'); + for (const [value, label] of metricValues(overview)) { + const metric = el('div', undefined, 'metric'); + metric.append(el('strong', value), el('span', label)); + metrics.append(metric); + } + return metrics; +} + +function surface(heading, rows) { + const section = el('section', undefined, 'surface'); + section.append(el('h2', heading), ...rows); + return section; +} + +/** The Overview's nodes, and its thumbnail row (painted by its own read). */ +function overviewNodes(overview, { scope, navigate, openStatistics }) { + const go = section => () => navigate(`#/system/${section}`); + const review = operationRow({ scope, title: ambiguousTitle(overview.needs_review), + description: Number.isSafeInteger(overview.needs_review?.pending) + ? 'Review metadata before assigning a positive / negative pair.' : NOT_AVAILABLE, + label: 'Review metadata', onClick: () => {} }); + // D93 (Q6): the review queue (Needs review) is not part of the workspace yet. + review.querySelector('button').setAttribute('aria-disabled', 'true'); + const [sourcesTitle, sourcesDescription] = sourcesCopy(overview.sources); + const thumbnails = operationRow({ scope, title: THUMBNAILS_LOADING, description: 'Generate only missing previews when necessary.', + label: 'Review thumbnails', onClick: go('sources') }); + const health = surface('Library health', [ + review, + operationRow({ scope, title: 'Backup status', description: backupText(overview.backup), label: 'View backups', + onClick: go('backups') }), + // Until the thumbnail review dialog (S5), Review thumbnails opens Sources, where thumbnail generation lives. + thumbnails, + ]); + const preferences = surface('Workspace preferences', [ + operationRow({ scope, title: 'One display configuration', description: 'Grid and density settings are shared across views.', + label: 'Appearance', onClick: go('appearance') }), + operationRow({ scope, title: sourcesTitle, description: sourcesDescription, label: 'Manage sources', onClick: go('sources') }), + operationRow({ scope, title: 'Statistics', description: 'Runs and ratings by category, with library totals.', label: 'Explore', + onClick: openStatistics }), + ]); + const columns = el('div', undefined, 'two-col'); + columns.append(health, preferences); + return { nodes: [metricsNode(overview), columns], thumbnails }; +} + +/** + * The Overview section's mount (`mountSystem`'s `sections.overview`): returns its loading line, which the server's + * answer replaces in place (the concept's metrics and columns sit directly in `.settings-layout`). A read that a newer + * one superseded, or one answering after the section left, paints nothing. + */ +export function mountOverview(container, { api, scope, navigate = navigateTo, notify = showWorkspaceToast, + latestOnly = window.ApiClient.latestOnly, STALE = window.ApiClient.STALE }) { + const overviewRead = latestOnly({ lifetime: scope.signal }); + const statisticsRead = latestOnly({ lifetime: scope.signal }); + const thumbnailsRead = latestOnly({ lifetime: scope.signal }); + let shown = []; + // The thumbnail status (undefined while read, null when it failed), painted into the row whenever both are there. + let thumbnailStatus; + let thumbnailRow = null; + let sources = null; + const paintThumbnails = () => { + if (thumbnailRow) thumbnailRow.querySelector('h3').textContent = thumbnailTitle(thumbnailStatus, sources); + }; + + function swap(nodes) { + const anchor = shown.find(node => node.isConnected); + if (anchor) anchor.before(...nodes); + for (const node of shown) node.remove(); + shown = nodes; + return nodes; + } + + async function openStatistics(button) { + try { + const statistics = await statisticsRead(signal => api.get(STATISTICS_URL, { signal, cache: 'no-store' })); + if (statistics === STALE || scope.signal.aborted) return; + openContentDialog({ scope, title: 'Usage statistics', trigger: button, nodes: statisticsNodes(statistics) }); + } catch (error) { + if (!scope.signal.aborted) notify(STATISTICS_FAILED_TEXT); + } + } + + function loading() { + const status = el('p', LOADING_TEXT, 'small'); + status.setAttribute('role', 'status'); + return [status]; + } + + async function load() { + try { + const overview = await overviewRead(signal => api.get(OVERVIEW_URL, { signal, cache: 'no-store' })); + if (overview === STALE || scope.signal.aborted) return; + const { nodes, thumbnails } = overviewNodes(overview, { scope, navigate, openStatistics }); + thumbnailRow = thumbnails; + sources = overview.sources; + swap(nodes); + paintThumbnails(); + } catch (error) { + if (scope.signal.aborted) return; + const alert = el('p', LOAD_FAILED_TEXT); + alert.setAttribute('role', 'alert'); + const retry = createButton('Retry'); + scope.listen(retry, 'click', () => { + swap(loading()); + load(); + loadThumbnails(); + }); + const panel = el('section', undefined, 'surface'); + panel.append(alert, retry); + swap([panel]); + } + } + async function loadThumbnails() { + let status; + try { + status = await thumbnailsRead(signal => api.get(THUMBNAILS_URL, { signal, cache: 'no-store' })); + } catch (error) { + status = null; + } + if (status === STALE || scope.signal.aborted) return; + thumbnailStatus = status ?? null; + paintThumbnails(); + } + const first = swap(loading()); + load(); + loadThumbnails(); + return first; +} diff --git a/web/js/workspace/system-state.mjs b/web/js/workspace/system-state.mjs new file mode 100644 index 0000000..2c24e67 --- /dev/null +++ b/web/js/workspace/system-state.mjs @@ -0,0 +1,201 @@ +/** + * The System draft store (#249 System S2 Task 5): every System section edits drafts behind the one Save row (the + * concept's `.save-row`, app.js renderSystem 705), in three domains: + * - `browser`: preferences kept in this browser (Appearance); + * - `host`: the server configuration, saved with the `revision` its read returned (`expected_revision`, #220); + * - `runtime`: values applied to the running server only (the log level, Q10b). + * + * `save()` writes browser, then host, then runtime, and only the domains with changes. A domain that fails keeps its + * drafts dirty and its keys `failed()`, so the section highlights them; the others are saved. A host save refused + * because the configuration changed elsewhere (`settings-changed`) is a `conflict` and keeps the draft. A host save before + * `load('host')` returned a revision fails without a request: a blind write would merge over a configuration changed + * elsewhere (S1 reads a null `expected_revision` as "merge into what is on disk"). `reloadHost()` reads the host values + * again (System calls it on mount); a host conflict reloads them itself, so the next save carries the current revision. + * Drafts live for + * the page session (Q10a): a section left and opened again shows them, and a reload of its values keeps them. + * + * Adapters: `readBrowser()` and `readRuntime()` return the values; `readHost()` returns `{values, revision}`. + * `writeBrowser(changes)` returns `{persisted}` (false: kept for this tab only), `writeHost(changes, + * {expected_revision})` the settings response (`{success, persisted, revision}`), `writeRuntime(changes)` anything; + * a write that throws failed. + */ +export { SYSTEM_SECTIONS, isSystemSection } from './state.mjs'; + +/** D92: the Save row's status copy; never the concept's prototype text. */ +export const SETTINGS_STATUS = Object.freeze({ + clean: 'No unsaved changes.', + dirty: 'Unsaved changes.', + browser: 'Saved in this browser.', + host: 'Saved to the server configuration.', + runtime: 'Applied until the server restarts.', + partial: 'Some changes were not saved. Review the highlighted settings.', + tabOnly: 'Saved for this tab only; browser storage is unavailable.', + conflict: 'Settings changed elsewhere. Reload this section to see them.', +}); + +export const DRAFT_DOMAINS = Object.freeze(['browser', 'host', 'runtime']); +const CONFLICT_CODE = 'settings-changed'; +const ADAPTERS = ['readBrowser', 'readHost', 'readRuntime', 'writeBrowser', 'writeHost', 'writeRuntime']; + +/** The status copy of one `save()` outcome (`{browser, host, runtime}`, each clean/saved/tab-only/applied/failed/conflict). */ +export function statusText(outcome) { + const results = DRAFT_DOMAINS.map(domain => outcome?.[domain]); + if (results.includes('conflict')) return SETTINGS_STATUS.conflict; + if (results.includes('failed')) return SETTINGS_STATUS.partial; + const saved = []; + if (outcome?.browser === 'saved') saved.push(SETTINGS_STATUS.browser); + if (outcome?.browser === 'tab-only') saved.push(SETTINGS_STATUS.tabOnly); + if (outcome?.host === 'saved') saved.push(SETTINGS_STATUS.host); + if (outcome?.runtime === 'applied') saved.push(SETTINGS_STATUS.runtime); + return saved.join(' ') || SETTINGS_STATUS.clean; +} + +function requireDomain(domain) { + if (!DRAFT_DOMAINS.includes(domain)) throw new TypeError(`Unknown settings domain: ${domain}`); +} + +/** Create the store; see the module comment for the adapters. */ +export function createSystemDrafts(adapters) { + for (const name of ADAPTERS) { + if (typeof adapters?.[name] !== 'function') throw new TypeError(`System drafts need ${name}`); + } + const baseline = { browser: {}, host: {}, runtime: {} }; + const drafts = { browser: new Map(), host: new Map(), runtime: new Map() }; + const failedKeys = { browser: new Set(), host: new Set(), runtime: new Set() }; + const listeners = new Set(); + let revision = null; + let outcome = null; + let saving = null; + + const notify = () => { for (const listener of [...listeners]) listener(); }; + const changes = domain => Object.fromEntries(drafts[domain]); + + /** Read a domain's saved values; drafts are kept, and a draft equal to the new value is clean. */ + async function load(domain) { + requireDomain(domain); + let values; + if (domain === 'host') { + const read = await adapters.readHost(); + values = read?.values ?? {}; + revision = read?.revision ?? null; + } else { + values = await (domain === 'browser' ? adapters.readBrowser() : adapters.readRuntime()); + } + baseline[domain] = { ...(values ?? {}) }; + for (const [key, value] of drafts[domain]) if (Object.is(baseline[domain][key], value)) drafts[domain].delete(key); + notify(); + return { ...baseline[domain] }; + } + + function get(domain, key) { + requireDomain(domain); + return drafts[domain].has(key) ? drafts[domain].get(key) : baseline[domain][key]; + } + + function set(domain, key, value) { + requireDomain(domain); + if (Object.is(baseline[domain][key], value)) drafts[domain].delete(key); + else drafts[domain].set(key, value); + failedKeys[domain].delete(key); + outcome = null; + notify(); + } + + function dirty(domain) { + if (domain !== undefined) { + requireDomain(domain); + return drafts[domain].size > 0; + } + return DRAFT_DOMAINS.some(name => drafts[name].size > 0); + } + + function discard(domain) { + for (const name of domain === undefined ? DRAFT_DOMAINS : [domain]) { + requireDomain(name); + drafts[name].clear(); + failedKeys[name].clear(); + } + outcome = null; + notify(); + } + + /** The values of one domain as written: they become its saved values, and drafts still equal to them are clean. */ + function commit(domain, written) { + for (const [key, value] of written) { + baseline[domain][key] = value; + if (drafts[domain].has(key) && Object.is(drafts[domain].get(key), value)) drafts[domain].delete(key); + } + } + + function fail(domain, written) { + for (const [key] of written) failedKeys[domain].add(key); + } + + async function saveDomain(domain) { + const written = new Map(drafts[domain]); + if (written.size === 0) return 'clean'; + const values = Object.fromEntries(written); + try { + if (domain === 'browser') { + const result = await adapters.writeBrowser(values); + commit(domain, written); + return result?.persisted === false ? 'tab-only' : 'saved'; + } + if (domain === 'host') { + if (typeof revision !== 'string') { fail(domain, written); return 'failed'; } + const result = await adapters.writeHost(values, { expected_revision: revision }); + if (result?.code === CONFLICT_CODE) { fail(domain, written); return 'conflict'; } + if (!result?.success || result.persisted !== true) { fail(domain, written); return 'failed'; } + if (typeof result.revision === 'string') revision = result.revision; + commit(domain, written); + return 'saved'; + } + await adapters.writeRuntime(values); + commit(domain, written); + return 'applied'; + } catch (error) { + fail(domain, written); + return error?.code === CONFLICT_CODE ? 'conflict' : 'failed'; + } + } + + /** Read the host values again; a failed read leaves host saves refused until one succeeds. */ + const reloadHost = () => load('host').then(() => true, () => false); + + /** Save the dirty domains in order; a Save pressed while one runs joins it. */ + function save() { + saving ??= (async () => { + for (const domain of DRAFT_DOMAINS) failedKeys[domain].clear(); + const result = {}; + for (const domain of DRAFT_DOMAINS) result[domain] = await saveDomain(domain); + // Settings changed elsewhere: read them (and their revision), keeping the drafts (Q10). + if (result.host === 'conflict') await reloadHost(); + return result; + })().then(result => { + saving = null; + outcome = result; + notify(); + return result; + }); + return saving; + } + + return { + load, + reloadHost, + get, + set, + dirty, + discard, + save, + changes, + saving: () => saving !== null, + failed: domain => { requireDomain(domain); return new Set(failedKeys[domain]); }, + /** The Save row's text: the last save's outcome until the next edit, else unsaved or clean. */ + statusText: () => (outcome ? statusText(outcome) : dirty() ? SETTINGS_STATUS.dirty : SETTINGS_STATUS.clean), + subscribe(listener) { + listeners.add(listener); + return () => listeners.delete(listener); + }, + }; +} diff --git a/web/js/workspace/system.mjs b/web/js/workspace/system.mjs new file mode 100644 index 0000000..0fa7658 --- /dev/null +++ b/web/js/workspace/system.mjs @@ -0,0 +1,148 @@ +/** + * Native System (#249 System S2 Task 5): the concept's `renderSystem` shell (app.js 698–706) at `#/system/
`. + * Under the page heading it renders the section nav (`.section-nav`, one button per section, the shown one + * `aria-pressed`; a press navigates to its hash), then `.settings-layout` with the section's body and the Save row + * (`span.small#settings-status` + "Save preferences"). D92: no "Sample configuration" chip, the status copy is + * truthful and "Save preferences" is aria-disabled while nothing is unsaved. A section not delivered yet shows the + * per-section hand-off to its legacy tool (D94), never mock placeholders. Overview is native (system-overview.mjs). + * + * Switching sections keeps the shell (and focus on the pressed button) and disposes the old section's scope, so its + * listeners and timers never outlive it. The drafts are module-level (Q10): they live for the page session. + */ +import { createButton, createScope } from '../shared/toolkit/core.js'; +import { SYSTEM_SECTIONS, navigateTo } from './state.mjs'; +import { createSystemDrafts, DRAFT_DOMAINS } from './system-state.mjs'; +import { sectionNav } from './system-controls.mjs'; +import { healthNotice, mountOverview } from './system-overview.mjs'; + +const SETTINGS_URL = '/prompt_manager/settings'; +const ADMIN = Object.freeze({ href: '/prompt_manager/admin', label: 'Open existing administration' }); +const GALLERY = Object.freeze({ href: '/prompt_manager/gallery', label: 'Open existing image gallery' }); +/** + * D94: where each section's capabilities live until its PR (the plan's legacy capability coverage): the admin page's + * dashboard, Settings, Maintenance, Backup, Restore, Scan, Logs, Diagnostics and Auto-tag tools, and the gallery's + * settings, thumbnails, duplicates and cache tools. + */ +export const SECTION_HANDOFFS = Object.freeze({ + appearance: [GALLERY, ADMIN], + sources: [ADMIN, GALLERY], + integrations: [ADMIN], + tagging: [ADMIN], + maintenance: [ADMIN, GALLERY], + backups: [ADMIN], + diagnostics: [ADMIN], +}); +export const NOT_DELIVERED = 'This setting is not available here yet.'; + +let drafts = null; + +function el(tag, text, className) { + const node = document.createElement(tag); + if (text !== undefined) node.textContent = text; + if (className) node.className = className; + return node; +} + +/** A domain whose settings no delivered section edits yet: nothing can be drafted, and a write would say it failed. */ +const undelivered = async () => { throw new Error(NOT_DELIVERED); }; + +/** The page session's drafts (Q10); the host domain reads and saves the server settings with their revision (#220). */ +export function systemDrafts(api) { + drafts ??= createSystemDrafts({ + readBrowser: () => ({}), + readRuntime: () => ({}), + writeBrowser: undelivered, + writeRuntime: undelivered, + async readHost() { + const result = await api.get(SETTINGS_URL, { cache: 'no-store' }); + // The store reads a missing value as none and a missing revision as unreadable (no host save). + return { values: result.settings, revision: result.revision }; + }, + writeHost: (changes, { expected_revision }) => api.post(SETTINGS_URL, { ...changes, expected_revision }), + }); + return drafts; +} + +/** D94: the shared hand-off panel, linking each legacy tool that still owns this section. */ +function handoff(section) { + const panel = el('section', undefined, 'surface handoff'); + panel.append(el('h2', 'Your current tools are ready'), + el('p', 'This workspace is being introduced in stages. Continue in the existing tool with your saved data.')); + for (const tool of SECTION_HANDOFFS[section]) { + const link = el('a', tool.label, 'workspace-link'); + link.href = window.ApiClient.resolveUrl(tool.href); + panel.append(link); + } + return [panel]; +} + +/** D92: highlight the controls of the settings the last save could not save. */ +function markFailed(layout, store) { + const failed = new Set(DRAFT_DOMAINS.flatMap(domain => [...store.failed(domain)])); + for (const control of layout.querySelectorAll('[data-setting]')) { + if (failed.has(control.dataset.setting)) control.setAttribute('aria-invalid', 'true'); + else control.removeAttribute('aria-invalid'); + } +} + +/** + * The sections delivered so far: Overview (S2 Task 6), and Maintenance's notice with "Run health checks" above the + * hand-off to the maintenance tools that are not native yet (D123). + */ +export const DELIVERED_SECTIONS = Object.freeze({ + overview: mountOverview, + maintenance: (container, { api, activity, scope }) => [healthNotice({ api, activity, scope }), ...handoff('maintenance')], +}); + +/** + * Mount System under the heading `renderHeading` wrote in `main`. Returns `{show(section)}`, which shows another + * section in place. `scope` is the page's view scope; `sections` maps a section id to a delivered section's mount + * (`(container, {api, activity, scope, drafts}) => nodes`), and every other section hands off. `activity` is the page's + * Activity store, which follows the jobs a section starts. + * + * The host values are read on mount (and by the store again after a conflict), so a save carries the current revision; + * until a read returned one, a host save fails without a request (system-state.mjs). + */ +export function mountSystem(main, api, scope, { section, activity, sections = DELIVERED_SECTIONS, navigate = navigateTo }) { + const store = systemDrafts(api); + store.reloadHost(); + const nav = sectionNav({ label: 'System sections', sections: SYSTEM_SECTIONS, active: section, scope, + onSelect: id => navigate(`#/system/${id}`) }); + const layout = el('div', undefined, 'settings-layout'); + const saveRow = el('div', undefined, 'save-row'); + const status = el('span', undefined, 'small'); + status.id = 'settings-status'; + const save = createButton('Save preferences', 'ca-primary'); + saveRow.append(status, save); + layout.append(saveRow); + main.append(nav, layout); + + const paintStatus = () => { + status.textContent = store.statusText(); + if (store.dirty() && !store.saving()) save.removeAttribute('aria-disabled'); + else save.setAttribute('aria-disabled', 'true'); + markFailed(layout, store); + }; + scope.cleanup(store.subscribe(paintStatus)); + scope.listen(save, 'click', () => { + if (save.getAttribute('aria-disabled') === 'true') return; + const saving = store.save(); + paintStatus(); + saving.then(() => { if (!scope.signal.aborted) paintStatus(); }); + }); + + let sectionScope = null; + scope.cleanup(() => sectionScope?.dispose()); + function show(next) { + sectionScope?.dispose(); + sectionScope = createScope(); + for (const button of nav.children) button.setAttribute('aria-pressed', String(button.dataset.value === next)); + while (layout.firstChild !== saveRow) layout.firstChild.remove(); + const mount = sections[next]; + const nodes = mount ? mount(layout, { api, activity, scope: sectionScope, drafts: store }) : handoff(next); + layout.prepend(...nodes); + paintStatus(); + } + show(section); + return { show }; +} diff --git a/web/js/workspace/workspace-dialogs.mjs b/web/js/workspace/workspace-dialogs.mjs index 39b35e6..da53853 100644 --- a/web/js/workspace/workspace-dialogs.mjs +++ b/web/js/workspace/workspace-dialogs.mjs @@ -11,7 +11,8 @@ * Rescan; no "Import portable recipes" (D15). Rescan shows its running state (aria-disabled "Rescanning…"). D55: the * concept's own "Scan an output folder → Review scan" row follows as "Scan for metadata". * - "Review metadata scan" (a1428–1432 `scan-review`, D55): Sources and Scope in a `dl.meta-grid`, the concept's - * sentence, a backup `.notice` ("Back up now") and Cancel / Start scan; no sample-plan note. + * sentence, a backup `.notice` ("Back up now") and Cancel / Start scan; no sample-plan note. It is one caller of + * `openReviewDialog`, the review every record-changing System operation shares, with its `backupNotice`. * - "Copy prompt" (a1223–1241): the select-to-copy fallback when the clipboard is unavailable (D22). * - One-value input (a1138 `simpleInput`): a "Category"-style field (`input#action-input`) and Cancel / a primary * action; used by the Images selection bar's "Set category" (#249 D8) and the toolbar's Save view. @@ -367,6 +368,98 @@ const BACKUP_BUSY_LABEL = 'Backing up…'; export const SCAN_SCOPE_LABELS = { new: 'New and unresolved files', all: 'All files' }; export const SCAN_ALL_FILES_HINT = 'Also read the files already linked to their prompt again.'; +/** The scan review's backup box copy (D55). */ +export const SCAN_BACKUP_COPY = Object.freeze({ heading: 'Back up the database first (recommended)', + text: 'Downloads a copy of the prompts database before the scan changes any records.' }); + +/** + * The D55 backup box: the concept's System `.notice` (`div` > `h3` + `p`, then `.row-actions` with "Back up now"). + * `onBackup` runs the backup and toasts its outcome; until it settles the button reads "Backing up…" and is + * aria-disabled, so a second press is ignored. It never closes the dialog it sits in. + */ +export function backupNotice({ copy, onBackup }) { + if (typeof copy?.heading !== 'string' || !copy.heading || typeof copy.text !== 'string' || !copy.text) { + throw new TypeError('The backup box needs its heading and text'); + } + requireCallbacks({ onBackup }); + const notice = el('div', undefined, 'notice'); + const text = el('div'); + text.append(el('h3', copy.heading), el('p', copy.text)); + const actions = el('div', undefined, 'row-actions'); + const backup = createButton(BACKUP_LABEL); + actions.append(backup); + notice.append(text, actions); + backup.addEventListener('click', async () => { + if (backup.getAttribute('aria-disabled') === 'true') return; + backup.setAttribute('aria-disabled', 'true'); + backup.textContent = BACKUP_BUSY_LABEL; + try { + await onBackup(); + } finally { + backup.removeAttribute('aria-disabled'); + backup.textContent = BACKUP_LABEL; + } + }); + return notice; +} + +function requireGrid(grid) { + if (!Array.isArray(grid) || grid.length === 0 || grid.some(row => !Array.isArray(row) || row.length !== 2 || + typeof row[0] !== 'string' || !row[0] || typeof row[1] !== 'string')) { + throw new TypeError('The review needs its [term, value] grid'); + } +} + +/** + * The shared record-changing review (the concept's `scan-review`, maintenance plan and restore preview): an optional + * `h3` heading, the `dl.meta-grid` of `[term, value]` rows, page-built `options` nodes, the optional `p.small` + * sentence, the D55 backup box when `backup` is `{copy, onBackup}`, then `.dialog-actions` with `cancelLabel` and the + * primary (`ca-primary`, or the concept's `danger` when `primary.danger`). The primary closes the dialog, then calls + * `onStart()`. Returns `{setValue(term, text)}` for a grid value that follows an option. + */ +export function openReviewDialog({ scope, title, heading = null, grid, options = [], sentence = null, backup = null, + cancelLabel, primary, onStart, trigger = null }) { + requireScope(scope); + requireCallbacks({ onStart }); + for (const [name, text] of Object.entries({ title, cancelLabel, label: primary?.label })) { + if (typeof text !== 'string' || !text) throw new TypeError(`The review needs its ${name}`); + } + requireGrid(grid); + if (!Array.isArray(options) || options.some(node => !node || typeof node !== 'object')) { + throw new TypeError('The review options must be nodes'); + } + const nodes = []; + if (heading) nodes.push(el('h3', heading)); + const list = el('dl', undefined, 'meta-grid'); + const values = new Map(); + for (const [term, value] of grid) { + const entry = el('div'); + const dd = el('dd', value); + values.set(term, dd); + entry.append(el('dt', term), dd); + list.append(entry); + } + nodes.push(list, ...options); + if (sentence) nodes.push(el('p', sentence, 'small')); + if (backup) nodes.push(backupNotice(backup)); + const cancel = createButton(cancelLabel); + const start = createButton(primary.label, primary.danger ? 'danger' : 'ca-primary'); + cancel.addEventListener('click', () => finish()); + start.addEventListener('click', () => { + finish(); + onStart(); + }); + const actions = el('div', undefined, 'dialog-actions'); + actions.append(cancel, start); + nodes.push(actions); + show(title, nodes, { scope, trigger }); + return { + setValue(term, text) { + const dd = values.get(term); + if (dd) dd.textContent = text; + }, + }; +} /** * The concept's "Review metadata scan" (`scan-review`, D55). `sources` is the number of output folders the scan reads. @@ -379,18 +472,9 @@ export function openScanReviewDialog({ scope, sources, onBackup, onStart, trigge requireScope(scope); requireCallbacks({ onBackup, onStart }); if (!Number.isSafeInteger(sources) || sources < 0) throw new TypeError('The scan review needs its source count'); - const grid = el('dl', undefined, 'meta-grid'); - const values = {}; - for (const [term, value] of [['Sources', `${sources.toLocaleString('en-US')} output ${sources === 1 ? 'folder' : 'folders'}`], - ['Scope', SCAN_SCOPE_LABELS.new]]) { - const entry = el('div'); - values[term] = el('dd', value); - entry.append(el('dt', term), values[term]); - grid.append(entry); - } // The concept's System toggle: a `.setting-row` whose checkbox is the Toolkit switch (concept `hydrateSwitches`). const toggle = createSwitch('', '', 'scan-all-files'); - const allFiles = toggle.querySelector('input'); + const allFiles = toggle.children[0]; allFiles.checked = false; const option = el('div', undefined, 'setting-row'); const optionText = el('div'); @@ -402,35 +486,10 @@ export function openScanReviewDialog({ scope, sources, onBackup, onStart, trigge optionText.append(optionLabel, hint); option.append(optionText, toggle); const chosenScope = () => (allFiles.checked ? 'all' : 'new'); - allFiles.addEventListener('change', () => { values.Scope.textContent = SCAN_SCOPE_LABELS[chosenScope()]; }); - const notice = el('div', undefined, 'notice'); - const copy = el('div'); - copy.append(el('h3', 'Back up the database first (recommended)'), - el('p', 'Downloads a copy of the prompts database before the scan changes any records.')); - const backupActions = el('div', undefined, 'row-actions'); - const backup = createButton(BACKUP_LABEL); - backupActions.append(backup); - notice.append(copy, backupActions); - backup.addEventListener('click', async () => { - if (backup.getAttribute('aria-disabled') === 'true') return; - backup.setAttribute('aria-disabled', 'true'); - backup.textContent = BACKUP_BUSY_LABEL; - try { - await onBackup(); - } finally { - backup.removeAttribute('aria-disabled'); - backup.textContent = BACKUP_LABEL; - } - }); - const cancel = createButton('Cancel'); - const start = createButton('Start scan', 'ca-primary'); - cancel.addEventListener('click', () => finish()); - start.addEventListener('click', () => { - const files = chosenScope(); - finish(); - onStart({ scope: files }); - }); - const actions = el('div', undefined, 'dialog-actions'); - actions.append(cancel, start); - show('Review metadata scan', [grid, option, el('p', SCAN_REVIEW_SENTENCE, 'small'), notice, actions], { scope, trigger }); + const review = openReviewDialog({ scope, trigger, title: 'Review metadata scan', + grid: [['Sources', `${sources.toLocaleString('en-US')} output ${sources === 1 ? 'folder' : 'folders'}`], + ['Scope', SCAN_SCOPE_LABELS.new]], + options: [option], sentence: SCAN_REVIEW_SENTENCE, backup: { copy: SCAN_BACKUP_COPY, onBackup }, + cancelLabel: 'Cancel', primary: { label: 'Start scan' }, onStart: () => onStart({ scope: chosenScope() }) }); + allFiles.addEventListener('change', () => review.setValue('Scope', SCAN_SCOPE_LABELS[chosenScope()])); } diff --git a/web/js/workspace/workspace.css b/web/js/workspace/workspace.css index 7117376..c7088b3 100644 --- a/web/js/workspace/workspace.css +++ b/web/js/workspace/workspace.css @@ -82,6 +82,15 @@ main:focus-visible { outline: 2px solid var(--ca-focus); outline-offset: 4px; } .tool-card p { flex: 1; margin: 0 0 12px; } .workspace-link { display: inline-flex; align-items: center; min-height: 44px; padding: 8px 0; font-size: 12px; overflow-wrap: anywhere; } .handoff .workspace-link { margin-top: 12px; } +/* D94: a System section not delivered yet links every legacy tool that still owns it, side by side. */ +.handoff .workspace-link + .workspace-link { margin-inline-start: 18px; /* D94 */ } + +/* D92: System's Save row is aria-disabled while nothing is unsaved (or a save runs), and a setting the last save could + not save is highlighted, as its "Review the highlighted settings" status says. */ +.save-row button[aria-disabled="true"] { opacity: .55; cursor: not-allowed; /* D92 */ } +.setting-row [aria-invalid="true"] { outline: 2px solid var(--red); outline-offset: 2px; /* D92 */ } +/* D93: an Overview row whose action is not available yet (the review queue, Q6) keeps its button, disabled. */ +.operation .row-actions button[aria-disabled="true"] { opacity: .55; cursor: not-allowed; /* D93 */ } @media (max-width: 700px) { .stat-grid { gap: 8px; } .stat { padding: 16px; } From b0a025a326b523ab88ebb3b41bc71e204e4a91b9 Mon Sep 17 00:00:00 2001 From: Vito Date: Fri, 23 Oct 2026 05:17:10 -0700 Subject: [PATCH 2/4] fix(resolver): read NaN as null in the scan's legacy-parser fallback (#304) ComfyUI stamps "is_changed": [NaN] on nodes that always re-run, the PromptManager node among them. The legacy gallery and admin pages replaced NaN with null before JSON.parse, so they traced these files' prompts. The scan's fallback parsed records strictly, so every NaN-stamped file the resolver could not decide (hires-fix and refiner chains, custom latent nodes) fell to "Needs review" although legacy linked its prompt. parse_page_json parses as the legacy pages did, but only the NaN number reads as null: the word inside prompt text is kept. Infinity still makes the record unreadable, as it did in the pages. --- py/api/admin.py | 11 +-- tests/test_generation_params.py | 45 +++++++++++++ tests/test_pair_scan.py | 116 ++++++++++++++++++++++++++++++++ utils/generation_params.py | 25 +++++++ 4 files changed, 193 insertions(+), 4 deletions(-) diff --git a/py/api/admin.py b/py/api/admin.py index 6410489..2b58b59 100644 --- a/py/api/admin.py +++ b/py/api/admin.py @@ -41,7 +41,7 @@ try: PLACEHOLDERS, GenerationParamsError, extract_generation_params, - parse_js_json, + parse_page_json, ) from ...utils.prompt_pair_resolver import resolve_prompt_pair except ImportError: @@ -51,7 +51,7 @@ except ImportError: PLACEHOLDERS, GenerationParamsError, extract_generation_params, - parse_js_json, + parse_page_json, ) from utils.prompt_pair_resolver import resolve_prompt_pair @@ -1603,11 +1603,14 @@ class AdminRoutesMixin: @staticmethod def _graph_prompt_text(metadata): - """The positive prompt the browser parser traces from the graphs, or None.""" + """The positive prompt the browser parser traces from the graphs, or None. + + Records parse as the legacy pages parsed them (NaN reads as null). + """ records = {} for key in ("prompt", "workflow"): value = metadata.get(key) - records[key] = parse_js_json(value) if isinstance(value, str) else value + records[key] = parse_page_json(value) if isinstance(value, str) else value try: positive = extract_generation_params(records)["positivePrompt"] except GenerationParamsError: diff --git a/tests/test_generation_params.py b/tests/test_generation_params.py index 990d3a6..507cea2 100644 --- a/tests/test_generation_params.py +++ b/tests/test_generation_params.py @@ -5,7 +5,10 @@ returns for the same input (tests/test_generation_params_js_parity.py checks that directly when Node is available). """ +import json import os +import shutil +import subprocess # nosec B404 - runs a fixed Node one-liner import sys import unittest @@ -17,6 +20,7 @@ from utils.generation_params import ( # noqa: E402 extract_generation_params, extract_from_records, parse_js_json, + parse_page_json, ) try: @@ -219,6 +223,47 @@ class ParsesLikeJavaScript(unittest.TestCase): run("seed-beyond-double-precision")["seed"], 4974052037430000001.0 ) + def test_page_parse_reads_nan_numbers_as_null(self): + # gallery.js and admin.js replace NaN with null before JSON.parse. + self.assertEqual( + parse_page_json('{"a": NaN, "b": [NaN], "t": "NaN"}'), + {"a": None, "b": [None], "t": "NaN"}, + ) + self.assertEqual( + parse_page_json('{"seed": 9007199254740993}'), {"seed": 9007199254740992.0} + ) + for text in ('{"a": Infinity}', "[-Infinity]", "not json", "", None): + with self.subTest(text=text): + self.assertIsNone(parse_page_json(text)) + + @unittest.skipUnless(shutil.which("node"), "Node is not installed") + def test_page_parse_equals_the_legacy_pages_for_nan_numbers(self): + # gallery.js/admin.js: two NaN regexes, then JSON.parse (Infinity throws). + texts = [ + '{"1": {"is_changed": [NaN]}, "2": {"is_changed": NaN, "v": 1.5}}', + '{"a":NaN,"b":[NaN, 2]}', + '{"a": Infinity}', + '{"seed": 9007199254740993}', + ] + script = ( + "const texts = JSON.parse(require('fs').readFileSync(0, 'utf8'));" + "console.log(JSON.stringify(texts.map((t) => { try { return {v: JSON.parse(" + "t.replace(/:\\s*NaN\\b/g, ': null').replace(/\\bNaN\\b/g, 'null'))}; }" + " catch (e) { return null; } })));" + ) + completed = subprocess.run( # nosec B603 B607 - fixed argv, no shell + ["node", "-e", script], + input=json.dumps(texts), + capture_output=True, + text=True, + check=True, + timeout=60, + ) + expected = json.loads(completed.stdout) + for text, page in zip(texts, expected): + with self.subTest(text=text): + self.assertEqual(parse_page_json(text), page and page["v"]) + def test_duplicate_keys_keep_the_last_value(self): self.assertEqual(parse_js_json('{"a": 1, "a": 2}'), {"a": 2}) diff --git a/tests/test_pair_scan.py b/tests/test_pair_scan.py index 8b65fb6..b8d914d 100644 --- a/tests/test_pair_scan.py +++ b/tests/test_pair_scan.py @@ -165,6 +165,122 @@ class TestPairScan(AdminAPITestCase): ) self.assertEqual(self.api.db.get_recent_prompts()["prompts"], []) + def _nan_scanned(self, name, prompt_graph): + """Scan ``prompt_graph`` as ComfyUI writes it: ``is_changed`` is NaN.""" + stamped = { + node_id: {**node, "is_changed": [float("nan")]} + for node_id, node in prompt_graph.items() + } + text = json.dumps(stamped) + self.assertIn("NaN", text) + return self._scanned_text(name, text) + + def _scanned_text(self, name, prompt_text): + meta = {"prompt": prompt_text} + return self.api._ingest_scanned_file(self.output_dir / name, meta) + + @staticmethod + def _hires_fix_graph(positive): + """A PromptManager prompt on a base sampler and a hires-fix sampler. + + The base latent comes from a custom batch node and the hires pass from + a custom image filter: no sampler branch traces, so the scan links the + prompt the legacy pages show. + """ + return { + "1": { + "class_type": "CheckpointLoaderSimple", + "inputs": {"ckpt_name": "m.safetensors"}, + }, + "2": { + "class_type": "MyTriggerWords", + "inputs": {"trigger_words": ["1", 2]}, + }, + "3": { + "class_type": "PromptManager", + "inputs": { + "text": positive, + "prepend_text": ["2", 0], + "append_text": "", + "clip": ["1", 1], + }, + }, + "4": { + "class_type": "CLIPTextEncode", + "inputs": {"text": "blur", "clip": ["1", 1]}, + }, + "5": {"class_type": "MyLatentBatch", "inputs": {"batch_size": 6}}, + "6": { + "class_type": "KSampler", + "inputs": { + "seed": 7, + "model": ["1", 0], + "positive": ["3", 0], + "negative": ["4", 0], + "latent_image": ["5", 0], + }, + }, + "7": { + "class_type": "VAEDecode", + "inputs": {"samples": ["6", 0], "vae": ["1", 2]}, + }, + "8": {"class_type": "MyImageFilter", "inputs": {"images": ["7", 0]}}, + "9": { + "class_type": "VAEEncode", + "inputs": {"pixels": ["8", 0], "vae": ["1", 2]}, + }, + "14": { + "class_type": "KSampler", + "inputs": { + "seed": 7, + "denoise": 0.25, + "model": ["1", 0], + "positive": ["3", 0], + "negative": ["4", 0], + "latent_image": ["9", 0], + }, + }, + "15": { + "class_type": "VAEDecode", + "inputs": {"samples": ["14", 0], "vae": ["1", 2]}, + }, + "16": {"class_type": "SaveImage", "inputs": {"images": ["15", 0]}}, + } + + def test_hires_fix_prompt_links_with_or_without_nan_is_changed(self): + # Legacy parity: the legacy pages read NaN as null before parsing, so they + # showed this prompt; the scan left NaN-stamped files in review. + for name, scan in (("plain.png", None), ("stamped.png", "nan")): + with self.subTest(name=name): + g = self._hires_fix_graph("cat on a sofa") + if scan: + status = self._nan_scanned(name, g) + else: + status = self._scanned(name, g) + self.assertIn(status, ("added", "linked")) + self.assertEqual(self._texts(), ["cat on a sofa"]) + + def test_untraced_single_sampler_links_the_legacy_prompt_with_nan(self): + g = self._hires_fix_graph("cat on a sofa") + for node_id in ("7", "8", "9", "14", "15"): + del g[node_id] + g["16"]["inputs"]["images"] = ["6", 0] + self.assertEqual(self._nan_scanned("single.png", g), "added") + self.assertEqual(self._texts(), ["cat on a sofa"]) + + def test_nan_inside_prompt_text_is_kept(self): + # Only the NaN number reads as null; legacy's regex also rewrote text. + g = self._hires_fix_graph("NaN cat, value: NaN") + self.assertEqual(self._nan_scanned("text.png", g), "added") + self.assertEqual(self._texts(), ["NaN cat, value: NaN"]) + + def test_infinity_is_still_unreadable_like_the_legacy_pages(self): + text = json.dumps(self._hires_fix_graph("cat")).replace( + '"seed": 7', '"seed": Infinity' + ) + self.assertIn(self._scanned_text("inf.png", text), ("ambiguous", "unresolved")) + self.assertEqual(self._texts(), []) + def test_raw_png_workflow_is_stored_as_object_and_repeat_is_unchanged(self): path = self.output_dir / "workflow.png" workflow = {"nodes": [{"id": 1, "type": "PromptManagerPair"}]} diff --git a/utils/generation_params.py b/utils/generation_params.py index e65ecd1..42b7c9b 100644 --- a/utils/generation_params.py +++ b/utils/generation_params.py @@ -28,6 +28,7 @@ __all__ = [ "extract_generation_params", "extract_from_records", "parse_js_json", + "parse_page_json", ] PLACEHOLDERS = MappingProxyType( @@ -953,6 +954,30 @@ def parse_js_json(text): return None +def _nan_as_null(name): + """The legacy pages' NaN cleaning; Infinity still throws in ``JSON.parse``.""" + if name == "NaN": + return None + raise ValueError(f"{name} is not JSON") + + +def parse_page_json(text): + """A metadata record as the legacy gallery and admin pages parsed it. + + They replace ``NaN`` with ``null`` and then ``JSON.parse`` (ComfyUI writes + ``"is_changed": [NaN]`` for nodes that always re-run, PromptManager among + them). Only the NaN number reads as null here: the pages' regex also + rewrote the word inside strings, which changed prompt text. None where the + parse throws. + """ + if not isinstance(text, str): + return None + try: + return json.loads(text, parse_constant=_nan_as_null, parse_int=_double_integer) + except (ValueError, RecursionError): + return None + + def extract_from_records(prompt_text, workflow_text): """``extract_generation_params`` for raw ``prompt``/``workflow`` record text. From 88e5b78207bc1581103c16b698cc2e6afd8e5974 Mon Sep 17 00:00:00 2001 From: Vito Date: Fri, 23 Oct 2026 05:42:59 -0700 Subject: [PATCH 3/4] fix(gallery): check duplicate deletes in the scanned root and refresh the listing (#305) The duplicate cleanup workflow failed in CI with the deleted file still served (200). The delete was real: the scan reads only the output root, and the browser fixture's input root holds a different shared/same.png. The test checked a bare /images/serve/ URL, which tries every gallery root, so after the output copy went it answered with the input copy. That only happened when the kernel stamped a bulk copy a fraction of a millisecond before the wall-clock stamp on shared/same.png, which made shared/same.png a Delete candidate (CI trace: 0000 .8605533, same .8606777, 0001 .8618972). The workflow now checks each file in its own root (?root=N), checks the input copy survives, and a new workflow forces the CI order through the scan response so the collision runs every time. The server log in that failure also showed the gallery listing a file the cleanup had deleted: delete_duplicate_images left the 30 s listing cache, so the gallery kept counting the file. A successful delete now invalidates the gallery cache. --- py/api/admin.py | 3 + tests/browser/workflows.spec.mjs | 100 +++++++++++++++++++++++-------- tests/test_api_admin.py | 17 ++++++ 3 files changed, 96 insertions(+), 24 deletions(-) diff --git a/py/api/admin.py b/py/api/admin.py index 2b58b59..dc4e9bc 100644 --- a/py/api/admin.py +++ b/py/api/admin.py @@ -804,6 +804,9 @@ class AdminRoutesMixin: result = await self._run_in_executor( self._delete_duplicate_images_sync, image_paths, output_dir ) + if result["deleted_count"]: + # The cached listing would keep (and count) the deleted files. + self.invalidate_gallery_cache() response_data = { "success": True, diff --git a/tests/browser/workflows.spec.mjs b/tests/browser/workflows.spec.mjs index 0ee4620..ad42f37 100644 --- a/tests/browser/workflows.spec.mjs +++ b/tests/browser/workflows.spec.mjs @@ -354,39 +354,48 @@ test('prompt deletion requires confirmation and can retry after a network failur } }); -test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => { +// The duplicate scan reads the output root only, but the fixture's input root holds a different +// shared/same.png. A bare /images/serve/ URL tries every gallery root in turn, so once the +// output copy is deleted it answers 200 with the input copy: check each file in its own root. +const serveIn = (root, path) => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}?root=${root}`; +const rootOf = url => new URL(url, 'http://gallery.test').searchParams.get('root'); + +// Which copy the scan keeps depends on file times: the fixture stamps shared/same.png with the +// wall clock, and the kernel can stamp its bulk copies a fraction of a millisecond earlier. +// listOrder rewrites the finished scan's groups so a test can pick the order it needs. +async function openDuplicateScan(page, request, listOrder = null) { await expect(await request.post('/__test__/reset', { data: { extra_images: 2 } })).toBeOK(); + if (listOrder) { + await page.route('**/prompt_manager/scan_duplicates/stream', async route => { + const response = await route.fetch(); + const body = (await response.text()).replace(/^data: (.*)$/gm, (line, json) => { + const event = JSON.parse(json); + if (event.type !== 'complete') return line; + for (const group of event.duplicates) group.images = listOrder(group.images); + return `data: ${JSON.stringify(event)}`; + }); + await route.fulfill({ response, body }); + }); + } await page.goto('/prompt_manager/gallery'); await expect(page.locator('#galleryGrid .image-item')).toHaveCount(4); const originals = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original)); + const output = rootOf(originals.find(url => url.includes('/bulk/'))); + const input = rootOf(originals.find(url => url.includes('/shared/same.png') && rootOf(url) !== output)); + expect(output).not.toBeNull(); + expect(input).not.toBeNull(); await page.locator('#settingsBtn').click(); await page.locator('#scanDuplicatesBtn').click(); await expect(page.locator('#duplicatesCount')).toHaveText('1'); const candidates = page.locator('.duplicate-checkbox'); await expect(candidates).toHaveCount(2); - const selected = await candidates.first().getAttribute('data-image-path'); - const retained = await candidates.last().getAttribute('data-image-path'); - const serve = path => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}`; - await candidates.first().check(); - let deletes = 0; - page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; }); - confirmNextDialog(page, false, 'permanently delete 1 duplicate image files'); - await page.locator('#removeDuplicatesBtn').click(); - expect(deletes).toBe(0); - await expect(await request.get(serve(selected))).toBeOK(); - await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 }); - confirmNextDialog(page, true, 'permanently delete 1 duplicate image files'); - await page.locator('#removeDuplicatesBtn').click(); - await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible(); - await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled(); - await expect(await request.get(serve(selected))).toBeOK(); - confirmNextDialog(page, true, 'permanently delete 1 duplicate image files'); - const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images')); - await page.locator('#removeDuplicatesBtn').click(); - expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 }); - await expect(page.locator('#duplicatesModal')).toBeHidden(); - expect((await request.get(serve(selected))).status()).toBe(404); - await expect(await request.get(serve(retained))).toBeOK(); + return { originals, output, input, candidates }; +} + +async function expectOnlySelectedGone(page, request, { originals, output, input }, selected, retained) { + expect((await request.get(serveIn(output, selected))).status()).toBe(404); + await expect(await request.get(serveIn(output, retained))).toBeOK(); + await expect(await request.get(serveIn(input, 'shared/same.png'))).toBeOK(); await page.reload(); await expect(page.locator('#galleryGrid .image-item')).toHaveCount(3); const remaining = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original)); @@ -395,4 +404,47 @@ test('duplicate cleanup cancels safely and deletes only the selected file after expect(originals).toContain(url); await expect(await request.get(url)).toBeOK(); } +} + +test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => { + const scan = await openDuplicateScan(page, request); + const { output, candidates } = scan; + const selected = await candidates.first().getAttribute('data-image-path'); + const retained = await candidates.last().getAttribute('data-image-path'); + await candidates.first().check(); + let deletes = 0; + page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; }); + confirmNextDialog(page, false, 'permanently delete 1 duplicate image files'); + await page.locator('#removeDuplicatesBtn').click(); + expect(deletes).toBe(0); + await expect(await request.get(serveIn(output, selected))).toBeOK(); + await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 }); + confirmNextDialog(page, true, 'permanently delete 1 duplicate image files'); + await page.locator('#removeDuplicatesBtn').click(); + await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible(); + await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled(); + await expect(await request.get(serveIn(output, selected))).toBeOK(); + confirmNextDialog(page, true, 'permanently delete 1 duplicate image files'); + const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images')); + await page.locator('#removeDuplicatesBtn').click(); + expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 }); + await expect(page.locator('#duplicatesModal')).toBeHidden(); + await expectOnlySelectedGone(page, request, scan, selected, retained); +}); + +test('deleting a duplicate whose path another gallery root also holds removes only the scanned copy', async ({ page, request }) => { + // The order CI saw: a bulk copy stamped first, so shared/same.png is the first Delete candidate. + const ciOrder = ['bulk/0000.png', 'shared/same.png', 'bulk/0001.png']; + const rank = image => ciOrder.indexOf(image.path.replace(/\\/g, '/')); + const scan = await openDuplicateScan(page, request, images => [...images].sort((a, b) => rank(a) - rank(b))); + const selected = await scan.candidates.first().getAttribute('data-image-path'); + const retained = await scan.candidates.last().getAttribute('data-image-path'); + expect(selected.replace(/\\/g, '/')).toBe('shared/same.png'); + await scan.candidates.first().check(); + confirmNextDialog(page, true, 'permanently delete 1 duplicate image files'); + const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images')); + await page.locator('#removeDuplicatesBtn').click(); + expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 }); + await expect(page.locator('#duplicatesModal')).toBeHidden(); + await expectOnlySelectedGone(page, request, scan, selected, retained); }); diff --git a/tests/test_api_admin.py b/tests/test_api_admin.py index 61d703b..b34c0df 100644 --- a/tests/test_api_admin.py +++ b/tests/test_api_admin.py @@ -868,6 +868,23 @@ class TestDeleteDuplicatesEndpoint(AdminAPITestCase): self.assertFalse(target.exists()) self.assertTrue(thumb_dir.exists()) + async def test_deleted_files_leave_the_cached_gallery_listing(self): + # The listing is cached for 30 s; a delete must not leave the gallery + # listing (and counting) a file that is gone. + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + for name in ("keep.png", "gone.png"): + (self.output_dir / name).write_bytes(b"same bytes") + + async def listed(): + resp = await self.client.get("/prompt_manager/images/output") + data = await resp.json() + return data["total"], sorted(i["filename"] for i in data["images"]) + + self.assertEqual(await listed(), (2, ["gone.png", "keep.png"])) + resp = await self._post({"image_paths": ["gone.png"]}) + self.assertEqual((await resp.json())["deleted_count"], 1) + self.assertEqual(await listed(), (1, ["keep.png"])) + def test_unexpected_error_per_file_is_reported_without_path(self): self.api._delete_one_duplicate = lambda *a: (_ for _ in ()).throw( PermissionError(13, "Permission denied", os.path.join(self.tmpdir, "x.png")) From add035d94061956f957b6533848e00e8dce94d8d Mon Sep 17 00:00:00 2001 From: Vito Date: Fri, 23 Oct 2026 06:08:44 -0700 Subject: [PATCH 4/4] fix(images): release a collection viewer image Viewer.js has not built yet (#293) * fix(images): release a collection viewer image Viewer.js has not built yet Leaving a viewer image while it was still loading (Previous, Next, the arrow keys, Retry) deferred the inline Viewer.js destroy() until the image's view settled, failed or a 10 s fallback ran. Before its image loads Viewer.js has not built, so nothing could settle: the stage dropped the image, the late load ran Viewer.js's build() on an image without a parent, and getComputedStyle(null) threw a page error ("Argument 1 is not an object" in Firefox, "must be an instance of Element" in WebKit). Under CI load this failed "collection viewer follows filtered loaded order beyond mounted cards". A viewer that is not ready yet is now destroyed at once, which cancels its pending build, and its image src is released. A built viewer with a still-loading view keeps the existing deferral. * fix(images): build Viewer.js only on a loaded viewer image Destroying an unbuilt inline Viewer.js at once (429b3a8) is not safe in one state. Viewer.js's own cancel removes its image listeners only while `image.complete` is false, so an image that had finished downloading with its load event still queued kept them. Closing the viewer then released the src, the queued load built Viewer.js anyway, and its first view found no listed image: "can't access property querySelector, c is undefined" (workspace-history.spec.mjs:222 on Firefox in CI). The viewer now constructs Viewer.js from its own load listener, which the media scope removes when the image is left. Before the load no Viewer.js exists, so leaving the image only releases its src; after it Viewer.js sees a complete image and only its zero-delay build is pending, which destroy() cancels. Transforms report Viewer.js as unavailable only when it did not load at all. * fix(images): describe transforms by the Viewer.js an image was built with 1086436 creates Viewer.js only once the image has loaded, so the "Viewer.js could not load" reason moved from the instance to the live window.Viewer. An image shown natively while Viewer.js was missing then offered transforms as soon as Viewer.js reappeared, which full-inspector.spec.mjs:854 ("collection viewer preserves original download and rooted retries") caught on every project. Each rebuild now captures the Viewer.js it builds with. The reason, the native fallback and the later construction all use that capture. --- tests/browser/workspace-images.spec.mjs | 44 +++++++++++++++++++++++ tests/js/workspace-media-viewer.test.js | 47 +++++++++++++++++++++---- web/js/workspace/media-viewer.mjs | 34 ++++++++++++------ 3 files changed, 107 insertions(+), 18 deletions(-) diff --git a/tests/browser/workspace-images.spec.mjs b/tests/browser/workspace-images.spec.mjs index 4f544cf..5727a80 100644 --- a/tests/browser/workspace-images.spec.mjs +++ b/tests/browser/workspace-images.spec.mjs @@ -463,6 +463,50 @@ test('collection viewer ignores delayed media callbacks and keeps inspection ret await page.evaluate(()=>{location.hash='/overview';});await expect(page.locator('.viewer-container,#viewer-dialog')).toHaveCount(0); }); +test('collection viewer leaving an image Viewer.js has not built yet releases it at once',async({page})=>{ + await page.goto(workspace);await expect(cards(page)).toHaveCount(2); + let release,held;const waiting=new Promise(resolve=>{held=resolve;});const pending=new Promise(resolve=>{release=resolve;}); + await page.route('**/images/serve/shared/same.png?*',async route=>{ + const url=new URL(route.request().url()); + if(url.searchParams.get('root')==='0'&&url.searchParams.has('_pm_viewer')) {held();await pending;} + await route.continue().catch(()=>{}); + }); + await cards(page).first().getByRole('button',{name:'Open same.png',exact:true}).click();await waiting; + const preview=page.locator('dialog#viewer-dialog'); + await page.evaluate(()=>{window.unbuiltImage=document.querySelector('#viewer-stage > img');}); + await preview.getByRole('button',{name:'Next media',exact:true}).click(); + await expect(preview.getByRole('button',{name:'Zoom in',exact:true})).toHaveAttribute('aria-disabled','false'); + // The left image is off the stage. Viewer.js has not built it, so the viewer is destroyed and the src released at + // once: its late load must never build Viewer.js into an image without a parent (getComputedStyle(null)). + release(); + await expect.poll(()=>page.evaluate(()=>[window.unbuiltImage.isConnected,window.unbuiltImage.hasAttribute('src'),!!window.unbuiltImage.viewer])).toEqual([false,false,false]); + await page.evaluate(()=>{location.hash='/overview';});await expect(page.locator('.viewer-container,#viewer-dialog')).toHaveCount(0); +}); + +test('collection viewer closed while its image finishes loading never builds Viewer.js afterwards',async({page})=>{ + await page.goto(workspace);await expect(cards(page)).toHaveCount(2); + let held;const waiting=new Promise(resolve=>{held=resolve;}); + // The image is held for the whole test: its late load is delivered below, at the moment Viewer.js cannot cancel. + await page.route('**/images/serve/shared/same.png?*',async route=>{ + const url=new URL(route.request().url()); + if(url.searchParams.get('root')==='0'&&url.searchParams.has('_pm_viewer')) {held();return;} + await route.continue().catch(()=>{}); + }); + await cards(page).first().getByRole('button',{name:'Open same.png',exact:true}).click();await waiting; + const preview=page.locator('dialog#viewer-dialog'); + // Viewer.js's own cancel removes its image listeners only while `image.complete` is false. An image that has + // finished downloading but whose load event is still queued is already complete: close in that state, then deliver it. + await page.evaluate(()=>{window.finishingImage=document.querySelector('#viewer-stage > img');Object.defineProperty(window.finishingImage,'complete',{get:()=>true});}); + await page.keyboard.press('Escape');await expect(preview).toHaveCount(0); + const after=await page.evaluate(async()=>{ + const image=window.finishingImage;image.dispatchEvent(new Event('load')); + // Viewer.js builds an inline viewer in a zero-delay timeout after its image loads; this runs after it. + await new Promise(resolve=>setTimeout(resolve)); + return {src:image.hasAttribute('src'),built:!!image.parentNode?.querySelector('.viewer-container')}; + }); + expect(after).toEqual({src:false,built:false}); +}); + test('collection viewer describes video transforms and releases native playback on navigation',async({page,request})=>{ await expect(await request.post('/__test__/reset',{data:{include_video:true}})).toBeOK();await page.goto(workspace); await page.locator('article[data-image-key="0:clip.webm"] .image-open').click(); diff --git a/tests/js/workspace-media-viewer.test.js b/tests/js/workspace-media-viewer.test.js index 749694a..d375903 100644 --- a/tests/js/workspace-media-viewer.test.js +++ b/tests/js/workspace-media-viewer.test.js @@ -124,18 +124,51 @@ test('real inspection navigation preserves slideshow across two advances and def } finally {view.dispose();controller.dispose();parentScope.dispose();globalThis.setTimeout=oldSetTimeout;globalThis.clearTimeout=oldClearTimeout;f.restore();} }); -test('an inline Viewer still loading its image is destroyed only after it has viewed it', async () => { +test('Viewer.js is built only on a loaded image and an unbuilt viewer is released at once', async () => { const {reportedLocationAdapter:adapter} = await import('../../web/js/workspace/history-media.mjs'); - const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c')]; + const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c'),occurrenceMedia('2',0,'d')]; const instances = []; - // Viewer.js's inline destroy() leaves an in-flight view running; its callback then needs a live instance. - f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.viewed = false; instances.push(this); } + // On a loaded image Viewer.js schedules its inline build; until it runs `ready` is false and destroy() cancels it. + f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.ready = false; this.viewed = false; instances.push(this); } destroy() { this.destroyed++; } resize() {} }; const globals = {ResizeObserver:globalThis.ResizeObserver, requestAnimationFrame:globalThis.requestAnimationFrame, cancelAnimationFrame:globalThis.cancelAnimationFrame}; Object.assign(globalThis, {ResizeObserver:class { observe() {} disconnect() {} }, requestAnimationFrame:() => 1, cancelAnimationFrame() {}}); const view = module.createMediaViewer({container:f.container,parentScope,inspectionController:controller,adapter,sender:{open(){}}}); try { - view.open({items,selectedKey:'c'.repeat(64)}); + const stageImage = () => f.walk(f.container,n => n.tagName === 'img')[0]; + view.open({items,selectedKey:'d'.repeat(64)}); + // Viewer.js's own cancel misses an image whose load event is still queued, so none exists before the load. + const unloaded = stageImage(), dropped = [];unloaded.removeAttribute = key => dropped.push(key); + assert.equal(instances.length, 0, 'no Viewer.js before the image loads'); + f.walk(f.container,n => n.attributes['aria-label'] === 'Previous media')[0].fire('click'); + assert.deepEqual(dropped, ['src'], 'an image left before it loads is released at once'); + unloaded.fire('load'); + assert.equal(instances.length, 0, 'its late load builds no Viewer.js'); + stageImage().fire('load'); + const [loading] = instances, image = loading.node, removed = []; + image.removeAttribute = key => removed.push(key); + // Next drops the image whose build is pending; a deferred destroy would let that build run there. + f.walk(f.container,n => n.attributes['aria-label'] === 'Next media')[0].fire('click'); + stageImage().fire('load');assert.equal(instances.length, 2, 'the next image builds its own Viewer.js once loaded'); + assert.equal(loading.destroyed, 1, 'an unbuilt viewer is destroyed at once'); + assert.deepEqual(removed, ['src'], 'and its image is released at once'); + loading.options.viewed.call(loading); + assert.equal(loading.destroyed, 1, 'never destroyed twice'); + } finally { view.dispose(); parentScope.dispose(); f.restore(); Object.assign(globalThis, globals); } +}); + +test('an inline Viewer still loading its image is destroyed only after it has viewed it', async () => { + const {reportedLocationAdapter:adapter} = await import('../../web/js/workspace/history-media.mjs'); + const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c')]; + const instances = []; + // A built (`ready`) inline Viewer.js's destroy() leaves an in-flight view running; its callback then needs a live instance. + f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.ready = true; this.viewed = false; instances.push(this); } + destroy() { this.destroyed++; } resize() {} }; + const globals = {ResizeObserver:globalThis.ResizeObserver, requestAnimationFrame:globalThis.requestAnimationFrame, cancelAnimationFrame:globalThis.cancelAnimationFrame}; + Object.assign(globalThis, {ResizeObserver:class { observe() {} disconnect() {} }, requestAnimationFrame:() => 1, cancelAnimationFrame() {}}); + const view = module.createMediaViewer({container:f.container,parentScope,inspectionController:controller,adapter,sender:{open(){}}}); + try { + view.open({items,selectedKey:'c'.repeat(64)});f.walk(f.container,n => n.tagName === 'img')[0].fire('load'); const [loading] = instances, image = loading.node, removed = []; image.removeAttribute = key => removed.push(key); f.walk(f.container,n => n.attributes['aria-label'] === 'Close image viewer')[0].fire('click'); @@ -148,7 +181,7 @@ test('an inline Viewer still loading its image is destroyed only after it has vi loading.options.viewed.call(loading); assert.equal(loading.destroyed, 1, 'never destroyed twice'); - view.open({items,selectedKey:'c'.repeat(64)}); + view.open({items,selectedKey:'c'.repeat(64)});f.walk(f.container,n => n.tagName === 'img')[0].fire('load'); const shown = instances[1]; shown.viewed = true; shown.options.viewed.call(shown); f.walk(f.container,n => n.attributes['aria-label'] === 'Close image viewer')[0].fire('click'); assert.equal(shown.destroyed, 1, 'a viewed image is destroyed at once'); @@ -236,7 +269,7 @@ test('viewer uses adapter.mediaUrl for image and video', async () => { const img = f.walk(f.container,n => n.tagName === 'img')[0]; assert.equal(img.src, image.url, 'the blob URL is used exactly'); assert.doesNotMatch(img.src, /\?/); - assert.equal(instances[0].node, img, 'Viewer.js shows the blob image'); + img.fire('load');assert.equal(instances[0].node, img, 'Viewer.js shows the blob image'); const download = f.walk(f.container,n => n.attributes['aria-label'] === 'Download original')[0]; assert.equal(download.href, image.url);assert.equal(download.download, 'a.png'); f.walk(f.container,n => n.attributes['aria-label'] === 'Next media')[0].fire('click'); diff --git a/web/js/workspace/media-viewer.mjs b/web/js/workspace/media-viewer.mjs index 4427dd6..bd1b323 100644 --- a/web/js/workspace/media-viewer.mjs +++ b/web/js/workspace/media-viewer.mjs @@ -35,7 +35,7 @@ export function createMediaViewer({api,container,parentScope,inspectionControlle if (scope.signal.aborted) return; active?.close(false); const {items,selectedKey,trigger = document.activeElement} = options; - let mounted = false, initial, entry, currentKey, viewer = null, mediaScope = null, closed = false; + let mounted = false, initial, entry, currentKey, viewer = null, withoutViewer = false, mediaScope = null, closed = false; let originalDownload = null, inspectionState = inspectionController.state; const session = createViewerSession({adapter,items,initialKey:selectedKey,onChange:(state,selected) => { if (!mounted) { initial = [state,selected]; return; } @@ -115,7 +115,9 @@ export function createMediaViewer({api,container,parentScope,inspectionControlle mediaScope?.dispose();mediaScope = createScope();viewer = null; for (const child of [...stage.children]) if (child !== failure) child.remove(); failure.hidden = true; - const mediaLocal = mediaScope, key = session.state.key; + const mediaLocal = mediaScope, key = session.state.key, ViewerJs = window.Viewer; + // Transforms follow the Viewer.js this media was built with, which builds once the image has loaded. + withoutViewer = !ViewerJs; const current = () => !local.signal.aborted && !mediaLocal.signal.aborted && session.state.key === key; // An adapter's own media URL (a session-owned blob:) is used exactly, so a retry reuses it; a server // original gets a fresh cache-buster on every build. @@ -142,15 +144,23 @@ export function createMediaViewer({api,container,parentScope,inspectionControlle },{once:true}); } else { node.alt = selected.filename;node.decoding = 'async'; - node.hidden = !!window.Viewer; + node.hidden = !!ViewerJs; // With Viewer.js the src is released after the viewer (see release below). - if (!window.Viewer) { + if (!ViewerJs) { mediaLocal.listen(node,'load',() => {if (current()) session.ready(key);}); mediaLocal.cleanup(() => node.removeAttribute('src')); } } node.src = src;stage.append(node); - if (!selected.is_video && window.Viewer) { + if (!selected.is_video && ViewerJs) { + // Viewer.js is built on an image that has loaded. Its own cancel removes its image listeners only + // while `image.complete` is false, so an image left with its load event still queued would build a + // viewer into the released image after destroy() (#293). Until then no viewer exists to cancel. + let mountedViewer = false; + mediaLocal.listen(node,'load',() => { if (current()) { mountedViewer = true;mountViewer(); } },{once:true}); + mediaLocal.cleanup(() => { if (!mountedViewer) node.removeAttribute('src'); }); + } + function mountViewer() { let fitFrame = 0, fitWidth, fitHeight, viewSettled = false, released = false, fallback = 0; const release = () => { if (released) return; @@ -166,7 +176,7 @@ export function createMediaViewer({api,container,parentScope,inspectionControlle fitWidth = width;fitHeight = height;instance.resize(); }); }; - const instance = new window.Viewer(node,{inline:true,toolbar:false,navbar:false,title:false,button:false, + const instance = new ViewerJs(node,{inline:true,toolbar:false,navbar:false,title:false,button:false, keyboard:false,fullscreen:false,transition:false,viewed() { viewSettled = true; if (mediaLocal.signal.aborted) { release(); return; } @@ -176,10 +186,12 @@ export function createMediaViewer({api,container,parentScope,inspectionControlle const observer = new ResizeObserver(scheduleFit);observer.observe(stage); viewer = instance;mediaLocal.cleanup(() => { observer.disconnect();cancelAnimationFrame(fitFrame); - // Inline Viewer.js keeps viewing a still-loading image after destroy(), and that view - // rebuilds its list from images that still have a src; keep both until the view - // settles, fails or times out. - if (viewSettled) { release(); return; } + // Before Viewer.js has built (its zero-delay build is pending), destroy() cancels that build; + // deferring it would let the build run into the image the stage has just dropped. + // Once built, inline Viewer.js keeps viewing a still-loading image after destroy(), and that + // view rebuilds its list from images that still have a src; keep both until the view settles, + // fails or times out. + if (viewSettled || !instance.ready) { release(); return; } node.addEventListener('error',release,{once:true}); fallback = setTimeout(release,PENDING_VIEW_DESTROY_MS); }); @@ -188,7 +200,7 @@ export function createMediaViewer({api,container,parentScope,inspectionControlle } function updateControls(state) { const unavailable = inspectionState.refreshRequired ? inspectionState.message : state.phase === 'error' ? 'Media could not be loaded. Retry this file.' : entry.is_video ? - 'Image transforms are unavailable for video.' : !viewer ? + 'Image transforms are unavailable for video.' : withoutViewer ? 'Image transforms are unavailable because Viewer.js could not load.' : !state.mediaReady ? 'Image transforms are available when the image is ready.' : ''; reason.textContent = unavailable;reason.hidden = !unavailable; for (const node of transforms) {