diff --git a/TESTING.md b/TESTING.md index b36d73f9..ddef2ca1 100644 --- a/TESTING.md +++ b/TESTING.md @@ -2460,6 +2460,29 @@ files to the system trash. **No code path unlinks or overwrites a media file.** back records of files that are in the trash (Missing-file links cleans them up). The legacy gallery lists every file on disk, removed or not. Hard links have one mark per path, and on FAT/exFAT/SMB a same-size replacement can stay hidden. + Since #501 the thumbnail jobs (System › Thumbnails' counts and its missing, rebuild and + convert runs, and the previews after a scan) and the legacy `GET /scan_output_dir` + listing skip hidden files too; an on-demand preview the legacy gallery asks for is still + made, and a one-image auto-tag (`/autotag/single` + `/autotag/apply`) is an explicit + write that shows the file again. System › Maintenance **Missing-file links** also + forgets the marks that hide nothing any more (the file is gone from a folder that still + exists, or its bytes changed), counted in its scope ("K removed images whose files are + gone or changed") and job line, judged only under a source that is readable and lists + something (an unplugged drive's empty mount point keeps its marks), never for a source + no longer configured, and re-checked at apply (`plan-changed`). The sidebar's Images, + Favorites and Needs review counts and the collection counts are read again when an + `image_delete` job ends in the tab that followed it (`JOB_END_COUNTS`, `state.mjs`). + Tests: `tests/test_image_removal_gaps.py`, the sidebar case in + `tests/browser/workspace-images-delete.spec.mjs`. +- **Trash folders inside an output root** (#501, `utils/media_folders.py`): no walker + descends a folder below a root named `.Trash`, `.Trash-`, `.Trashes` or + `$RECYCLE.BIN` (any case), as none descends `thumbnails`: the workspace and legacy + listings (`_walk_media`, `_settled_entry`), the metadata scan and duplicate find + (`_collect_media_files`), the legacy auto-tag listing (`_collect_scan_files`) and the + delete review (`trash-folder`). The image monitor tests every folder of the new file's + absolute path, so it also ignores outputs written into a source that itself lies + inside such a folder. Tests: `tests/test_media_folders.py`, + `tests/test_image_removal_gaps.py`. - **Trash backends** (`utils/trash.py`, no dependency): Linux and BSD implement the freedesktop Trash specification (home trash on the same mount, else the mount's `.Trash/$uid` or `.Trash-$uid`), reserve the name with `O_EXCL` and move with @@ -2476,8 +2499,12 @@ files to the system trash. **No code path unlinks or overwrites a media file.** predicted (`URLForDirectory` answers 3328/`ENOENT` until macOS makes it). A refused move keeps the file and logs the `NSError`; a move macOS reports done but whose place cannot be shown (no result URL, or one inside an output root) goes under `errors` as - `trash-unconfirmed`, keeps its records and stops the job. Windows reports `no-trash` - until #500; library-only removal works everywhere. Tests use `tests/fake_trash.py`; the + `trash-unconfirmed`, keeps its records and stops the job. Windows + never uses the Recycle Bin (#500, maintainer ruling (a)): files larger than the bin's + capacity are deleted permanently and IFileOperation's `PreDeleteItem` guard cannot detect + it (native probe runs 40951037575, 40953347904), so every file answers `windows-recycle` + ("Moving files to the Recycle Bin isn't available on Windows: files larger than the bin's + capacity would be deleted permanently."). Library-only removal works everywhere. Tests use `tests/fake_trash.py`; the browser test server installs it, so no test touches a real trash. The native proof, `tests/test_trash_macos_native.py`, runs only with `PM_NATIVE_TRASH=macos` in portability.yml's `native-trash` job (`macos-latest` and `macos-15-intel`): the startup @@ -2610,7 +2637,8 @@ Category". It is built only from the concept's Images parts: input[type=checkbox]`); Home, End and the arrow keys still move between titles. - `div#library-bulkbar` sits between the toolbar and the caption, empty until something is checked, then the concept's `.bulkbar`: "N selected", Set category, Clear (D151 - adds Add tags before Set category, in the Images bar's order). The + adds Add tags before Set category, in the Images bar's order; D152 adds Delete… before + Clear, and Restore alone on the archived shelf). The selection is a set of prompt ids over the loaded shelf: windowed-out cards keep their state, Load more keeps it, and any shelf reload (search, category, sort, tags, a write) and leaving the page clear it. Clicking a checkbox toggles it and sets the @@ -2691,6 +2719,97 @@ during a write); `tests/browser/library-concept.spec.mjs` (the bar comparison no Add tags); units `tests/js/workspace-library-selection.test.js`; Python `tests/test_bulk_library_tags.py`. A Python change: ComfyUI needs a restart. +### Deleting and archiving selected prompts (#443) + +**D152** (beyond the concept, on D148's bar; the concept archives a prompt from its recipe +editor's "Archive…", `app.js` 976, reviewed in `maintenancePlan`, 953–958, and maps the legacy +delete to a "reference-safe archive", 783): the Prompt library deletes or archives the +selected prompts in one reviewed step, replacing the legacy admin page's bulk and single +Delete, which refuses a whole batch when any prompt belongs to a saved pair (maintainer +rulings of 2026-10-28): + +- The bar reads "N selected", [Select all N loaded], Add tags, Set category, **Delete…** + (the concept's `danger` class, as its Images bar's "Remove links…"), Clear. +- Delete… posts `{prompt_ids, expected_epoch}` to `POST + /prompt_manager/library/delete/preview`, which classifies each id in this order: not + found (missing or not a Library parent), protected (tagged `__protected__`), already + archived, archive (the positive or negative of any saved pair, archived pair or not), + delete (everything else). From the press until the review opens, every bar action, Clear + too, is `aria-disabled` and Delete… reads "Reviewing…" (a double press sends one preview). +- The review is the shared record-changing review (`openReviewDialog`), titled "Delete + selected prompts", "Archive selected prompts" or "Delete or archive selected prompts", its + `dl.meta-grid` rows Delete permanently ("N prompts not in a saved pair") and Archive ("N prompts in a saved + pair: hidden, restorable", never negative variants: a pair's prompt with none is archived too) (always), Protected (kept) with up to + three titles, Already archived, No longer exist, and, when something is deleted, Image + links removed and Backup ("Create and verify before deleting", or "Off: no safety backup + (System › Backup & export › Recovery policy)"); then `p.small` sentences: why paired + prompts are archived and how to restore them, what a delete removes and that it cannot be + undone except by restoring the safety backup (which returns the whole library to that + moment; kept among the last 10), that a later scan, new image or node run with the same + text adds a deleted prompt again, and how to unprotect. The primary is `danger`: "Delete D + prompts", "Archive A prompts" or "Delete D and archive A prompts". When nothing can change, + or the disk has no room for the promised backup, the review is close-only with the reason. + The concept's "Review maintenance plan" title, its demo note, Scope/Protected/Media on + disk rows and `ca-primary` start are replaced by these. +- Start posts the reviewed lists `{archive_ids, delete_ids, expected_epoch, backup}` to `POST + /prompt_manager/library/delete` while Delete… reads "Deleting…" (or "Archiving…"). The + server takes a `records` hold ("Delete prompts (Library)", refused 409 `busy` beside a scan, + restore, maintenance, move or Optimize), checks the disk (507 `insufficient-space`) and, + when a delete remains and the reviewed `backup` was "on" or the policy is on now, takes a + verified safety backup (500 `backup-failed`, nothing changed). One `BEGIN IMMEDIATE` + transaction then deletes only reviewed deletes still unpaired whose prompt row, tags, + title and image links are unchanged since the backup (exactly the legacy rows: image + link rows, cascaded tags, Library state and pair links of images, released workflow + documents; files never), archives the paired ones and any reviewed delete now paired, and + counts the rest (`already_archived`, `not_found`, `changed`). Archiving writes a + `prompt_archive` row, the prompt's `updated_at` and, when its stored hash is stale or + missing and no other row holds the canonical one, the canonical hash, so scans, the + monitor and node saves find the archived row instead of adding a copy. A changed epoch + answers 409 `storage-changed`; nothing is written. +- Outcomes: "Deleted D prompts." "Archived A prompts." with already archived, no longer + exist, changed and kept-protected sentences and "Safety backup saved: NAME (System › + Backup & export)."; the shelf, categories, cached Library views and sidebar counts reload. + Storage restored: "The library was restored or moved since this page loaded. Nothing was + deleted or archived."; busy: "NAME is running. Nothing was changed; try again when it + finishes."; a failed backup or a short disk: the server's sentence; no answer, 502, 503 or + 504: "The server did not confirm the change. The Library was reloaded: check the shelf and + Show archived prompts."; other refusals: "The selected prompts could not be changed. Try + again.", the selection kept. +- **Show archived prompts (N)** (heading actions, before Export, `aria-pressed`; N the + archived prompts matching the filters, `archived_total`; the shelf counterpart of #235's + Show archived) lists only archived prompts (`GET /prompt_manager/library?shelf=archived`, + a cursor never crosses shelves), each card with an "Archived" `.chip`, the caption "N + archived prompts · M negative variants hidden with them", the bar "N selected", Restore, + Clear, and "No archived prompts." / "No archived prompts match." when empty. The pressed + state stays with the shelf visit, never in a saved view; Browse and a saved view open the + shelf. Restore posts `{prompt_ids, expected_epoch}` to `POST + /prompt_manager/library/restore` (every requested archived id, protected or not): "Restored + N prompts to the Prompt library." +- An archived prompt's page shows the concept's `.notice` ("Archived" / "This prompt is archived + because it is in a saved pair. It and its N negative variants are hidden from the Prompt library; nothing was + removed.", or "It is hidden from…" when it has none) with Restore; + everything else on the page works. New prompt or Add negative on an archived prompt's + text keeps it archived and toasts "This text is an archived prompt. Restore it to show it + in the Prompt library again." +- Archived prompts: hidden from the shelf, its search, `total`, the sidebar count, Tags & + categories' prompt counts (a value only archived prompts hold stays listed with 0) and + System › Overview and the restore review's positive counts; still counted by rename and + delete impacts and usage statistics; still shown as another prompt's negative variant, on + recorded runs and Images, by the legacy pages and the ComfyUI node's search; still reached + by run counting, scans and the monitor; never pruned by maintenance (a new protection, + "archived in the Library"); exported with `archived: true`. Older builds ignore the table + and show them again; the rows stay. +- Deviation from legacy: a selected `__protected__` prompt is never deleted or archived (the + legacy delete removed it); it is listed in the review and the toast. + +Tests: `tests/browser/workspace-library-delete.spec.mjs` (mixed review and result on the +server, Cancel, Show archived prompts and Restore from the bar and the notice, double press, +failed backup, busy, unconfirmed, restore fence); `tests/browser/library-delete-concept.spec.mjs` +(the review against the concept's `maintenancePlan`, the bar's Delete… against the concept's +"Remove links…"); units `tests/js/workspace-library-delete.test.js`; Python +`tests/test_prompt_archive_schema.py`, `tests/test_bulk_library_delete.py`, +`tests/test_library_delete_routes.py`. A Python change: ComfyUI needs a restart. + ### No 1–5 prompt rating in the workspace (#447) **D153** (maintainer ruling 2026-10-28: the star is only for favorites; the concept keeps @@ -3915,8 +4034,8 @@ the per-section hand-off panel, was retired by S12 (#329): every section is nati - **D104** (Backup & export, S9, Q4): Recovery policy keeps the concept's "Proposed" chip. "Create a safety backup before maintenance" is the server's `safety_backup` (`maintenance.safety_backup`) policy. Its line says what it does ("Keep a verified copy - of the database on disk before maintenance, cleanups and LoRA imports change records. A - restore always keeps one."); the concept's "Preview a verified backup before applying + of the database on disk before maintenance, cleanups, LoRA imports and prompt deletes + change records. A restore always keeps one."); the concept's "Preview a verified backup before applying operations that change records." describes nothing the server does. It is drafted behind the Save row and saved with `expected_revision` (a configuration changed elsewhere is the conflict status); it is disabled with "Server settings could not be read." when the read failed. "Scheduled diff --git a/database/image_removal.py b/database/image_removal.py index 3360433c..58099e03 100644 --- a/database/image_removal.py +++ b/database/image_removal.py @@ -144,6 +144,23 @@ def removed_from_library(conn, path) -> bool: return False +def mark_hides(marks: dict, locator: str, path) -> bool: + """Whether a removal mark in *marks* hides the file at *path* (blocking). + + *marks* is ``{locator: (size, mtime_ns)}``. The mark of *locator* hides + the file while ``os.lstat(path)`` still has the size and ``mtime_ns`` it + was removed with; a file that cannot be looked at is not hidden. + """ + mark = marks.get(locator) + if mark is None: + return False + try: + info = os.lstat(path) + except OSError: + return False + return (info.st_size, info.st_mtime_ns) == tuple(mark) + + def write_mark(conn, locator, size, mtime_ns): _write_mark(conn, locator, size, mtime_ns, utc_now_iso()) diff --git a/database/library.py b/database/library.py index 49f18ac9..25ba6247 100644 --- a/database/library.py +++ b/database/library.py @@ -7,6 +7,7 @@ from contextlib import contextmanager from .library_folders import folder_condition, list_library_folders from .library_tokens import library_edit_token +from .prompt_archive import ARCHIVED, SHELVED _ELIGIBLE = """(EXISTS(SELECT 1 FROM prompt_pairs ep WHERE ep.positive_prompt_id=p.id) OR EXISTS(SELECT 1 FROM generated_images eg WHERE eg.prompt_id=p.id) @@ -106,6 +107,12 @@ ARCHIVE_SCOPES = { } +# Shelves (#443): the parents on the shelf, or the archived ones. +PARENT_SCOPES = {"active": SHELVED, "archived": ARCHIVED} +# A positive pair owner that is on the shelf, as a condition on alias pp. +SHELVED_PAIR = SHELVED.replace("p.id", "pp.positive_prompt_id") + + def _escape(value: str) -> str: return value.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") @@ -270,6 +277,8 @@ class LibraryOperations: rows = conn.execute( "SELECT p.id,p.text,ls.title,p.category,p.rating,p.notes,p.created_at," "p.run_count,p.last_used_at," + + ARCHIVED + + " AS archived," + (f"{sort_expression} AS _sort," if sort_expression else "") + "(SELECT COUNT(*) FROM prompt_pairs v WHERE v.positive_prompt_id=p.id " "AND v.archived=0) AS variant_count," @@ -293,6 +302,7 @@ class LibraryOperations: # Usage is shown, never compared: the token hashes owner fields only. row["run_count"] = row["run_count"] if type(row["run_count"]) is int else 0 row["last_used_at"] = utc_iso(row["last_used_at"]) + row["archived"] = bool(row["archived"]) row["edit_token"] = library_edit_token(epoch, row) return result @@ -310,10 +320,13 @@ class LibraryOperations: sort: str = "newest", limit: int = 40, after: dict | None = None, + shelf: str = "active", ) -> dict: _limit(limit) if sort not in _ORDERS: raise ValueError("Unknown library sort order") + if not isinstance(shelf, str) or shelf not in PARENT_SCOPES: + raise ValueError("shelf must be active or archived") where, params = _filter( text, category, [] if tags is None else tags, tag_mode, folder_prefixes ) @@ -328,15 +341,18 @@ class LibraryOperations: ceiling = after["ceiling"] where += " AND p.id<=?" params.append(ceiling) - # One SELECT counts the filtered parents and their active variants. - total, variant_total = conn.execute( - "SELECT COUNT(*),COALESCE(SUM((SELECT COUNT(*) FROM prompt_pairs v " - "WHERE v.positive_prompt_id=p.id AND v.archived=0)),0) FROM " - + _FROM - + " WHERE " - + where, + # One SELECT counts the filtered parents of this scope, their active + # variants, and the filtered archived parents (Show archived, #443). + scope = PARENT_SCOPES[shelf] + total, variant_total, archived_total = conn.execute( + f"SELECT COALESCE(SUM({scope}),0)," + f"COALESCE(SUM(CASE WHEN {scope} THEN (SELECT COUNT(*) " + "FROM prompt_pairs v WHERE v.positive_prompt_id=p.id " + "AND v.archived=0) ELSE 0 END),0)," + f"COALESCE(SUM({ARCHIVED}),0) FROM " + _FROM + " WHERE " + where, params, ).fetchone() + where += f" AND {scope}" expression, direction = _ORDERS[sort] if after is not None: operator = ">" if direction == "ASC" else "<" @@ -369,6 +385,7 @@ class LibraryOperations: "items": rows, "total": total, "variant_total": variant_total, + "archived_total": archived_total, # Echoed so a client can tell this read applied its tag match. "tag_mode": tag_mode, "has_more": has_more, @@ -386,6 +403,7 @@ class LibraryOperations: sort: str = "newest", limit: int = 40, after: dict | None = None, + shelf: str = "active", ) -> dict: with self.library_snapshot(include_ceiling=True) as (conn, epoch, ceiling): return self._list_library( @@ -400,6 +418,7 @@ class LibraryOperations: sort=sort, limit=limit, after=after, + shelf=shelf, ) def _is_library_positive(self, conn: sqlite3.Connection, positive_id: int) -> bool: @@ -644,7 +663,9 @@ class LibraryOperations: owners = {} order = [] for row in conn.execute( - "SELECT p.id,p.text,ls.title,p.category,p.rating,p.notes FROM " + "SELECT p.id,p.text,ls.title,p.category,p.rating,p.notes," + + ARCHIVED + + " AS archived FROM " + _FROM + " WHERE " + _ELIGIBLE @@ -657,6 +678,7 @@ class LibraryOperations: "tags": [], "rating": row["rating"], "notes": row["notes"], + "archived": bool(row["archived"]), "variants": [], } order.append(row["id"]) @@ -701,8 +723,9 @@ class LibraryOperations: """System › Overview counts from one snapshot, writing nothing. ``positive_prompts`` and ``negative_variants`` are the unfiltered - Library's ``total`` and ``variant_total``: eligible parents and active - pairs (every pair's positive is eligible). ``images`` counts indexed + Library's ``total`` and ``variant_total``: eligible parents on the + shelf (not archived, #443) and their active pairs (every pair's + positive is eligible). ``images`` counts indexed ``generated_images`` rows, not files on disk. An image has no rating of its own (it shows its linked prompt's), so ``rated_images`` counts the indexed images whose linked prompt is rated. ``review_pending`` is @@ -710,8 +733,14 @@ class LibraryOperations: """ with self.library_snapshot() as (conn, _epoch, _ceiling): row = conn.execute( - "SELECT (SELECT COUNT(*) FROM prompts p WHERE " + _ELIGIBLE + ")," - "(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0)," + "SELECT (SELECT COUNT(*) FROM prompts p WHERE " + + _ELIGIBLE + + " AND " + + SHELVED + + ")," + "(SELECT COUNT(*) FROM prompt_pairs pp WHERE pp.archived=0 AND " + + SHELVED_PAIR + + ")," "(SELECT COUNT(*) FROM generated_images)," "(SELECT COUNT(*) FROM generated_images gi JOIN prompts r " "ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL)," diff --git a/database/library_archive.py b/database/library_archive.py new file mode 100644 index 00000000..c44aa5e0 --- /dev/null +++ b/database/library_archive.py @@ -0,0 +1,289 @@ +"""Delete selected Library prompts; archive the paired ones (#443, D152). + +A selected Library parent is classified, in this order, as ``not_found`` +(missing, or not a Library parent), ``protected`` (tagged ``__protected__``; +never changed), ``already_archived``, ``archive`` (the positive or negative of +any saved pair, archived pair or not: archiving hides it from the shelf and +removes nothing) or ``delete`` (in no pair: deleted exactly as the legacy +delete does, through ``delete_prompt_rows``). Every command checks the +storage epoch first and runs in one ``BEGIN IMMEDIATE`` transaction. + +The write executes what the user reviewed: it is given the reviewed archive +and delete lists, and deletes only reviewed deletes the route backed up +(``candidates``) whose fingerprint is unchanged since that backup. +""" + +import sqlite3 +from typing import Iterable + +from .library import _ELIGIBLE +from .library_commands import _validate_bulk_ids, _validate_epoch +from .models import utc_now_iso +from .prompt_archive import ARCHIVED as _ARCHIVED +from .prompt_archive import TABLE as ARCHIVE +from .workflow_documents import documents_of_rows, release_documents + +try: + from ..utils.hashing import generate_prompt_hash +except ImportError: + from utils.hashing import generate_prompt_hash + +# Ids bound per statement: the pair check binds each id twice, which stays +# under SQLite's historical 999-variable limit (maintenance.py ID_BATCH). +ID_BATCH = 400 +_PROTECTED = ( + "EXISTS(SELECT 1 FROM prompt_tags pt JOIN tags t ON t.id=pt.tag_id " + "WHERE pt.prompt_id=p.id AND t.name='__protected__')" +) +_PAIRED = ( + "EXISTS(SELECT 1 FROM prompt_pairs pp " + "WHERE pp.positive_prompt_id=p.id OR pp.negative_prompt_id=p.id)" +) +# Classes in the order a prompt is tested; the first that holds wins. +CLASSES = ("not_found", "protected", "already_archived", "archive", "delete") + + +def _batches(ids: list) -> Iterable[list]: + for start in range(0, len(ids), ID_BATCH): + yield ids[start : start + ID_BATCH] + + +def delete_prompt_rows(conn: sqlite3.Connection, prompt_ids: list[int]) -> int: + """Delete prompts and their image link rows; return the prompts deleted. + + The legacy delete's statements (``delete_prompt``, ``bulk_delete_prompts``) + in the caller's transaction: image link rows first (``image_prompt_pairs`` + cascades, review decisions keep a NULL image), then the prompts (tags and + Library state cascade), then the workflow documents no row references. + Image files are never touched. The caller checks pairs first. + """ + documents = documents_of_rows(conn, "prompt_id", prompt_ids) + deleted = 0 + for chunk in _batches(list(prompt_ids)): + marks = ",".join("?" for _ in chunk) + conn.execute( + f"DELETE FROM generated_images WHERE prompt_id IN ({marks})", # nosec + chunk, + ) + deleted += conn.execute( + f"DELETE FROM prompts WHERE id IN ({marks})", chunk # nosec B608 + ).rowcount + release_documents(conn, documents) + return deleted + + +class LibraryArchiveOperations: + """PromptDatabase mixin: classify, archive or delete, and restore parents.""" + + def _classify_library_delete( + self, conn: sqlite3.Connection, prompt_ids: list[int] + ) -> dict: + found = {} + for chunk in _batches(prompt_ids): + marks = ",".join("?" for _ in chunk) + for row in conn.execute( + f"SELECT p.id,{_PROTECTED},{_ARCHIVED},{_PAIRED} " # nosec B608 + f"FROM prompts p WHERE p.id IN ({marks}) AND {_ELIGIBLE}", + chunk, + ): + protected, archived, paired = row[1], row[2], row[3] + if protected: + found[row[0]] = "protected" + elif archived: + found[row[0]] = "already_archived" + else: + found[row[0]] = "archive" if paired else "delete" + classes = {name: [] for name in CLASSES} + for pid in prompt_ids: + classes[found.get(pid, "not_found")].append(pid) + return classes + + @staticmethod + def _image_links(conn: sqlite3.Connection, prompt_ids: list[int]) -> int: + total = 0 + for chunk in _batches(prompt_ids): + marks = ",".join("?" for _ in chunk) + total += conn.execute( + "SELECT COUNT(*) FROM generated_images " # nosec B608 + f"WHERE prompt_id IN ({marks})", + chunk, + ).fetchone()[0] + return total + + @staticmethod + def _fingerprints(conn: sqlite3.Connection, prompt_ids: list[int]) -> dict: + """What a safety backup holds of each prompt a delete would remove. + + The prompt row (usage included), its tag ids, its Library title and + its image link rows (count and newest id), as a JSON-safe list. + """ + prints = {} + for chunk in _batches(prompt_ids): + marks = ",".join("?" for _ in chunk) + for row in conn.execute( + "SELECT p.id,p.text,p.hash,p.category,p.rating,p.notes," # nosec + "p.updated_at,p.run_count,p.last_used_at," + "(SELECT group_concat(tag_id) FROM (SELECT tag_id FROM prompt_tags " + "WHERE prompt_id=p.id ORDER BY tag_id))," + "(SELECT title FROM prompt_library_state WHERE prompt_id=p.id)," + "(SELECT COUNT(*) FROM generated_images WHERE prompt_id=p.id)," + "(SELECT MAX(id) FROM generated_images WHERE prompt_id=p.id) " + f"FROM prompts p WHERE p.id IN ({marks})", + chunk, + ): + prints[row[0]] = [ + value if isinstance(value, (int, str, type(None))) else repr(value) + for value in tuple(row)[1:] + ] + return prints + + @staticmethod + def _checked_ids(prompt_ids: list[int], expected_epoch: str) -> list[int]: + _validate_epoch(expected_epoch) + _validate_bulk_ids(prompt_ids) + return list(dict.fromkeys(prompt_ids)) + + def classify_library_delete( + self, prompt_ids: list[int], *, expected_epoch: str + ) -> dict: + """What a delete of ``prompt_ids`` would do now (reads only). + + Returns the id lists per class, in request order, ``image_links`` + (the image link rows the ``delete`` class would remove) and + ``fingerprints`` of the ``delete`` class (``_fingerprints``). + """ + prompt_ids = self._checked_ids(prompt_ids, expected_epoch) + with self.library_snapshot() as (conn, _epoch, _ceiling): + epoch = self._library_epoch(conn, expected_epoch) + classes = self._classify_library_delete(conn, prompt_ids) + return { + "library_epoch": epoch, + **classes, + "image_links": self._image_links(conn, classes["delete"]), + "fingerprints": self._fingerprints(conn, classes["delete"]), + } + + @staticmethod + def _repair_hash(conn: sqlite3.Connection, prompt_id: int) -> None: + """Give an archived prompt its canonical hash, so ingestion finds it. + + Scans, the monitor and node saves resolve prompts by hash; a stale or + missing one would make them insert a visible copy (Q3). When another + row already holds the canonical hash, lookups find that row and no new + copy is made, so nothing is written. + """ + text, stored = conn.execute( + "SELECT text,hash FROM prompts WHERE id=?", (prompt_id,) + ).fetchone() + canonical = generate_prompt_hash(text) if isinstance(text, str) else None + if canonical is None or stored == canonical: + return + if conn.execute( + "SELECT 1 FROM prompts WHERE hash=? AND id!=?", (canonical, prompt_id) + ).fetchone(): + return + conn.execute("UPDATE prompts SET hash=? WHERE id=?", (canonical, prompt_id)) + + def archive_or_delete_library_prompts( + self, + archive_ids: list[int], + delete_ids: list[int], + *, + expected_epoch: str, + candidates: dict | None, + ) -> dict: + """Archive and delete what the user reviewed, all or nothing. + + ``archive_ids`` and ``delete_ids`` are the reviewed classes. + ``candidates`` maps each reviewed delete the route backed up to its + fingerprint then (None: every reviewed delete, unchecked; tests and + callers without a backup). A reviewed delete is deleted only when it + still classifies ``delete`` and its fingerprint is unchanged; one that + is now paired, and a reviewed archive now unpaired, are archived (the + restorable direction); anything else is kept and counted. + """ + _validate_epoch(expected_epoch) + for ids in (archive_ids, delete_ids): + if not isinstance(ids, list) or any( + type(pid) is not int or not 1 <= pid <= 2**63 - 1 for pid in ids + ): + raise ValueError("prompt ids must be lists of positive integers") + reviewed_delete = set(delete_ids) + if not archive_ids and not delete_ids: + raise ValueError("Nothing to delete or archive") + if reviewed_delete & set(archive_ids): + raise ValueError("A prompt cannot be both archived and deleted") + prompt_ids = list(dict.fromkeys([*archive_ids, *delete_ids])) + with self._pair_transaction() as conn: + epoch = self._library_epoch(conn, expected_epoch) + classes = self._classify_library_delete(conn, prompt_ids) + now_prints = self._fingerprints(conn, classes["delete"]) + deletes, archives, changed = [], list(classes["archive"]), 0 + for pid in classes["delete"]: + if pid not in reviewed_delete: + archives.append(pid) # reviewed as archive, now unpaired + elif candidates is None or candidates.get(pid) == now_prints[pid]: + deletes.append(pid) + else: + changed += 1 + changed += len(classes["protected"]) + image_links = self._image_links(conn, deletes) + now = utc_now_iso() + for pid in archives: + self._repair_hash(conn, pid) + conn.executemany( + f"INSERT INTO {ARCHIVE}(prompt_id, archived_at) VALUES (?, ?)", + [(pid, now) for pid in archives], + ) + # A changed row is the newest edit the restore fingerprint reads. + conn.executemany( + "UPDATE prompts SET updated_at=? WHERE id=?", + [(now, pid) for pid in archives], + ) + deleted = delete_prompt_rows(conn, deletes) + return { + "library_epoch": epoch, + "deleted": deleted, + "archived": len(archives), + "already_archived": len(classes["already_archived"]), + "not_found": len(classes["not_found"]), + "changed": changed, + "image_links": image_links, + } + + def restore_library_prompts( + self, prompt_ids: list[int], *, expected_epoch: str + ) -> dict: + """Put archived prompts back on the shelf, all or nothing. + + Every requested id with an archive row is restored, whatever its + protection or Library parent state: restoring only un-hides it. + """ + prompt_ids = self._checked_ids(prompt_ids, expected_epoch) + with self._pair_transaction() as conn: + epoch = self._library_epoch(conn, expected_epoch) + restored, existing = [], set() + for chunk in _batches(prompt_ids): + marks = ",".join("?" for _ in chunk) + for pid, archived in conn.execute( + f"SELECT p.id,{_ARCHIVED} FROM prompts p " # nosec B608 + f"WHERE p.id IN ({marks})", + chunk, + ): + existing.add(pid) + if archived: + restored.append(pid) + now = utc_now_iso() + conn.executemany( + f"DELETE FROM {ARCHIVE} WHERE prompt_id=?", [(pid,) for pid in restored] + ) + conn.executemany( + "UPDATE prompts SET updated_at=? WHERE id=?", + [(now, pid) for pid in restored], + ) + return { + "library_epoch": epoch, + "restored": len(restored), + "unchanged": len(existing) - len(restored), + "not_found": len(prompt_ids) - len(existing), + } diff --git a/database/maintenance.py b/database/maintenance.py index a76d2d9c..27002761 100644 --- a/database/maintenance.py +++ b/database/maintenance.py @@ -15,6 +15,8 @@ change: ``cleanup_orphaned_images``), counted only under a source that can be read now, as the health check's ``missing_files`` judges them (``health.link_states``). It deletes database rows only, never a file. + It also forgets the removal marks (#480) that hide nothing any more: the + file is gone or changed, under a readable source (#501, ``_stale_marks``). ``preview_maintenance`` reads one snapshot on its own ``mode=ro`` connection (``health.readonly_uri``) and closes it in ``finally``; files are looked at @@ -41,6 +43,8 @@ from pathlib import Path from .execution_contexts import LIVE_HINT_CTES from .health import BATCH_ROWS, link_states, readable_roots, readonly_uri +from .image_removal_schema import TABLE as MARKS_TABLE +from .image_removal_schema import clear_marks, has_marks_table from .pairs import PairDependencyError from .workflow_documents import documents_of_rows, release_documents @@ -85,6 +89,12 @@ PROTECTIONS = ( "with Library state (a title or an explicit parent)", "with Library state (a title or an explicit parent)", ), + ( + # #443: an archived prompt is restorable, never pruned. + "EXISTS (SELECT 1 FROM prompt_archive pa WHERE pa.prompt_id = p.id)", + "archived in the Library", + "archived in the Library", + ), ( "p.id IN (SELECT prompt_id FROM live_hint_prompts)", "used by recorded runs", @@ -93,7 +103,8 @@ PROTECTIONS = ( ) # Exempt references: images, pairs, the __protected__ tag, Library parent -# state, and live recorded-run hints (computed once, see LIVE_HINT_CTES). +# state, the Library archive, and live recorded-run hints (computed once, +# see LIVE_HINT_CTES). ORPHANED_PROMPTS_SQL = ( "WITH " + LIVE_HINT_CTES @@ -187,6 +198,8 @@ class MaintenanceDraft: protected: dict media: dict details: list = field(default_factory=list) + # missing_links only: locators of removal marks whose file is gone (#501). + marks: list = field(default_factory=list) def public(self): """The plan fields a response may carry (no fingerprint, no targets).""" @@ -289,6 +302,90 @@ def _delete_ids(conn, table, ids): return removed +def _mark_rows(conn, only=None): + """``[(locator, size, mtime_ns)]`` of the removal marks (all, or those + among *only*), sorted by locator.""" + if not has_marks_table(conn): + return [] + sql = f"SELECT locator, size, mtime_ns FROM {MARKS_TABLE}" # nosec B608 + if only is None: + return sorted(tuple(row) for row in conn.execute(sql)) + found = [] + for batch in _batches(sorted(only)): + found.extend( + tuple(row) + for row in conn.execute( + f"{sql} WHERE locator IN ({_marks(batch)})", batch # nosec B608 + ) + ) + return sorted(found) + + +def _lists_something(folder): + """Whether *folder* holds at least one entry (an unplugged drive's mount + point, or an unbound volume, is an empty folder that reads fine).""" + try: + with os.scandir(folder) as entries: + return next(entries, None) is not None + except OSError: + return False + + +def _name_listed(path): + """Whether *path*'s folder lists its name, compared under ``normcase``. + + A locator is ``normcase``d; on a case-sensitive NTFS folder ``lstat`` of + the lower-cased spelling fails for a file that is there. + """ + folder, name = os.path.split(path) + wanted = os.path.normcase(name) + try: + with os.scandir(folder) as entries: + return any(os.path.normcase(entry.name) == wanted for entry in entries) + except OSError: + return True # cannot tell: keep the mark + + +def _stale_mark(locator, size, mtime_ns): + """Whether the file of a mark is gone or has other bytes (blocking).""" + try: + info = os.lstat(locator) + except (FileNotFoundError, NotADirectoryError): + return os.path.isdir(os.path.dirname(locator)) and not _name_listed(locator) + except OSError: + return False + return (info.st_size, info.st_mtime_ns) != (size, mtime_ns) + + +def _stale_marks(marks, roots): + """``[(locator, owner)]`` of the marks that hide nothing any more (#501). + + *marks* are ``(locator, size, mtime_ns)`` rows. A mark is stale when its + file was changed, or is gone from a folder that still exists. Only marks + under a source that can be read now and lists something are judged (an + unplugged drive's marks, or its empty mount point's, stay); any doubt + keeps the mark. *owner* is the index of the longest source holding it. + Locators are ``normcase(realpath)`` paths, so the sources are too. + """ + readable = readable_roots(roots) + prefixes = [ + os.path.normcase(os.path.realpath(root)).rstrip("\\/") + os.sep + for root in roots + ] + judged = {} + stale = [] + for locator, size, mtime_ns in marks: + owners = [i for i, prefix in enumerate(prefixes) if locator.startswith(prefix)] + if not owners: + continue # under no source now (a source removed from the settings) + owner = max(owners, key=lambda i: len(prefixes[i])) + if owner not in judged: + judged[owner] = readable[owner] and _lists_something(roots[owner]) + if judged[owner] and _stale_mark(locator, size, mtime_ns): + stale.append((locator, owner)) + return stale + + def _read_snapshot(db_path, read): """``read(conn)`` on one ``mode=ro`` snapshot; the connection always closes.""" conn = sqlite3.connect(readonly_uri(db_path), uri=True) @@ -431,8 +528,11 @@ class MaintenanceOperations: if operation == "workflow_storage": return self._plan_workflow_storage() if operation == "missing_links": - rows = _read_snapshot(self.model.db_path, self._link_rows) - return self._plan_missing_links(rows, roots) + rows, marks = _read_snapshot( + self.model.db_path, + lambda conn: (self._link_rows(conn), _mark_rows(conn)), + ) + return self._plan_missing_links(rows, roots, _stale_marks(marks, roots)) reader = ( self._plan_duplicate_prompts if operation == "duplicate_prompts" @@ -563,13 +663,15 @@ class MaintenanceOperations: ) @staticmethod - def _links_fingerprint(candidates): + def _links_fingerprint(candidates, stale=()): fingerprint = _Fingerprint("missing_links") for row, owner in candidates: fingerprint.add([row[0], row[1], row[3], owner]) + for locator, owner in stale: + fingerprint.add(["mark", locator, owner]) return fingerprint.hexdigest() - def _plan_missing_links(self, rows, roots): + def _plan_missing_links(self, rows, roots, stale=()): readable = readable_roots(roots) candidates, unavailable, outside = [], 0, 0 for row, owner, state in link_states(rows, roots, readable): @@ -597,29 +699,47 @@ class MaintenanceOperations: ) ) count = len(candidates) + parts = [] + if count: + parts.append( + _plural(count, "link to a missing file", "links to missing files") + ) + if stale: + parts.append( + _plural( + len(stale), + "removed image whose file is gone or changed", + "removed images whose files are gone or changed", + ) + ) + details = [ + { + "label": _relative(row[1], roots[owner]), + "detail": f"{_source_name(roots[owner], owner)} · the file is missing", + } + for row, owner in candidates + ] + [ + { + "label": _relative(locator, os.path.realpath(roots[owner])), + "detail": ( + f"{_source_name(roots[owner], owner)} · removed image, " + "the file is gone or changed" + ), + } + for locator, owner in stale + ] return MaintenanceDraft( operation="missing_links", - fingerprint=self._links_fingerprint(candidates), + fingerprint=self._links_fingerprint(candidates, stale), targets=[row[0] for row, _owner in candidates], scope={ - "count": count, - "label": ( - _plural(count, "link to a missing file", "links to missing files") - if count - else "No links to missing files" - ), + "count": count + len(stale), + "label": "; ".join(parts) or "No links to missing files", }, protected={"count": unavailable + outside, "reasons": reasons}, media={"deletes_files": False, "label": LINKS_ONLY}, - details=[ - { - "label": _relative(row[1], roots[owner]), - "detail": ( - f"{_source_name(roots[owner], owner)} · the file is missing" - ), - } - for row, owner in candidates[:MAX_DETAILS] - ], + details=details[:MAX_DETAILS], + marks=[locator for locator, _owner in stale], ) def _plan_optimize(self): @@ -669,14 +789,18 @@ class MaintenanceOperations: # ── apply ───────────────────────────────────────────────────────── - def apply_maintenance(self, operation, fingerprint, targets, roots=()) -> dict: + def apply_maintenance( + self, operation, fingerprint, targets, roots=(), marks=() + ) -> dict: """Apply a reviewed plan exactly, or raise PlanChanged (blocking). Inside ``BEGIN IMMEDIATE`` the fingerprint is computed again on the write connection; when it differs nothing is written. ``targets`` and ``roots`` are the plan's. Returns ``{changed, rows}``: how many records changed and up to ``MAX_RESULT_ROWS`` descriptions of them - (``{prompt_id[, kept_id]}`` or ``{image_path}``). + (``{prompt_id[, kept_id]}`` or ``{image_path}``). ``marks`` are a + missing-links plan's stale removal marks; that apply also returns + ``marks``, how many it forgot. """ if operation not in MAINTENANCE_OPERATIONS or operation == "workflow_storage": raise ValueError(f"Unknown maintenance operation {operation!r}") @@ -689,9 +813,12 @@ class MaintenanceOperations: appliers = { "duplicate_prompts": self._apply_duplicate_prompts, "unused_components": self._apply_unused_components, - "missing_links": self._apply_missing_links, } with self._pair_transaction() as conn: + if operation == "missing_links": + return self._apply_missing_links( + conn, fingerprint, targets, roots, list(marks) + ) return appliers[operation](conn, fingerprint, targets, roots) def _apply_duplicate_prompts(self, conn, fingerprint, targets, _roots): @@ -720,7 +847,7 @@ class MaintenanceOperations: ] return {"changed": changed, "rows": rows} - def _apply_missing_links(self, conn, fingerprint, targets, roots): + def _apply_missing_links(self, conn, fingerprint, targets, roots, marks): rows = [] for batch in _batches(sorted(targets)): rows.extend( @@ -737,10 +864,14 @@ class MaintenanceOperations: if state != "missing": # back, or its source went away raise PlanChanged("missing_links") candidates.append((row, owner)) - if self._links_fingerprint(candidates) != fingerprint: + stale = _stale_marks(_mark_rows(conn, marks) if marks else [], roots) + if [locator for locator, _owner in stale] != marks: # a mark went or is valid + raise PlanChanged("missing_links") + if self._links_fingerprint(candidates, stale) != fingerprint: raise PlanChanged("missing_links") changed = _delete_ids(conn, "generated_images", [row[0] for row in rows]) return { "changed": changed, + "marks": clear_marks(conn, marks), "rows": [{"image_path": row[1]} for row in rows[:MAX_RESULT_ROWS]], } diff --git a/database/models.py b/database/models.py index d64cbcba..4c506962 100644 --- a/database/models.py +++ b/database/models.py @@ -45,6 +45,7 @@ except ImportError: from utils.hashing import register_prompt_functions from .image_categories import image_categories_schema_error, migrate_image_categories from .image_removal_schema import image_removals_schema_error, migrate_image_removals +from .prompt_archive import migrate_prompt_archive, prompt_archive_schema_error from .workflow_documents import ( DOCUMENT_ID_COLUMNS, migrate_workflow_documents, @@ -367,6 +368,8 @@ class PromptModel: migrate_autotag_state(conn) # Image categories (#332): independent of every table above. migrate_image_categories(conn) + # The Library parent archive (#443): independent of every table above. + migrate_prompt_archive(conn) # Removal marks (#480): independent of every table above. migrate_image_removals(conn) # Last: the older generated_images rebuilds run only before file_path. @@ -1245,6 +1248,11 @@ class PromptModel: ) if categories_error: return False, categories_error + archive_error = prompt_archive_schema_error( + conn, allow_missing=not require_library_state + ) + if archive_error: + return False, archive_error removals_error = image_removals_schema_error( conn, allow_missing=not require_library_state ) diff --git a/database/operations.py b/database/operations.py index da5d0ae8..51607876 100644 --- a/database/operations.py +++ b/database/operations.py @@ -25,6 +25,7 @@ from .review import ReviewOperations from .review_decisions import ReviewDecisionOperations from .saved_views import SavedViewOperations from .library_commands import LibraryCommandOperations +from .library_archive import LibraryArchiveOperations, delete_prompt_rows from .vocabulary import VocabularyOperations from .execution_history import ExecutionHistoryOperations from .execution_contexts import ExecutionContextOperations @@ -449,6 +450,7 @@ class PromptDatabase( PairOperations, LibraryOperations, LibraryCommandOperations, + LibraryArchiveOperations, VocabularyOperations, ExecutionHistoryOperations, ExecutionContextOperations, @@ -2351,16 +2353,10 @@ class PromptDatabase( Returns: Number of prompts actually deleted. """ - count = 0 with self._pair_transaction() as conn: self._ensure_prompts_unpaired(conn, prompt_ids) - documents = documents_of_rows(conn, "prompt_id", prompt_ids) - for pid in prompt_ids: - conn.execute("DELETE FROM generated_images WHERE prompt_id = ?", (pid,)) - cursor = conn.execute("DELETE FROM prompts WHERE id = ?", (pid,)) - if cursor.rowcount > 0: - count += 1 - release_documents(conn, documents) + # The workspace Library's delete runs the same statements (#443). + count = delete_prompt_rows(conn, list(dict.fromkeys(prompt_ids))) conn.commit() return count diff --git a/database/prompt_archive.py b/database/prompt_archive.py new file mode 100644 index 00000000..9f824361 --- /dev/null +++ b/database/prompt_archive.py @@ -0,0 +1,107 @@ +"""Library parent archive (#443, D152): prompts hidden from the shelf, restorable. + +A row means the Library parent is archived; no row means it is on the shelf. +Nothing else about the prompt changes: its text, metadata, tags, run count, +image links, pairs and recorded runs stay. Restore deletes the row. Deleting +the prompt (any path) drops the row by cascade. + +A table of its own, not a column: ``prompt_library_state`` is validated by its +exact definition, so a new column there would make older builds refuse the +upgraded database, while older builds ignore an unknown table (they show the +archived prompts again and keep the rows). Optional in old backups, required +once a database is current. +""" + +import re +import sqlite3 + +TABLE = "prompt_archive" +MAX_STAMP_BYTES = 64 +# What ``utc_now_iso`` writes starts with a date; anything else was not written here. +_STAMP_GLOB = "[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T*" +_DDL = ( + f"CREATE TABLE {TABLE} (prompt_id INTEGER PRIMARY KEY, " + "archived_at TEXT NOT NULL, " + "FOREIGN KEY(prompt_id) REFERENCES prompts(id) ON DELETE CASCADE)" +) +_COLUMNS = (("prompt_id", "INTEGER", 0, 1), ("archived_at", "TEXT", 1, 0)) +_FOREIGN_KEYS = {("prompt_id", "prompts", "id", "CASCADE")} +# A predicate on ``p`` (the prompts alias): the parent is not archived. +SHELVED = f"NOT EXISTS(SELECT 1 FROM {TABLE} pa WHERE pa.prompt_id=p.id)" +ARCHIVED = f"EXISTS(SELECT 1 FROM {TABLE} pa WHERE pa.prompt_id=p.id)" + + +def _compact(sql: str) -> str: + return re.sub(r"\s+", "", sql or "").lower() + + +def _structure_error(conn: sqlite3.Connection, allow_missing: bool) -> str | None: + objects = conn.execute( + "SELECT type,sql FROM sqlite_master WHERE name=? " + "OR (type IN ('index','trigger') AND tbl_name=?)", + (TABLE, TABLE), + ).fetchall() + if not objects: + return None if allow_missing else "Missing prompt archive schema" + tables = [sql for kind, sql in objects if kind == "table"] + if len(tables) != 1 or len(objects) != 1: + return "Prompt archive name is reserved by an unsupported schema object" + if _compact(tables[0]) != _compact(_DDL): + return "Prompt archive has an invalid table definition" + columns = tuple( + (row[1], row[2].upper(), row[3], row[5]) + for row in conn.execute(f"PRAGMA table_info({TABLE})") + ) + if columns != _COLUMNS: + return "Prompt archive has invalid columns" + foreign_keys = { + (row[3], row[2], row[4], row[6]) + for row in conn.execute(f"PRAGMA foreign_key_list({TABLE})") + } + if foreign_keys != _FOREIGN_KEYS: + return "Prompt archive has invalid foreign key definitions" + return None + + +def prompt_archive_schema_error( + conn: sqlite3.Connection, *, allow_missing: bool = True +) -> str | None: + """Reject an unsupported archive schema or values; reads only. + + A missing table is acceptable for backups that predate it unless + ``allow_missing`` is false. + """ + error = _structure_error(conn, allow_missing) + if error is not None: + return error + if not conn.execute( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (TABLE,) + ).fetchone(): + return None + if conn.execute(f"PRAGMA foreign_key_check({TABLE})").fetchone(): + return "Prompt archive contains foreign key violations" + if conn.execute( + f"SELECT 1 FROM {TABLE} WHERE typeof(archived_at)!='text' " # nosec B608 + f"OR length(CAST(archived_at AS BLOB))>{MAX_STAMP_BYTES} " + f"OR archived_at NOT GLOB '{_STAMP_GLOB}' LIMIT 1" + ).fetchone(): + return "Prompt archive contains invalid values" + return None + + +def migrate_prompt_archive(conn: sqlite3.Connection) -> None: + """Install the table atomically; validate before and after (no backfill).""" + conn.execute("SAVEPOINT prompt_archive_schema") + try: + error = _structure_error(conn, True) + if error: + raise sqlite3.DatabaseError(error) + conn.execute(_DDL.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1)) + error = _structure_error(conn, False) + if error: + raise sqlite3.DatabaseError(error) + conn.execute("RELEASE prompt_archive_schema") + except BaseException: + conn.execute("ROLLBACK TO prompt_archive_schema") + conn.execute("RELEASE prompt_archive_schema") + raise diff --git a/database/restore_stage.py b/database/restore_stage.py index 704435bf..1a3a3762 100644 --- a/database/restore_stage.py +++ b/database/restore_stage.py @@ -149,6 +149,7 @@ def library_summary(conn: sqlite3.Connection) -> Dict[str, int]: collections (#331). """ from .library import _ELIGIBLE + from .prompt_archive import SHELVED, TABLE as ARCHIVE has_state = conn.execute( "SELECT 1 FROM sqlite_master WHERE type='table' AND " @@ -157,9 +158,15 @@ def library_summary(conn: sqlite3.Connection) -> Dict[str, int]: has_pairs = conn.execute( "SELECT 1 FROM sqlite_master WHERE type='table' AND name='prompt_pairs'" ).fetchone() + has_archive = conn.execute( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (ARCHIVE,) + ).fetchone() if has_state and has_pairs: + # Archived parents (#443) are off the shelf, as in library_counts. positives = conn.execute( - "SELECT COUNT(*) FROM prompts p WHERE " + _ELIGIBLE + "SELECT COUNT(*) FROM prompts p WHERE " + + _ELIGIBLE + + (" AND " + SHELVED if has_archive else "") ).fetchone()[0] else: positives = _count(conn, "prompts") diff --git a/database/vocabulary.py b/database/vocabulary.py index 5eb22c34..971112bd 100644 --- a/database/vocabulary.py +++ b/database/vocabulary.py @@ -20,7 +20,8 @@ counted, hashed into the token and renamed with the prompts and pairs. import sqlite3 -from .library import _ELIGIBLE, UNTAGGED, _escape, _limit +from .library import _ELIGIBLE, SHELVED_PAIR, UNTAGGED, _escape, _limit +from .prompt_archive import SHELVED from .library_commands import _TOKEN, LibraryCommandError, _validate_epoch from .library_tokens import canonical_token from .models import utc_now_iso @@ -40,17 +41,21 @@ _IMAGE_FILES = ( def _tag_rows(page: str, order: str = "page.id") -> str: """Count rows for the registry rows selected by ``page`` (id, name).""" - # Eligible parent components carrying a tag in the page, plus the - # positive parent of every active pair carrying it (always eligible). + # Eligible parent components on the shelf carrying a tag in the page, + # plus the positive parent of every active pair carrying it (always + # eligible) when that parent is on the shelf: archived parents (#443) + # are hidden with their pairs, so Browse lists what was counted. return ( "WITH page AS (" + page + ")," "owners(tag_id,pid) AS (" "SELECT pt.tag_id,p.id FROM prompts p " "JOIN prompt_tags pt ON pt.prompt_id=p.id " "WHERE pt.tag_id IN (SELECT id FROM page) AND " + _ELIGIBLE + " " + "AND " + SHELVED + " " "UNION SELECT ppt.tag_id,pp.positive_prompt_id FROM prompt_pairs pp " "JOIN prompt_pair_tags ppt ON ppt.pair_id=pp.id " - "WHERE pp.archived=0 AND ppt.tag_id IN (SELECT id FROM page))," + "WHERE pp.archived=0 AND " + SHELVED_PAIR + " " + "AND ppt.tag_id IN (SELECT id FROM page))," "images(tag_id,n) AS (" + _IMAGE_FILES + " " "WHERE mit.tag_id IN (SELECT id FROM page) GROUP BY mit.tag_id) " "SELECT page.id,page.name,COUNT(DISTINCT owners.pid) AS prompt_count," @@ -79,8 +84,10 @@ _RANKED_TAG_PAGE = ( "WITH owners(tag_id,pid) AS (" "SELECT pt.tag_id,p.id FROM prompts p " "JOIN prompt_tags pt ON pt.prompt_id=p.id WHERE " + _ELIGIBLE + " " + "AND " + SHELVED + " " "UNION SELECT ppt.tag_id,pp.positive_prompt_id FROM prompt_pairs pp " - "JOIN prompt_pair_tags ppt ON ppt.pair_id=pp.id WHERE pp.archived=0)," + "JOIN prompt_pair_tags ppt ON ppt.pair_id=pp.id WHERE pp.archived=0 " + "AND " + SHELVED_PAIR + ")," "counted AS (SELECT t.id,t.name,COUNT(DISTINCT o.pid) AS prompt_count " "FROM tags t LEFT JOIN owners o ON o.tag_id=t.id WHERE {search} " "GROUP BY t.id)," @@ -94,7 +101,13 @@ _RANKED_TAG_PAGE = ( "ORDER BY page.prompt_count {direction},page.id" ) _UNTAGGED_COUNT = ( - "SELECT COUNT(*) FROM prompts p WHERE (" + _ELIGIBLE + ") AND " + UNTAGGED + "SELECT COUNT(*) FROM prompts p WHERE (" + + _ELIGIBLE + + ") AND " + + UNTAGGED + # Untagged prompts opens the shelf, which hides archived parents (#443). + + " AND " + + SHELVED ) _TAG_ONE = _tag_rows("SELECT id,name FROM tags WHERE id=?") # Every owner kind a tag can sit on, for the delete confirmation. @@ -134,7 +147,9 @@ def _validate_tag_id(tag_id: object) -> None: # Category values in Tags & categories (#332 §7): the prompt-backed values -# (nonblank values on eligible parents or on active pairs, grouped exactly, so +# (nonblank values on eligible parents or on active pairs; ``prompt_count`` +# counts only parents on the shelf, so a value only archived parents (#443) +# hold stays listed with 0, as a tag does; grouped exactly, so # whitespace and case variants stay distinct; /library/options lists the same # prompt-backed set with its own query) plus the values only image rows hold. # A prompt-backed value is ordered by its oldest owning positive; an image-only @@ -143,15 +158,17 @@ def _validate_tag_id(tag_id: object) -> None: # files missing on disk. No FULL OUTER JOIN, so it runs on SQLite 3.31+. _IMAGE_ONLY_FIRST_PID = 2**63 - 1 _CATEGORY_ROWS = ( - "WITH owners(value,pid) AS (" - "SELECT p.category,p.id FROM prompts p WHERE p.category IS NOT NULL " - "AND TRIM(p.category)!='' AND " + _ELIGIBLE + " " - "UNION SELECT pp.category,pp.positive_prompt_id FROM prompt_pairs pp " + "WITH owners(value,pid,shelved) AS (" + "SELECT p.category,p.id," + SHELVED + " FROM prompts p " + "WHERE p.category IS NOT NULL AND TRIM(p.category)!='' AND " + _ELIGIBLE + " " + "UNION SELECT pp.category,pp.positive_prompt_id," + SHELVED_PAIR + " " + "FROM prompt_pairs pp " "WHERE pp.archived=0 AND pp.category IS NOT NULL AND TRIM(pp.category)!='')," "image_values(value,image_count) AS (" "SELECT category,COUNT(*) FROM image_categories GROUP BY category)," "grouped(value,first_pid,prompt_count) AS (" - "SELECT value,MIN(pid),COUNT(DISTINCT pid) FROM owners GROUP BY value " + "SELECT value,MIN(pid),COUNT(DISTINCT CASE WHEN shelved THEN pid END) " + "FROM owners GROUP BY value " "UNION ALL SELECT value," + str(_IMAGE_ONLY_FIRST_PID) + ",0 FROM image_values " "WHERE value NOT IN (SELECT value FROM owners))," "counted AS (SELECT g.value,g.first_pid,g.prompt_count," diff --git a/py/api/__init__.py b/py/api/__init__.py index f15cce0d..0955bb0c 100644 --- a/py/api/__init__.py +++ b/py/api/__init__.py @@ -19,6 +19,7 @@ Split into domain-specific mixins for maintainability: - SystemRoutesMixin: System health check, Overview and usage statistics - LibraryLocationRoutesMixin: where the library lives, Move library, Keep it here - MaintenanceRoutesMixin: reviewed maintenance plans (preview, apply) +- LibraryDeleteRoutesMixin: the Library's Delete… (archive paired) and Restore - WorkflowStorageRoutesMixin: store each workflow once, then Reclaim space - DuplicateFilesMixin: duplicate image files as find and removal jobs - LoraImportRoutesMixin: the reviewed, staged LoRA reference import job @@ -43,6 +44,7 @@ from .prompts import PromptRoutesMixin from .revalidation import revalidated_response from .pairs import PairRoutesMixin from .library import LibraryRoutesMixin +from .library_delete import LibraryDeleteRoutesMixin from .vocabulary import VocabularyRoutesMixin from .annotations import AnnotationRoutesMixin from .review import ReviewRoutesMixin @@ -315,6 +317,7 @@ async def _gzip_middleware(request, handler): class PromptManagerAPI( PairRoutesMixin, LibraryRoutesMixin, + LibraryDeleteRoutesMixin, VocabularyRoutesMixin, AnnotationRoutesMixin, ReviewRoutesMixin, @@ -700,6 +703,7 @@ class PromptManagerAPI( self._register_pair_routes(routes) self._register_library_routes(routes) + self._register_library_delete_routes(routes) self._register_vocabulary_routes(routes) self._register_annotation_routes(routes) self._register_review_routes(routes) diff --git a/py/api/admin.py b/py/api/admin.py index a6b123d2..a9f820e2 100644 --- a/py/api/admin.py +++ b/py/api/admin.py @@ -36,10 +36,13 @@ except ImportError: try: from ...utils import exit_signal + from ...utils.media_folders import is_skipped_folder except ImportError: from utils import exit_signal + from utils.media_folders import is_skipped_folder try: + from ...database.image_removal import mark_hides from ...database.pair_ingestion import SCAN_MARKS, ingest_resolved_pair from ...database.run_attribution import ( RecordedOutput, @@ -60,6 +63,7 @@ try: decide_file_prompt, ) except ImportError: + from database.image_removal import mark_hides from database.pair_ingestion import SCAN_MARKS, ingest_resolved_pair from database.run_attribution import ( RecordedOutput, @@ -118,10 +122,11 @@ DUPLICATE_FIND_LEGACY_TITLE = "Find duplicate image files (gallery page)" def _collect_media_files(output_dirs, extensions): """Media files under ``output_dirs`` (blocking), bounded and symlink-free. - Walks each root without following symlinks, skips ``thumbnails`` - directories and symlinked files, never descends more than MAX_SCAN_DEPTH - levels and stops after MAX_SCAN_FILES files in total. Extensions match - case-insensitively and each path is listed once. + Walks each root without following symlinks, skips ``thumbnails`` and + trash folders (``utils.media_folders``) and symlinked files, never + descends more than MAX_SCAN_DEPTH levels and stops after MAX_SCAN_FILES + files in total. Extensions match case-insensitively and each path is + listed once. """ wanted = {ext.lower() for ext in extensions} found = [] @@ -136,7 +141,7 @@ def _collect_media_files(output_dirs, extensions): dirnames[:] = sorted( d for d in dirnames - if d != "thumbnails" + if not is_skipped_folder(d) and depth < MAX_SCAN_DEPTH and not os.path.islink(os.path.join(current, d)) ) @@ -893,11 +898,8 @@ class AdminRoutesMixin: stat = os.stat(path) except OSError: continue - mark = marks.get(os.path.normcase(os.path.realpath(str(path)))) - if mark is not None and tuple(mark) == ( - stat.st_size, - stat.st_mtime_ns, - ): + locator = os.path.normcase(os.path.realpath(str(path))) + if locator in marks and mark_hides(marks, locator, path): continue key = _physical_file_key(path, stat) if key not in seen: diff --git a/py/api/autotag_routes.py b/py/api/autotag_routes.py index edc8ca82..d3ff1629 100644 --- a/py/api/autotag_routes.py +++ b/py/api/autotag_routes.py @@ -23,6 +23,11 @@ from .prompts import thumbnail_url_for from . import thumbnail_settings from .. import model_gate +try: + from ...utils.media_folders import is_skipped_folder +except ImportError: + from utils.media_folders import is_skipped_folder + from aiohttp import web _INTERNAL_ERROR = "An internal error occurred. Check server logs for details." @@ -1319,8 +1324,9 @@ class AutotagRoutesMixin: def _collect_scan_files(output_path: Path): """Media files under ``output_path`` from a bounded, symlink-free walk. - Descends at most ``_SCAN_MAX_DEPTH`` levels, skips ``thumbnails`` - trees and symlinked files, and stops after ``_SCAN_MAX_FILES`` files. + Descends at most ``_SCAN_MAX_DEPTH`` levels, skips ``thumbnails`` and + trash folders (``utils.media_folders``) and symlinked files, and stops + after ``_SCAN_MAX_FILES`` files. Returns ``(paths, truncated)``. """ found = [] @@ -1330,7 +1336,7 @@ class AutotagRoutesMixin: dirnames[:] = sorted( name for name in dirnames - if name != "thumbnails" and depth + 1 <= _SCAN_MAX_DEPTH + if not is_skipped_folder(name) and depth + 1 <= _SCAN_MAX_DEPTH ) for name in sorted(filenames): if Path(name).suffix.lower() not in _MEDIA_EXTENSIONS: @@ -1357,6 +1363,10 @@ class AutotagRoutesMixin: files, truncated = await self._run_in_executor( self._collect_scan_files, output_path ) + # #501: a file removed from the library is not offered for tagging. + files = await self._run_in_executor( + self._without_removed, output_path, files + ) has_thumbnails = (output_path / "thumbnails").is_dir() images = [] diff --git a/py/api/image_delete.py b/py/api/image_delete.py index 5abe4dbc..862cda26 100644 --- a/py/api/image_delete.py +++ b/py/api/image_delete.py @@ -47,12 +47,16 @@ from .maintenance import MAINTENANCE_PREVIEW_LEASE, _fail_job, _failure, backup_ try: from ...database import image_removal_schema + from ...database.image_removal import mark_hides from ...database.annotation_schema import MAX_LOCATOR_LENGTH from ...utils import trash + from ...utils.media_folders import is_trash_folder except ImportError: from database import image_removal_schema + from database.image_removal import mark_hides from database.annotation_schema import MAX_LOCATOR_LENGTH from utils import trash + from utils.media_folders import is_trash_folder IMAGE_DELETE = "delete_images" MODES = ("library", "trash") @@ -61,7 +65,6 @@ TITLES = { "trash": "Delete images (move to the trash)", } MAX_DETAIL_ROWS = 50 -TRASH_FOLDERS = frozenset({".trash", ".trashes", "$recycle.bin"}) # The system moved the file but its place in a trash could not be shown (#499). TRASH_UNCONFIRMED = "trash-unconfirmed" RECORDS_FAILED_CODE = "records-failed" @@ -94,6 +97,10 @@ REASONS = { TRASH_UNCONFIRMED: ( "Moved by macOS, but its place in the trash could not be confirmed." ), + trash.WINDOWS_RECYCLE: ( + "Moving files to the Recycle Bin isn't available on Windows: files larger " + "than the bin's capacity would be deleted permanently." + ), "not-attempted": "Not attempted: the job stopped after an error.", } _PROBLEM_CODES = { @@ -107,19 +114,12 @@ SOURCES_CHANGED = "The sources changed since the review; nothing was changed." RECORDS_FAILED = "The image records could not be removed; the job stopped." -def _is_trash_folder(name: str) -> bool: - folded = name.casefold() - return folded in TRASH_FOLDERS or ( - folded.startswith(".trash-") and folded[7:].isdigit() - ) - - def lexical_problem(relative: str) -> str | None: """The refusal code a root-relative POSIX path earns before any file access.""" parts = relative.split("/") if "thumbnails" in parts[:-1]: return "thumbnail" - if any(_is_trash_folder(part) for part in parts[:-1]): + if any(is_trash_folder(part) for part in parts[:-1]): return "trash-folder" if os.path.splitext(parts[-1])[1].lower() not in DELETABLE_MEDIA_EXTENSIONS: return "not-media" @@ -472,37 +472,49 @@ class ImageDeleteMixin: if not marks: return set() located = self._workspace_locators(listings) - hidden = set() - for locator, (size, mtime_ns) in marks.items(): - entry = located.get(locator) - if entry is None: - continue - try: - info = os.lstat(entry.path) - except OSError: - continue - if (info.st_size, info.st_mtime_ns) == (size, mtime_ns): - hidden.add(entry.normalized) - return hidden + return { + entry.normalized + for locator, entry in ((locator, located.get(locator)) for locator in marks) + if entry is not None and mark_hides(marks, locator, entry.path) + } def _removed_paths(self, paths) -> set: """The absolute paths among *paths* a removal mark hides (blocking).""" marks = self._removal_marks() if not marks: return set() - hidden = set() - for path in paths: - mark = marks.get(os.path.normcase(os.path.realpath(str(path)))) - if mark is None: - continue - try: - info = os.lstat(path) - except OSError: - continue - if (info.st_size, info.st_mtime_ns) == tuple(mark): - hidden.add(str(path)) + return { + str(path) + for path in paths + if mark_hides(marks, os.path.normcase(os.path.realpath(str(path))), path) + } + + def _removal_check_under(self, root): + """``hidden(path) -> bool`` for files a walk of *root* found (blocking). + + A walk never follows a link below *root*, so a file's locator is + ``normcase(join(realpath(root), relative))``: no ``realpath`` per + file, and an ``lstat`` only for a marked one. Without marks it + answers False at once. + """ + marks = self._removal_marks() + if not marks: + return lambda _path: False + root = os.fspath(root) + real = os.path.realpath(root) + + def hidden(path): + relative = os.path.relpath(os.fspath(path), root) + locator = os.path.normcase(os.path.join(real, relative)) + return mark_hides(marks, locator, path) + return hidden + def _without_removed(self, root, paths) -> list: + """*paths* found by a walk of *root*, less the hidden ones (blocking).""" + removed = self._removal_check_under(root) + return [path for path in paths if not removed(path)] + def parse_mode(value): """The apply's ``mode``, or None when it is not one of MODES.""" diff --git a/py/api/images.py b/py/api/images.py index 38286e2e..eb5636d9 100644 --- a/py/api/images.py +++ b/py/api/images.py @@ -66,6 +66,7 @@ try: from ...database.models import normalize_image_path from ...database.run_attribution import recorded_output_lookup from ...utils.card_seed import card_seed, header_params, parsed_positive + from ...utils.media_folders import in_skipped_folder, is_skipped_folder from ...utils.parallel import map_parallel from ...utils.json_strict import dumps_strict from ...utils.inspection_response import ( @@ -91,6 +92,7 @@ except ImportError: from database.models import normalize_image_path from database.run_attribution import recorded_output_lookup from utils.card_seed import card_seed, header_params, parsed_positive + from utils.media_folders import in_skipped_folder, is_skipped_folder from utils.parallel import map_parallel from utils.json_strict import dumps_strict from utils.inspection_response import ( @@ -915,7 +917,8 @@ def _name_suffix(name): def _walk_media(output_path, max_files): """Yield the ``os.DirEntry`` of each media file under *output_path*. - Symlinks are never followed, ``thumbnails/`` is skipped, descent stops + Symlinks are never followed, ``thumbnails/`` and trash folders are + skipped (``utils.media_folders``, #501), descent stops at MAX_SCAN_DEPTH and at most *max_files* entries are produced. The order is ``os.walk``'s (top-down, subfolders in ``os.scandir`` order, files by name), which the gallery's ties in mtime keep. Each entry's type comes @@ -927,8 +930,6 @@ def _walk_media(output_path, max_files): produced = 0 while stack: top, rel_parts = stack.pop() - if "thumbnails" in rel_parts: - continue dirs, files = [], [] try: with os.scandir(top) as entries: @@ -952,7 +953,7 @@ def _walk_media(output_path, max_files): if len(rel_parts) >= MAX_SCAN_DEPTH: continue for entry in reversed(dirs): - if not entry.is_symlink(): + if not entry.is_symlink() and not is_skipped_folder(entry.name): stack.append((entry.path, rel_parts + (entry.name,))) @@ -976,7 +977,7 @@ def _settled_entry(root, relative): *folders, name = relative.split(os.sep) if _name_suffix(name).lower() not in MEDIA_EXTENSIONS: return None - if "thumbnails" in folders or len(folders) > MAX_SCAN_DEPTH: + if in_skipped_folder(folders) or len(folders) > MAX_SCAN_DEPTH: return None folder = os.fspath(root) for part in ("", *folders): @@ -3075,7 +3076,9 @@ class ImageRoutesMixin: no symlinks), each preview folder is listed once, and at most MAX_THUMBNAIL_FILES targets are queued per batch (a job plans again after each batch); ``remaining`` counts the targets beyond that cap. - Sources in *exclude* (worked on by an earlier batch) are left out. + Sources in *exclude* (worked on by an earlier batch) are left out, and + so are files removed from the library (#501: a removal mark hides + them, and their previews went with their records). ``stop()`` true ends the walk before the next file. """ settings = writer_settings(settings) @@ -3084,11 +3087,14 @@ class ImageRoutesMixin: plan = {key: 0 for key in _PLAN_COUNTS} plan.update(targets=[], fresh_files=[]) folders = {} + removed = self._removal_check_under(output_path) for src in _iter_media_files(output_path, MAX_GALLERY_FILES): if stop is not None and stop(): break if exclude and src in exclude: continue + if removed(src): + continue rel_no_ext = src.relative_to(output_path).with_suffix("") stem = f"{rel_no_ext.name}{thumbnail_settings.THUMB_SUFFIX}" dst = thumbnails_dir / rel_no_ext.parent / (stem + settings.extension) diff --git a/py/api/jobs.py b/py/api/jobs.py index 52f016be..7196d43b 100644 --- a/py/api/jobs.py +++ b/py/api/jobs.py @@ -123,6 +123,9 @@ MODEL_DOWNLOAD_JOB_KIND = "model_download" DUPLICATES_JOB_KIND = "duplicates" DUPLICATE_GROUP_NAMES = ("groups_found",) DUPLICATE_CLEANUP_GROUP_NAMES = ("removed", "kept", "errors") +# The workspace Library's Delete… (#443, py/api/library_delete.py): a +# ``records`` hold around its safety backup and write. +PROMPT_DELETE_JOB_KIND = "prompt_delete" IMAGE_DELETE_JOB_KIND = "image_delete" # #480 IMAGE_DELETE_GROUP_NAMES = ("trashed", "removed", "not_changed", "errors") # The System health check (py/api/system.py): a read-only ``read`` job with diff --git a/py/api/library.py b/py/api/library.py index be699e70..02ca0088 100644 --- a/py/api/library.py +++ b/py/api/library.py @@ -157,6 +157,13 @@ def _archive_scope(request: web.Request) -> str: return value +def _shelf(request: web.Request) -> str: + value = request.query.get("shelf", "active") + if value not in ("active", "archived"): + raise ValueError("shelf must be active or archived") + return value + + def _missing() -> web.Response: return web.json_response( {"success": False, "error": "Positive prompt not found"}, status=404 @@ -332,6 +339,9 @@ class LibraryRoutesMixin: "tags": sorted(set(request.query.getall("tag", []))), "tag_mode": request.query.get("tag_mode", "all"), "sort": request.query.get("sort", "newest"), + # #443: which shelf; part of the cursor scope, so a cursor never + # crosses shelves. Not the variant ``archive`` scope of /{id}. + "shelf": _shelf(request), } limit = _page_limit(request) cursor = request.query.get("cursor") diff --git a/py/api/library_delete.py b/py/api/library_delete.py new file mode 100644 index 00000000..34038e04 --- /dev/null +++ b/py/api/library_delete.py @@ -0,0 +1,263 @@ +"""Delete or archive selected Library prompts, and restore archived ones (#443). + +The workspace Prompt library's Delete… (D152). The legacy ``POST +/prompt_manager/bulk/delete`` and ``DELETE /prompt_manager/delete/{id}`` are +separate and unchanged. + +- ``POST /prompt_manager/library/delete/preview {prompt_ids, expected_epoch}`` + classifies the selection (reads only, under a lease): the ids to ``delete`` + (in no saved pair), to ``archive`` (paired), the ``protected`` ones (tagged + ``__protected__``, kept), counts of ``already_archived`` and ``not_found``, + the ``image_links`` a delete removes, the safety ``backup`` policy and, + when a backup would be taken, the ``disk`` it needs. +- ``POST /prompt_manager/library/delete {archive_ids, delete_ids, + expected_epoch, backup}`` executes that review under a ``records`` hold + (refused 409 ``busy`` beside a scan, restore, maintenance, move or + Optimize). When a delete remains and a backup was promised (the reviewed + ``backup`` was "on" or the policy is on now) it checks the disk (507 + ``insufficient-space``) and takes a verified safety backup (500 + ``backup-failed``); then one transaction deletes the reviewed deletes whose + fingerprint is unchanged since the backup and archives the paired ones. +- ``POST /prompt_manager/library/restore {prompt_ids, expected_epoch}`` + removes the archive mark (under a lease). + +A changed storage epoch answers 409 ``storage-changed``; nothing is written. +""" + +import sqlite3 + +from aiohttp import web + +try: + from ...database.library_commands import LibraryCommandError + from ...database.safety_backup import SafetyBackupFailed +except ImportError: + from database.library_commands import LibraryCommandError + from database.safety_backup import SafetyBackupFailed + +from . import maintenance as _maintenance +from .admin import _prune_safety_backups +from .job_routes import busy_response +from .jobs import PROMPT_DELETE_JOB_KIND, JobBusy +from .prompts import parse_bulk_prompt_ids + +PREVIEW_URL = "/prompt_manager/library/delete/preview" +DELETE_URL = "/prompt_manager/library/delete" +RESTORE_URL = "/prompt_manager/library/restore" +DELETE_TITLE = "Delete prompts (Library)" +_SELECTION_KEYS = frozenset({"prompt_ids", "expected_epoch"}) +_DELETE_KEYS = frozenset({"archive_ids", "delete_ids", "expected_epoch", "backup"}) +STORAGE_CHANGED = "Prompt storage was restored; refresh the library" + + +def _failure(status, code, error, **extra): + return web.json_response( + {"success": False, "code": code, "error": error, **extra}, status=status + ) + + +def _invalid(message): + return _failure(400, "invalid-request", message) + + +def _storage_changed(exc): + return _failure(409, exc.code, STORAGE_CHANGED, library_epoch=exc.epoch) + + +async def _body(request, keys): + """The JSON object with exactly ``keys``, or raise ValueError.""" + try: + data = await request.json() + except ValueError: + raise ValueError("Expected a JSON object") from None + if not isinstance(data, dict): + raise ValueError("Expected a JSON object") + if set(data) != keys: + raise ValueError("Request must contain exactly " + ", ".join(sorted(keys))) + return data + + +def _reviewed_ids(value, name): + """A reviewed id list: possibly empty, else as ``parse_bulk_prompt_ids``.""" + if value == []: + return [] + try: + return parse_bulk_prompt_ids(value) + except ValueError as exc: + raise ValueError(str(exc).replace("prompt_ids", name)) from None + + +class LibraryDeleteRoutesMixin: + """Mixin providing the workspace Library's delete, archive and restore.""" + + def _register_library_delete_routes(self, routes): + routes.post(PREVIEW_URL)(self.preview_library_delete) + routes.post(DELETE_URL)(self.delete_library_prompts) + routes.post(RESTORE_URL)(self.restore_library_prompts) + + async def preview_library_delete(self, request): + try: + data = await _body(request, _SELECTION_KEYS) + ids = parse_bulk_prompt_ids(data["prompt_ids"]) + except ValueError as exc: + return _invalid(str(exc)) + db = self.db + policy = self._safety_backup_policy() + try: + async with self._jobs.lease("Library delete review"): + review = await self._run_in_executor( + db.classify_library_delete, + ids, + expected_epoch=data["expected_epoch"], + ) + disk = None + if review["delete"] and policy == "on": + public, short, _size = await self._run_in_executor( + _maintenance.disk_plan, + db.model.db_path, + backup=True, + vacuum=False, + ) + disk = { + **public, + "message": ( + None + if short is None + else _maintenance.short_space_message(short) + ), + } + except JobBusy as busy: + return busy_response(busy) + except LibraryCommandError as exc: + return _storage_changed(exc) + except ValueError as exc: + return _invalid(str(exc)) + except (OSError, sqlite3.Error): + self.logger.exception("Library delete review failed") + return _failure(500, "review-failed", "The delete could not be reviewed") + return web.json_response( + { + "success": True, + "library_epoch": review["library_epoch"], + "delete": review["delete"], + "archive": review["archive"], + "protected": review["protected"], + "already_archived": len(review["already_archived"]), + "not_found": len(review["not_found"]), + "image_links": review["image_links"], + "backup": policy, + "disk": disk, + } + ) + + async def delete_library_prompts(self, request): + try: + data = await _body(request, _DELETE_KEYS) + archive_ids = _reviewed_ids(data["archive_ids"], "archive_ids") + delete_ids = _reviewed_ids(data["delete_ids"], "delete_ids") + if not archive_ids and not delete_ids: + raise ValueError("Nothing to delete or archive") + if len(archive_ids) + len(delete_ids) > 1000: + raise ValueError("At most 1000 prompts can be changed at once") + if set(archive_ids) & set(delete_ids): + raise ValueError("A prompt cannot be both archived and deleted") + if data["backup"] not in ("on", "off"): + raise ValueError("backup must be on or off") + except ValueError as exc: + return _invalid(str(exc)) + busy = self._jobs.blocker(PROMPT_DELETE_JOB_KIND, admission="records") + if busy is not None: + return busy_response(JobBusy(busy)) + try: + async with self._jobs.hold( + PROMPT_DELETE_JOB_KIND, DELETE_TITLE, admission="records" + ) as job: + response = await self._held_library_delete( + self.db, data, archive_ids, delete_ids + ) + if response.status >= 400: + job.finish("failed") + return response + except JobBusy as busy: + return busy_response(busy) + + async def _held_library_delete(self, db, data, archive_ids, delete_ids): + epoch = data["expected_epoch"] + promised = data["backup"] == "on" or self._safety_backup_policy() == "on" + backup = None + try: + candidates = None + if delete_ids: + review = await self._run_in_executor( + db.classify_library_delete, delete_ids, expected_epoch=epoch + ) + candidates = review["fingerprints"] + if candidates and promised: + _disk, short, _size = await self._run_in_executor( + _maintenance.disk_plan, + db.model.db_path, + backup=True, + vacuum=False, + ) + if short is not None: + return _failure( + 507, + "insufficient-space", + _maintenance.short_space_message(short), + ) + backup = await self._run_in_executor( + _maintenance.take_safety_backup, db.model + ) + result = await self._run_in_executor( + db.archive_or_delete_library_prompts, + archive_ids, + delete_ids, + expected_epoch=epoch, + candidates=candidates if candidates is not None else {}, + ) + except LibraryCommandError as exc: + return _storage_changed(exc) + except SafetyBackupFailed as error: + self.logger.error("Safety backup before a Library delete failed") + return _failure( + 500, "backup-failed", _maintenance.backup_failed_message(error) + ) + except ValueError as exc: + return _invalid(str(exc)) + except (OSError, sqlite3.Error): + self.logger.exception("Library delete failed") + return _failure(500, "delete-failed", "The prompts could not be changed") + finally: + if backup is not None: + await self._prune_after_delete(db) + return web.json_response({"success": True, **result, "backup": backup}) + + async def _prune_after_delete(self, db): + try: + await self._run_in_executor(_prune_safety_backups, db.model.db_path) + except Exception: # pruning old copies never fails the delete + self.logger.exception("Could not prune old safety backups") + + async def restore_library_prompts(self, request): + try: + data = await _body(request, _SELECTION_KEYS) + ids = parse_bulk_prompt_ids(data["prompt_ids"]) + except ValueError as exc: + return _invalid(str(exc)) + try: + async with self._jobs.lease("Library restore"): + result = await self._run_in_executor( + self.db.restore_library_prompts, + ids, + expected_epoch=data["expected_epoch"], + ) + except JobBusy as busy: + return busy_response(busy) + except LibraryCommandError as exc: + return _storage_changed(exc) + except ValueError as exc: + return _invalid(str(exc)) + except (OSError, sqlite3.Error): + self.logger.exception("Library restore failed") + return _failure(500, "restore-failed", "The prompts could not be restored") + return web.json_response({"success": True, **result}) diff --git a/py/api/maintenance.py b/py/api/maintenance.py index 4a518efd..e7d34020 100644 --- a/py/api/maintenance.py +++ b/py/api/maintenance.py @@ -281,7 +281,7 @@ def short_space_message(short): ) -def _changed_line(operation, count): +def _changed_line(operation, count, marks=0): if operation == "optimize": return "Optimized the database" if operation == "duplicate_prompts": @@ -291,7 +291,11 @@ def _changed_line(operation, count): noun = "unused prompt component" if count == 1 else "unused prompt components" return f"Removed {count:,} {noun}" noun = "link to a missing file" if count == 1 else "links to missing files" - return f"Removed {count:,} {noun}" + line = f"Removed {count:,} {noun}" + if marks: + gone = "removed image" if marks == 1 else "removed images" + line += f"; forgot {marks:,} {gone} whose files are gone or changed" + return line def take_safety_backup(model, label="backup"): @@ -419,6 +423,7 @@ class MaintenanceRoutesMixin: draft.public(), { "targets": draft.targets, + "marks": draft.marks, "roots": roots, "protected": draft.protected["count"], "rows": draft.scope["count"], @@ -653,6 +658,7 @@ class MaintenanceRoutesMixin: plan.fingerprint, payload["targets"], payload["roots"], + payload.get("marks", ()), ) except PlanChanged: return _fail_job(job, "plan-changed", PLAN_CHANGED) @@ -664,7 +670,7 @@ class MaintenanceRoutesMixin: ) job.record_counted("changed", result["changed"], rows) job.record_counted("protected", payload["protected"]) - line = _changed_line(plan.operation, result["changed"]) + line = _changed_line(plan.operation, result["changed"], result.get("marks", 0)) job.log_line(line) job.update(progress=100, status_line=f"{line} · {backup_note(backup)}") diff --git a/tests/browser/library-concept.spec.mjs b/tests/browser/library-concept.spec.mjs index be8bf28e..d7387aaf 100644 --- a/tests/browser/library-concept.spec.mjs +++ b/tests/browser/library-concept.spec.mjs @@ -44,7 +44,12 @@ export function mapLibraryConcept(reference) { // "4 runs" and "Used today". The two-line clamp is // applied below through `clampedCards`, measured on the concept with `CLAMP_CSS` (its samples wrap at 390px). footer:card.footer})); - return {...reference,library:{...library,toolbar, + // D152 (#443): Show archived prompts (N) comes first in the heading actions, a plain button like Export (the concept + // fixture has nothing archived, so N is 0). + const actions=reference.heading.actions; + const plain=actions.buttons.find(button=>!button.primary); + const heading={...reference.heading,actions:{...actions,buttons:[{...plain,text:'Show archived prompts (0)'},...actions.buttons]}}; + return {...reference,heading,library:{...library,toolbar, // D35: the caption uses the singular for one. caption:{...library.caption,top:library.caption.top-rise,spans:library.caption.spans.map(span=>({...span,text:countCopy(span.text)}))}, ...(({grid,cards:clamped})=>({grid:{...grid,top:grid.top-rise},cards:clamped}))(clampedCards({grid:library.grid,cards})), diff --git a/tests/browser/library-delete-concept.spec.mjs b/tests/browser/library-delete-concept.spec.mjs new file mode 100644 index 00000000..6f373449 --- /dev/null +++ b/tests/browser/library-delete-concept.spec.mjs @@ -0,0 +1,147 @@ +import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs'; +import { test, expect } from './fixtures.mjs'; +import { SIZES, THEMES, seedConcept, openConcept, openProduction, diffInventory, mutate } from './concept-inventory.mjs'; + +// #443 D152: the Library's Delete… review against the concept's own reviewed archive: the recipe editor's "Archive…" +// (app.js 976), which opens `maintenancePlan("archive ")` (953–958, 1309–1310). The concept's shell, grid tokens, +// `p.small` and actions are kept; its title, demo note, rows, note text and `ca-primary` start are replaced by D152's, +// and the start takes the concept's `danger` class, measured on the editor's own "Archive…". The bar's Delete… is +// compared with the concept Images bar's danger action, "Remove links…" (app.js 432). +test.use({ locale: 'en-US', timezoneId: 'UTC' }); + +test.beforeEach(async ({ page, request, browserName }) => { + await startCoverage(page, browserName); + await seedConcept(request); + page.conceptErrors = []; + page.on('pageerror', error => page.conceptErrors.push(error.message)); +}); +test.afterEach(async ({ page, browserName }) => { + await saveCoverage(page, browserName); + expect(page.conceptErrors).toEqual([]); +}); + +/** Runs in the page: the open review dialog's title, body tree, grid, notes and actions with the concept's tokens. */ +function measureReview() { + const dialog = document.querySelector('dialog#modal[open]'); + const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null; + const name = node => [node.localName, ...[...node.classList].sort()].join('.'); + const text = node => node?.textContent.trim() ?? null; + const body = dialog.querySelector('#modal-body'); + const grid = body.querySelector(':scope > dl.meta-grid'); + const notes = [...body.querySelectorAll(':scope > p.small')]; + const actions = body.querySelector(':scope > .dialog-actions'); + return { + title: text(dialog.querySelector('#modal-title')), + tree: [...body.children].map(name).filter(item => item !== 'p.demo-note' && item !== 'h3'), + grid: { rows: [...grid.children].map(row => [text(row.querySelector('dt')), text(row.querySelector('dd'))]), + ...css(grid, 'display', 'gridTemplateColumns', 'columnGap', 'rowGap'), + dt: css(grid.querySelector('dt'), 'fontSize', 'color', 'textTransform'), dd: css(grid.querySelector('dd'), 'fontSize', 'color') }, + notes: notes.map(note => ({ text: text(note), ...css(note, 'fontSize', 'color', 'marginTop') })), + actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'), + buttons: [...actions.children].map(item => ({ text: text(item), className: item.className, + ...css(item, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) }, + }; +} + +/** Runs in the page: a button's class and tokens (the concept's danger button, or production's). */ +function measureButton(selector) { + const node = document.querySelector(selector); + const style = getComputedStyle(node); + return { className: node.className, ...Object.fromEntries(['backgroundColor', 'color', 'fontSize', 'fontWeight', 'paddingTop', + 'paddingLeft', 'borderTopColor', 'borderTopLeftRadius', 'height'].map(key => [key, key === 'height' + ? `${Math.round(node.getBoundingClientRect().height * 100) / 100}px` : style[key]])) }; +} + +async function conceptReview(page, theme) { + await openConcept(page, 'prompt/alpine', theme); + await page.locator('#main .heading-actions').getByRole('button', { name: 'Edit metadata', exact: true }).click(); + const archive = page.locator('dialog#modal[open] .dialog-actions button', { hasText: 'Archive…' }); + await expect(archive).toBeVisible();await page.mouse.move(0, 0); + const danger = await page.evaluate(measureButton, 'dialog#modal[open] .dialog-actions button.danger'); + await archive.click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Review maintenance plan'); + await page.mouse.move(0, 0); + return { danger, review: await page.evaluate(measureReview) }; +} + +async function seedMixed(request) { + // One unpaired prompt beside the concept's paired ones, so the review deletes one and archives one. + await expect(await request.post('/prompt_manager/save', { data: { text: 'Unpaired study', category: 'Studies' } })).toBeOK(); + const items = (await (await request.get('/prompt_manager/library?limit=100')).json()).items; + return { unpaired: items.find(item => item.text === 'Unpaired study'), paired: items.find(item => item.variant_count > 0) }; +} + +async function productionReview(page, request, theme) { + const { unpaired, paired } = await seedMixed(request); + await openProduction(page, 'library', theme); + for (const item of [unpaired, paired]) await page.locator(`article[data-prompt-id="${item.id}"] .image-overlays input`).check(); + await page.locator('#library-bulkbar .bulkbar').getByRole('button', { name: 'Delete…', exact: true }).click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Delete or archive selected prompts'); + await page.mouse.move(0, 0); + return page.evaluate(measureReview); +} + +/** D152's changes to the concept's review; everything else must equal the concept. */ +function mapDeleteConcept({ danger, review }) { + const [cancel] = review.actions.buttons; + const note = review.notes.at(-1); + return { + ...review, + title: 'Delete or archive selected prompts', + tree: ['dl.meta-grid', 'p.small', 'p.small', 'div.dialog-actions'], + grid: { ...review.grid, rows: [['Delete permanently', '1 prompt not in a saved pair'], + ['Archive', '1 prompt in a saved pair: hidden, restorable'], + ['Image links removed', '0 images will no longer show these prompts (files stay on disk)'], + ['Backup', 'Create and verify before deleting']] }, + notes: [ + { ...note, text: 'Prompts in a saved pair, as its positive or its negative, are archived instead of deleted, so the ' + + 'pair and its runs stay intact. Archived prompts cannot be deleted. To bring them back, press Show archived prompts, ' + + 'select them and press Restore.' }, + // A second paragraph takes the concept's own `p + p` margin (style.css 93-95). + { ...note, marginTop: '10px', text: "Deleting removes each prompt's text, title, category, tags, rating, notes, run count, last use and " + + 'image links. Deleted prompts cannot be brought back one by one: the only way back is to restore the safety ' + + 'backup taken first (System › Backup & export), which returns the whole library to that moment. It is kept ' + + 'among the last 10 safety backups. Image files, favorites, collections and recorded runs stay. A later scan, a ' + + 'new image or a node run with the same text adds it again as a new prompt.' }, + ], + actions: { ...review.actions, tree: [review.actions.tree[0], 'button.danger'], buttons: [cancel, { text: 'Delete 1 and archive 1 prompt', className: danger.className, + backgroundColor: danger.backgroundColor, color: danger.color, fontSize: danger.fontSize, paddingTop: danger.paddingTop, + paddingLeft: danger.paddingLeft, borderTopColor: danger.borderTopColor }] }, + }; +} + +for (const theme of THEMES) for (const [width, height] of SIZES) { + test(`the Delete… review matches the concept's reviewed archive ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => { + await page.setViewportSize({ width, height }); + const concept = await conceptReview(page, theme); + expect(concept.review.tree).toEqual(['dl.meta-grid', 'p.small', 'div.dialog-actions']); + await page.screenshot({ path: testInfo.outputPath(`concept-archive-review-${theme}-${width}.png`) }); + const production = await productionReview(page, request, theme); + await page.screenshot({ path: testInfo.outputPath(`production-delete-review-${theme}-${width}.png`) }); + expect(diffInventory(mapDeleteConcept(concept), production)).toEqual([]); + }); +} + +test('the Delete… review comparison detects mutations', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const concept = await conceptReview(page, 'dark'); + await productionReview(page, request, 'dark'); + const expected = mapDeleteConcept(concept); + const paths = async () => diffInventory(expected, await page.evaluate(measureReview)).map(item => item.path); + expect(await paths()).toEqual([]); + await mutate(page, { css: '#modal-body .dialog-actions .danger { padding-left: 2px }' }); + expect(await paths()).toContain('actions.buttons.1.paddingLeft'); +}); + +test('the bar\'s Delete… matches the concept Images bar\'s danger action', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + await openConcept(page, 'images', 'dark'); + await page.locator('#image-grid .media-card').first().locator('input[data-select]').check(); + await expect(page.locator('#bulkbar .bulkbar')).toBeVisible();await page.mouse.move(0, 0); + const reference = await page.evaluate(measureButton, '#bulkbar .bulkbar button.danger'); + const { paired } = await seedMixed(request); + await openProduction(page, 'library', 'dark'); + await page.locator(`article[data-prompt-id="${paired.id}"] .image-overlays input`).check(); + await expect(page.locator('#library-bulkbar .bulkbar')).toBeVisible();await page.mouse.move(0, 0); + expect(await page.evaluate(measureButton, '#library-bulkbar .bulkbar button.danger')).toEqual(reference); +}); diff --git a/tests/browser/system-backups-concept.spec.mjs b/tests/browser/system-backups-concept.spec.mjs index 8655d621..a2c06b1d 100644 --- a/tests/browser/system-backups-concept.spec.mjs +++ b/tests/browser/system-backups-concept.spec.mjs @@ -87,8 +87,8 @@ function measureBackups(mainSelector) { } const SCHEDULE_REASON = 'Daily and Weekly are not available yet: back up with Back up whenever you choose.'; -const SAFETY_DESCRIPTION = 'Keep a verified copy of the database on disk before maintenance, cleanups and LoRA imports change ' + - 'records. A restore always keeps one.'; +const SAFETY_DESCRIPTION = 'Keep a verified copy of the database on disk before maintenance, cleanups, LoRA imports and ' + + 'prompt deletes change records. A restore always keeps one.'; /** * D103: "Back up" (the concept's "Simulate backup"), the status line from /backup/status, "Export JSON" (its "Export diff --git a/tests/browser/workspace-images-delete.spec.mjs b/tests/browser/workspace-images-delete.spec.mjs index bf727731..7051166b 100644 --- a/tests/browser/workspace-images-delete.spec.mjs +++ b/tests/browser/workspace-images-delete.spec.mjs @@ -73,6 +73,19 @@ for (const mode of ['library', 'trash']) { }); } +// #501: the sidebar's Images count (read once at mount) follows a finished delete without a reload. +test('a finished delete re-reads the sidebar Images count', async ({ page }) => { + await openImages(page); + const badge = page.locator('#primary-nav a[data-route="images"] .nav-count'); + await expect(badge).toHaveText(/^\d[\d,]*$/); + const before = Number((await badge.textContent()).replace(/,/g, '')); + await check(page, 0); + const { primary } = await deleteChecked(page, 'library'); + await primary.click(); + await expect(toast(page)).toContainText('Removed 1 image from the library.'); + await expect(badge).toHaveText((before - 1).toLocaleString('en-US')); +}); + test('Cancel changes nothing', async ({ page }) => { const before = await openImages(page); await check(page, 0); diff --git a/tests/browser/workspace-library-delete.spec.mjs b/tests/browser/workspace-library-delete.spec.mjs new file mode 100644 index 00000000..59b72af8 --- /dev/null +++ b/tests/browser/workspace-library-delete.spec.mjs @@ -0,0 +1,215 @@ +import {startCoverage,saveCoverage} from '../../tools/vendor/browser-coverage.mjs'; +import {test,expect} from './fixtures.mjs'; +import {armUnseenRestores,unseenRestore} from './unseen-restore.mjs'; + +// #443 D152: Delete… on the Library bar deletes unpaired prompts, archives paired ones (restorable), keeps protected ones. +const route='/prompt_manager/workspace#/library'; +const PREVIEW='/prompt_manager/library/delete/preview'; +const DELETE='/prompt_manager/library/delete'; +const cards=page=>page.locator('article[data-prompt-id]'); +const card=(page,id)=>page.locator(`article[data-prompt-id="${id}"]`); +const box=(page,id)=>card(page,id).locator('.image-overlays input[type="checkbox"]'); +const bar=page=>page.locator('#library-bulkbar > .bulkbar'); +const toast=page=>page.locator('.toast'); +const review=page=>page.getByRole('dialog'); +const showArchived=page=>page.locator('#workspace-main .heading-actions').getByRole('button',{name:/^Show archived prompts/}); + +async function items(request,shelf='active') { + const query=shelf==='archived'?'&shelf=archived':''; + return (await(await request.get(`/prompt_manager/library?limit=100${query}`)).json()).items; +} +const byText=(list,text)=>list.find(item=>item.text===text); +function recordPosts(page,path) { + const posts=[]; + page.on('request',item=>{if(item.method()==='POST'&&new URL(item.url()).pathname===path)posts.push(item.postDataJSON());}); + return posts; +} +async function openReview(page) { + await bar(page).getByRole('button',{name:'Delete…',exact:true}).click(); + await expect(review(page)).toBeVisible(); +} + +test.beforeEach(async({page,request,browserName})=>{ + await startCoverage(page,browserName);await expect(await request.post('/__test__/reset',{data:{native_library:true,extra_prompts:2}})).toBeOK(); + page.libraryErrors=[];page.on('pageerror',error=>page.libraryErrors.push(error.message)); +}); +test.afterEach(async({page,browserName})=>{await saveCoverage(page,browserName);expect(page.libraryErrors).toEqual([]);}); + +test('Delete… reviews the selection, deletes the unpaired, archives the paired and keeps the protected',async({page,request})=>{ + const before=await items(request); + const dog=byText(before,'Dog landscape'),cat=byText(before,'Cat portrait'),guarded=byText(before,'Library extra 0000'); + await expect(await request.post('/prompt_manager/bulk/tags',{data:{prompt_ids:[guarded.id],tags:['__protected__']}})).toBeOK(); + const previews=recordPosts(page,PREVIEW),writes=recordPosts(page,DELETE); + await page.goto(route);await expect(cards(page)).toHaveCount(before.length); + for(const item of [dog,cat,guarded]) await box(page,item.id).check(); + await expect(bar(page).getByRole('button')).toHaveText([`Select all ${before.length} loaded`,'Add tags','Set category','Delete…','Clear']); + await expect(bar(page).getByRole('button',{name:'Delete…',exact:true})).toHaveClass(/danger/); + await openReview(page); + await expect(review(page)).toHaveAccessibleName('Delete or archive selected prompts'); + const grid=review(page).locator('dl.meta-grid > div'); + await expect(grid.locator('dt')).toHaveText(['Delete permanently','Archive','Protected (kept)','Image links removed','Backup']); + await expect(grid.locator('dd')).toHaveText(['1 prompt not in a saved pair','1 prompt in a saved pair: hidden, restorable', + '1: Library extra 0000','0 images will no longer show these prompts (files stay on disk)','Create and verify before deleting']); + await expect(review(page).locator('p.small')).toHaveCount(3); + await expect(review(page).getByRole('button',{name:'Delete 1 and archive 1 prompt',exact:true})).toHaveClass(/danger/); + // Cancel writes nothing and releases the bar. + await review(page).getByRole('button',{name:'Cancel',exact:true}).click(); + await expect(review(page)).toHaveCount(0);expect(writes).toEqual([]); + await expect(bar(page).getByRole('button',{name:'Delete…',exact:true})).not.toHaveAttribute('aria-disabled','true'); + await openReview(page); + await review(page).getByRole('button',{name:'Delete 1 and archive 1 prompt',exact:true}).click(); + await expect(toast(page)).toHaveText(/^Deleted 1 prompt\. Archived 1 prompt\. 1 protected prompt was kept\. Safety backup saved: \S+\.backup_\S+ \(System › Backup & export\)\.$/); + await expect(page.locator('#library-bulkbar')).toBeEmpty(); + await expect(card(page,dog.id)).toHaveCount(0);await expect(card(page,cat.id)).toHaveCount(0);await expect(card(page,guarded.id)).toHaveCount(1); + const epoch=(await(await request.get('/prompt_manager/library/options')).json()).library_epoch; + // One preview per review opened (Cancel, then Delete…). + const reviewed={prompt_ids:[dog.id,cat.id,guarded.id],expected_epoch:epoch}; + expect(previews).toEqual([reviewed,reviewed]); + expect(writes).toEqual([{archive_ids:[cat.id],delete_ids:[dog.id],expected_epoch:epoch,backup:'on'}]); + const after=await items(request); + expect(byText(after,'Dog landscape')).toBeUndefined();expect(byText(after,'Cat portrait')).toBeUndefined(); + expect((await items(request,'archived')).map(item=>item.id)).toEqual([cat.id]); + await expect(showArchived(page)).toHaveText('Show archived prompts (1)'); +}); + +test('Show archived prompts lists archived prompts, and Restore on its bar returns them',async({page,request})=>{ + const before=await items(request);const cat=byText(before,'Cat portrait'),dual=byText(before,'Dual role portrait'); + await page.goto(route);await expect(cards(page)).toHaveCount(before.length); + await box(page,cat.id).check();await box(page,dual.id).check(); + await openReview(page); + await expect(review(page)).toHaveAccessibleName('Archive selected prompts'); + // The archive reason is the saved pair, for one or many prompts (live note on #470): never negative variants. + await expect(review(page).locator('dl.meta-grid > div').filter({hasText:'Archive'}).locator('dd')).toHaveText('2 prompts in a saved pair: hidden, restorable'); + await review(page).getByRole('button',{name:'Archive 2 prompts',exact:true}).click(); + await expect(toast(page)).toHaveText('Archived 2 prompts.'); + await expect(cards(page)).toHaveCount(before.length-2); + await expect(showArchived(page)).toHaveAttribute('aria-pressed','false'); + await showArchived(page).click(); + await expect(showArchived(page)).toHaveAttribute('aria-pressed','true'); + await expect(cards(page)).toHaveCount(2); + await expect(page.locator('.section-caption > span').first()).toHaveText(/^2 archived prompts · \d+ negative variants? hidden with them$/); + await expect(card(page,cat.id).locator('.card-title .chip',{hasText:'Archived'})).toHaveCount(1); + await box(page,cat.id).check(); + await expect(bar(page).getByRole('button')).toHaveText(['Select all 2 loaded','Restore','Clear']); + await bar(page).getByRole('button',{name:'Restore',exact:true}).click(); + await expect(toast(page)).toHaveText('Restored 1 prompt to the Prompt library.'); + await expect(cards(page)).toHaveCount(1); + await showArchived(page).click(); + await expect(cards(page)).toHaveCount(before.length-1); + await expect(card(page,cat.id)).toHaveCount(1); + expect((await items(request,'archived')).map(item=>item.id)).toEqual([dual.id]); + // An empty archived view says so. + await showArchived(page).click();await box(page,dual.id).check(); + await bar(page).getByRole('button',{name:'Restore',exact:true}).click(); + await expect(page.locator('.library-status')).toHaveText('No archived prompts.'); +}); + +test('an archived prompt\'s page says so and restores it from its notice',async({page,request})=>{ + const cat=byText(await items(request),'Cat portrait'); + const epoch=(await(await request.get('/prompt_manager/library/options')).json()).library_epoch; + await expect(await request.post(DELETE,{data:{archive_ids:[cat.id],delete_ids:[],expected_epoch:epoch,backup:'off'}})).toBeOK(); + await page.goto(`/prompt_manager/workspace#/library/${cat.id}`); + const notice=page.locator('.library-archived-notice'); + await expect(notice.locator('h2')).toHaveText('Archived'); + // The notice says why (the saved pair) and counts the prompt's negative variants (live note on #470). + expect(cat.variant_count).toBeGreaterThan(1); + await expect(notice.locator('p').first()).toHaveText(`This prompt is archived because it is in a saved pair. It and its ${cat.variant_count} negative variants are hidden from the Prompt library; nothing was removed.`); + await notice.getByRole('button',{name:'Restore this prompt to the Prompt library'}).click(); + await expect(toast(page)).toHaveText('Restored 1 prompt to the Prompt library.'); + await expect(notice).toHaveCount(0); + expect(byText(await items(request),'Cat portrait')?.id).toBe(cat.id); +}); + +// #443: a starved renderer (headless WebKit under CPU load, a background tab) can hold animation frames for +// seconds; a loaded page of prompts is shown without waiting for one, on first load and after a delete. +async function holdAnimationFrames(page) { + await page.addInitScript(()=>{ + const request=window.requestAnimationFrame.bind(window),cancel=window.cancelAnimationFrame.bind(window); + const held=new Map(),real=new Map();let holding=true,next=0; + window.requestAnimationFrame=callback=>{const id=--next;if(holding)held.set(id,callback);else real.set(id,request(callback));return id;}; + window.cancelAnimationFrame=id=>{held.delete(id);if(real.has(id))cancel(real.get(id));real.delete(id);}; + window.__releaseAnimationFrames=()=>{holding=false;for(const [id,callback] of held)real.set(id,request(callback));held.clear();}; + }); +} +test('a held animation frame does not hide the loaded prompts, on first load or after a delete',async({page,request})=>{ + const before=await items(request);const dog=byText(before,'Dog landscape'); + await holdAnimationFrames(page); + await page.goto(route);await expect(cards(page)).toHaveCount(before.length); + await box(page,dog.id).check();await openReview(page); + await review(page).getByRole('button',{name:'Delete 1 prompt',exact:true}).click(); + await expect(toast(page)).toHaveText(/^Deleted 1 prompt\./); + await expect(cards(page)).toHaveCount(before.length-1);await expect(card(page,dog.id)).toHaveCount(0); + await page.evaluate(()=>window.__releaseAnimationFrames()); + await expect(cards(page)).toHaveCount(before.length-1); +}); + +test('a double press sends one preview, and the bar holds while it is reviewed',async({page,request})=>{ + const dog=byText(await items(request),'Dog landscape'); + const previews=recordPosts(page,PREVIEW); + let release;const held=new Promise(resolve=>{release=resolve;}); + await page.route(`**${PREVIEW}`,async item=>{await held;await item.fallback();}); + await page.goto(route);await box(page,dog.id).check(); + const remove=bar(page).getByRole('button',{name:/^(Delete…|Reviewing…)$/}); + await remove.click(); + await expect(remove).toHaveText('Reviewing…');await expect(remove).toHaveAttribute('aria-disabled','true'); + await expect(bar(page).getByRole('button',{name:'Clear',exact:true})).toHaveAttribute('aria-disabled','true'); + await remove.click({force:true}); + release(); + await expect(review(page)).toHaveAccessibleName('Delete selected prompts'); + expect(previews).toHaveLength(1); + await review(page).getByRole('button',{name:'Cancel',exact:true}).click(); + await expect(bar(page).getByRole('button',{name:'Delete…',exact:true})).not.toHaveAttribute('aria-disabled','true'); +}); + +test('a restore since the page loaded, a failed backup, a busy job and an unconfirmed answer each say what happened',async({page,request})=>{ + const before=await items(request);const dog=byText(before,'Dog landscape'); + await armUnseenRestores(page); + await page.goto(route);await expect(cards(page)).toHaveCount(before.length); + // Failed backup: nothing changed, the selection stays. + await page.route(`**${DELETE}`,item=>item.fulfill({status:500,contentType:'application/json', + body:JSON.stringify({success:false,code:'backup-failed',error:'The safety backup could not be taken and verified; nothing was changed.'})})); + await box(page,dog.id).check();await openReview(page); + await review(page).getByRole('button',{name:'Delete 1 prompt',exact:true}).click(); + await expect(toast(page)).toHaveText('The safety backup failed. The safety backup could not be taken and verified; nothing was changed.'); + await expect(box(page,dog.id)).toBeChecked(); + // Busy: names the job, nothing changed. + await page.unroute(`**${DELETE}`); + await page.route(`**${DELETE}`,item=>item.fulfill({status:409,contentType:'application/json', + body:JSON.stringify({success:false,code:'busy',error:'busy',running:{id:'abc',title:'Scan'}})})); + await openReview(page);await review(page).getByRole('button',{name:'Delete 1 prompt',exact:true}).click(); + await expect(toast(page)).toHaveText('Scan is running. Nothing was changed; try again when it finishes.'); + // Unconfirmed: the shelf reloads. + await page.unroute(`**${DELETE}`); + await page.route(`**${DELETE}`,item=>item.abort()); + await openReview(page);await review(page).getByRole('button',{name:'Delete 1 prompt',exact:true}).click(); + await expect(toast(page)).toHaveText('The server did not confirm the change. The Library was reloaded: check the shelf and Show archived prompts.'); + await expect(page.locator('#library-bulkbar')).toBeEmpty(); + await page.unroute(`**${DELETE}`); + // A restore the page has not seen: the review opened before it is refused on the server. + await box(page,dog.id).check();await openReview(page); + const backup=await request.get('/prompt_manager/backup');await expect(backup).toBeOK();const buffer=await backup.body(); + const restore=await unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'library.db',mimeType:'application/octet-stream',buffer}}})); + await expect(restore).toBeOK(); + await review(page).getByRole('button',{name:'Delete 1 prompt',exact:true}).click(); + await expect(toast(page)).toHaveText('The library was restored or moved since this page loaded. Nothing was deleted or archived.'); + expect(byText(await items(request),'Dog landscape')).toBeDefined(); +}); + +test('a failed review keeps the selection, and a selection nothing can change opens the close-only review',async({page,request})=>{ + const before=await items(request);const guarded=byText(before,'Library extra 0000'); + await expect(await request.post('/prompt_manager/bulk/tags',{data:{prompt_ids:[guarded.id],tags:['__protected__']}})).toBeOK(); + await page.goto(route);await expect(cards(page)).toHaveCount(before.length); + await box(page,guarded.id).check(); + await page.route(`**${PREVIEW}`,item=>item.fulfill({status:500,contentType:'application/json',body:JSON.stringify({success:false,code:'review-failed',error:'x'})}),{times:1}); + await bar(page).getByRole('button',{name:'Delete…',exact:true}).click(); + await expect(toast(page)).toHaveText('The selected prompts could not be reviewed. Try again.'); + await expect(box(page,guarded.id)).toBeChecked(); + await expect(bar(page).getByRole('button',{name:'Delete…',exact:true})).not.toHaveAttribute('aria-disabled','true'); + await openReview(page); + await expect(review(page)).toHaveAccessibleName('Delete selected prompts'); + await expect(review(page).locator('p.small').last()).toHaveText('None of the selected prompts can be deleted or archived.'); + await expect(review(page).locator('.dialog-actions > button')).toHaveText(['Close']); + await review(page).getByRole('button',{name:'Close',exact:true}).click(); + await expect(bar(page).getByRole('button',{name:'Delete…',exact:true})).not.toHaveAttribute('aria-disabled','true'); + expect(byText(await items(request),'Library extra 0000')).toBeDefined(); +}); diff --git a/tests/browser/workspace-library-editor.spec.mjs b/tests/browser/workspace-library-editor.spec.mjs index 42def0d0..725b5427 100644 --- a/tests/browser/workspace-library-editor.spec.mjs +++ b/tests/browser/workspace-library-editor.spec.mjs @@ -493,13 +493,15 @@ test('a restore seen finishing keeps New prompt open and reloads the Library aft test('library heading offers Export then + New prompt, and New opens the saved prompt history',async({page,request})=>{ await page.goto(`${workspace}#/library`);await expect(page.locator('article[data-prompt-id]').first()).toBeVisible(); const actions=headingActions(page).locator('button'); - expect(await visibleNames(actions)).toEqual(['Export','+ New prompt']); + // D152 (#443): Show archived prompts (N) comes first, a plain toggle. + expect(await visibleNames(actions)).toEqual(['Show archived prompts (0)','Export','+ New prompt']); + await expect(actions.nth(0)).toHaveAttribute('aria-pressed','false');await expect(actions.nth(0)).not.toHaveClass(/ca-primary/); // One image is counted in the singular. const f=page.fixture;await expect(page.locator(`article[data-prompt-id="${f.positive_id}"]`).getByText('1 image',{exact:true})).toHaveCount(1); - await expect(actions.nth(0)).toHaveAccessibleName('Export full Library JSON');await expect(actions.nth(0)).not.toHaveClass(/ca-primary/); - await expect(actions.nth(1)).toHaveClass(/\bca-primary\b/); - for(const index of [0,1])expect((await actions.nth(index).boundingBox()).height).toBeGreaterThanOrEqual(32); - await actions.nth(1).click();const dialog=editorDialog(page);await expect(dialog.getByRole('heading',{name:'New prompt',exact:true})).toBeVisible();await ready(dialog); + await expect(actions.nth(1)).toHaveAccessibleName('Export full Library JSON');await expect(actions.nth(1)).not.toHaveClass(/ca-primary/); + await expect(actions.nth(2)).toHaveClass(/\bca-primary\b/); + for(const index of [0,1,2])expect((await actions.nth(index).boundingBox()).height).toBeGreaterThanOrEqual(32); + await actions.nth(2).click();const dialog=editorDialog(page);await expect(dialog.getByRole('heading',{name:'New prompt',exact:true})).toBeVisible();await ready(dialog); await field(dialog,'Title').fill('Harbor at dawn');await field(dialog,'Positive prompt').fill('A quiet harbor at dawn');await field(dialog,'Negative prompt (optional)').fill('fog, noise'); await button(dialog,'Save').click();await expect(dialog).toHaveCount(0); const created=(await(await request.get('/prompt_manager/library?text=quiet%20harbor')).json()).items[0]; @@ -633,7 +635,11 @@ test('restored storage after a confirmed save or before an archive offers an exp const response=await route.fetch();await expect(await restore()).toBeOK();return route.fulfill({response}); }); await page.getByRole('button',{name:'Edit metadata',exact:true}).click();let dialog=editorDialog(page);await ready(dialog); - await field(dialog,'Notes').fill('Lost to a restore');await button(dialog,'Save').click(); + await field(dialog,'Notes').fill('Lost to a restore'); + // The save's answer waits for a whole restore in the route above; wait for that answer, not the 5 s expect + // default (#501 merge queue: under load the restore outlasted it on WebKit). + const saved=page.waitForResponse(item=>item.request().method()==='PATCH'&&new URL(item.url()).pathname===`/prompt_manager/library/${f.positive_id}`); + await button(dialog,'Save').click();await saved; await expect(note(dialog)).toHaveText('Saved; Library refresh failed because prompt storage was restored. Reload the Library to see current data.'); expect(await actionNames(dialog)).toEqual(['Reload Library','Create text revision','Archive…','Cancel','Save']); for(const name of ['Create text revision','Archive…'])await expect(button(dialog,name)).toBeDisabled(); diff --git a/tests/browser/workspace-library-selection.spec.mjs b/tests/browser/workspace-library-selection.spec.mjs index d92eeb76..517973db 100644 --- a/tests/browser/workspace-library-selection.spec.mjs +++ b/tests/browser/workspace-library-selection.spec.mjs @@ -62,7 +62,8 @@ test('Set category writes the selected prompts on the server, reloads the shelf await box(page,first.id).check();await expect(bar(page).locator('strong')).toHaveText('1 selected'); await box(page,second.id).check();await expect(bar(page).locator('strong')).toHaveText('2 selected'); // Two of three loaded are checked, so Select all N loaded follows the count (D151, #444 ruling). - await expect(bar(page).getByRole('button')).toHaveText([`Select all ${items.length} loaded`,'Add tags','Set category','Clear']); + // D152 (#443) adds Delete… before Clear. + await expect(bar(page).getByRole('button')).toHaveText([`Select all ${items.length} loaded`,'Add tags','Set category','Delete…','Clear']); await setCategory(page,' Studio set '); await expect(toast(page)).toHaveText('Category “Studio set” set on 2 prompts.'); await expect(dialog(page)).toHaveCount(0); @@ -287,7 +288,7 @@ test('Ctrl/Cmd+A and Select all N loaded select every loaded prompt, windowed on let release;const held=new Promise(resolve=>{release=resolve;}); await page.route('**/prompt_manager/bulk/tags',async route=>{await held;await route.continue();},{times:1}); await addTags(page,'everyone'); - for(const name of ['Add tags','Set category',`Select all ${all} loaded`]) await expect(bar(page).getByRole('button',{name,exact:true})).toHaveAttribute('aria-disabled','true'); + for(const name of ['Add tags','Set category','Delete…','Clear',`Select all ${all} loaded`]) await expect(bar(page).getByRole('button',{name,exact:true})).toHaveAttribute('aria-disabled','true'); release(); await expect(toast(page)).toHaveText(`Added “everyone” to ${all-1} prompts.`); expect(writes).toHaveLength(1);expect(new Set(writes[0].prompt_ids).size).toBe(all-1); diff --git a/tests/browser/workspace-system-backups.spec.mjs b/tests/browser/workspace-system-backups.spec.mjs index 3a0404d4..fecbdbcc 100644 --- a/tests/browser/workspace-system-backups.spec.mjs +++ b/tests/browser/workspace-system-backups.spec.mjs @@ -11,8 +11,8 @@ const STATUS_URL = '/prompt_manager/backup/status'; const PREVIEW_URL = '/prompt_manager/restore/preview'; const APPLY_URL = '/prompt_manager/restore/apply'; const WHEN = { dateStyle: 'medium', timeStyle: 'short' }; -const SAFETY_DESCRIPTION = 'Keep a verified copy of the database on disk before maintenance, cleanups and LoRA imports change ' + - 'records. A restore always keeps one.'; +const SAFETY_DESCRIPTION = 'Keep a verified copy of the database on disk before maintenance, cleanups, LoRA imports and ' + + 'prompt deletes change records. A restore always keeps one.'; test.use({ locale: 'en-US', timezoneId: 'UTC' }); diff --git a/tests/browser/workspace-system-diagnostics.spec.mjs b/tests/browser/workspace-system-diagnostics.spec.mjs index 9f811362..2f4cc07a 100644 --- a/tests/browser/workspace-system-diagnostics.spec.mjs +++ b/tests/browser/workspace-system-diagnostics.spec.mjs @@ -274,11 +274,18 @@ test('level filter is minimum-level over all six choices', async ({ page, reques for (const line of (await lines(page).textContent()).split('\n')) expect(line).toMatch(/^\d\d:\d\d:\d\d [A-Z]+ +\S/); expect(urls.at(-1)).toBe('?limit=100'); for (const minimum of LEVELS) { + // Wait for this level's answer: the previous level's lines also hold the CRITICAL line, so seeing it proved + // nothing, and under load the filtered answer painted after the check (#501 merge queue, firefox). + const answered = page.waitForResponse(item => new URL(item.url()).pathname === LOGS_URL + && new URL(item.url()).search === `?limit=100&level=${minimum}`); await level.selectOption(minimum); + await answered; await expect.poll(() => urls.at(-1)).toBe(`?limit=100&level=${minimum}`); await expect(lines(page)).toContainText('CRITICAL The database could not be opened'); - const shown = await shownLevels(page); - expect(shown.every(name => LEVELS.indexOf(name) >= LEVELS.indexOf(minimum)), `${minimum}: ${shown}`).toBe(true); + await expect.poll(async () => { + const shown = await shownLevels(page); + return shown.every(name => LEVELS.indexOf(name) >= LEVELS.indexOf(minimum)) || `${minimum}: ${shown}`; + }).toBe(true); if (minimum === 'DEBUG') await expect(lines(page)).toContainText('DEBUG Thumbnail cache lookup'); if (minimum === 'WARNING') { await expect(lines(page)).not.toContainText('Database backup verified'); diff --git a/tests/fixtures/legacy_contract_snapshot.json b/tests/fixtures/legacy_contract_snapshot.json index 4d3dc248..1bed157e 100644 --- a/tests/fixtures/legacy_contract_snapshot.json +++ b/tests/fixtures/legacy_contract_snapshot.json @@ -132,6 +132,9 @@ "POST /prompt_manager/jobs/thumbnails", "POST /prompt_manager/jobs/{job_id}/cancel", "POST /prompt_manager/library", + "POST /prompt_manager/library/delete", + "POST /prompt_manager/library/delete/preview", + "POST /prompt_manager/library/restore", "POST /prompt_manager/library/{positive_id}/variants", "POST /prompt_manager/logs/config", "POST /prompt_manager/logs/truncate", diff --git a/tests/js/workspace-images-delete.test.js b/tests/js/workspace-images-delete.test.js index 90782294..d03e8c81 100644 --- a/tests/js/workspace-images-delete.test.js +++ b/tests/js/workspace-images-delete.test.js @@ -65,6 +65,8 @@ test('sentences follow the mode and the backup policy', () => { assert.deepEqual(deleteSentences(plan(), 'trash'), [SENTENCES.trash, USE_LIBRARY_ONLY]); const nowhere = plan({ trash: { count: 0 }, trash_refused: { count: 3, rows: [] } }); assert.deepEqual(deleteSentences(nowhere, 'trash'), [SENTENCES.trash, NO_TRASH_HERE, USE_LIBRARY_ONLY]); + // #500: no "yet" (Windows is unavailable by design); each file's reason says why. + assert.equal(NO_TRASH_HERE, 'Moving to the trash is not available for these files; each one lists why.'); assert.deepEqual(deleteSentences(plan({ trash_refused: { count: 0 } }), 'trash'), [SENTENCES.trash]); }); diff --git a/tests/js/workspace-library-delete.test.js b/tests/js/workspace-library-delete.test.js new file mode 100644 index 00000000..b80eb68e --- /dev/null +++ b/tests/js/workspace-library-delete.test.js @@ -0,0 +1,150 @@ +const test = require('node:test'); +const assert = require('node:assert/strict'); + +// #443 D152: the Prompt library's Delete… review (delete unpaired, archive paired, keep protected) and Restore. +const source = import('../../web/js/workspace/library-delete.mjs'); + +const preview = (patch = {}) => ({success: true, library_epoch: 'e', delete: [], archive: [], protected: [], + already_archived: 0, not_found: 0, image_links: 0, backup: 'on', disk: null, ...patch}); + +test('the routes are the workspace ones, never the legacy bulk delete', async () => { + const {PREVIEW_URL, DELETE_URL, RESTORE_URL} = await source; + assert.equal(PREVIEW_URL, '/prompt_manager/library/delete/preview'); + assert.equal(DELETE_URL, '/prompt_manager/library/delete'); + assert.equal(RESTORE_URL, '/prompt_manager/library/restore'); +}); + +test('the title, primary and body follow what the review will do', async () => { + const {reviewOf, reviewTitle, primaryLabel, deleteBody} = await source; + const both = reviewOf(preview({delete: [1, 2], archive: [3]})); + assert.equal(reviewTitle(both), 'Delete or archive selected prompts'); + assert.equal(primaryLabel(both), 'Delete 2 and archive 1 prompt'); + assert.deepEqual(deleteBody(both), {archive_ids: [3], delete_ids: [1, 2], expected_epoch: 'e', backup: 'on'}); + const onlyDelete = reviewOf(preview({delete: [1]})); + assert.equal(reviewTitle(onlyDelete), 'Delete selected prompts'); + assert.equal(primaryLabel(onlyDelete), 'Delete 1 prompt'); + const onlyArchive = reviewOf(preview({archive: [4, 5]})); + assert.equal(reviewTitle(onlyArchive), 'Archive selected prompts'); + assert.equal(primaryLabel(onlyArchive), 'Archive 2 prompts'); +}); + +test('the grid shows each row only when it applies, with protected titles in selection order', async () => { + const {reviewOf, reviewGrid} = await source; + const names = {7: 'Seven', 8: 'Eight', 9: 'Nine', 10: 'Ten'}; + const review = reviewOf(preview({delete: [1], archive: [2], protected: [7, 8, 9, 10], already_archived: 1, + not_found: 2, image_links: 3})); + assert.deepEqual(reviewGrid(review, id => names[id]), [ + ['Delete permanently', '1 prompt not in a saved pair'], + ['Archive', '1 prompt in a saved pair: hidden, restorable'], + ['Protected (kept)', '4: Seven, Eight, Nine and 1 more'], + ['Already archived', '1'], + ['No longer exist', '2'], + ['Image links removed', '3 images will no longer show these prompts (files stay on disk)'], + ['Backup', 'Create and verify before deleting'], + ]); + const archiveOnly = reviewOf(preview({archive: [2]})); + assert.deepEqual(reviewGrid(archiveOnly).map(([term]) => term), ['Delete permanently', 'Archive']); + const off = reviewOf(preview({delete: [1], backup: 'off'})); + assert.equal(reviewGrid(off).at(-1)[1], 'Off: no safety backup (System › Backup & export › Recovery policy)'); +}); + +test('the Archive row names the saved pair, never negative variants, for one or many prompts', async () => { + const {reviewOf, reviewGrid} = await source; + // A saved-pair prompt with 0 negative variants (live prompt 7901) is archive-only for its pair, not for variants. + const one = reviewGrid(reviewOf(preview({archive: [2]}))).find(([term]) => term === 'Archive'); + assert.deepEqual(one, ['Archive', '1 prompt in a saved pair: hidden, restorable']); + const many = reviewGrid(reviewOf(preview({archive: [2, 3, 4]}))).find(([term]) => term === 'Archive'); + assert.deepEqual(many, ['Archive', '3 prompts in a saved pair: hidden, restorable']); + for (const [, value] of [one, many]) assert.doesNotMatch(value, /variant/); +}); + +test('the sentences explain archive, irreversibility per backup policy, re-creation and protection', async () => { + const {reviewOf, reviewSentences} = await source; + const [archive, remove, guarded] = reviewSentences(reviewOf(preview({delete: [1], archive: [2], protected: [3]}))); + // Live test (#470): the archive reason is the saved pair, which a prompt with 0 negative variants has too. + assert.match(archive, /^Prompts in a saved pair, as its positive or its negative, are archived instead of deleted, so the pair and its runs stay intact\./); + assert.doesNotMatch(archive, /variant/); + assert.match(archive, /Archived prompts cannot be deleted\. To bring them back, press Show archived prompts/); + assert.match(remove, /the only way back is to restore the safety backup taken first \(System › Backup & export\), which returns the whole library to that moment\. It is kept among the last 10 safety backups\./); + assert.match(remove, /A later scan, a new image or a node run with the same text adds it again as a new prompt\.$/); + assert.equal(guarded, 'Protected prompts are tagged __protected__; remove the tag to delete them.'); + const [off] = reviewSentences(reviewOf(preview({delete: [1], backup: 'off'}))); + assert.match(off, /Deleted prompts cannot be brought back\. Image files/); + assert.doesNotMatch(off, /safety backup/); +}); + +test('a review that cannot start says why: nothing to change, or no room for the backup', async () => { + const {reviewOf, blockedReason, COPY} = await source; + assert.equal(blockedReason(reviewOf(preview({protected: [1]}))), COPY.none); + assert.equal(blockedReason(reviewOf(preview({delete: [1], disk: {sufficient: false, message: 'Not enough. Nothing was changed.'}}))), + 'Not enough. Nothing was changed.'); + assert.match(blockedReason(reviewOf(preview({delete: [1], disk: {sufficient: false}}))), /not enough free space/); + assert.equal(blockedReason(reviewOf(preview({delete: [1], disk: {sufficient: true}}))), null); + assert.equal(blockedReason(reviewOf(preview({archive: [1]}))), null); +}); + +test('a delete outcome counts each part, singular and plural, and names the backup', async () => { + const {reviewOf, deleteOutcome} = await source; + const review = reviewOf(preview({delete: [1, 2], archive: [3], protected: [9]})); + assert.deepEqual(deleteOutcome({success: true, deleted: 2, archived: 3, already_archived: 0, not_found: 0, changed: 0, + backup: 'prompts.db.backup_x'}, null, review), {committed: true, reload: true, + toast: 'Deleted 2 prompts. Archived 3 prompts. 1 protected prompt was kept. Safety backup saved: prompts.db.backup_x (System › Backup & export).'}); + assert.equal(deleteOutcome({success: true, deleted: 1, archived: 0, already_archived: 1, not_found: 1, changed: 1, + backup: null}, null, reviewOf(preview())).toast, + 'Deleted 1 prompt. 1 was already archived. 1 no longer exists. 1 changed since the review and was kept.'); + assert.equal(deleteOutcome({success: true, deleted: 0, archived: 0, already_archived: 2, not_found: 3, changed: 2, + backup: null}, null, reviewOf(preview({protected: [1, 2]}))).toast, + 'Nothing was deleted or archived. 2 were already archived. 3 no longer exist. 2 changed since the review and were kept. 2 protected prompts were kept. The Library was reloaded.'); + assert.equal(deleteOutcome({success: true, deleted: 0, archived: 0}, null, reviewOf(preview())).committed, false); +}); + +test('a failed delete says nothing changed, or that it may have, and keeps or reloads the selection', async () => { + const {deleteOutcome, COPY} = await source; + assert.deepEqual(deleteOutcome(null, {status: 409, code: 'storage-changed'}), {toast: COPY.storage, committed: false, reload: true}); + assert.equal(COPY.storage, 'The library was restored or moved since this page loaded. Nothing was deleted or archived.'); + assert.deepEqual(deleteOutcome(null, {status: 409, code: 'busy', running: {title: 'Scan'}}), + {toast: 'Scan is running. Nothing was changed; try again when it finishes.', committed: false, reload: false}); + assert.deepEqual(deleteOutcome(null, {status: 500, code: 'backup-failed', message: 'The safety backup could not be taken and verified; nothing was changed.'}), + {toast: 'The safety backup failed. The safety backup could not be taken and verified; nothing was changed.', committed: false, reload: false}); + assert.deepEqual(deleteOutcome(null, {status: 507, code: 'insufficient-space', message: 'Not enough free disk space. Nothing was changed.'}), + {toast: 'Not enough free disk space. Nothing was changed.', committed: false, reload: false}); + for (const failure of [new TypeError('Failed to fetch'), {status: 502}, {status: 503}, {status: 504}]) { + assert.deepEqual(deleteOutcome(null, failure), {toast: COPY.unconfirmed, committed: true, reload: true}); + } + assert.deepEqual(deleteOutcome(null, {status: 400}), {toast: COPY.failed, committed: false, reload: false}); +}); + +test('a failed preview reloads only after a restore; Restore outcomes have their own words', async () => { + const {previewOutcome, restoreOutcome, COPY} = await source; + assert.deepEqual(previewOutcome({status: 409, code: 'storage-changed'}), {toast: COPY.storage, committed: false, reload: true}); + assert.deepEqual(previewOutcome({status: 500}), {toast: COPY.reviewFailed, committed: false, reload: false}); + assert.equal(previewOutcome({status: 409, code: 'busy', running: {title: 'Optimize'}}).toast, + 'Optimize is running. Nothing was changed; try again when it finishes.'); + assert.deepEqual(restoreOutcome({success: true, restored: 2, unchanged: 1, not_found: 1}, null), + {toast: 'Restored 2 prompts to the Prompt library. 1 was not archived. 1 no longer exists.', committed: true, reload: true}); + assert.deepEqual(restoreOutcome({success: true, restored: 0, unchanged: 2, not_found: 0}, null), + {toast: 'No prompt was restored. 2 were not archived.', committed: false, reload: true}); + assert.deepEqual(restoreOutcome(null, {status: 409, code: 'storage-changed'}), {toast: COPY.restoreStorage, committed: false, reload: true}); + assert.deepEqual(restoreOutcome(null, {status: 503}), {toast: COPY.restoreUnconfirmed, committed: true, reload: true}); + assert.deepEqual(restoreOutcome(null, {status: 400}), {toast: COPY.restoreFailed, committed: false, reload: false}); +}); + +test('reviewOf keeps only well-formed ids and counts from an answer', async () => { + const {reviewOf} = await source; + const review = reviewOf({library_epoch: 3, delete: [1, 'x', -2, 2], archive: null, protected: [1.5], already_archived: -1, + not_found: 'x', image_links: 2, backup: 'maybe'}); + assert.deepEqual(review, {epoch: null, delete: [1, 2], archive: [], protected: [], alreadyArchived: 0, notFound: 0, + imageLinks: 2, backup: 'on', disk: null}); +}); + +test('the archived notice says why (the saved pair) and names negative variants only when there are any', async () => { + const {archivedNoticeText} = await source; + assert.equal(archivedNoticeText(0), 'This prompt is archived because it is in a saved pair. It is hidden from the Prompt ' + + 'library; nothing was removed.'); + assert.equal(archivedNoticeText(1), 'This prompt is archived because it is in a saved pair. It and its 1 negative variant ' + + 'are hidden from the Prompt library; nothing was removed.'); + assert.equal(archivedNoticeText(3), 'This prompt is archived because it is in a saved pair. It and its 3 negative variants ' + + 'are hidden from the Prompt library; nothing was removed.'); + // An unknown count (an older server) never claims variants. + assert.equal(archivedNoticeText(undefined), archivedNoticeText(0)); +}); diff --git a/tests/test_autotag_engine.py b/tests/test_autotag_engine.py index ec3e468e..5f8b2bde 100644 --- a/tests/test_autotag_engine.py +++ b/tests/test_autotag_engine.py @@ -8,6 +8,7 @@ import contextlib import csv import os import shutil +import subprocess import sys import tempfile import types @@ -15,7 +16,18 @@ import unittest from pathlib import Path from unittest.mock import MagicMock, patch -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, _REPO_ROOT) + +# Run in a child interpreter: import py.autotag with a stub ComfyUI ``server`` +# and print every ML backend that the import pulled in. +_IMPORT_AUTOTAG_REPORT_BACKENDS = """ +import sys +from unittest.mock import MagicMock +sys.modules["server"] = MagicMock() +import py.autotag +print(*sorted({"onnxruntime", "llama_cpp"} & set(sys.modules))) +""" from PIL import Image @@ -142,17 +154,26 @@ class TestModelInjection(EngineTestCase): self.assertEqual(service._model_loader, service._load_model) def test_importing_autotag_does_not_import_ml_backends(self): - for heavy in ("onnxruntime", "llama_cpp"): - self.assertNotIn(heavy, sys.modules) + # A fresh interpreter: earlier tests in this process may have imported + # an installed backend, so only a clean import can prove this (#472). + result = subprocess.run( + [sys.executable, "-c", _IMPORT_AUTOTAG_REPORT_BACKENDS], + cwd=_REPO_ROOT, + capture_output=True, + text=True, + timeout=120, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stdout.split(), []) def test_default_wd14_loader_reports_missing_onnxruntime(self): _touch_model_files(self.models_dir, "wd14-vit") service = AutoTagService(models_dir=self.models_dir) - if "onnxruntime" in sys.modules: - self.skipTest("onnxruntime is installed in this environment") - with self.assertRaises(RuntimeError) as ctx: - service.load_model("wd14-vit") + # A None entry makes onnxruntime unimportable even where it is installed. + with _patch_modules({"onnxruntime": None}): + with self.assertRaises(RuntimeError) as ctx: + service.load_model("wd14-vit") self.assertIn("onnxruntime", str(ctx.exception)) self.assertFalse(service.is_model_loaded()) diff --git a/tests/test_autotag_packages.py b/tests/test_autotag_packages.py index fb910ce8..c5336439 100644 --- a/tests/test_autotag_packages.py +++ b/tests/test_autotag_packages.py @@ -1,6 +1,7 @@ """#424: each tagging backend reports the Python packages it cannot import. -``find_spec`` is patched throughout: no ML backend is installed or imported. +``find_spec`` is patched throughout, and a backend an earlier test imported is +hidden where it matters, so these pass with or without the ML packages (#472). """ import os @@ -75,8 +76,13 @@ class TestMissingPackages(unittest.TestCase): def test_find_spec_errors_count_as_missing(self): for error in (ValueError("no spec"), ImportError("broken"), OSError("io")): with self.subTest(error=type(error).__name__): - with patch.object( - autotag.importlib.util, "find_spec", side_effect=error + # Hide an onnxruntime an earlier test imported, so the check + # reaches find_spec (#472). + with ( + _patch_modules({"onnxruntime": None}), + patch.object( + autotag.importlib.util, "find_spec", side_effect=error + ), ): self.assertEqual(missing_packages("wd14-vit"), ["onnxruntime"]) diff --git a/tests/test_bulk_library_delete.py b/tests/test_bulk_library_delete.py new file mode 100644 index 00000000..35b9abe6 --- /dev/null +++ b/tests/test_bulk_library_delete.py @@ -0,0 +1,525 @@ +"""#443 (D152): delete selected Library prompts; paired ones are archived. + +The DB commands, and what every reader does with an archived parent. +""" + +import re +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from database.execution_history_schema import rotate_history_epoch +from database.library_commands import LibraryCommandError +from database.library_folders import folder_prefixes +from database.operations import PromptDatabase +from database.prompt_archive import TABLE as ARCHIVE +from utils.hashing import generate_prompt_hash + + +def save(db, text, **metadata): + return db.save_prompt(text, prompt_hash=generate_prompt_hash(text), **metadata) + + +def rows(db, sql, params=()): + with db.model.get_connection() as conn: + return [tuple(row) for row in conn.execute(sql, params).fetchall()] + + +def archived_ids(db): + return sorted(row[0] for row in rows(db, f"SELECT prompt_id FROM {ARCHIVE}")) + + +def prompt_ids(db): + return sorted(row[0] for row in rows(db, "SELECT id FROM prompts")) + + +_STAMP = re.compile(r"^\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}") + + +def _stable(value): + return "<time>" if isinstance(value, str) and _STAMP.match(value) else value + + +def table_dump(db): + """Every row of every table (times masked), to compare two databases.""" + tables = [ + row[0] + for row in rows( + db, + "SELECT name FROM sqlite_master WHERE type='table' " + "AND name NOT LIKE 'sqlite_%' AND name!='execution_history_metadata' " + "ORDER BY name", + ) + ] + dump = {} + for table in tables: + dump[table] = sorted( + (tuple(map(_stable, row)) for row in rows(db, f"SELECT * FROM {table}")), + key=repr, + ) + return dump + + +class LibraryDeleteCase(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.root = Path(self.temp.name) + self.db = PromptDatabase(str(self.root / "delete.db")) + self.epoch = self.db.get_library_options()["library_epoch"] + + def tearDown(self): + self.db.close_all() + self.temp.cleanup() + + def paired(self, positive, negative="blur"): + pair = self.db.save_prompt_pair(positive, negative) + found = self.db.get_prompt_pair(pair) + return found["positive_prompt_id"], found["negative_prompt_id"], pair + + def image(self, prompt_id, name): + path = self.root / name + path.write_bytes(b"png") + return self.db.link_image_to_prompt(prompt_id, str(path)) + + def apply(self, ids, epoch=None, review=None): + """Review ``ids`` (or use ``review``), then write what was reviewed.""" + review = review or self.classify(ids) + return self.db.archive_or_delete_library_prompts( + review["archive"], + review["delete"], + expected_epoch=epoch or self.epoch, + candidates=review["fingerprints"], + ) + + def classify(self, ids, epoch=None): + return self.db.classify_library_delete(ids, expected_epoch=epoch or self.epoch) + + def restore(self, ids, epoch=None): + return self.db.restore_library_prompts(ids, expected_epoch=epoch or self.epoch) + + +class TestClassifyAndApply(LibraryDeleteCase): + def test_each_class_in_order(self): + plain = save(self.db, "Plain") + protected = save(self.db, "Guarded", tags=["__protected__"]) + protected_paired, _neg, _pair = self.paired("Guarded pair") + self.db.bulk_add_tags([protected_paired], ["__protected__"]) + positive, negative_only, _pair = self.paired("Cat", "only negative") + both = save(self.db, "Both roles") + self.image(both, "both.png") # a parent through its image + self.db.save_prompt_pair("Other", "Both roles") + archived_pair_positive, _n, old_pair = self.paired("Old") + self.db.archive_prompt_pair(old_pair, True) + already, _n2, _p2 = self.paired("Already") + self.apply([already]) + self.image(plain, "plain.png") + ids = [ + plain, + protected, + protected_paired, + positive, + negative_only, + both, + archived_pair_positive, + already, + 99999, + ] + result = self.classify(ids) + self.assertEqual( + result, + { + "library_epoch": self.epoch, + "delete": [plain], + "archive": [positive, both, archived_pair_positive], + "protected": [protected, protected_paired], + "already_archived": [already], + "not_found": [negative_only, 99999], + "image_links": 1, + "fingerprints": result["fingerprints"], + }, + ) + self.assertEqual(list(result["fingerprints"]), [plain]) + + def test_apply_archives_paired_deletes_unpaired_and_skips_protected(self): + plain = save(self.db, "Plain", tags=["red"]) + protected = save(self.db, "Guarded", tags=["__protected__"]) + positive, negative, pair = self.paired("Cat") + self.image(plain, "a.png") + self.image(positive, "b.png") + before = rows(self.db, "SELECT updated_at FROM prompts WHERE id=?", (positive,)) + result = self.apply([plain, protected, positive, 99999]) + self.assertEqual( + result, + { + "library_epoch": self.epoch, + "deleted": 1, + "archived": 1, + "already_archived": 0, + "not_found": 0, + "changed": 0, + "image_links": 1, + }, + ) + self.assertEqual(prompt_ids(self.db), sorted([protected, positive, negative])) + self.assertEqual(archived_ids(self.db), [positive]) + # Archive keeps everything: the pair, its image link and the prompt row. + self.assertEqual( + len(rows(self.db, "SELECT 1 FROM prompt_pairs WHERE id=?", (pair,))), 1 + ) + self.assertEqual( + len( + rows( + self.db, + "SELECT 1 FROM generated_images WHERE prompt_id=?", + (positive,), + ) + ), + 1, + ) + after = rows(self.db, "SELECT updated_at FROM prompts WHERE id=?", (positive,)) + self.assertNotEqual(before, after) + # The image file stays on disk. + self.assertTrue((self.root / "a.png").exists()) + + def test_delete_has_exactly_the_legacy_row_effects(self): + twin = PromptDatabase(str(self.root / "twin.db")) + self.addCleanup(twin.close_all) + twin_epoch = twin.get_library_options()["library_epoch"] + for db in (self.db, twin): + a = save(db, "Alpha", tags=["x", "y"], category="c") + save(db, "Beta") + path = self.root / "shared.png" + path.write_bytes(b"png") + db.link_image_to_prompt(a, str(path), {"workflow": {"nodes": []}}) + self.db.bulk_delete_prompts([a]) + twin.archive_or_delete_library_prompts( + [], [a], expected_epoch=twin_epoch, candidates=None + ) + self.assertEqual(table_dump(self.db), table_dump(twin)) + + def test_already_archived_and_restore(self): + positive, _neg, _pair = self.paired("Cat") + self.apply([positive]) + again = self.classify([positive]) + self.assertEqual( + (again["archive"], again["already_archived"]), ([], [positive]) + ) + # Archived between the review and the write: counted, not archived twice. + other, _n, _p = self.paired("Dog") + review = self.classify([other]) + self.apply([other]) + late = self.apply([], review=review) + self.assertEqual((late["archived"], late["already_archived"]), (0, 1)) + plain = save(self.db, "Plain") + result = self.restore([positive, plain, 99999]) + self.assertEqual( + result, + { + "library_epoch": self.epoch, + "restored": 1, + "unchanged": 1, + "not_found": 1, + }, + ) + self.assertEqual(archived_ids(self.db), [other]) + + def test_the_write_executes_only_what_was_reviewed(self): + plain = save(self.db, "Plain") + guarded = save(self.db, "Guarded later") + touched = save(self.db, "Touched later") + review = self.classify([plain, guarded, touched]) + # Between the review and the write: one becomes protected, one is used. + self.db.bulk_add_tags([guarded], ["__protected__"]) + self.db.record_prompt_use(touched) + result = self.apply([], review=review) + self.assertEqual((result["deleted"], result["changed"]), (1, 2)) + self.assertEqual(prompt_ids(self.db), sorted([guarded, touched])) + self.assertEqual(archived_ids(self.db), []) + + def test_a_new_image_link_after_the_review_keeps_the_prompt(self): + plain = save(self.db, "Plain") + review = self.classify([plain]) + self.image(plain, "late.png") + self.assertEqual(self.apply([], review=review)["changed"], 1) + self.assertIn(plain, prompt_ids(self.db)) + + def test_reviewed_deletes_without_a_backup_are_kept(self): + plain = save(self.db, "Plain") + review = self.classify([plain]) + result = self.db.archive_or_delete_library_prompts( + [], [plain], expected_epoch=self.epoch, candidates={} + ) + self.assertEqual((result["deleted"], result["changed"]), (0, 1)) + self.assertEqual(review["delete"], [plain]) + + def test_a_reviewed_delete_that_became_paired_is_archived(self): + plain = save(self.db, "Plain") + review = self.classify([plain]) + self.db.save_prompt_pair("Plain", "late negative") + result = self.apply([], review=review) + self.assertEqual((result["deleted"], result["archived"]), (0, 1)) + + def test_a_reviewed_archive_that_lost_its_pair_is_archived_not_deleted(self): + positive, _negative, pair = self.paired("Cat") + review = self.classify([positive]) + self.db.delete_prompt_pair(pair) + result = self.apply([], review=review) + self.assertEqual((result["deleted"], result["archived"]), (0, 1)) + self.assertEqual(archived_ids(self.db), [positive]) + + def test_one_id_in_both_lists_or_no_ids_is_refused(self): + plain = save(self.db, "Plain") + for archive, delete in (([plain], [plain]), ([], [])): + with self.subTest(archive=archive, delete=delete): + with self.assertRaises(ValueError): + self.db.archive_or_delete_library_prompts( + archive, delete, expected_epoch=self.epoch, candidates=None + ) + + def test_archiving_repairs_a_missing_or_stale_hash(self): + missing, _n1, _p1 = self.paired("Hashless") + stale, _n2, _p2 = self.paired("Stale hash") + with self.db.model.get_connection() as conn: + conn.execute("UPDATE prompts SET hash=NULL WHERE id=?", (missing,)) + conn.execute("UPDATE prompts SET hash='stale' WHERE id=?", (stale,)) + self.apply([missing, stale]) + for pid, text in ((missing, "Hashless"), (stale, "Stale hash")): + self.assertEqual( + rows(self.db, "SELECT hash FROM prompts WHERE id=?", (pid,)), + [(generate_prompt_hash(text),)], + ) + # So every ingestion lookup finds the archived row. + self.assertEqual(save(self.db, text), pid) + + def test_a_canonical_hash_another_row_holds_is_left_alone(self): + stale, _n, _p = self.paired("Twin text") + with self.db.model.get_connection() as conn: + conn.execute("UPDATE prompts SET hash='stale' WHERE id=?", (stale,)) + twin = save(self.db, "twin text") + self.apply([stale]) + self.assertEqual( + rows(self.db, "SELECT hash FROM prompts WHERE id=?", (stale,)), [("stale",)] + ) + self.assertEqual(save(self.db, "Twin text"), twin) + + def test_restore_unhides_protected_and_non_parent_archived_prompts(self): + guarded, _n, _p = self.paired("Guarded") + self.apply([guarded]) + self.db.bulk_add_tags([guarded], ["__protected__"]) + negative_only = save(self.db, "Only a negative") + self.db.save_prompt_pair("Owner", "Only a negative") + with self.db.model.get_connection() as conn: + conn.execute( + f"INSERT INTO {ARCHIVE} VALUES (?, '2026-10-28T00:00:00+00:00')", + (negative_only,), + ) + result = self.restore([guarded, negative_only]) + self.assertEqual(result["restored"], 2) + self.assertEqual(archived_ids(self.db), []) + + def test_restored_storage_changes_nothing(self): + plain = save(self.db, "Plain") + positive, _neg, _pair = self.paired("Cat") + rotate_history_epoch(self.db.model.get_connection()) + for call in ( + lambda: self.apply([plain, positive]), + lambda: self.classify([plain]), + lambda: self.restore([positive]), + ): + with self.assertRaises(LibraryCommandError) as caught: + call() + self.assertEqual(caught.exception.code, "storage-changed") + self.assertIn(plain, prompt_ids(self.db)) + self.assertEqual(archived_ids(self.db), []) + + def test_a_failure_mid_write_rolls_everything_back(self): + plain = save(self.db, "Plain") + positive, _neg, _pair = self.paired("Cat") + with self.db.model.get_connection() as conn: + conn.execute( + "CREATE TRIGGER fail_delete BEFORE DELETE ON prompts " + "BEGIN SELECT RAISE(ABORT, 'boom'); END" + ) + with self.assertRaises(sqlite3.DatabaseError): + self.apply([positive, plain]) + self.assertIn(plain, prompt_ids(self.db)) + self.assertEqual(archived_ids(self.db), []) + + def test_many_ids_are_handled_in_batches(self): + ids = [save(self.db, f"Many {i}") for i in range(1000)] + result = self.apply(ids) + self.assertEqual(result["deleted"], 1000) + + def test_rejects_invalid_arguments(self): + plain = save(self.db, "Plain") + for ids, epoch in (([], self.epoch), ([True], self.epoch), ([plain], "x")): + for call in (self.apply, self.classify, self.restore): + with self.subTest(ids=ids, epoch=epoch, call=call.__name__): + with self.assertRaises(ValueError): + call(ids, epoch) + + +class TestArchivedReaders(LibraryDeleteCase): + """What each reader does with an archived parent (design §4.4).""" + + def setUp(self): + super().setUp() + self.kept, _kn, _kp = self.paired("Kept cat", "kept blur") + self.gone, _gn, _gp = self.paired("Gone cat", "gone blur") + for pid, category, tag in ((self.kept, "keep", "k"), (self.gone, "gone", "g")): + self.db.update_prompt_metadata(pid, category=category, tags=[tag]) + self.apply([self.gone]) + + def ids_of(self, page): + return [item["id"] for item in page["items"]] + + def test_listing_search_and_totals(self): + active = self.db.list_library() + self.assertEqual(self.ids_of(active), [self.kept]) + self.assertEqual( + (active["total"], active["variant_total"], active["archived_total"]), + (1, 1, 1), + ) + archived = self.db.list_library(shelf="archived") + self.assertEqual(self.ids_of(archived), [self.gone]) + self.assertEqual((archived["total"], archived["variant_total"]), (1, 1)) + self.assertEqual(self.ids_of(self.db.list_library(text="Gone")), []) + self.assertEqual( + self.ids_of(self.db.list_library(text="Gone", shelf="archived")), + [self.gone], + ) + self.assertEqual(self.ids_of(self.db.list_library(tags=["g"])), []) + self.assertEqual(self.db.list_library(category="gone")["archived_total"], 1) + with self.assertRaises(ValueError): + self.db.list_library(shelf="all") + + def test_the_444_filters_and_tag_counts_hide_it(self): + # #444's folder filter, tag match and tag ranking keep #443's shelves. + bare, _bn, _bp = self.paired("Bare cat", "bare blur") + self.apply([bare]) + folder = self.root / "out" / "cats" + folder.mkdir(parents=True) + for pid, name in ((self.kept, "kept.png"), (self.gone, "gone.png")): + self.image(pid, str(Path("out") / "cats" / name)) + prefixes = folder_prefixes([str(self.root / "out")], "cats") + self.assertEqual( + self.ids_of(self.db.list_library(folder_prefixes=prefixes)), [self.kept] + ) + self.assertEqual( + self.ids_of( + self.db.list_library(folder_prefixes=prefixes, shelf="archived") + ), + [self.gone], + ) + self.assertEqual( + self.ids_of(self.db.list_library(tags=["k", "g"], tag_mode="any")), + [self.kept], + ) + self.assertEqual(self.ids_of(self.db.list_library(tag_mode="untagged")), []) + self.assertEqual( + self.ids_of(self.db.list_library(tag_mode="untagged", shelf="archived")), + [bare], + ) + tags = self.db.list_vocabulary_tags(sort="prompts") + self.assertEqual( + {row["name"]: row["prompt_count"] for row in tags["tags"]}, + {"k": 1, "g": 0}, + ) + self.assertEqual(tags["untagged_count"], 0) + self.assertEqual( + self.db.list_vocabulary_tags(search="g", sort="prompts")["tags"][0][ + "prompt_count" + ], + 0, + ) + + def test_detail_and_options(self): + self.assertTrue(self.db.get_library_positive(self.gone)["positive"]["archived"]) + self.assertFalse( + self.db.get_library_positive(self.kept)["positive"]["archived"] + ) + # Suggestions keep archived parents' categories: the archived view + # filters by them. + self.assertEqual(self.db.get_library_options()["categories"], ["gone", "keep"]) + + def test_counts(self): + self.assertEqual(self.db.library_counts()["positive_prompts"], 1) + self.assertEqual(self.db.library_counts()["negative_variants"], 1) + tags = { + row["name"]: row["prompt_count"] + for row in self.db.list_vocabulary_tags()["tags"] + } + self.assertEqual((tags["k"], tags["g"]), (1, 0)) + categories = { + row["value"]: row["prompt_count"] + for row in self.db.list_vocabulary_categories()["categories"] + } + # A value only archived parents hold stays listed with 0, as a tag does. + self.assertEqual(categories, {"keep": 1, "gone": 0}) + + def test_restore_summary_and_export(self): + from database.restore_stage import library_summary + + summary = library_summary(self.db.model.get_connection()) + self.assertEqual(summary["positives"], 1) + exported = { + item["positive_text"]: item["archived"] + for item in self.db.export_library()["positives"] + } + self.assertEqual(exported, {"Kept cat": False, "Gone cat": True}) + + def test_usage_statistics_keep_archived_runs(self): + self.db.record_prompt_use(self.gone, 3) + runs = {row["category"]: row["runs"] for row in self.db.usage_by_category()} + self.assertEqual(runs["gone"], 3) + + def test_runs_node_saves_and_new_prompt_find_the_archived_row(self): + count = len(prompt_ids(self.db)) + self.assertTrue(self.db.record_prompt_use(self.gone, 2)) + existing = self.db.get_prompt_by_hash(generate_prompt_hash("Gone cat")) + self.assertEqual(existing["id"], self.gone) + self.assertEqual(save(self.db, "Gone cat"), self.gone) + self.db.save_prompt_pair("Gone cat", "new blur") + self.assertEqual(len(prompt_ids(self.db)), count + 1) # only the new negative + self.assertEqual(archived_ids(self.db), [self.gone]) + + def test_legacy_reads_still_list_it_and_legacy_delete_still_refuses(self): + from database.pairs import PairDependencyError + + texts = [p["text"] for p in self.db.get_recent_prompts(limit=50)["prompts"]] + self.assertIn("Gone cat", texts) + with self.assertRaises(PairDependencyError): + self.db.bulk_delete_prompts([self.gone]) + + def test_as_another_parents_negative_it_stays_a_visible_variant(self): + owner, _neg, pair = self.paired("Owner cat", "Gone cat") + detail = self.db.get_library_positive(owner) + self.assertEqual([v["id"] for v in detail["variants"]], [pair]) + self.assertEqual(self.ids_of(self.db.list_library(text="Gone")), [owner]) + + def test_maintenance_never_prunes_an_archived_prompt(self): + lonely = save(self.db, "Lonely") + with self.db.model.get_connection() as conn: + conn.execute( + f"INSERT INTO {ARCHIVE} VALUES (?, '2026-10-28T00:00:00+00:00')", + (lonely,), + ) + orphaned = [row[0] for row in conn.execute(self.orphaned_sql()).fetchall()] + self.assertNotIn(lonely, orphaned) + + @staticmethod + def orphaned_sql(): + from database.maintenance import ORPHANED_PROMPTS_SQL + + return ORPHANED_PROMPTS_SQL + + def test_bulk_metadata_writes_still_reach_an_archived_parent(self): + result = self.db.bulk_set_library_category( + [self.gone], "moved", expected_epoch=self.epoch + ) + self.assertEqual(result["updated"], 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_history_reads.py b/tests/test_history_reads.py index 73994f3c..29814572 100644 --- a/tests/test_history_reads.py +++ b/tests/test_history_reads.py @@ -400,6 +400,32 @@ class HistoryReadTests(unittest.TestCase): pass self.assertFalse(conn.in_transaction) + def test_a_deleted_negative_is_unavailable_and_a_negative_only_one_is_not(self): + # #443: a Library delete can remove an unpaired prompt a run used as + # its negative. The run's prompt ids are live hints, so a deleted one + # reads unavailable; a negative-only prompt stays available. + db = self.database() + self.saved(db) + self.saved(db, "negative", 2) + start = self.capture(db) + with db.recorded_history_snapshot(): + context = db.get_history_run(start.run_id)["contexts"][0] + self.assertTrue(context["navigation"]["negative_available"]) + with db.model.get_connection() as conn: + conn.execute( + "INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) " + "VALUES(9,1,2)" + ) + with db.recorded_history_snapshot(): + context = db.get_history_run(start.run_id)["contexts"][0] + self.assertTrue(context["navigation"]["negative_available"]) + with db.model.get_connection() as conn: + conn.execute("DELETE FROM prompt_pairs WHERE id=9") + conn.execute("DELETE FROM prompts WHERE id=2") + with db.recorded_history_snapshot(): + context = db.get_history_run(start.run_id)["contexts"][0] + self.assertFalse(context["navigation"]["negative_available"]) + def test_detail_preserves_full_text_snapshots_hints_and_archive(self): db = self.database() self.saved(db) diff --git a/tests/test_image_delete_routes.py b/tests/test_image_delete_routes.py index 464b3b91..7d291928 100644 --- a/tests/test_image_delete_routes.py +++ b/tests/test_image_delete_routes.py @@ -306,6 +306,13 @@ class TestFileCheck(ImageDeleteTestCase): file_check(str(self.output_dir / "ok.png"), root, "ok.png")[0] ) + def test_windows_reason_says_why_plainly(self): + self.assertEqual( + image_delete.REASONS["windows-recycle"], + "Moving files to the Recycle Bin isn't available on Windows: files " + "larger than the bin's capacity would be deleted permanently.", + ) + def test_reasons_cover_every_code(self): from utils import trash @@ -392,7 +399,7 @@ class TestNativeBackend(ImageDeleteTestCase): native_trash = sys.platform != "win32" if not native_trash: self.assertEqual(plan["trash"], {"count": 0}) - self.assertEqual(plan["trash_refused"]["counts"], {"no-trash": 1}) + self.assertEqual(plan["trash_refused"]["counts"], {"windows-recycle": 1}) body = await self.post(APPLY, {"token": plan["token"], "mode": "library"}, 202) job = self.api._jobs.get(body["job"]["id"]) await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS) diff --git a/tests/test_image_removal_gaps.py b/tests/test_image_removal_gaps.py new file mode 100644 index 00000000..cb961720 --- /dev/null +++ b/tests/test_image_removal_gaps.py @@ -0,0 +1,251 @@ +"""Removal-mark gaps after #480, and trash folders inside output roots (#501). + +- Thumbnail jobs and the legacy auto-tag listing skip a library-removed file. +- Missing-file links also forgets the marks of files that are gone (only under + a source that can be read now). +- Every walker skips a trash folder that sits inside an output root. +""" + +import asyncio +import os +import shutil +import sqlite3 +from pathlib import Path +from unittest.mock import patch + +try: + from tests.disk_waits import DISK_BOUND_SECONDS + from tests.test_image_delete_routes import ImageDeleteTestCase +except ImportError: # discovered with tests/ as the top-level directory + from disk_waits import DISK_BOUND_SECONDS + from test_image_delete_routes import ImageDeleteTestCase + +from py.api import images as images_module +from py.api.admin import _collect_media_files +from py.api.autotag_routes import AutotagRoutesMixin +from utils.image_monitor import ImageGenerationHandler + +MAINTENANCE_PREVIEW = "/prompt_manager/maintenance/preview" +MAINTENANCE_APPLY = "/prompt_manager/maintenance/apply" + + +class _GapCase(ImageDeleteTestCase): + def marks(self): + conn = sqlite3.connect(self.api.db.model.db_path) + try: + return [ + row[0] for row in conn.execute("SELECT locator FROM image_removals") + ] + finally: + conn.close() + + def revision(self): + conn = sqlite3.connect(self.api.db.model.db_path) + try: + return conn.execute( + "SELECT revision FROM image_removals_meta WHERE singleton_id=1" + ).fetchone()[0] + finally: + conn.close() + + +class TestThumbnailJobsSkipHiddenFiles(_GapCase): + async def test_missing_and_rebuild_plans_leave_a_removed_file_out(self): + self.write("hidden.png", b"hidden bytes", mtime=1_000_000) + self.write("shown.png", b"shown bytes", mtime=2_000_000) + await self.delete(["hidden.png"], "library") + thumbs = self.output_dir / "thumbnails" + for mode in ("missing", "rebuild"): + with self.subTest(mode=mode): + plan = self.api._plan_thumbnails(self.output_dir, thumbs, mode=mode) + sources = [src.name for src, _dst, _video in plan["targets"]] + self.assertEqual(sources, ["shown.png"]) + self.assertEqual(plan["missing"], 1) + + async def test_a_changed_file_gets_its_preview_again(self): + self.write("hidden.png", b"hidden bytes", mtime=1_000_000) + await self.delete(["hidden.png"], "library") + self.write("hidden.png", b"new and longer bytes", mtime=3_000_000) + plan = self.api._plan_thumbnails( + self.output_dir, self.output_dir / "thumbnails" + ) + self.assertEqual([t[0].name for t in plan["targets"]], ["hidden.png"]) + + +class TestLegacyAutotagListing(_GapCase): + async def test_scan_output_dir_leaves_a_removed_file_out(self): + self.write("hidden.png", b"hidden bytes", mtime=1_000_000) + self.write("shown.png", b"shown bytes", mtime=2_000_000) + await self.delete(["hidden.png"], "library") + with patch.object( + self.api, "_find_comfyui_output_dir", return_value=str(self.output_dir) + ): + response = await self.client.get("/prompt_manager/scan_output_dir") + body = await response.json() + self.assertEqual([image["filename"] for image in body["images"]], ["shown.png"]) + + +class TestMissingFileLinksPrunesMarks(_GapCase): + async def links_review(self, status=200): + return await self.post( + MAINTENANCE_PREVIEW, {"operation": "missing_links", "options": {}}, status + ) + + async def apply(self, token, status=202): + body = await self.post(MAINTENANCE_APPLY, {"token": token}, status) + if status != 202: + return body + job = self.api._jobs.get(body["job"]["id"]) + await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS) + return job + + async def removed(self, name): + self.write("other.png", b"another output") # the source lists something + self.write(name, f"bytes of {name}".encode(), mtime=1_000_000) + await self.delete([name], "library") + self.assertEqual(len(self.marks()), 1) + + async def test_the_mark_of_a_gone_file_is_counted_and_forgotten(self): + await self.removed("gone.png") + # Not an unlink by PromptManager: the user deleted it elsewhere. + os.remove(self.output_dir / "gone.png") + plan = await self.links_review() + self.assertEqual(plan["scope"]["count"], 1) + self.assertEqual( + plan["scope"]["label"], "1 removed image whose file is gone or changed" + ) + before = self.revision() + job = await self.apply(plan["token"]) + self.assertEqual(job.state, "finished", job.summary()) + self.assertEqual(self.marks(), []) + self.assertGreater(self.revision(), before) + + async def test_the_mark_of_a_present_file_stays(self): + await self.removed("kept.png") + plan = await self.links_review() + self.assertEqual(plan["scope"]["count"], 0) + self.assertEqual(plan["scope"]["label"], "No links to missing files") + self.assertEqual(len(self.marks()), 1) + + async def test_a_mark_under_an_unreadable_source_stays(self): + await self.removed("away.png") + os.remove(self.output_dir / "away.png") + with patch( + "database.maintenance.readable_roots", + side_effect=lambda r: [False] * len(r), + ): + plan = await self.links_review() + self.assertEqual(plan["scope"]["count"], 0) + self.assertEqual(len(self.marks()), 1) + + async def test_the_mark_of_a_changed_file_is_forgotten(self): + await self.removed("changed.png") + self.write("changed.png", b"other and longer bytes", mtime=4_000_000) + plan = await self.links_review() + self.assertEqual(plan["scope"]["count"], 1) + job = await self.apply(plan["token"]) + self.assertEqual(job.state, "finished", job.summary()) + self.assertEqual(self.marks(), []) + + async def test_marks_under_an_empty_source_stay(self): + # An unplugged drive leaves its mount point: an empty, readable folder. + await self.removed("mounted.png") + for child in list(self.output_dir.iterdir()): + if child.is_dir(): + shutil.rmtree(child) + else: + os.remove(child) + plan = await self.links_review() + self.assertEqual(plan["scope"]["count"], 0) + self.assertEqual(len(self.marks()), 1) + + async def test_a_name_the_folder_still_lists_is_not_gone(self): + # A case-sensitive NTFS folder: lstat of the normcased locator fails. + await self.removed("Listed.png") + real_lstat = os.lstat + locator = self.marks()[0] + + def lstat(path, *args, **kwargs): + if os.fspath(path) == locator: + raise FileNotFoundError(path) + return real_lstat(path, *args, **kwargs) + + with ( + patch("database.maintenance.os.lstat", side_effect=lstat), + patch("database.maintenance.os.path.normcase", side_effect=str.lower), + ): + from database.maintenance import _stale_marks + + stale = _stale_marks([(locator, 1, 1)], [str(self.output_dir)]) + self.assertEqual(stale, []) + + async def test_a_file_back_before_the_apply_changes_the_plan(self): + await self.removed("back.png") + path = self.output_dir / "back.png" + saved = Path(self.tmpdir) / "saved.png" + shutil.copy2(path, saved) + os.remove(path) + plan = await self.links_review() + shutil.copy2(saved, path) + job = await self.apply(plan["token"]) + self.assertEqual(job.state, "failed", job.summary()) + self.assertEqual(len(self.marks()), 1) + + +class TestWalkersSkipTrashFolders(_GapCase): + def seed(self): + self.write("kept.png", b"kept") + self.write( + "x.png", b"trashed x", folder=self.output_dir / ".Trash-1000" / "files" + ) + self.write( + "y.png", b"trashed y", folder=self.output_dir / "sub" / "$RECYCLE.BIN" + ) + self.write("z.png", b"trashed z", folder=self.output_dir / ".Trashes") + + def test_walk_media_and_the_settled_entry(self): + self.seed() + walked = sorted( + Path(entry.path).name + for entry in images_module._walk_media(self.output_dir, 100) + ) + self.assertEqual(walked, ["kept.png"]) + for relative in ( + os.path.join(".Trash-1000", "files", "x.png"), + os.path.join("sub", "$RECYCLE.BIN", "y.png"), + ): + with self.subTest(relative=relative): + self.assertIsNone( + images_module._settled_entry(self.output_dir, relative) + ) + + def test_scan_duplicate_and_autotag_walkers(self): + self.seed() + collected = [ + Path(p).name for p in _collect_media_files([self.output_dir], [".png"]) + ] + self.assertEqual(collected, ["kept.png"]) + found, _truncated = AutotagRoutesMixin._collect_scan_files(self.output_dir) + self.assertEqual([p.name for p in found], ["kept.png"]) + + def test_the_image_monitor(self): + monitor = ImageGenerationHandler.__new__(ImageGenerationHandler) + monitor.supported_extensions = (".png",) + base = str(self.output_dir) + self.assertTrue(monitor.is_image_file(os.path.join(base, "a.png"))) + for folder in (".Trash-1000", "$Recycle.Bin", ".Trashes", ".trash"): + with self.subTest(folder=folder): + self.assertFalse( + monitor.is_image_file(os.path.join(base, folder, "files", "a.png")) + ) + + async def test_the_workspace_listing(self): + self.seed() + self.assertEqual(await self.listed(), ["kept.png"]) + + def test_a_root_named_like_a_trash_folder_still_lists(self): + root = Path(self.tmpdir) / ".Trash-1000" + (root / "inner").mkdir(parents=True) + (root / "inner" / "a.png").write_bytes(b"a") + walked = [Path(e.path).name for e in images_module._walk_media(root, 10)] + self.assertEqual(walked, ["a.png"]) diff --git a/tests/test_library.py b/tests/test_library.py index ef7566c0..24540f24 100644 --- a/tests/test_library.py +++ b/tests/test_library.py @@ -345,7 +345,8 @@ class TestLibrary(unittest.TestCase): self.assertEqual( (parent["title"], parent["notes"]), ("Portrait", "keep\nnotes") ) - self.assertNotIn("archived", parent) + # The parent's own archive state (#443), never a pair's. + self.assertIs(parent["archived"], False) self.assertRegex(parent["edit_token"], r"^[0-9a-f]{64}$") self.assertEqual(page["items"][1]["title"], " \t\u00a0\u3000") self.assertEqual(self.db.list_library(text="Portrait")["total"], 1) @@ -1520,6 +1521,7 @@ class TestLibrary(unittest.TestCase): "tags", "rating", "notes", + "archived", "variants", }, ) diff --git a/tests/test_library_delete_routes.py b/tests/test_library_delete_routes.py new file mode 100644 index 00000000..6bf68d88 --- /dev/null +++ b/tests/test_library_delete_routes.py @@ -0,0 +1,353 @@ +"""#443 (D152): the Library delete, archive and restore routes, and ingestion after.""" + +import json +import os +from unittest.mock import patch + +from tests.test_api_admin import AdminAPITestCase # first: it stubs ComfyUI +from database.execution_history_schema import rotate_history_epoch # noqa: E402 +from database.prompt_archive import TABLE as ARCHIVE # noqa: E402 +from database.safety_backup import SafetyBackupFailed # noqa: E402 +from py.config import PromptManagerConfig # noqa: E402 +from utils.hashing import generate_prompt_hash # noqa: E402 + +PREVIEW = "/prompt_manager/library/delete/preview" +DELETE = "/prompt_manager/library/delete" +RESTORE = "/prompt_manager/library/restore" + + +def graph(positive, negative="blur"): + return { + "1": {"class_type": "CLIPTextEncode", "inputs": {"text": positive}}, + "2": {"class_type": "CLIPTextEncode", "inputs": {"text": negative}}, + "3": { + "class_type": "KSampler", + "inputs": {"positive": ["1", 0], "negative": ["2", 0]}, + }, + } + + +class LibraryDeleteRouteCase(AdminAPITestCase): + def epoch(self): + return self.api.db.get_library_options()["library_epoch"] + + def save(self, text, **metadata): + return self.api.db.save_prompt( + text, prompt_hash=generate_prompt_hash(text), **metadata + ) + + def paired(self, positive, negative="blur"): + pair = self.api.db.save_prompt_pair(positive, negative) + return self.api.db.get_prompt_pair(pair)["positive_prompt_id"] + + def rows(self, sql, params=()): + with self.api.db.model.get_connection() as conn: + return [tuple(row) for row in conn.execute(sql, params).fetchall()] + + def ids(self): + return sorted(row[0] for row in self.rows("SELECT id FROM prompts")) + + def archived(self): + return sorted(row[0] for row in self.rows(f"SELECT prompt_id FROM {ARCHIVE}")) + + def backups(self): + folder = os.path.dirname(self.api.db.model.db_path) + return sorted(name for name in os.listdir(folder) if ".backup_" in name) + + async def post(self, url, body): + response = await self.client.request("POST", url, json=body) + return response.status, await response.json() + + async def review(self, ids): + status, body = await self.post( + PREVIEW, {"prompt_ids": ids, "expected_epoch": self.epoch()} + ) + self.assertEqual(status, 200, body) + return body + + async def confirm(self, review, backup=None): + return await self.post( + DELETE, + { + "archive_ids": review["archive"], + "delete_ids": review["delete"], + "expected_epoch": review["library_epoch"], + "backup": backup or review["backup"], + }, + ) + + def policy(self, on): + previous = PromptManagerConfig.SAFETY_BACKUP + PromptManagerConfig.SAFETY_BACKUP = on + self.addCleanup(setattr, PromptManagerConfig, "SAFETY_BACKUP", previous) + + +class TestLibraryDeleteRoutes(LibraryDeleteRouteCase): + async def test_preview_then_delete_archives_deletes_and_backs_up(self): + self.policy(True) + plain = self.save("Plain") + guarded = self.save("Guarded", tags=["__protected__"]) + positive = self.paired("Cat") + review = await self.review([plain, guarded, positive, 99999]) + self.assertEqual( + { + key: review[key] + for key in ("delete", "archive", "protected", "not_found") + }, + { + "delete": [plain], + "archive": [positive], + "protected": [guarded], + "not_found": 1, + }, + ) + self.assertEqual(review["backup"], "on") + self.assertTrue(review["disk"]["sufficient"]) + status, body = await self.confirm(review) + self.assertEqual(status, 200, body) + self.assertEqual( + (body["deleted"], body["archived"], body["changed"]), (1, 1, 0) + ) + self.assertEqual(self.backups(), [body["backup"]]) + self.assertNotIn(os.sep, body["backup"]) + self.assertNotIn(plain, self.ids()) + self.assertEqual(self.archived(), [positive]) + # The backup still holds the deleted prompt. + import sqlite3 + + path = os.path.join(os.path.dirname(self.api.db.model.db_path), body["backup"]) + with sqlite3.connect(path) as copy: + self.assertEqual( + copy.execute( + "SELECT COUNT(*) FROM prompts WHERE id=?", (plain,) + ).fetchone()[0], + 1, + ) + copy.close() + # The hold ended: nothing is listed as running. + self.assertEqual(self.api._jobs.running(), []) + + async def test_archive_only_takes_no_backup(self): + self.policy(True) + positive = self.paired("Cat") + review = await self.review([positive]) + self.assertIsNone(review["disk"]) + status, body = await self.confirm(review) + self.assertEqual((status, body["archived"], body["backup"]), (200, 1, None)) + self.assertEqual(self.backups(), []) + + async def test_policy_off_takes_none_unless_the_review_promised_one(self): + plain = self.save("Plain") + other = self.save("Other") + self.policy(False) + review = await self.review([plain]) + self.assertEqual((review["backup"], review["disk"]), ("off", None)) + status, body = await self.confirm(review) + self.assertEqual((status, body["deleted"], body["backup"]), (200, 1, None)) + # Reviewed with the policy on, turned off before the confirm: kept. + review = await self.review([other]) + status, body = await self.confirm(review, backup="on") + self.assertEqual(status, 200) + self.assertIsNotNone(body["backup"]) + + async def test_a_failed_backup_changes_nothing(self): + self.policy(True) + plain = self.save("Plain") + positive = self.paired("Cat") + review = await self.review([plain, positive]) + with patch( + "py.api.maintenance.take_safety_backup", + side_effect=SafetyBackupFailed("x"), + ): + status, body = await self.confirm(review) + self.assertEqual((status, body["code"]), (500, "backup-failed")) + self.assertIn("nothing was changed", body["error"]) + self.assertIn(plain, self.ids()) + self.assertEqual(self.archived(), []) + + async def test_a_short_disk_refuses_before_any_change(self): + self.policy(True) + plain = self.save("Plain") + from py.api import maintenance + + with patch.object(maintenance, "_free_bytes", return_value=0): + review = await self.review([plain]) + self.assertFalse(review["disk"]["sufficient"]) + self.assertIn("Nothing was changed", review["disk"]["message"]) + status, body = await self.confirm(review) + self.assertEqual((status, body["code"]), (507, "insufficient-space")) + self.assertIn(plain, self.ids()) + + async def test_busy_beside_a_record_changing_job(self): + plain = self.save("Plain") + review = await self.review([plain]) + async with self.api._jobs.hold("scan", "Scan", admission="records"): + status, body = await self.confirm(review) + self.assertEqual((status, body["code"]), (409, "busy")) + self.assertEqual(body["running"]["title"], "Scan") + self.assertIn(plain, self.ids()) + + async def test_restored_storage_answers_409_on_every_route(self): + plain = self.save("Plain") + review = await self.review([plain]) + rotate_history_epoch(self.api.db.model.get_connection()) + status, body = await self.confirm(review) + self.assertEqual((status, body["code"]), (409, "storage-changed")) + for url in (PREVIEW, RESTORE): + status, body = await self.post( + url, {"prompt_ids": [plain], "expected_epoch": review["library_epoch"]} + ) + self.assertEqual((status, body["code"]), (409, "storage-changed")) + self.assertIn(plain, self.ids()) + + async def test_restore_route(self): + positive = self.paired("Cat") + await self.confirm(await self.review([positive])) + status, body = await self.post( + RESTORE, {"prompt_ids": [positive, 99999], "expected_epoch": self.epoch()} + ) + self.assertEqual(status, 200) + self.assertEqual((body["restored"], body["not_found"]), (1, 1)) + self.assertEqual(self.archived(), []) + + async def test_bodies_are_validated(self): + plain = self.save("Plain") + epoch = self.epoch() + selection = {"prompt_ids": [plain], "expected_epoch": epoch} + for url, body in ( + (PREVIEW, {**selection, "extra": 1}), + (PREVIEW, {"prompt_ids": [], "expected_epoch": epoch}), + (PREVIEW, {"prompt_ids": [True], "expected_epoch": epoch}), + (PREVIEW, {"prompt_ids": list(range(1, 1002)), "expected_epoch": epoch}), + (PREVIEW, {"prompt_ids": [plain], "expected_epoch": "nope"}), + (RESTORE, {"prompt_ids": [plain]}), + (DELETE, {**selection}), + ( + DELETE, + { + "archive_ids": [], + "delete_ids": [], + "expected_epoch": epoch, + "backup": "on", + }, + ), + ( + DELETE, + { + "archive_ids": [plain], + "delete_ids": [plain], + "expected_epoch": epoch, + "backup": "on", + }, + ), + ( + DELETE, + { + "archive_ids": [], + "delete_ids": [plain], + "expected_epoch": epoch, + "backup": "maybe", + }, + ), + ( + DELETE, + { + "archive_ids": list(range(1, 600)), + "delete_ids": list(range(600, 1100)), + "expected_epoch": epoch, + "backup": "off", + }, + ), + ): + with self.subTest(url=url, body=body): + status, answer = await self.post(url, body) + self.assertEqual(status, 400, answer) + self.assertFalse(answer["success"]) + for url in (PREVIEW, DELETE, RESTORE): + response = await self.client.request( + "POST", url, data="[1]", headers={"Content-Type": "application/json"} + ) + self.assertEqual(response.status, 400) + self.assertIn(plain, self.ids()) + + async def test_the_legacy_bulk_delete_is_unchanged(self): + plain = self.save("Plain") + positive = self.paired("Cat") + response = await self.client.request( + "POST", "/prompt_manager/bulk/delete", json={"prompt_ids": [positive]} + ) + self.assertEqual(response.status, 409) + response = await self.client.request( + "POST", "/prompt_manager/bulk/delete", json={"prompt_ids": [plain]} + ) + self.assertEqual((await response.json())["deleted_count"], 1) + self.assertEqual(self.archived(), []) + + async def test_shelf_parameter_and_cursor_scope(self): + kept = [self.paired(f"Kept {i}") for i in range(3)] + gone = [self.paired(f"Gone {i}") for i in range(3)] + await self.confirm(await self.review(gone)) + response = await self.client.request( + "GET", "/prompt_manager/library?limit=2&shelf=active" + ) + page = await response.json() + self.assertEqual(page["archived_total"], 3) + self.assertTrue(set(item["id"] for item in page["items"]) <= set(kept)) + response = await self.client.request( + "GET", + "/prompt_manager/library?limit=2&shelf=archived" + f"&cursor={page['next_cursor']}", + ) + self.assertEqual(response.status, 400) + response = await self.client.request( + "GET", "/prompt_manager/library?shelf=archived" + ) + archived = await response.json() + self.assertEqual(sorted(item["id"] for item in archived["items"]), gone) + self.assertTrue(all(item["archived"] for item in archived["items"])) + response = await self.client.request("GET", "/prompt_manager/library?shelf=all") + self.assertEqual(response.status, 400) + + +class TestIngestionAfterArchive(LibraryDeleteRouteCase): + def scan(self, name, positive, negative="blur"): + meta = {"prompt": json.dumps(graph(positive, negative))} + path = self.output_dir / name + path.write_bytes(b"png") + return self.api._ingest_scanned_file(path, meta) + + async def test_a_scan_after_archive_creates_no_duplicate_and_keeps_it_archived( + self, + ): + self.assertEqual(self.scan("first.png", "Sunset"), "added") + positive = self.api.db.get_prompt_by_hash(generate_prompt_hash("Sunset"))["id"] + await self.confirm(await self.review([positive])) + before = self.ids() + self.scan("second.png", "Sunset") + self.scan("third.png", " sunset ", "other negative") + self.assertEqual( + [i for i in self.ids() if i not in before], + [ + self.api.db.get_prompt_by_hash(generate_prompt_hash("other negative"))[ + "id" + ] + ], + ) + self.assertEqual(self.archived(), [positive]) + linked = self.rows( + "SELECT COUNT(*) FROM generated_images WHERE prompt_id=?", (positive,) + ) + self.assertEqual(linked, [(3,)]) + response = await self.client.request("GET", "/prompt_manager/library") + self.assertNotIn( + positive, [item["id"] for item in (await response.json())["items"]] + ) + + async def test_a_scan_after_delete_creates_the_prompt_again(self): + plain = self.save("Lonely sunset") + await self.confirm(await self.review([plain])) + self.assertEqual(self.scan("again.png", "Lonely sunset"), "added") + again = self.api.db.get_prompt_by_hash(generate_prompt_hash("Lonely sunset")) + self.assertIsNotNone(again) + self.assertNotEqual(again["id"], plain) + self.assertEqual(self.archived(), []) diff --git a/tests/test_media_folders.py b/tests/test_media_folders.py new file mode 100644 index 00000000..d866aa93 --- /dev/null +++ b/tests/test_media_folders.py @@ -0,0 +1,41 @@ +"""The shared folder skip every media walker uses (#501).""" + +import unittest + +from utils.media_folders import in_skipped_folder, is_skipped_folder, is_trash_folder + + +class TrashFolderNameTests(unittest.TestCase): + def test_trash_folder_names_in_any_case(self): + for name in ( + ".Trash", + ".trash", + ".Trash-1000", + ".trash-0", + ".Trashes", + "$RECYCLE.BIN", + "$Recycle.Bin", + ): + with self.subTest(name=name): + self.assertTrue(is_trash_folder(name)) + self.assertTrue(is_skipped_folder(name)) + + def test_other_names_are_not_trash(self): + for name in (".Trash-abc", ".Trash-", "trash", "Recycle", "RECYCLE.BIN", "x"): + with self.subTest(name=name): + self.assertFalse(is_trash_folder(name)) + + def test_thumbnails_is_skipped_exactly(self): + self.assertTrue(is_skipped_folder("thumbnails")) + self.assertFalse(is_skipped_folder("Thumbnails")) + self.assertFalse(is_trash_folder("thumbnails")) + + def test_folder_parts(self): + self.assertTrue(in_skipped_folder(("a", ".Trash-1000", "files"))) + self.assertTrue(in_skipped_folder(("thumbnails",))) + self.assertFalse(in_skipped_folder(("a", "b"))) + self.assertFalse(in_skipped_folder(())) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_prompt_archive_schema.py b/tests/test_prompt_archive_schema.py new file mode 100644 index 00000000..c23180fd --- /dev/null +++ b/tests/test_prompt_archive_schema.py @@ -0,0 +1,196 @@ +"""#443: the Library parent archive table, its migration and its validation.""" + +import shutil +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from database.models import PromptModel +from database.operations import PromptDatabase +from database.prompt_archive import ( + TABLE, + migrate_prompt_archive, + prompt_archive_schema_error, +) +from utils.hashing import generate_prompt_hash + + +def _tables(conn): + return { + row[0] + for row in conn.execute("SELECT name FROM sqlite_master WHERE type='table'") + } + + +class TestPromptArchiveSchema(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.path = str(Path(self.temp.name) / "archive.db") + self.db = PromptDatabase(self.path) + + def tearDown(self): + self.db.close_all() + self.temp.cleanup() + + def conn(self): + return self.db.model.get_connection() + + def save(self, text): + return self.db.save_prompt(text, prompt_hash=generate_prompt_hash(text)) + + def test_startup_creates_the_table_and_it_validates(self): + self.assertIn(TABLE, _tables(self.conn())) + self.assertIsNone(prompt_archive_schema_error(self.conn(), allow_missing=False)) + + def test_migration_is_idempotent(self): + conn = self.conn() + migrate_prompt_archive(conn) + migrate_prompt_archive(conn) + self.assertIsNone(prompt_archive_schema_error(conn, allow_missing=False)) + + def test_a_missing_table_is_accepted_only_when_allowed(self): + conn = sqlite3.connect(":memory:") + self.addCleanup(conn.close) + self.assertIsNone(prompt_archive_schema_error(conn)) + self.assertEqual( + prompt_archive_schema_error(conn, allow_missing=False), + "Missing prompt archive schema", + ) + + def test_deleting_a_prompt_drops_its_archive_row(self): + pid = self.save("Archived then deleted") + conn = self.conn() + conn.execute( + f"INSERT INTO {TABLE}(prompt_id, archived_at) VALUES (?, ?)", + (pid, "2026-10-28T00:00:00+00:00"), + ) + conn.commit() + self.db.delete_prompt(pid) + self.assertEqual(conn.execute(f"SELECT COUNT(*) FROM {TABLE}").fetchone()[0], 0) + + def _broken(self, *statements): + conn = sqlite3.connect(":memory:") + self.addCleanup(conn.close) + conn.execute("CREATE TABLE prompts (id INTEGER PRIMARY KEY)") + for statement in statements: + conn.execute(statement) + return conn + + def test_unsupported_structures_are_refused(self): + good = ( + f"CREATE TABLE {TABLE} (prompt_id INTEGER PRIMARY KEY, " + "archived_at TEXT NOT NULL, " + "FOREIGN KEY(prompt_id) REFERENCES prompts(id) ON DELETE CASCADE)" + ) + cases = { + "other definition": ( + f"CREATE TABLE {TABLE} (prompt_id INTEGER PRIMARY KEY, " + "archived_at TEXT)", + ), + "no cascade": ( + f"CREATE TABLE {TABLE} (prompt_id INTEGER PRIMARY KEY, " + "archived_at TEXT NOT NULL, " + "FOREIGN KEY(prompt_id) REFERENCES prompts(id))", + ), + "extra index": (good, f"CREATE INDEX x ON {TABLE}(archived_at)"), + "trigger": ( + good, + f"CREATE TRIGGER t AFTER INSERT ON {TABLE} BEGIN SELECT 1; END", + ), + "view name": (f"CREATE VIEW {TABLE} AS SELECT 1",), + } + for name, statements in cases.items(): + with self.subTest(name): + self.assertIsNotNone( + prompt_archive_schema_error(self._broken(*statements)) + ) + + def test_invalid_values_and_dangling_rows_are_refused(self): + good = ( + f"CREATE TABLE {TABLE} (prompt_id INTEGER PRIMARY KEY, " + "archived_at TEXT NOT NULL, " + "FOREIGN KEY(prompt_id) REFERENCES prompts(id) ON DELETE CASCADE)" + ) + for name, row in { + "number": "(1, 5)", + "blob": "(1, x'00')", + "long": f"(1, '{'x' * 65}')", + "dangling": "(99, '2026-10-28T00:00:00+00:00')", + }.items(): + with self.subTest(name): + conn = self._broken(good, "INSERT INTO prompts(id) VALUES (1)") + conn.execute(f"INSERT INTO {TABLE} VALUES {row}") + self.assertIsNotNone(prompt_archive_schema_error(conn)) + + def test_a_refused_structure_stops_the_migration_without_changes(self): + conn = self._broken( + f"CREATE TABLE {TABLE} (prompt_id INTEGER PRIMARY KEY, archived_at TEXT)" + ) + with self.assertRaises(sqlite3.DatabaseError): + migrate_prompt_archive(conn) + sql = conn.execute( + "SELECT sql FROM sqlite_master WHERE name=?", (TABLE,) + ).fetchone()[0] + self.assertNotIn("NOT NULL", sql) + + +class TestPromptArchiveBackups(unittest.TestCase): + """Restore validation accepts an older backup and refuses a damaged table.""" + + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.root = Path(self.temp.name) + self.db = PromptDatabase(str(self.root / "live.db")) + + def tearDown(self): + self.db.close_all() + self.temp.cleanup() + + def copy(self, name): + target = self.root / name + self.db.model.backup_database(str(target)) + return str(target) + + def test_a_backup_without_the_table_verifies_and_migrates(self): + path = self.copy("old.db") + with sqlite3.connect(path) as conn: + conn.execute(f"DROP TABLE {TABLE}") + conn.close() + ok, reason = self.db.model.verify_database_file(path) + self.assertTrue(ok, reason) + model = PromptModel(path) + try: + self.assertIn(TABLE, _tables(model.get_connection())) + finally: + model.close_all() + + def test_a_backup_with_a_damaged_table_is_refused(self): + path = self.copy("bad.db") + with sqlite3.connect(path) as conn: + conn.execute(f"DROP TABLE {TABLE}") + conn.execute(f"CREATE TABLE {TABLE} (prompt_id INTEGER, archived_at TEXT)") + conn.close() + ok, reason = self.db.model.verify_database_file(path) + self.assertFalse(ok) + self.assertIn("archive", reason.lower()) + + def test_an_archived_prompt_survives_a_backup_copy(self): + pid = self.db.save_prompt("Kept", prompt_hash=generate_prompt_hash("Kept")) + conn = self.db.model.get_connection() + conn.execute( + f"INSERT INTO {TABLE}(prompt_id, archived_at) VALUES (?, ?)", + (pid, "2026-10-28T00:00:00+00:00"), + ) + conn.commit() + path = self.copy("copy.db") + shutil.copy(path, path + ".2") + with sqlite3.connect(path + ".2") as other: + self.assertEqual( + other.execute(f"SELECT prompt_id FROM {TABLE}").fetchall(), [(pid,)] + ) + other.close() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_trash_freedesktop.py b/tests/test_trash_freedesktop.py index 29e9ff8c..db54baf0 100644 --- a/tests/test_trash_freedesktop.py +++ b/tests/test_trash_freedesktop.py @@ -286,6 +286,17 @@ class UnavailableBackendTests(unittest.TestCase): ) self.assertIsInstance(trash.default_backend("linux"), trash.FreedesktopBackend) + def test_windows_says_why_the_recycle_bin_is_not_used(self): + # #500 ruling (a): the Recycle Bin can delete permanently, so it is never used. + backend = trash.default_backend("win32") + self.assertEqual( + backend.available("C:/x/a.png", ["C:/x"]), trash.WINDOWS_RECYCLE + ) + with self.assertRaises(trash.TrashRefused) as caught: + backend.move("C:/x/a.png", ["C:/x"]) + self.assertEqual(caught.exception.code, trash.WINDOWS_RECYCLE) + self.assertIn(trash.WINDOWS_RECYCLE, trash.CODES) + def test_module_functions_use_the_replaceable_backend(self): fake = mock.Mock() fake.available.return_value = "network" diff --git a/tools/vendor/frontend-coverage-baseline.json b/tools/vendor/frontend-coverage-baseline.json index 7c18907f..8b5daf2e 100644 --- a/tools/vendor/frontend-coverage-baseline.json +++ b/tools/vendor/frontend-coverage-baseline.json @@ -73,7 +73,7 @@ }, "metadata-autocomplete.js": { "lines": 98, - "branches": 82, + "branches": 83, "functions": 85, "statements": 98 }, @@ -371,6 +371,12 @@ "functions": 98, "statements": 98 }, + "workspace/library-delete.mjs": { + "lines": 98, + "branches": 95, + "functions": 92, + "statements": 98 + }, "workspace/library-editor-state.mjs": { "lines": 96, "branches": 92, @@ -397,8 +403,8 @@ }, "workspace/library-selection.mjs": { "lines": 98, - "branches": 86, - "functions": 84, + "branches": 87, + "functions": 89, "statements": 98 }, "workspace/library-state.mjs": { @@ -410,7 +416,7 @@ "workspace/library.mjs": { "lines": 96, "branches": 85, - "functions": 96, + "functions": 97, "statements": 96 }, "workspace/local-inspection.mjs": { @@ -553,7 +559,7 @@ }, "workspace/system-backups.mjs": { "lines": 95, - "branches": 88, + "branches": 89, "functions": 94, "statements": 95 }, diff --git a/utils/image_monitor.py b/utils/image_monitor.py index d2fc1c19..18c62c1f 100644 --- a/utils/image_monitor.py +++ b/utils/image_monitor.py @@ -45,6 +45,7 @@ from .foreign_runs import ( store_foreign, ) from .hashing import generate_prompt_hash +from .media_folders import in_skipped_folder from .metadata_extractor import ComfyUIMetadataExtractor from .parallel import note_refusal, refused_at_shutdown from .logging_config import get_logger @@ -428,11 +429,12 @@ class ImageGenerationHandler(FileSystemEventHandler): Returns: True if the file has a supported image extension and is not - inside a thumbnails directory, False otherwise + inside a thumbnails or trash folder, False otherwise """ - # Skip files in thumbnails directory - those are derivatives, not - # generated images - if "/thumbnails/" in filepath or "\\thumbnails\\" in filepath: + # Previews are derivatives, and a trash folder inside an output + # folder holds thrown-away files (#501): neither is a new output. + folders = filepath.replace("\\", "/").split("/")[:-1] + if in_skipped_folder(folders): return False return filepath.lower().endswith(self.supported_extensions) diff --git a/utils/media_folders.py b/utils/media_folders.py new file mode 100644 index 00000000..6052a001 --- /dev/null +++ b/utils/media_folders.py @@ -0,0 +1,31 @@ +"""The folders no media walker lists below an output root (#501). + +``thumbnails`` holds PromptManager's own previews. A trash folder that sits +inside an output root (``D:\\$RECYCLE.BIN`` when ``D:\\`` is a root, +``.Trash-1000`` at a mount point, ``.Trashes`` on a macOS volume) holds files +the user threw away: they are not outputs, so they are never listed, scanned, +linked, tagged or previewed. Every walker asks these predicates instead of +testing names itself; only folders *below* a root are tested. +""" + +TRASH_FOLDER_NAMES = frozenset({".trash", ".trashes", "$recycle.bin"}) +PREVIEW_FOLDER = "thumbnails" + + +def is_trash_folder(name: str) -> bool: + """A trash folder name, in any case: ``.Trash``, ``.Trash-<uid>``, + ``.Trashes`` or ``$RECYCLE.BIN``.""" + folded = name.casefold() + if folded in TRASH_FOLDER_NAMES: + return True + return folded.startswith(".trash-") and folded[7:].isdigit() + + +def is_skipped_folder(name: str) -> bool: + """A folder no walker descends: ``thumbnails`` (exact) or a trash folder.""" + return name == PREVIEW_FOLDER or is_trash_folder(name) + + +def in_skipped_folder(parts) -> bool: + """Whether any of the folder names *parts* is skipped.""" + return any(is_skipped_folder(part) for part in parts) diff --git a/utils/trash.py b/utils/trash.py index 5134b3a2..40a5a756 100644 --- a/utils/trash.py +++ b/utils/trash.py @@ -16,7 +16,9 @@ same mount, else the file's mount's ``.Trash/$uid`` (only under a real, sticky ``renameat2(RENAME_NOREPLACE)``; without that call the move is refused. macOS uses :class:`utils.trash_macos.MacTrashBackend` (#499): NSFileManager's ``trashItemAtURL`` through ctypes, after refusing what it cannot show to be -recoverable. Windows has no backend yet (:class:`UnavailableBackend`, #500). +recoverable. Windows never uses the Recycle Bin (#500: files over its capacity +are deleted permanently, undetectably), so it answers ``windows-recycle`` +through :class:`UnavailableBackend`. ``BACKEND`` is the one replaceable seam; tests install a fake. """ @@ -39,6 +41,8 @@ LOCKED = "locked" FAILED = "failed" NO_PERMISSION = "no-permission" CLOUD_SYNC = "cloud-sync" +# Windows never uses the Recycle Bin (#500): it can delete permanently. +WINDOWS_RECYCLE = "windows-recycle" CODES = frozenset( { NO_TRASH, @@ -49,6 +53,7 @@ CODES = frozenset( FAILED, NO_PERMISSION, CLOUD_SYNC, + WINDOWS_RECYCLE, } ) @@ -204,13 +209,16 @@ def _owned_folder(path: str, uid: int, *, create: bool) -> bool: class UnavailableBackend: - """No trash on this system: every move is refused, nothing is touched.""" + """No trash on this system: every move is refused with *code*, untouched.""" + + def __init__(self, code: str = NO_TRASH): + self.code = code def available(self, path: str, roots) -> str | None: - return NO_TRASH + return self.code def move(self, path: str, roots) -> str: - raise TrashRefused(NO_TRASH) + raise TrashRefused(self.code) class FreedesktopBackend: @@ -372,17 +380,21 @@ class FreedesktopBackend: def default_backend(platform: str = sys.platform): - """The backend for *platform*; Windows has none yet (#500). + """The backend for *platform* (#480, #499, #500). macOS gets :class:`utils.trash_macos.MacTrashBackend`, which loads its - Foundation bridge only on first use (#499). + Foundation bridge only on first use (#499). Windows never uses the Recycle + Bin (#500). """ if platform == "darwin": from . import trash_macos return trash_macos.MacTrashBackend() if platform == "win32" or platform.startswith("cygwin"): - return UnavailableBackend() + # #500, ruling (a): Windows deletes files larger than the Recycle Bin's + # capacity permanently, and IFileOperation's PreDeleteItem guard cannot + # tell (native probe runs 40951037575, 40953347904). + return UnavailableBackend(WINDOWS_RECYCLE) return FreedesktopBackend() diff --git a/web/js/workspace/images-delete.mjs b/web/js/workspace/images-delete.mjs index 7f53015e..28b92219 100644 --- a/web/js/workspace/images-delete.mjs +++ b/web/js/workspace/images-delete.mjs @@ -32,7 +32,7 @@ export const REVIEW_FAILED = 'The review failed. Nothing was changed.'; export const SOURCE_CHANGED = 'The sources changed. Refresh the images and try again.'; export const STORAGE_CHANGED = 'Prompt storage was restored. Reopen Delete.'; export const USE_LIBRARY_ONLY = 'To remove these files from the library anyway, choose “Remove from the library only”.'; -export const NO_TRASH_HERE = 'Moving to the trash is not available on this system yet.'; +export const NO_TRASH_HERE = 'Moving to the trash is not available for these files; each one lists why.'; const SENTENCES = Object.freeze({ libraryBackup: 'Removed images stay hidden unless the file is modified or replaced. Their tags, category, collections ' + 'and favorite can only be recovered by restoring the safety backup, which also undoes every later change.', diff --git a/web/js/workspace/index.mjs b/web/js/workspace/index.mjs index 389470ab..f75483fb 100644 --- a/web/js/workspace/index.mjs +++ b/web/js/workspace/index.mjs @@ -1,7 +1,8 @@ import '../api-client.js'; import { createButton, createScope } from '../shared/toolkit/core.js'; import { routes, mountRouter, summarizeStats, routeTitle, loadNavCounts, navigateTo, COLLECTION_ROUTE, - LIBRARY_WRITE_COUNTS, NAV_COUNT_ROUTES, FAVORITE_WRITE_COUNTS, FAVORITE_COUNT_DELAY_MS, REVIEW_WRITE_COUNTS } from './state.mjs'; + LIBRARY_WRITE_COUNTS, NAV_COUNT_ROUTES, FAVORITE_WRITE_COUNTS, FAVORITE_COUNT_DELAY_MS, REVIEW_WRITE_COUNTS, + JOB_END_COUNTS } from './state.mjs'; import { mountWorkspaceTheme } from './theme.mjs'; import { forgetImagesVisit, mountImages, onFavoriteChange } from './images.mjs'; import { forgetLibraryVisit, mountLibrary, mountNewPromptAction, onLibraryWrite } from './library.mjs'; @@ -196,16 +197,20 @@ function mountWorkspace() { scope.cleanup(() => window.removeEventListener(type, revive, true)); } scope.listen(document, 'visibilitychange', () => { if (document.visibilityState === 'visible') revive(); }); - // A metadata scan that ends may have added items to Needs review: re-read its count once per ended scan. - const runningScans = new Set(); + // A job that ends re-reads the counts it can change, once per ended job: a metadata scan may have added items to + // Needs review; deleting images (#480) hides or trashes them, with their favorites, review items and collection + // memberships (#501). + const runningJobs = new Map(); scope.cleanup(activity.subscribe(() => { - let ended = false; + const ended = new Set(); for (const job of activity.jobs()) { - if (job.kind !== 'scan') continue; - if (job.state === 'running') runningScans.add(job.id); - else if (runningScans.delete(job.id)) ended = true; + if (!JOB_END_COUNTS[job.kind]) continue; + if (job.state === 'running') runningJobs.set(job.id, job.kind); + else if (runningJobs.delete(job.id)) ended.add(job.kind); } - if (ended) refreshCounts(REVIEW_WRITE_COUNTS); + if (!ended.size) return; + refreshCounts([...new Set([...ended].flatMap(kind => JOB_END_COUNTS[kind]))]); + if (ended.has('image_delete') && !leaving) collections.load(); })); refreshCounts(NAV_COUNT_ROUTES); // S9: a finished restore replaced the library. The cached visits are dropped, the counts and saved views read diff --git a/web/js/workspace/library-delete.mjs b/web/js/workspace/library-delete.mjs new file mode 100644 index 00000000..51de4868 --- /dev/null +++ b/web/js/workspace/library-delete.mjs @@ -0,0 +1,207 @@ +import { countOf } from './plural.mjs'; +import { openCloseOnlyReview, openReviewDialog } from './workspace-dialogs.mjs'; + +/** + * The Prompt library's Delete… review and outcomes (#443, D152, beyond the concept). The concept maps the legacy prompt + * delete to a "reference-safe archive" reviewed in its `maintenancePlan` dialog; the workspace offers Delete… on the + * D148 bar, whose review (the shared `openReviewDialog`) says what happens to each selected prompt: deleted (in no saved + * pair), archived (paired: hidden, restorable) or kept (tagged `__protected__`). The server executes the reviewed + * lists (`POST /prompt_manager/library/delete`), taking a verified safety backup before any delete when the policy is + * on. Restore (`/library/restore`) puts archived prompts back on the shelf. + */ +export const PREVIEW_URL = '/prompt_manager/library/delete/preview'; +export const DELETE_URL = '/prompt_manager/library/delete'; +export const RESTORE_URL = '/prompt_manager/library/restore'; +export const REVIEWING = 'Reviewing…'; +export const DELETE_LABEL = 'Delete…'; +export const RESTORE_LABEL = 'Restore'; +// Answers that may follow a committed write: no status, or a gateway's. +const UNCONFIRMED_STATUSES = new Set([502, 503, 504]); +const NOTHING = 'Nothing was deleted or archived.'; +export const COPY = Object.freeze({ + reviewFailed: 'The selected prompts could not be reviewed. Try again.', + storage: `The library was restored or moved since this page loaded. ${NOTHING}`, + unconfirmed: 'The server did not confirm the change. The Library was reloaded: check the shelf and Show archived prompts.', + failed: 'The selected prompts could not be changed. Try again.', + nothing: `${NOTHING} The Library was reloaded.`, + none: 'None of the selected prompts can be deleted or archived.', + restoreStorage: 'The library was restored or moved since this page loaded. No prompt was returned to the Library.', + restoreUnconfirmed: 'Returning the prompts could not be confirmed. The Library was reloaded.', + restoreFailed: 'The prompts could not be returned to the Library. Try again.', + archivedText: 'This text is an archived prompt. Restore it to show it in the Prompt library again.', +}); + +const count = value => (Number.isSafeInteger(value) && value >= 0 ? value : 0); +const ids = value => (Array.isArray(value) ? value.filter(id => Number.isSafeInteger(id) && id > 0) : []); +const were = (value, singular, plural) => `${value.toLocaleString('en-US')} ${value === 1 ? singular : plural}`; + +/** The preview's classes as counts and id lists, whatever else the answer holds. */ +export function reviewOf(preview) { + return { + epoch: typeof preview?.library_epoch === 'string' ? preview.library_epoch : null, + delete: ids(preview?.delete), archive: ids(preview?.archive), protected: ids(preview?.protected), + alreadyArchived: count(preview?.already_archived), notFound: count(preview?.not_found), + imageLinks: count(preview?.image_links), backup: preview?.backup === 'off' ? 'off' : 'on', + disk: preview?.disk && typeof preview.disk === 'object' ? preview.disk : null, + }; +} + +export function reviewTitle(review) { + if (!review.archive.length) return 'Delete selected prompts'; + if (!review.delete.length) return 'Archive selected prompts'; + return 'Delete or archive selected prompts'; +} + +/** "Title A, Title B, Title C and 2 more": the protected prompts the page has loaded, in selection order. */ +export function protectedNames(review, titleOf) { + const names = review.protected.map(titleOf).filter(name => typeof name === 'string' && name); + const shown = names.slice(0, 3).join(', '); + const more = review.protected.length - Math.min(3, names.length); + return more > 0 ? `${shown}${shown ? ' and ' : ''}${more.toLocaleString('en-US')} more` : shown; +} + +/** An archived prompt's page notice: why it is archived, and its negative variants only when it has any. */ +export function archivedNoticeText(variantCount) { + const hidden = Number.isSafeInteger(variantCount) && variantCount > 0 + ? `It and its ${countOf(variantCount, 'negative variant')} are hidden` + : 'It is hidden'; + return `This prompt is archived because it is in a saved pair. ${hidden} from the Prompt library; nothing was removed.`; +} + +/** The review's `[term, value]` rows (design §7.2); rows that do not apply are left out. */ +export function reviewGrid(review, titleOf = () => null) { + const rows = [ + ['Delete permanently', `${countOf(review.delete.length, 'prompt')} not in a saved pair`], + // The archive reason is the saved pair (a pair's positive or negative), with or without negative variants. + ['Archive', `${countOf(review.archive.length, 'prompt')} in a saved pair: hidden, restorable`], + ]; + if (review.protected.length) { + const names = protectedNames(review, titleOf); + rows.push(['Protected (kept)', `${review.protected.length.toLocaleString('en-US')}${names ? `: ${names}` : ''}`]); + } + if (review.alreadyArchived) rows.push(['Already archived', review.alreadyArchived.toLocaleString('en-US')]); + if (review.notFound) rows.push(['No longer exist', review.notFound.toLocaleString('en-US')]); + if (review.delete.length) { + rows.push(['Image links removed', `${countOf(review.imageLinks, 'image')} will no longer show these prompts (files stay on disk)`]); + rows.push(['Backup', review.backup === 'on' ? 'Create and verify before deleting' + : 'Off: no safety backup (System › Backup & export › Recovery policy)']); + } + return rows; +} + +/** The review's sentences, in order, each when it applies. */ +export function reviewSentences(review) { + const sentences = []; + if (review.archive.length) { + sentences.push('Prompts in a saved pair, as its positive or its negative, are archived instead of deleted, so the ' + + 'pair and its runs stay intact. Archived prompts cannot be deleted. To bring them back, press Show archived prompts, ' + + 'select them and press Restore.'); + } + if (review.delete.length) { + const undo = review.backup === 'on' + ? ' Deleted prompts cannot be brought back one by one: the only way back is to restore the safety backup taken ' + + 'first (System › Backup & export), which returns the whole library to that moment. It is kept among the last ' + + '10 safety backups.' + : ' Deleted prompts cannot be brought back.'; + sentences.push("Deleting removes each prompt's text, title, category, tags, rating, notes, run count, last use and " + + `image links.${undo} Image files, favorites, collections and recorded runs stay. A later scan, a new image or a ` + + 'node run with the same text adds it again as a new prompt.'); + } + if (review.protected.length) sentences.push('Protected prompts are tagged __protected__; remove the tag to delete them.'); + return sentences; +} + +export function primaryLabel(review) { + const deletes = review.delete.length, archives = review.archive.length; + if (deletes && archives) return `Delete ${deletes.toLocaleString('en-US')} and archive ${countOf(archives, 'prompt')}`; + return deletes ? `Delete ${countOf(deletes, 'prompt')}` : `Archive ${countOf(archives, 'prompt')}`; +} + +/** Why a review cannot start, or null: nothing to change, or no room for the promised backup. */ +export function blockedReason(review) { + if (!review.delete.length && !review.archive.length) return COPY.none; + if (review.delete.length && review.disk && review.disk.sufficient === false) { + return typeof review.disk.message === 'string' && review.disk.message + ? review.disk.message : 'There is not enough free space for the safety backup. Nothing was changed.'; + } + return null; +} + +/** + * Open the review for `review` (`reviewOf`). `titleOf(id)` names a loaded prompt. A review that cannot start is the + * close-only review (`onDismiss` runs at once, as nothing is pending); otherwise Start calls `onStart()` and any other + * close `onDismiss()`. + */ +export function openDeleteReview({ scope, trigger, review, titleOf, onStart, onDismiss }) { + const title = reviewTitle(review), grid = reviewGrid(review, titleOf); + const options = reviewSentences(review).map(text => Object.assign(document.createElement('p'), { className: 'small', textContent: text })); + const blocked = blockedReason(review); + if (blocked) { + onDismiss(); + openCloseOnlyReview({ scope, trigger, title, grid, options, sentence: blocked }); + return; + } + openReviewDialog({ scope, trigger, title, grid, options, cancelLabel: 'Cancel', + primary: { label: primaryLabel(review), danger: true }, onStart, onDismiss }); +} + +/** The write's body: the reviewed lists, the epoch and the reviewed backup promise. */ +export function deleteBody(review) { + return { archive_ids: review.archive, delete_ids: review.delete, expected_epoch: review.epoch, backup: review.backup }; +} + +function failureOutcome(failure, copy) { + if (failure?.code === 'storage-changed') return { toast: copy.storage, committed: false, reload: true }; + if (failure?.code === 'busy') { + const name = failure.running?.title || 'Another job'; + return { toast: `${name} is running. Nothing was changed; try again when it finishes.`, committed: false, reload: false }; + } + const status = failure?.status; + if (!Number.isInteger(status) || UNCONFIRMED_STATUSES.has(status)) return { toast: copy.unconfirmed, committed: true, reload: true }; + // The server's own refusals that name their reason (backup-failed, insufficient-space) say "Nothing was changed". + if (['backup-failed', 'insufficient-space'].includes(failure?.code) && typeof failure.message === 'string' && failure.message) { + const lead = failure.code === 'backup-failed' ? 'The safety backup failed. ' : ''; + return { toast: `${lead}${failure.message}`, committed: false, reload: false }; + } + return { toast: copy.failed, committed: false, reload: false }; +} + +/** What a settled delete means: the toast, whether it may have `committed` and whether the shelf must `reload`. */ +export function deleteOutcome(result, failure, review) { + if (failure !== null || !result?.success) { + return failureOutcome(failure, { storage: COPY.storage, unconfirmed: COPY.unconfirmed, failed: COPY.failed }); + } + const deleted = count(result.deleted), archived = count(result.archived); + const parts = []; + if (deleted) parts.push(`Deleted ${countOf(deleted, 'prompt')}.`); + if (archived) parts.push(`Archived ${countOf(archived, 'prompt')}.`); + if (!parts.length) parts.push(NOTHING); + const already = count(result.already_archived), missing = count(result.not_found), changed = count(result.changed); + if (already) parts.push(`${were(already, 'was', 'were')} already archived.`); + if (missing) parts.push(`${missing.toLocaleString('en-US')} no longer ${missing === 1 ? 'exists' : 'exist'}.`); + if (changed) parts.push(`${changed.toLocaleString('en-US')} changed since the review and ${changed === 1 ? 'was' : 'were'} kept.`); + const kept = review?.protected?.length || 0; + if (kept) parts.push(`${countOf(kept, 'protected prompt')} ${kept === 1 ? 'was' : 'were'} kept.`); + if (typeof result.backup === 'string' && result.backup) parts.push(`Safety backup saved: ${result.backup} (System › Backup & export).`); + if (!deleted && !archived) parts.push('The Library was reloaded.'); + return { toast: parts.join(' '), committed: deleted + archived > 0, reload: true }; +} + +/** What a settled restore means (the same shape as `deleteOutcome`). */ +export function restoreOutcome(result, failure) { + if (failure !== null || !result?.success) { + return failureOutcome(failure, { storage: COPY.restoreStorage, unconfirmed: COPY.restoreUnconfirmed, failed: COPY.restoreFailed }); + } + const restored = count(result.restored), unchanged = count(result.unchanged), missing = count(result.not_found); + const parts = [restored ? `Restored ${countOf(restored, 'prompt')} to the Prompt library.` : 'No prompt was restored.']; + if (unchanged) parts.push(`${unchanged.toLocaleString('en-US')} ${unchanged === 1 ? 'was' : 'were'} not archived.`); + if (missing) parts.push(`${missing.toLocaleString('en-US')} no longer ${missing === 1 ? 'exists' : 'exist'}.`); + return { toast: parts.join(' '), committed: restored > 0, reload: true }; +} + +/** A failed preview: a restored storage reloads the shelf; anything else keeps the selection. */ +export function previewOutcome(failure) { + if (failure?.code === 'storage-changed') return { toast: COPY.storage, committed: false, reload: true }; + if (failure?.code === 'busy') return failureOutcome(failure, {}); + return { toast: COPY.reviewFailed, committed: false, reload: false }; +} diff --git a/web/js/workspace/library-selection.mjs b/web/js/workspace/library-selection.mjs index 27f2b883..9a7af623 100644 --- a/web/js/workspace/library-selection.mjs +++ b/web/js/workspace/library-selection.mjs @@ -6,6 +6,8 @@ import { NO_TAGS, parseBulkTags } from './images-tagging-bulk.mjs'; import { openInputDialog } from './workspace-dialogs.mjs'; import { countOf } from './plural.mjs'; import { showWorkspaceToast } from './toast.mjs'; +import { COPY as DELETE_COPY, DELETE_LABEL, DELETE_URL, PREVIEW_URL, REVIEWING, RESTORE_LABEL, RESTORE_URL, deleteBody, deleteOutcome, + openDeleteReview, previewOutcome, restoreOutcome, reviewOf } from './library-delete.mjs'; /** * The Prompt library's selection and its bulk actions (D148, D151, beyond the concept): "Add tags" (#442, D151) and "Set @@ -19,6 +21,10 @@ import { showWorkspaceToast } from './toast.mjs'; * Apply posts `{prompt_ids, category|tags, expected_epoch}` to `POST /prompt_manager/bulk/category` or `/bulk/tags`: * the epoch the shelf was read under, so a restored storage answers 409 `storage-changed` and nothing is written. * Add tags writes prompt tags only, never image tags or auto-tags. + * + * Delete… (#443, D152) previews the selection, opens its review (`library-delete.mjs`) and posts the reviewed lists: + * unpaired prompts are deleted, paired ones archived, protected ones kept. On the archived shelf (`getShelf()` is + * 'archived') the bar offers Restore instead of the other actions. */ export const CATEGORY_URL = '/prompt_manager/bulk/category'; export const PROMPT_CATEGORY_DIALOG = Object.freeze({ ...CATEGORY_DIALOG, title: 'Categorize selected prompts' }); @@ -123,29 +129,34 @@ export function categoryProblem(value) { * to `focusFallback` when it was on the bar the reload hides. */ export function mountLibrarySelection({ scope, api, grid, getItems, getEpoch, getCategories, focusFallback, onCommitted, - onReload }) { + onReload, getShelf = () => 'active', nameOf = () => null }) { let checked = Object.freeze(new Set()), anchor = null, shiftBox = null, writing = false; const element = el('div'); element.id = 'library-bulkbar'; const bulk = el('div', undefined, 'bulkbar'), count = el('strong'); const addTags = createButton('Add tags'); const setCategory = createButton('Set category'); + const deleteButton = createButton(DELETE_LABEL, 'danger'); + const restoreButton = createButton(RESTORE_LABEL); const clearButton = createButton('Clear'); const selectLoaded = createButton(''); - const actions = [addTags, setCategory, selectLoaded]; - bulk.append(count, addTags, setCategory, clearButton); + // Every bar action is aria-disabled while a review or a write runs (D148, D151, D152), Clear too. + const actions = [addTags, setCategory, deleteButton, restoreButton, selectLoaded, clearButton]; + let shownShelf = null; scope.listen(selectLoaded, 'click', () => { if (writing) return; selectAll(); // The pressed offer left the bar: focus moves to its first action, as after D135's offer. - addTags.focus({ preventScroll: true }); + (shownShelf === 'archived' ? restoreButton : addTags).focus({ preventScroll: true }); }); scope.listen(addTags, 'click', () => openTags()); scope.listen(setCategory, 'click', () => openCategory()); - scope.listen(clearButton, 'click', () => { clear(); focusFallback.focus({ preventScroll: true }); }); + scope.listen(deleteButton, 'click', () => openDelete()); + scope.listen(restoreButton, 'click', () => restoreSelected()); + scope.listen(clearButton, 'click', () => { if (writing) return; clear(); focusFallback.focus({ preventScroll: true }); }); // Escape in the bar clears the selection; the bar hides, so focus moves to the page heading as after Clear. scope.listen(element, 'keydown', event => { - if (event.key !== 'Escape' || !checked.size) return; + if (event.key !== 'Escape' || !checked.size || writing) return; event.preventDefault(); clear(); focusFallback.focus({ preventScroll: true }); @@ -154,6 +165,11 @@ export function mountLibrarySelection({ scope, api, grid, getItems, getEpoch, ge const ids = () => getItems().map(item => item.id); function render() { if (!checked.size) { bulk.remove(); return; } + const shelf = getShelf() === 'archived' ? 'archived' : 'active'; + if (shelf !== shownShelf) { + shownShelf = shelf; + bulk.replaceChildren(count, ...(shelf === 'archived' ? [restoreButton] : [addTags, setCategory, deleteButton]), clearButton); + } const label = selectionLabel(checked.size); if (count.textContent !== label) count.textContent = label; const keys = ids(); @@ -240,20 +256,62 @@ export function mountLibrarySelection({ scope, api, grid, getItems, getEpoch, ge openInputDialog({ scope, trigger: addTags, ...PROMPT_TAGS_DIALOG, suggestions, validate: tagsProblem, onApply: text => write(TAGS_URL, { ...chosen, tags: parseBulkTags(text) }, 'tags') }); } - /** The write carries no page signal: once sent it completes even if the page is left. A failure keeps the selection. */ - async function write(url, body, kind) { + /** Hold every bar action while a review or write runs; `label` is what Delete… reads meanwhile. */ + function lock(label = null) { writing = true; for (const button of actions) button.setAttribute('aria-disabled', 'true'); + if (label !== null) deleteButton.textContent = label; + } + function unlock() { + writing = false; + for (const button of actions) button.removeAttribute('aria-disabled'); + deleteButton.textContent = DELETE_LABEL; + } + /** Delete… (#443): preview the selection, then review it; Start posts what the review showed. */ + async function openDelete() { + const chosen = target(DELETE_COPY.reviewFailed); + if (!chosen) return; + lock(REVIEWING); + let preview = null, failure = null; + try { + preview = await api.post(PREVIEW_URL, chosen, { signal: scope.signal }); + } catch (error) { + failure = error; + } + if (scope.signal.aborted) return; + if (failure !== null || !preview?.success) { + unlock(); + const outcome = previewOutcome(failure); + showWorkspaceToast(outcome.toast); + if (outcome.reload) onReload(); + return; + } + const review = { ...reviewOf(preview), epoch: chosen.expected_epoch }; + // The review is open: the bar stays held behind it, under its own name again. + deleteButton.textContent = DELETE_LABEL; + openDeleteReview({ scope, trigger: deleteButton, review, titleOf: nameOf, + onDismiss: unlock, + onStart: () => write(DELETE_URL, deleteBody(review), 'delete', (result, error) => deleteOutcome(result, error, review), + review.delete.length ? 'Deleting…' : 'Archiving…') }); + } + /** Restore (#443): the selected archived prompts go back on the shelf. */ + function restoreSelected() { + const chosen = target(DELETE_COPY.restoreFailed); + if (!chosen) return; + write(RESTORE_URL, chosen, 'restore', restoreOutcome); + } + /** The write carries no page signal: once sent it completes even if the page is left. A failure keeps the selection. */ + async function write(url, body, kind, outcomeOf = (result, failure) => writeOutcome(result, failure, kind), label = null) { + lock(label); let result = null, failure = null; try { result = await api.post(url, body); } catch (error) { failure = error; } finally { - writing = false; - for (const button of actions) button.removeAttribute('aria-disabled'); + unlock(); } - const outcome = writeOutcome(result, failure, kind); + const outcome = outcomeOf(result, failure); // A left page still invalidates what it cached, so a return never shows the old categories. if (outcome.committed) onCommitted(kind, result?.success ? result : { phase: 'unconfirmed' }); if (scope.signal.aborted) return; @@ -274,7 +332,9 @@ export function mountLibrarySelection({ scope, api, grid, getItems, getEpoch, ge selectAll, /** A loaded batch: the count and the offer follow it. */ listingChanged: () => { if (checked.size) render(); }, - /** Whether a Set category write is still in flight. */ + /** Whether a review or a write is still in flight. */ writing: () => writing, + /** The shelf changed (Show archived prompts): the bar's actions follow it. */ + shelfChanged: () => { shownShelf = null; render(); }, }; } diff --git a/web/js/workspace/library-state.mjs b/web/js/workspace/library-state.mjs index 596f82f7..ebaa2760 100644 --- a/web/js/workspace/library-state.mjs +++ b/web/js/workspace/library-state.mjs @@ -19,7 +19,12 @@ function parameters(filters,limit) { if(typeof folder!=='string' || folder.length>1024) throw new TypeError('Invalid library folder'); if(tagMode!=='all') query.set('tag_mode',tagMode); if(folder) query.set('folder',folder); - query.set('sort',sort);return query; + query.set('sort',sort); + // #443 D152: the archived shelf; the default shelf sends nothing, as before. + const shelf=filters.shelf || 'active'; + if(!['active','archived'].includes(shelf)) throw new TypeError('Invalid library shelf'); + if(shelf==='archived') query.set('shelf','archived'); + return query; } /** The Library's tag match values (#444); the first is the default, sent as no parameter. */ export const LIBRARY_TAG_MODES=Object.freeze(['all','any','untagged']); @@ -31,7 +36,7 @@ function validatePage(page,cursor,query) { if(!page || !Array.isArray(page.items) || page.items.some(item=>!parent(item)) || !Number.isSafeInteger(page.total) || page.total<0 || !Number.isSafeInteger(page.variant_total) || page.variant_total<0 || typeof page.has_more!=='boolean' || !(page.next_cursor===null || (typeof page.next_cursor==='string' && page.next_cursor.length)) || page.has_more!==(page.next_cursor!==null) || (page.has_more && cursor===page.next_cursor)) throw new TypeError('Invalid library page'); } export function createLibraryStream({api,onChange=()=>{},restore}) { - let state=Object.freeze({items:Object.freeze([]),total:0,variantTotal:null,cursor:null,hasMore:true,loading:false,error:null,epoch:null}); + let state=Object.freeze({items:Object.freeze([]),total:0,variantTotal:null,archivedTotal:null,cursor:null,hasMore:true,loading:false,error:null,epoch:null}); let query=parameters({},40),request,disposed=false,seen=new Set(); if(restore) { query=parameters(restore.filters,40);const saved=restore.state; @@ -57,11 +62,13 @@ export function createLibraryStream({api,onChange=()=>{},restore}) { const incoming=[];const added=new Set(); for(const item of page.items) if(!seen.has(item.id) && !added.has(item.id)) {added.add(item.id);incoming.push(Object.freeze({...item}));} for(const value of added) seen.add(value); - request=null;emit({items:incoming.length?Object.freeze([...state.items,...incoming]):state.items,epoch:state.epoch ?? epoch,total:page.total,variantTotal:page.variant_total,cursor:page.next_cursor,hasMore:page.has_more,loading:false,error:null});return incoming.length; + // #443: the archived parents matching the filters (Show archived prompts); null from an older server. + const archivedTotal=Number.isSafeInteger(page.archived_total) && page.archived_total>=0?page.archived_total:null; + request=null;emit({items:incoming.length?Object.freeze([...state.items,...incoming]):state.items,epoch:state.epoch ?? epoch,total:page.total,variantTotal:page.variant_total,archivedTotal,cursor:page.next_cursor,hasMore:page.has_more,loading:false,error:null});return incoming.length; } catch(error) {if(!disposed && current===request) {request=null;emit({loading:false,error});}return 0;} } return {get state(){return state;},loadMore,async reset(filters={}) { - if(disposed) return 0;const next=parameters(filters,40);request?.abort();request=null;query=next;seen=new Set();emit({items:Object.freeze([]),total:0,variantTotal:null,cursor:null,hasMore:true,loading:false,error:null,epoch:null});return loadMore(); + if(disposed) return 0;const next=parameters(filters,40);request?.abort();request=null;query=next;seen=new Set();emit({items:Object.freeze([]),total:0,variantTotal:null,archivedTotal:state.archivedTotal ?? null,cursor:null,hasMore:true,loading:false,error:null,epoch:null});return loadMore(); },dispose(){disposed=true;request?.abort();request=null;}}; } /** diff --git a/web/js/workspace/library.mjs b/web/js/workspace/library.mjs index 82c12dc2..712f8fd7 100644 --- a/web/js/workspace/library.mjs +++ b/web/js/workspace/library.mjs @@ -17,6 +17,7 @@ import {libraryFilters,libraryViewState} from './saved-views.mjs'; import {openRefineDialog,openContentDialog} from './workspace-dialogs.mjs'; import {mountLibrarySelection} from './library-selection.mjs'; import {fallBackToOriginal,replaceOnError,showsThumbnail,thumbnailSource} from './thumbnail-support.mjs'; +import {COPY as DELETE_COPY,RESTORE_URL,archivedNoticeText,restoreOutcome} from './library-delete.mjs'; let shelfVisit=null; // Every confirmed receipt advances this; a shelf mounted before it never restores its cached stream. @@ -109,6 +110,8 @@ export function rememberLibraryResult(result) { function outcomeMessage(result,outcome) { const {kind,receipt}=result.commit; if(kind==='metadata') return 'Saved.'; + // #443: a save that reused an archived prompt leaves it archived; say how to bring it back. + if((kind==='create' || kind==='add-negative') && result.detail?.positive?.archived===true) return DELETE_COPY.archivedText; if(kind==='archive-pair') return ARCHIVED_MESSAGE; if(kind!=='create' && kind!=='add-negative') return null; if(receipt.positive_created) return 'Saved.'; @@ -160,7 +163,8 @@ export function mountLibrary(main,api,parentScope,{positiveId,savedState=null,on const visit=saved?null:shelfVisit, mountedGeneration=writeGeneration; // An invalidated visit keeps only filters and anchor: the stream restarts and heights are measured again. const restorable=visit?.state?visit:null; - const filters={text:'',category:'',tags:[],sort:'newest',tagMode:'all',folder:'',...visit?.filters,...saved}; + // #443 D152: `shelf` ('active' or 'archived', Show archived prompts) lives in the visit's filters, never in a saved view. + const filters={text:'',category:'',tags:[],sort:'newest',tagMode:'all',folder:'',shelf:'active',...visit?.filters,...saved}; // D153: a sort the shelf no longer offers (an older "Highest rated" visit) reads as Recently added. if(!LIBRARY_SORTS.some(([value])=>value===filters.sort)) filters.sort='newest'; const section=el('section',undefined,'library-shelf');section.setAttribute('aria-label','Prompt library'); @@ -182,14 +186,22 @@ export function mountLibrary(main,api,parentScope,{positiveId,savedState=null,on // #437 D148: select prompts and set their category (the legacy admin's Set Category); a write reloads the shelf. let categoryNames=[]; const selection=mountLibrarySelection({scope,api,grid,getItems:()=>state?.items ?? [],getEpoch:()=>state?.epoch ?? null, - getCategories:()=>categoryNames,focusFallback:{focus:()=>focusHeading(main)}, + getCategories:()=>categoryNames,focusFallback:{focus:()=>focusHeading(main)},getShelf:()=>filters.shelf, + nameOf:id=>{const item=state?.items.find(entry=>entry.id===id);return item?displayTitle(item):null;}, onCommitted:(kind,receipt)=>invalidateLibraryAfterWrite({kind:`bulk-${kind}`,receipt}),onReload:()=>{loadCategories();reset();}}); const message=el('p','','library-status');message.setAttribute('role','status'); const continuation=el('div',undefined,'library-continuation'); const more=createButton('Load more prompts');const retry=createButton('Retry prompt library');retry.hidden=true; const actions=headingActions(main,section); const exportButton=createButton('Export');exportButton.setAttribute('aria-label','Export full Library JSON'); - const newPrompt=createButton('+ New prompt','ca-primary');actions.append(exportButton,newPrompt); + // #443 D152: the shelf's counterpart of Negative variants' Show archived (#235); N follows the filters. + const showArchived=createButton('Show archived prompts');showArchived.setAttribute('aria-pressed',String(filters.shelf==='archived')); + const newPrompt=createButton('+ New prompt','ca-primary');actions.append(showArchived,exportButton,newPrompt); + scope.listen(showArchived,'click',()=>{ + if(selection.writing()) return; + filters.shelf=filters.shelf==='archived'?'active':'archived';showArchived.setAttribute('aria-pressed',String(filters.shelf==='archived')); + reset();selection.shelfChanged(); + }); const exportStatus=el('p','','library-options-status');exportStatus.hidden=true;exportStatus.setAttribute('role','alert'); const editor=createNewPromptEditor(api,scope,main); scope.listen(newPrompt,'click',()=>editor.open({mode:'new',trigger:newPrompt})); @@ -244,6 +256,7 @@ export function mountLibrary(main,api,parentScope,{positiveId,savedState=null,on const content=el('div',undefined,'card-content'); const heading=el('div',undefined,'card-title');const open=createButton(name);open.dataset.history=''; heading.append(open,el('span',item.category || 'Uncategorized','chip')); + if(item.archived===true) heading.append(el('span','Archived','chip')); const prompts=el('div'); const blocks=mountPromptBlocks(prompts,{prompts:{positive:{status:'resolved',text:item.text,source:'library'}},sender,parentScope:scope,roles:['positive'],title:name}); const meta=el('div',undefined,'card-meta'); @@ -391,11 +404,16 @@ export function mountLibrary(main,api,parentScope,{positiveId,savedState=null,on // A loaded batch: the selection bar's count and Select all N loaded follow it (D151). if(batch) selection.listingChanged(); if(next.variantTotal!==null) { - const count=`${countOf(next.total,'positive prompt')} · ${countOf(next.variantTotal,'negative variant')}`; + const count=filters.shelf==='archived'?`${countOf(next.total,'archived prompt')} · ${countOf(next.variantTotal,'negative variant')} hidden with them` + :`${countOf(next.total,'positive prompt')} · ${countOf(next.variantTotal,'negative variant')}`; if(captionCount.textContent!==count) captionCount.textContent=count;caption.hidden=false; } + const archivedLabel=next.archivedTotal===null?'Show archived prompts':`Show archived prompts (${next.archivedTotal.toLocaleString('en-US')})`; + if(showArchived.textContent!==archivedLabel) showArchived.textContent=archivedLabel; + const filtered=Boolean(filters.text || filters.category || filters.tags.length || filters.folder || filters.tagMode!=='all'); + const empty=filters.shelf==='archived'?(filtered?'No archived prompts match.':'No archived prompts.'):'No matching prompts. Try changing the filters.'; const hidden=hiddenSave!==null && !next.loading && !next.error && !next.hasMore && !next.items.some(item=>item.id===hiddenSave)?' The saved prompt is hidden by the current filters.':''; - message.textContent=(next.error?'Prompt library could not be loaded. Retry.':next.loading?'Loading prompt library…':!next.items.length?'No matching prompts. Try changing the filters.':!next.hasMore?'You’re all caught up.':'')+hidden; + message.textContent=(next.error?'Prompt library could not be loaded. Retry.':next.loading?'Loading prompt library…':!next.items.length?empty:!next.hasMore?'You’re all caught up.':'')+hidden; message.setAttribute('role',next.error?'alert':'status');retry.hidden=!next.error;more.hidden=!next.hasMore || !!next.error;more.setAttribute('aria-disabled',String(next.loading)); // #464: a loaded page is laid out now, not on the next frame: a starved renderer (a background tab, headless // WebKit under load) can hold frames for seconds, which left a loaded shelf empty. Measurements still flush per frame. @@ -490,6 +508,26 @@ export function mountLibrary(main,api,parentScope,{positiveId,savedState=null,on if(current) current.apply(result);else loader.load(positiveId); focusHeading(main); } + /** The Archived notice: Restore takes the prompt off the archive, then the page reads it again. */ + function archivedNotice(content,shown) { + const notice=el('div',undefined,'notice library-archived-notice'),copy=el('div'),row=el('div',undefined,'row-actions'); + const status=el('p','','small');status.setAttribute('role','status');status.hidden=true; + const restore=createButton('Restore');restore.setAttribute('aria-label','Restore this prompt to the Prompt library'); + copy.append(el('h2','Archived'),el('p',archivedNoticeText(shown.positive.variant_count)),status);row.append(restore);notice.append(copy,row); + content.listen(restore,'click',async()=>{ + if(restore.getAttribute('aria-disabled')==='true') return; + restore.setAttribute('aria-disabled','true');status.hidden=true; + let result=null,failure=null; + try {result=await api.post(RESTORE_URL,{prompt_ids:[shown.positive.id],expected_epoch:shown.library_epoch});} catch(error) {failure=error;} + const outcome=restoreOutcome(result,failure); + if(outcome.committed) invalidateLibraryAfterWrite({kind:'bulk-restore',receipt:result?.success?result:{phase:'unconfirmed'}}); + if(content.signal.aborted) return; + restore.removeAttribute('aria-disabled'); + if(outcome.reload) {showWorkspaceToast(outcome.toast);detail=null;loader.load(positiveId);focusHeading(main);return;} + status.textContent=outcome.toast;status.hidden=false; + }); + return notice; + } /** Explicit exit after restored storage: dispose the recorded view, then remount with a fresh visit. */ function reloadView() { writeGeneration++;shelfVisit=shelfVisit && {filters:shelfVisit.filters,anchor:null}; @@ -667,6 +705,8 @@ export function mountLibrary(main,api,parentScope,{positiveId,savedState=null,on const gallery=el('section',undefined,'history-gallery');gallery.setAttribute('aria-label','Shared gallery');gallery.append(galleryHeading); const images=el('div',undefined,'library-linked-images');gallery.append(images); const recorded=el('div',undefined,'library-recorded-container'); + // #443 D152: an archived prompt's page says so first, with Restore (the concept's `.notice`). + if(detail.positive.archived===true) body.append(archivedNotice(content,detail)); body.append(summary,variants,gallery,recorded); const recordFor=pairId=>detail.variants.find(variant=>variant.id===pairId) || archivedList().find(variant=>variant.id===pairId) || (separate()?.id===pairId?separate():null); const archivedSelection=pairId=>archivedList().some(variant=>variant.id===pairId) || (separate()?.id===pairId && separate().archived===true); diff --git a/web/js/workspace/state.mjs b/web/js/workspace/state.mjs index 9af7739c..a2b056c9 100644 --- a/web/js/workspace/state.mjs +++ b/web/js/workspace/state.mjs @@ -129,6 +129,11 @@ export const LIBRARY_WRITE_COUNTS = Object.freeze(['library', 'tags']); export const FAVORITE_WRITE_COUNTS = Object.freeze(['favorites']); /** The count a review decision or a finished metadata scan can change. */ export const REVIEW_WRITE_COUNTS = Object.freeze(['review']); +/** Job kind -> the counts re-read once when such a job ends (a scan feeds Needs review; #501: a delete hides images). */ +export const JOB_END_COUNTS = Object.freeze({ + scan: REVIEW_WRITE_COUNTS, + image_delete: Object.freeze(['images', 'favorites', 'review']), +}); // Trailing delay that coalesces quick star toggles into one Favorites count read (#249 M1). export const FAVORITE_COUNT_DELAY_MS = 300; diff --git a/web/js/workspace/system-backups.mjs b/web/js/workspace/system-backups.mjs index a2893624..015f318d 100644 --- a/web/js/workspace/system-backups.mjs +++ b/web/js/workspace/system-backups.mjs @@ -56,8 +56,8 @@ export const HOST_UNAVAILABLE = 'Server settings could not be read.'; export const SCHEDULE_REASON = 'Daily and Weekly are not available yet: back up with Back up whenever you choose.'; export const WARNING_NOTE = 'Restores are staged, migrated and validated before replacing live data.'; // D104 (M2): what the safety-backup policy does (the concept's "Preview a verified backup…" line misstates it). -export const SAFETY_DESCRIPTION = 'Keep a verified copy of the database on disk before maintenance, cleanups and LoRA ' + - 'imports change records. A restore always keeps one.'; +export const SAFETY_DESCRIPTION = 'Keep a verified copy of the database on disk before maintenance, cleanups, LoRA ' + + 'imports and prompt deletes change records. A restore always keeps one.'; // I1: a third review while two are held (another window's): what the user can do about it. export const REVIEWS_FULL = 'Two restores are already being reviewed in another window. Discard one there, or try again ' + 'when it expires (15 minutes after it was reviewed).';