Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Vito Sansevero
|
7885433688 | ||
|
Vito Sansevero
|
ee091a26c7 | ||
|
Vito Sansevero
|
d6c404a197 | ||
|
Vito Sansevero
|
3ae1bd0538 | ||
|
Vito Sansevero
|
3f8c63f16e | ||
|
Vito Sansevero
|
07c2717422 | ||
|
Vito Sansevero
|
16330cad55 | ||
|
Vito Sansevero
|
fa98f02fa8 | ||
|
Vito Sansevero
|
b3d2e271fb | ||
|
Vito Sansevero
|
70581d6bca |
+16
-476
@@ -627,9 +627,8 @@ Recorded shell deviations:
|
||||
`a saved view row matches the concept's` saves one view in both documents and
|
||||
compares the rows; `tests/browser/workspace-saved-views.spec.mjs` covers saving,
|
||||
reopening, the defaults that replace rejected values, and failed reads and saves.
|
||||
- **D50** (retired by #249 PR F, ruling R5): SAVED VIEWS now lists the concept's
|
||||
"Needs review" row after Favorites, with its count; the comparison maps nothing out
|
||||
for it (only the concept's sample count "3" reads as the fixture's "0").
|
||||
- **D50**: no "Needs review" row in SAVED VIEWS until the tag-review slice (maintainer
|
||||
ruling, 2026-10-21). The comparison maps only that row out.
|
||||
- **D51** (maintainer ruling, 2026-10-21, beyond the concept): SAVED VIEWS starts with
|
||||
the concept's Favorites row (star icon, name, count). Its count is the Favorites
|
||||
listing's own `limit=1` total (`/images/output?mode=workspace&favorites=1`), follows
|
||||
@@ -1353,8 +1352,8 @@ targets of the longest legal path (1024 code points, even escaped as `\uXXXX`) f
|
||||
the selection bar's tagging shows is the concept's or the plan's (D85, D90, D91).
|
||||
- **D89** (plan D66, Activity for the `image-tags` job): the job's Activity item and its
|
||||
view are the scan's (D54, D56, D57, D59) with the job kind's own counters, from one
|
||||
per-kind table (`activity-kinds.mjs` since S5, D111; a kind without one gets no
|
||||
tiles, "Job log" and "Cancel job"). The titles are the server's: "Add
|
||||
per-kind table (`JOB_VIEWS` in `activity-store.mjs`; a kind without one keeps the
|
||||
scan's tiles under "Job log" and "Cancel job"). The titles are the server's: "Add
|
||||
tags to images" and "Auto-tag images".
|
||||
- The pill's verb is "Tagging" ("Tagging · 41% · 8,240 / 19,622").
|
||||
- The item line and the view's heading line read "Started HH:MM · N images · closing
|
||||
@@ -1387,7 +1386,7 @@ targets of the longest legal path (1024 code points, even escaped as `\uXXXX`) f
|
||||
finished · N tagged · M errors", "<title> cancelled · N tagged" or "<title> failed ·
|
||||
N tagged · M errors".
|
||||
- Tests: `tests/js/workspace-activity-store.test.js` ("image-tags jobs get their own
|
||||
tiles outcomes and pill verb", "unknown kinds get the generic view without the scan tiles");
|
||||
tiles outcomes and pill verb", "unknown kinds keep the generic view");
|
||||
`tests/browser/workspace-activity.spec.mjs` ("an image tagging job shows Tagged
|
||||
Unchanged Skipped Errors and opens tagged rows", "cancel tagging stops at a file
|
||||
boundary", "the report download names the kind", on fake `image-tags` jobs from
|
||||
@@ -1881,187 +1880,6 @@ npm run --prefix tools/vendor test:workflows -- inspect-concept.spec.mjs workspa
|
||||
npm run --prefix tools/vendor test:comfyui -- inspect-sender.spec.mjs
|
||||
```
|
||||
|
||||
### Needs review page and recorded deviations (#249 PR F)
|
||||
|
||||
`#/review` is the concept's Metadata review page (app.js `renderReview` a749–760) over
|
||||
the review routes (`web/js/workspace/review.mjs`, `review-state.mjs`): breadcrumb and
|
||||
document title "Review" (the concept's), heading "Metadata review" with its line and
|
||||
**Open activity** (`#/activity`), the `.notice` "N images need a decision" with the
|
||||
concept's sentence, then `.gallery-layout`: the queue's concept media cards
|
||||
(`image-card.mjs`, shared with Images; the image opens the shared viewer, Info selects
|
||||
the item, Tag opens the shared Tag image dialog with the review scope note) and
|
||||
`section.surface` "Resolve the association" (filename, Tag image, Sampler context,
|
||||
the chosen path's prompt blocks, Leave for later and primary Apply association). The
|
||||
first item is selected on load, as the concept's first image is. Sampler context lists
|
||||
"Sampler <id> · <positive> / <negative>" per candidate (the server's label), disables
|
||||
unusable ones, preselects the first workable one and ends with "Leave unresolved"
|
||||
(rulings R2/R3); "Leave unresolved" shows the ambiguous status in both blocks. Apply
|
||||
takes a fresh capture (`observe`) and posts the decision; Leave unresolved on a file
|
||||
under no current root posts without a capture. Both actions return to Images (R7); a
|
||||
recorded decision re-reads the row's count, as does every metadata scan that ends. A
|
||||
refused decision keeps the selection and shows the server's sentence (the
|
||||
association conflict's explanation included) as `p.warning-note` in the panel; a
|
||||
restored storage or a changed queue reads the queue again; a request with no answer
|
||||
keeps its capture and body, and the next Apply of the same choice resends it once.
|
||||
The SAVED VIEWS row "Needs review" (review icon) follows Favorites with the exact
|
||||
pending count (`GET /prompt_manager/review/items?limit=1`, its `total`).
|
||||
|
||||
- **D86** (plan D63, ruling R6): an empty queue shows only the notice, "No images need a
|
||||
decision" and "Ambiguous images are added when a metadata scan or a new generation
|
||||
finds more than one sampler path."
|
||||
- **D87** (plan D64): an ended metadata scan whose `scope.review_items` is above 0
|
||||
carries the concept's **Review metadata** (app.js a738 `go-review`): in its Activity
|
||||
item after the item's line, and in its job view's heading after Download report. The
|
||||
item's chip still names its state (D54), where the concept's says "Needs attention".
|
||||
- **D119**: the queue cards have no selection checkbox and
|
||||
no star (the page has no selection bar and the queue carries no favorite state); a
|
||||
file under no current root has an empty, `aria-disabled` image button and Tag; more
|
||||
than 40 items add the Images page's `.infinite-continuation` ("40 of 45 images",
|
||||
Load more images) after the layout; loading and a failed read are the page's status
|
||||
line (with Retry); a limited or unavailable file and the decision states are text in
|
||||
the panel's `p.warning-note`; the toasts "Association accepted." and "Left
|
||||
unresolved." replace the concept's prototype toast. The concept's "Sample review
|
||||
queue" chip and "sample" wording are prototype copy (D29). When a re-read (below)
|
||||
finds the open item gone, the next one opens with "<file> was decided elsewhere, so
|
||||
the next image is open."; a failed re-read keeps the queue and says "The review queue
|
||||
could not be read again. Reload the page to see the latest images." A queue card's
|
||||
title is its file name and its line the extension only (an undecided item names no
|
||||
prompt, and D11 omits the size and seed a queue item does not carry); its image's alt
|
||||
is the file name; the panel's Copy and Send buttons carry no prompt title in their
|
||||
names; the Tag image dialog opened from the queue heads with the file name. An
|
||||
unavailable file's empty media button keeps the concept's 1.33 media area, so its
|
||||
card lines up with the others. A long file name in the panel's `p.small` wraps inside
|
||||
the panel (`overflow-wrap: anywhere`), where the concept's short sample name never
|
||||
overflows. A queue card shows the Images listing's thumbnail when the server knows one
|
||||
(`thumbnail_url`, the same URL the Images page uses), else the original; a re-read
|
||||
rebuilds a card whose media, target or availability changed (a root came or went), so
|
||||
Open and Tag follow it. When an unanswered decision's item leaves the queue in a
|
||||
re-read, the decision was recorded: the page finishes as a recorded decision does (its
|
||||
toast, back to Images), not with "decided elsewhere".
|
||||
- **D120** (the concept has no live-update state): while the
|
||||
change stream is down, the heading carries the quiet status line "Live updates are
|
||||
paused: changes made elsewhere appear once the connection returns." (`p.small
|
||||
.review-live`, `role="status"`); the stream's next `ready` hides it.
|
||||
|
||||
**Live queue (F2).** An open Needs review page holds one server-sent event stream,
|
||||
`GET /prompt_manager/review/events` (`py/api/review_events.py`): `event: ready` when it
|
||||
opens, then `event: change` after each committed queue change (an item the scan or the
|
||||
monitor recorded, a spelling refresh, a decision from any tab; `database/review_changes.py`
|
||||
notifies after the commit, from whichever thread wrote it) and after a restore. `change`
|
||||
carries no data. `ready` carries the queue's version `{library_epoch, review_revision}`,
|
||||
read after the stream subscribed, and every list read answers the same pair
|
||||
(`review_revision` is the review index revision, which every queue change moves; a
|
||||
restore moves the epoch). **One read per visit (F3):** the page re-reads on `ready` only
|
||||
when the queue it shows is older than that version (an unknown version, `{}` when it
|
||||
cannot be read, re-reads); a `ready` heard while the first read runs is compared once that
|
||||
read answers. The test server counts a stream as open only once it has sent `ready`, so a
|
||||
browser test that changes the queue after that count is a `change`, never part of the
|
||||
version. The page re-reads on every `change`, keeping the open item, its choice and an
|
||||
unanswered decision; re-reads asked for while one runs collapse into one more. A change
|
||||
heard while a decision is in flight is owed: a refused or unanswered decision reads it
|
||||
once the panel is idle again (a recorded one leaves the page). Load more asked for during
|
||||
a re-read waits for it and continues after the queue it read; a re-read started during
|
||||
Load more takes that click over and reads one page deeper, and a failed re-read leaves
|
||||
Load more clickable. Apply fences on the epoch of the detail it decides: an observe under
|
||||
another storage releases its capture and reads the queue and the item again, and any
|
||||
re-read under another epoch reads the open item again. **Every re-read keeps the loaded depth (F3):** it
|
||||
reads page after page (40 each, the route's limit) until it holds as many items as Load
|
||||
more had shown, so the cards stay and the scroll position does not move; a restore met
|
||||
after the first page reads once more from the first page. The row's count follows the
|
||||
total the page shows (no count read per change). Nothing polls: a `: keepalive` comment
|
||||
every 25 s only lets the server drop a closed tab's subscription. The server sends the
|
||||
first change at once and collapses the notices of the next 0.4 s into one trailing
|
||||
`change` (`COALESCE_SECONDS`), so a busy scan costs an open page one re-read per 0.4 s at
|
||||
most. Shutdown (`on_shutdown`) ends every stream and answers a later one 503. A
|
||||
reconnect the browser gives up on (a 503, or a proxy's error while ComfyUI restarts) is
|
||||
opened again by the page after 3 s, doubling to 30 s, with D120's line shown while
|
||||
the stream is down; its `ready` re-syncs the queue. The stream lives only while the tab is
|
||||
visible: a hidden tab closes it and a visible one opens it again, and that `ready` reads
|
||||
only when the queue changed meanwhile. A duplicate recording and a refused decision change
|
||||
nothing and send nothing. Residual: each visible Needs review page holds one HTTP/1.1
|
||||
connection (browsers allow six per origin, shared with ComfyUI's editor), so several
|
||||
side-by-side visible windows on the page, during a metadata scan's job stream, can still
|
||||
queue the origin's other requests; a re-read of a deep queue costs one request per 40
|
||||
items shown.
|
||||
|
||||
Pinned by `tests/test_api_review_events.py`, `tests/js/workspace-review-state.test.js`
|
||||
("refresh …", "a refresh keeps the loaded depth …", "ready naming …") and
|
||||
`tests/browser/workspace-review.spec.mjs` ("an open page shows …", "a visit reads its
|
||||
queue once …", "a live re-read keeps the loaded depth and the scroll position", "a stream
|
||||
the browser gives up on …", "a hidden tab closes its stream …", "a re-read refreshes a
|
||||
card …"); the controller's owed, Load more and epoch cases are "I1: …", "I2: …", "M3:
|
||||
…", "M5: …" and "M7: …" in `workspace-review-state.test.js`, and the coalescing is "a
|
||||
burst of changes …" in `test_api_review_events.py`.
|
||||
Leaving the page mid-decision releases that decision's capture (the server may never see
|
||||
the request), and leaving during the observe releases the capture it then answers
|
||||
("leaving during the observe …").
|
||||
**Concept comparison (F3).** `tests/browser/review-concept.spec.mjs` measures the whole
|
||||
page (`reviewInventory` in `tests/browser/tagging-canonical-comparison.mjs`: every shown
|
||||
box below `main` with its place, fonts, edges and layout, a leaf's text, a button's name,
|
||||
a select's options) in the concept at `#review` and in production on the fixture's
|
||||
concept-shaped queue (`POST /__test__/review/reset {items: 3, concept: true}`: the
|
||||
concept's first three image names and sizes, and the concept's first prompt on the open
|
||||
item), and `mapConceptReview` applies only: D29 (no "Sample review queue" chip, no
|
||||
"sample" in the panel heading and Apply), D119 (above), D3 (the grid's
|
||||
`position: relative`), production's hooks and accessible names (`review-queue`,
|
||||
`review-panel`, the list roles, the region names) and sample data (the candidate options
|
||||
except Leave unresolved; widths and heights that follow from the changed copy, moved by
|
||||
exactly that much). It then matches at 1600×1100 and 390×844 in both themes on all four
|
||||
projects; the gallery layout follows the concept at 1600, 1200, 900 and 390px (310px,
|
||||
275px with a 15px gap, 280px, one column); the SAVED VIEWS rows match with Needs review
|
||||
after Favorites; the queue's Tag image dialog matches the concept's needs-review dialog
|
||||
(`needs-review-tagging.png`) under the C4 mapping; and the comparison is shown to notice
|
||||
a dropped notice border and reordered actions. The paired captures (populated queue in
|
||||
both themes and widths, conflict, limited item, unavailable file, empty queue) are test
|
||||
artifacts; no screenshot baseline is added. The comparison found one gap, the collapsed
|
||||
media area of an unavailable file's card, fixed above (`workspace-review.spec.mjs` "a
|
||||
limited item and an unavailable file …" pins it).
|
||||
|
||||
- **D3** and **D20** apply to the queue grid (its cards stay in the grid's flow; it is
|
||||
not virtualized) and to the panel's prompt blocks (their mount point is transparent).
|
||||
|
||||
Tests: `tests/browser/review-concept.spec.mjs` (concept comparison),
|
||||
`tests/js/workspace-review-state.test.js` (controller),
|
||||
`tests/js/workspace-state.test.js` (route and count),
|
||||
`tests/browser/workspace-review.spec.mjs` (queue, count, apply, leave, Tag, empty
|
||||
queue, Load more, Activity's Review metadata; items recorded by the real recorder in
|
||||
`tests/browser/workspace_review_fixture.py`), and the SAVED VIEWS rows in
|
||||
`shell-concept.spec.mjs` and `workspace-saved-views.spec.mjs`.
|
||||
|
||||
### Native image tagging and Needs review (#249 PRs A–F)
|
||||
|
||||
One place for the whole image-tagging slice; the details live in the sections named.
|
||||
|
||||
- **Routes.** Tagging: `POST /prompt_manager/image-annotations/observe`, `GET`/`DELETE
|
||||
/prompt_manager/image-annotations/capture/<token>`, `POST
|
||||
/prompt_manager/image-annotations/suggest`, `PUT /prompt_manager/image-annotations/tags`
|
||||
("Tag image dialog and recorded deviations"). Needs review: `GET
|
||||
/prompt_manager/review/items?limit=1..40&cursor=` (answers `library_epoch`,
|
||||
`review_revision`, `total`), `GET /prompt_manager/review/items/<id>`, `POST
|
||||
/prompt_manager/review/items/<id>/decision` and the change stream `GET
|
||||
/prompt_manager/review/events` ("Needs review decisions and queue reads", "Needs review
|
||||
page and recorded deviations"). Every legacy route keeps its payload; the only legacy
|
||||
change is `409 model-busy` from the legacy Auto-tag routes while another model
|
||||
operation holds the shared gate.
|
||||
- **S1 (fingerprint-current reads, ruled 2026-10-22).** The Images Tags filter, search and
|
||||
the inspector/viewer chips use a byte record only while the file's `(st_size,
|
||||
st_mtime_ns)` equals its verified fingerprint; writes hash the bytes. Residual risk
|
||||
under "Image annotations schema".
|
||||
- **Capture limits.** 256 MiB per file, 8 completed captures, 512 MiB reserved, 2
|
||||
capture/verify workers, 900 s idle, 1800 s hard; the browser fixtures release every
|
||||
capture on reset, and leaving the Needs review page mid-decision releases its capture.
|
||||
- **The record-changing job.** Bulk Add tags and Auto-tag selected run as the
|
||||
`image-tags` Activity job (PR D); like the metadata scan it is a record-changing job, so
|
||||
a start while another runs is refused (`busy`). Needs review items come from that scan
|
||||
and from the live monitor, and a decision changes links only when compatible
|
||||
(`association-conflict` otherwise, with zero writes).
|
||||
- **No downloads.** No automated test downloads or loads a model: the browser server
|
||||
injects `TestModel` (its `download_model` raises), native routes and jobs never
|
||||
download, and every tagging spec asserts no `/autotag/download/` request.
|
||||
- **Restart.** PRs A, B, D and E and PR F (the review change stream, the queue version)
|
||||
change Python: the maintainer restarts ComfyUI once after merging. HTML, CSS and JS
|
||||
changes need only a browser refresh.
|
||||
|
||||
### Native System (#249 System)
|
||||
|
||||
`#/system/<section>` is a native workspace route rendered by `web/js/workspace/system.mjs`
|
||||
@@ -2157,79 +1975,10 @@ keeps the drafts; a save refused for one source says the server's message as the
|
||||
and marks that row. After a save the source list is read again; a monitoring change
|
||||
waits for a restart (the running observer keeps watching until then, and the status
|
||||
says "Restart ComfyUI to apply: Watch for new outputs."). Thumbnail generation's
|
||||
Review options opens the concept's "Thumbnail maintenance" dialog (D99, below).
|
||||
Review options opens the legacy gallery in a new tab (D94) until its review dialog (S5).
|
||||
A list that cannot be read shows "Sources could not be loaded. Your saved data has not
|
||||
changed." and Retry, never an empty list.
|
||||
|
||||
Thumbnail generation is an Activity job (S5, `py/api/thumbnail_jobs.py`). `GET
|
||||
/prompt_manager/images/thumbnails/plan?mode=missing|rebuild&size=128..2048&quality=low|medium|high`
|
||||
counts the work (`targets`, `fresh`, `remaining` past the 10,000-file cap, the source
|
||||
labels) and writes nothing; a value out of range is a 400 `invalid-request` that starts
|
||||
nothing. `POST /prompt_manager/jobs/thumbnails {mode, size, quality}` starts the
|
||||
`thumbnails` job ("Thumbnail generation"): admission `guarded` (never beside a scan or
|
||||
another record-changing job, one at a time, a legacy clear refused while it runs),
|
||||
cancellable at a file boundary (a file not yet started is left alone; written previews
|
||||
stay), outcome groups Generated / Skipped (already fresh, listed without a log line
|
||||
each) / Errors. `size` is the longest edge and `quality` the encoder (low: JPEG 70, no
|
||||
optimize; medium: 85, optimized, what the legacy routes always wrote; high: 95,
|
||||
optimized). Video previews take the same choice: the OpenCV frame and the placeholder
|
||||
are saved with those JPEG settings, and the ffmpeg fallback (no OpenCV) maps them to
|
||||
`-q:v` 8 / 4 / 2 and fits the frame inside the edge keeping its aspect ratio (`-vf
|
||||
scale=E:E:force_original_aspect_ratio=decrease`, no longer a stretched square).
|
||||
`rebuild` writes fresh previews again, each through the same temp-file
|
||||
rename, so nothing is cleared first and a failed file keeps its old preview (a video
|
||||
whose frame cannot be decoded keeps its preview instead of getting the placeholder);
|
||||
originals are only read (`tests/test_thumbnail_jobs.py`). Cancel is also checked per
|
||||
file during the planning walk: a job cancelled there stops the walk at the next file,
|
||||
publishes no `start` frame and ends "Cancelled before any preview was written". Every
|
||||
exit sets a final status line: "N previews written · M errors" (plus "· R more on the
|
||||
next run" past the cap), "No previews to generate · N already up to date" (or "No
|
||||
images or videos to preview"), "Cancelled · N previews written", or the failure; a job
|
||||
with nothing to do never reads "Processing file 0 of 0" (`test_every_exit_sets_a_truthful_final_status_line`,
|
||||
and "a finished thumbnail job with nothing to do says so" in `workspace-activity.spec.mjs`).
|
||||
Windows: replacing a preview another process holds open (the gallery serving it, an
|
||||
image viewer) fails with WinError 5 or 32; that file counts as an error reading
|
||||
"Preview in use by another program; the old one was kept" (never the raw "[WinError
|
||||
5]"), and its old preview stays. Any other PermissionError reads "Permission denied
|
||||
writing the preview" (`test_a_locked_preview_is_a_readable_coded_error`).
|
||||
Thumbnail work (the job and the gallery's generate and clear holds) is `guarded`, but it
|
||||
writes only preview files and no records, so the image monitor's Needs review
|
||||
recording runs beside it instead of being deferred for a whole rebuild
|
||||
(`PREVIEW_ONLY_KINDS` in `py/api/review_recording.py`, `blocker(..., compatible=)`;
|
||||
`test_a_running_thumbnail_job_does_not_defer_monitor_recording`). Other `guarded` and
|
||||
all `records` jobs still defer it.
|
||||
|
||||
Deliberate change: the legacy gallery stream `POST /images/generate-thumbnails/progress`
|
||||
now starts this job (missing previews, its quality preset as the size, medium encoding)
|
||||
or attaches to the running one, and streams the job's unchanged `status`/`start`/
|
||||
`progress`/`file_error`/`complete` frames after a first `job {id, started}` frame that
|
||||
names the job and whether this stream started it. The stream reads the registry after
|
||||
its headers are sent, so a job that starts in that gap is attached to, not refused. A
|
||||
client that goes away no longer stops the work (`test_thumbnail_job_survives_client_disconnect`:
|
||||
one worker, 9 files in three batches; the stream sees its client go after the first
|
||||
batch and all 9 previews are written, where a stream that stops on disconnect leaves 1).
|
||||
The gallery's Cancel (its settings panel and the startup dialog) cancels by that id,
|
||||
and only a job its own stream started: a Cancel before the `job` frame keeps the stream
|
||||
open until it arrives, a double Cancel posts once, and a job another tab or System
|
||||
started keeps running while the gallery stops following it (`tests/js/gallery-thumbnails.test.js`).
|
||||
A job cancelled from Activity ends the stream with the error frame "Thumbnail
|
||||
generation was cancelled". The blocking
|
||||
`POST /images/generate-thumbnails` and the clear stay legacy holds.
|
||||
|
||||
The workspace starts it from "Thumbnail maintenance" (`thumbnail-review.mjs`, the
|
||||
concept's `thumbnail-review`), opened by Sources' Thumbnail generation → Review options
|
||||
and Overview's Review thumbnails. Scope is Missing thumbnails only (`missing`, the
|
||||
default) or Rebuild all (explicit opt-in) (`rebuild`); Quality is Medium · optimized /
|
||||
High · original / Low · fastest (`medium` / `high` / `low`), preselected from
|
||||
Appearance's Image loading quality (`gallerySettings.imageQuality`, the same values,
|
||||
Q3); Preview size limit is the longest edge, 512 by default, a whole number from 128 to
|
||||
2048. A size outside that range marks the field `aria-invalid`, the plan line says
|
||||
"Enter a size from 128 to 2048 pixels." and nothing is requested; Start reports the
|
||||
field's validity and keeps the dialog open. Start closes the dialog and starts the job
|
||||
through the Activity store, which toasts "Thumbnail generation started."; the job opens
|
||||
in Activity. A start refused because another job holds the slot toasts the D57 busy
|
||||
link toast and stays. The legacy gallery's thumbnail tools are unchanged.
|
||||
|
||||
Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per
|
||||
visit through `latestOnly()` on the section scope; a late answer after the section left
|
||||
paints nothing. Until it answers, the section shows "Loading the overview…"; a failure
|
||||
@@ -2257,7 +2006,7 @@ the connection's `total_changes` is 0 at close and the data reads back unchanged
|
||||
`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no
|
||||
reader can keep it byte-identical (`test_health_check_never_writes`).
|
||||
|
||||
Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D113; D121–D128 assigned):
|
||||
Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121–D128 assigned):
|
||||
|
||||
- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No
|
||||
unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server
|
||||
@@ -2270,15 +2019,13 @@ Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D
|
||||
and active saved negatives and the indexed `generated_images` rows (Q2). "Database
|
||||
integrity" is "Not checked" until a check ran this session, then "N issues" (checks
|
||||
that found issues, even beside a check that could not run), "Passed" or "Incomplete".
|
||||
The review row is "N ambiguous image links" (the Needs review queue's pending count,
|
||||
`needs_review.pending`) with the concept's line; its Review metadata action opens
|
||||
`#/review` (the concept's `go-review`). A count
|
||||
that is not an integer reads "Ambiguous image links" / "Not available yet." (Q6).
|
||||
"Ambiguous image links" reads "Not available yet." with its Review metadata action
|
||||
`aria-disabled` (CSS under D93) until the Needs review queue reports a count (Q6).
|
||||
"Backup status" reads "No retained backup record." until verified backups exist (S8).
|
||||
The thumbnail row is "Checking thumbnails…" until its own read answers, then "No
|
||||
missing thumbnails", "N missing thumbnails" (the pending count of
|
||||
`/images/thumbnails/status`) or "Thumbnail status unavailable"; Review thumbnails
|
||||
opens the thumbnail dialog (D99). "One display configuration" says "Grid
|
||||
opens Sources until the thumbnail dialog (S5). "One display configuration" says "Grid
|
||||
and density settings are shared across views." (no paging). The sources row is
|
||||
"N output sources" with "Monitoring and enrichment share one source list.", prefixed
|
||||
by "M of N configured sources available." or "None configured is available; ComfyUI's
|
||||
@@ -2339,43 +2086,6 @@ Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D
|
||||
legacy gallery's `/gallery/rescan` and toasts "Files refreshed. N files in M output
|
||||
folders." or "Files could not be refreshed. Try again.". The concept's "sample path"
|
||||
note is prototype copy and is dropped.
|
||||
- **D99** (Sources/Overview, S5): the concept's `thumbnail-review` dialog ("Thumbnail
|
||||
maintenance") with production copy. A `p.small[role=status]` line before the actions
|
||||
says what the server's read-only plan (`GET /images/thumbnails/plan`) counts for the
|
||||
chosen scope: "N previews to generate · M already up to date." or, for Rebuild all,
|
||||
"Every preview is regenerated: N files.", with "· R more on the next run" past the
|
||||
per-run cap; "Counting previews…" while it is read and "The previews could not be
|
||||
counted." when the read fails (Start still starts). Each scope is read once per
|
||||
dialog, since size and quality change no count. The primary is "Start" (the
|
||||
concept's "Simulate thumbnail job"). Overview's Review thumbnails opens the same
|
||||
dialog (the concept's row starts a sample "Thumbnail check" job directly). Maintenance's
|
||||
Thumbnail cache row takes the dialog with S7. `thumbnail dialog matches the concept
|
||||
from sources|overview` (system-concept.spec, both themes and sizes) maps the line and
|
||||
"Start"; `thumbnail dialog comparison detects mutations` proves they are the only
|
||||
difference and catches a renamed option, swapped options, swapped actions and a
|
||||
changed style.
|
||||
- **D111** (Activity, S5): each job kind's pill verb, toasts, line, tiles, outcomes,
|
||||
openable rows and labels come from one descriptor table (`activity-kinds.mjs`;
|
||||
`activity-store.mjs` re-exports it). The scan's (D56) and image tagging's (D89) are
|
||||
unchanged. A thumbnail job reads "Generating thumbnails · N% · a / b" in the pill,
|
||||
counts Generated (green), Skipped and Errors (red), lists Generated, Errors, Skipped
|
||||
(Generated and Skipped rows open the viewer), says "Thumbnails finished · N generated ·
|
||||
M errors" ("cancelled · N generated", "failed · …") and keeps "Job log" and "Cancel
|
||||
job". Every other kind (the health check, a kind this page does not know) gets the
|
||||
generic descriptor: title-based toasts and no tiles or Results, so it never shows the
|
||||
scan's zero tiles (S1 review M6, S2 review M3). A running job without a known total
|
||||
has an indeterminate `<progress>` (no `value`) and a pill without a percentage, whose
|
||||
mini bar (`aria-hidden`; the pill's label carries the state) is marked `indeterminate`
|
||||
and sweeps instead of showing an empty fill (a dimmed full bar under reduced motion). Only a
|
||||
running job the server can cancel shows Cancel. Tests:
|
||||
`tests/js/workspace-activity-kinds.test.js`, and in `workspace-activity.spec.mjs` "a
|
||||
thumbnail job shows Generated Skipped Errors, its pill verb and its toasts" and "a held
|
||||
legacy job and the health check show no tiles, no Results and no made-up percentage".
|
||||
- **D112** (Activity, S5): a legacy-page operation held on the registry (not
|
||||
cancellable, no outcome groups) reads as its server title ("Database restore (admin
|
||||
page)", "Thumbnail clear (gallery page)"), with no tiles, no Results, no Cancel, an
|
||||
indeterminate bar while it runs and "<title> finished." when it ends, whatever its
|
||||
kind (the gallery's clear is a `thumbnails` hold).
|
||||
- **D113** (Sources): the Worker threads range ends at the server's `cpu_count` (the
|
||||
concept's fixed `max=16`) and shows the server's real `worker_threads` (the concept's
|
||||
sample 4).
|
||||
@@ -2464,13 +2174,10 @@ and a source row, renames a label and the Add source button and swaps the `--lin
|
||||
token. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`,
|
||||
`{concept,production}-{statistics,health}-{dark,light}.png`,
|
||||
`{concept,production}-appearance-{dark,light}-{1600,390}.png`,
|
||||
`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`,
|
||||
`concept-thumbnail-review-{dark,light}-{1600,390}.png`,
|
||||
`production-thumbnail-review-{sources,overview}-{dark,light}-{1600,390}.png`) are test artifacts; there
|
||||
`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`) are test artifacts; there
|
||||
are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts,
|
||||
the health check and metric 4, a failed check, the thumbnail row against the status
|
||||
endpoint (painted after the rest when its read is held, unavailable when it fails), its Review thumbnails
|
||||
opening the thumbnail dialog and starting the job, the review row (Q6), the section links, the Statistics dialog against the
|
||||
endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the
|
||||
route, a load failure with Retry and late answers after leaving, and Appearance: new
|
||||
visits versus the current view and a saved view, the embedded gallery keeping Grid,
|
||||
metadata off and on request, the cache (counted up to a request barrier), hidden paths,
|
||||
@@ -2480,9 +2187,7 @@ the D97 rows and reset. `tests/browser/workspace-system-sources.spec.mjs` covers
|
||||
notice's Later and Review scan, a scoped scan from one source into Activity, Add source
|
||||
with its server check and Save, a refused folder, Remove down to auto-detect, an absent
|
||||
configured source, the three watch states, the monitoring and worker rows, Refresh
|
||||
files, the thumbnail dialog (its plan line per scope with the preselected quality, an
|
||||
invalid size that requests nothing, Start into Activity, Rebuild all, a failed plan and
|
||||
a refused start), a save refused by a running scan and a failed read. Its hooks are
|
||||
files, the thumbnail hand-off, a save refused by a running scan and a failed read. Its hooks are
|
||||
`tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected
|
||||
failures, an absent source, a deleted linked file, unthumbnailed files, a fake image
|
||||
monitor) on the
|
||||
@@ -2497,7 +2202,7 @@ node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-co
|
||||
tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js \
|
||||
tests/js/workspace-system-appearance.test.js tests/js/workspace-preferences.test.js \
|
||||
tests/js/workspace-inspection-state.test.js tests/js/workspace-images-state.test.js \
|
||||
tests/js/workspace-system-sources.test.js tests/js/workspace-thumbnail-review.test.js
|
||||
tests/js/workspace-system-sources.test.js
|
||||
python3 -m unittest tests.test_settings_sources tests.test_scan_scope -v
|
||||
npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \
|
||||
workspace-system-sources.spec.mjs \
|
||||
@@ -2678,167 +2383,6 @@ inspection's own stable stat (`tests/test_api_workspace_images.py`
|
||||
browser fixtures clear `media_image_tags`, `media_annotation_bytes` and
|
||||
`media_annotation_files` on reset and never the metadata row.
|
||||
|
||||
Review queue schema (#249, `database/review_schema.py`,
|
||||
`tests/test_review_schema.py`): the group `media_review_metadata` (one row:
|
||||
schema version, `review_index_revision`), `media_review_items` (one ambiguous
|
||||
resolver result per `(byte_id, resolver_version, evidence_hash)`, citing an
|
||||
annotation byte record `ON DELETE RESTRICT`, with the file's recorded
|
||||
`spelled_path` (the locator is case-folded on Windows, a listing target is
|
||||
not), 0..16 candidates, `limited`,
|
||||
`state` pending/selected/unresolved and a revision) and `media_review_decisions`
|
||||
(one row per item decision revision; `selected` exactly when a
|
||||
`candidate_hash` is present; `pair_id`/`image_id` navigation is set to NULL,
|
||||
never cascaded, when the pair or image is deleted, through
|
||||
`idx_media_review_decisions_pair`/`_image`). Evidence is canonical JSON
|
||||
(`canonical_evidence`: one object, sorted keys, no whitespace, ASCII only, no
|
||||
NaN/infinity, at most 8 MiB) and `evidence_hash` is its SHA-256. The group is
|
||||
valid only beside a valid annotation group (a review group without annotation
|
||||
tables is refused, catalog only, before the annotation install), installs like
|
||||
the annotation group (optional in old backups, required once current under
|
||||
`require_library_state`, SAVEPOINT install validated before and after), and
|
||||
validation re-canonicalizes every evidence document in one streamed pass. A
|
||||
startup runs that pass once (`test_startup_validates_stored_evidence_once`; the
|
||||
install trusts the annotation group it has just validated, and the SQL row
|
||||
predicates check type and size only, because the streamed pass proves the
|
||||
rest): 2.5 s to open a database of 40,000 items with 4 KiB of evidence each,
|
||||
was 7.7 s with two passes. Restore and `verify_database_file` keep the full
|
||||
validation, annotation group included. The browser fixtures clear
|
||||
`media_review_decisions` and `media_review_items` before the annotation tables,
|
||||
never the metadata row.
|
||||
|
||||
Needs review items (#249, `utils/review_capture.py`, `database/review.py`,
|
||||
`tests/test_review.py`, `tests/test_review_outcome.py`,
|
||||
`tests/test_review_choosable.py`): a file is queued only when its final outcome
|
||||
is ambiguous and unlinked and there is something to choose. Only scans produce
|
||||
items; the monitor links new outputs and clears pending items it resolves
|
||||
(maintainer rulings, 2026-10-23). The output scan (Activity's
|
||||
Scan for metadata and the legacy `/scan`) hands `record_ambiguous_file` a file
|
||||
the ingestion resolver finds `ambiguous` only when the browser-parser fallback
|
||||
then links nothing (status `ambiguous`) and no saved row links the file
|
||||
already; an ambiguous file a saved row links is reported `unchanged` ("Already
|
||||
linked to its prompt"), not Needs review. A file the fallback links, an exact
|
||||
pair the ingester stores, and `conflict` and `unresolved` files are not
|
||||
queued. Neither is an ambiguous file with nothing to choose: fewer than two of
|
||||
its candidates are `resolved` with non-blank positive text (`reviewable`). With
|
||||
none the file could not be traced; with exactly one, the untraced branches may
|
||||
still have made the file, so that text is not certain either. Both report
|
||||
`unresolved` ("could not be traced"), are never linked, and their pending,
|
||||
undecided items whose own evidence offers no choice (a limited item counts as
|
||||
a choice when it holds at least two candidates) leave the queue on the next
|
||||
scan or monitor pass (`settle_untraced_review_items`). The maintainer's
|
||||
second live check found 39 items whose 11 candidates were all "unresolved". The queue heals itself: after each batch, the scan removes the pending
|
||||
items of every listed file (read or skipped as known) that a saved row now
|
||||
links, and the monitor does the same for each ambiguous output it links
|
||||
(`settle_linked_review_items`; the review index revision moves). Only an
|
||||
undecided item goes (`pending`, revision 0, no decision row): a decision is
|
||||
never deleted or changed, so no schema state was added. The monitor never
|
||||
queues an output it linked; an unlinked file passed to `record_review_file`
|
||||
follows the scan's rule (a file the fallback would link is `skipped`). The
|
||||
maintainer's live check of #284/#288 found the queue at 10,232 against a review
|
||||
group of 4,677: every fallback-linked ambiguous file had been queued. It captures
|
||||
the file (scope `needs-review`), reads its metadata again from the capture,
|
||||
resolves it with the default profile and stores only an ambiguous result,
|
||||
bound to the capture's byte record (created or refreshed like an observe) and
|
||||
fenced by the capture's storage epoch. Evidence is the resolver reason plus one
|
||||
`{sampler_id, state, reason, positive, negative, hash}` per selectable
|
||||
candidate (nested ambiguous stages flattened, duplicates dropped; each side
|
||||
`{text, storage_text, provenance}`; `hash` = SHA-256 of the canonical candidate
|
||||
without it). More than 16 candidates, or evidence over 8 MiB, is a `limited`
|
||||
item without candidates. Recording is idempotent per `(byte_id,
|
||||
resolver_version, evidence_hash)` and never touches an existing item or its
|
||||
decisions; changed bytes add a new pending item and keep the old one. The
|
||||
capture is released before the write; no model is loaded and no pixel is
|
||||
decoded. Scan statuses, counters, outcome rows and the legacy SSE events are
|
||||
unchanged apart from the already-linked and nothing-to-choose cases above; the job's `scope` gains
|
||||
`review_items` (this scan's files that are in the pending queue after it, new
|
||||
or already there: the review group's ambiguous rows, less files a capture could
|
||||
not read or a decision already settled), `review_added` (new items),
|
||||
`review_settled` (items removed because their file is linked now),
|
||||
`review_untraced` (items removed because they offer nothing to choose) and
|
||||
`review_errors`. The log ends with `? N images added to Needs review`,
|
||||
`? N images already in Needs review`, `= N images left Needs review · now
|
||||
linked`, `= N images left Needs review · nothing to choose` and `! N images could not be added to Needs review` (failures, e.g.
|
||||
capture capacity), each only when N > 0. Recording captures in the background: it never takes the store's
|
||||
last worker or capture slot, which stay free for an interactive Tag dialog or
|
||||
Apply, and waits up to 2 s for room before counting the file as not added
|
||||
(`test_background_recording_leaves_a_worker_for_the_tag_dialog`). The monitor's
|
||||
pass runs after its own link, best effort: a failure is logged and the link
|
||||
stays. Recording follows the registry's `records`
|
||||
admission (#289): the scan's recording runs under its own `records` job, and
|
||||
the monitor's (one file, not a job) is refused while a `records` or `guarded`
|
||||
job or hold runs, except thumbnail work, which writes no records (S5 M7). It then records nothing, logs that the file waits for the
|
||||
next scan, and answers `deferred`; the next "new" scan reads the ambiguous file
|
||||
again (`test_monitor_recording_is_refused_beside_record_changing_jobs`). A new item moves `review_index_revision`. A
|
||||
rescan does not copy or hash an ambiguous file whose current size and
|
||||
`mtime_ns` equal the verified fingerprint of bytes an item already cites (a
|
||||
stat compare); the file still stays in the default "new" scope, because it is
|
||||
not a known image. The skip and a duplicate insert record the current spelling
|
||||
of the locator on its items (`refresh_review_spelling`, the review index
|
||||
revision moves, item revisions do not), so a case-only rename on Windows or
|
||||
macOS keeps its item applicable. Residual risks of the skip, like S1's: an
|
||||
in-place rewrite that keeps both size and `mtime_ns` is never re-queued (Apply
|
||||
still re-hashes, so it cannot link those bytes), and a file that is deleted or
|
||||
moved leaves its pending item counted in `total` (its target then formats as
|
||||
`unavailable` or fails to open, and Leave unresolved clears it).
|
||||
|
||||
Needs review decisions and queue reads (#249, `py/api/review.py`,
|
||||
`database/review.py`, `database/review_decisions.py`,
|
||||
`tests/test_api_review.py`): `GET /prompt_manager/review/items?limit=1..40&cursor=`
|
||||
answers `{success, library_epoch, total, items, next_cursor}` with pending items
|
||||
oldest first (each item adds the card's `url`, the Images listing's
|
||||
`/prompt_manager/images/serve/<path>?root=<i>&root_scope=<scope>` formatted without a
|
||||
stat, null when the file is under no current root, and `is_video` from its extension); `total` is the exact pending count. The cursor is an HMAC keyset
|
||||
on `(created_at, id)` bound to the storage epoch only (ruling M3, so "load more"
|
||||
keeps working while the monitor adds an item per generation): a malformed one
|
||||
is 400, a well-formed one this process cannot verify (tampered, or opened
|
||||
before a ComfyUI restart: the secret is per process) is 409 `review-conflict`
|
||||
with the current `total`, and one from before a restore is 409
|
||||
`storage-changed`. A decision or a new item leaves an open cursor valid: a
|
||||
decided item drops out of later pages and new items (stamped `utc_now_iso()`,
|
||||
microseconds) append after it. Residual risk: an item recorded after a wall
|
||||
clock step backwards sorts before an open cursor and shows only on reload.
|
||||
Targets are formatted from `spelled_path` against the current roots (the
|
||||
roots' real paths are the only file access); a file under no current root is
|
||||
`unavailable` with a null target. Every read and decision lists the roots and
|
||||
resolves real paths on a worker thread, never on the event loop. `GET .../items/{id}` adds `candidates`
|
||||
(`hash`, `sampler_id`, `state` resolved/unresolved, `reason`, `label` "<positive
|
||||
title> / <negative title>" from the saved prompt's Library title else the
|
||||
first line, and the role texts), `dependency_fingerprint` (SHA-256 of the
|
||||
file's legacy rows under every spelling of the same physical file, positive
|
||||
hashes and pair links with their negative hashes, as the ingester reads them) and `sha256`; a detail over 8 MiB, or a
|
||||
limited item, has no candidates. `POST .../items/{id}/decision
|
||||
{expected_epoch, expected_revision, capture_token, sha256, dependency_fingerprint,
|
||||
choice}` with a candidate hash (Apply) requires a capture of the item's file
|
||||
whose bytes are the item's, re-hashes the current file (`file-changed`
|
||||
otherwise, also for a fresh capture of new bytes), re-reads the capture and
|
||||
requires that its bytes are still ambiguous and still offer the chosen
|
||||
candidate (`review-conflict` otherwise; a resolver change that rewords the
|
||||
reason or reorders candidates leaves the item applicable, and
|
||||
`RESOLVER_VERSION` in `utils/prompt_pair_resolver.py` is the one version
|
||||
ingestion provenance and review share), then in one `_pair_transaction`
|
||||
checks the epoch, the item revision and state (`review-conflict` with the
|
||||
current item for a repeat), the re-hashed digest, the dependency fingerprint
|
||||
and the public ingester's compatible-only rules through
|
||||
`_ingest_resolved_pair_in` (any refusal is 409 `association-conflict` with
|
||||
zero writes, explained inline; Apply never reparents a link). Physical
|
||||
identity: the roots are real paths while the monitor links under the
|
||||
directory it watches, so a symlinked or junctioned root, `subst` or a mapped
|
||||
drive spells the same file otherwise. Every legacy row whose spelling ends with
|
||||
the file's name (the link summaries' unindexed suffix lookup) is resolved by
|
||||
`normcase(realpath)` against the locator, else `samefile`; two rows of the
|
||||
file refuse the choice, one is enriched under the spelling it was linked
|
||||
with, and a same-name row the request did not classify is a changed
|
||||
association (`TestPhysicalIdentity`, symlinked root and a portable fake
|
||||
junction), and writes
|
||||
the link with `kind:"user-selection"` provenance (decision id, reviewed digest,
|
||||
candidate hash, resolver version, role provenance), the decision row, the
|
||||
item's new state and revision and `review_index_revision`. `"unresolved"`
|
||||
records the decision only and accepts `capture_token: null`. A successful
|
||||
decision releases its capture. No decision changes run or usage counts or
|
||||
calls an execution writer; a Tag save never changes a review item. Errors carry
|
||||
fixed sentences: `review-failed` / `decision-failed` 500s never echo exception
|
||||
text or paths.
|
||||
|
||||
Capture store (#249, `utils/media_capture.py`, `tests/test_media_capture.py`):
|
||||
one explicit action captures one rooted media file into an owner-only
|
||||
`prompt-manager-captures-*` temp directory, streaming it in 1 MiB chunks while
|
||||
@@ -2855,12 +2399,8 @@ directories are `CaptureDenied`; the locator is
|
||||
`normcase(join(realpath(root), relative))`, so a symlinked root shares the
|
||||
listing's identity. Fixed limits: 256 MiB per file (stat and stream), 8
|
||||
completed captures, 512 MiB reserved, 2 capture/verify workers, 900 s idle,
|
||||
1800 s hard; interactive admission is nonblocking (`CaptureCapacity`), a
|
||||
`background=True` capture (Needs review recording) never takes the last worker
|
||||
or capture slot and waits up to `BACKGROUND_WAIT_SECONDS` (2 s) for room, and
|
||||
lifetimes are tested with an injected clock. The host's on_cleanup close is
|
||||
final: a recording still in flight gets a capture error instead of creating a
|
||||
new temp directory after it. An acquired capture outlives expiry and release
|
||||
1800 s hard; admission is nonblocking (`CaptureCapacity`) and lifetimes are
|
||||
tested with an injected clock. An acquired capture outlives expiry and release
|
||||
until its last use ends. Header-only dimensions (4096 px per side,
|
||||
16,777,216 px) and video mark Generate unsupported; `verify` re-hashes the
|
||||
current file without a copy. Every handle closes before unlink (a patched
|
||||
|
||||
-18
@@ -91,9 +91,6 @@ try:
|
||||
if host_app is not None and hasattr(host_app, "on_cleanup"):
|
||||
host_app.on_cleanup.append(api.cleanup_history_reads)
|
||||
host_app.on_cleanup.append(api.cleanup_annotation_captures)
|
||||
if host_app is not None and hasattr(host_app, "on_shutdown"):
|
||||
# Open Needs review change streams end before the server waits on them.
|
||||
host_app.on_shutdown.append(api.close_review_events)
|
||||
|
||||
except Exception as e:
|
||||
# Log to internal logging system without console spam
|
||||
@@ -125,21 +122,6 @@ try:
|
||||
_global_prompt_tracker = get_prompt_tracker(_global_db)
|
||||
_global_image_monitor = get_image_monitor(_global_db, _global_prompt_tracker)
|
||||
|
||||
# The Needs review pass for ambiguous new outputs runs through the API's
|
||||
# capture store (the one the scan and the Tag dialog use): a linked output
|
||||
# is never queued and its pending items leave the queue. It stops before
|
||||
# that store is closed at shutdown.
|
||||
_review_api = globals().get("api")
|
||||
if _review_api is not None:
|
||||
_global_image_monitor.set_review_recorder(_review_api.record_review_file)
|
||||
_review_host = getattr(config.server_instance, "app", None)
|
||||
if _review_host is not None and hasattr(_review_host, "on_cleanup"):
|
||||
|
||||
async def _stop_review_recording(app, monitor=_global_image_monitor):
|
||||
monitor.set_review_recorder(None)
|
||||
|
||||
_review_host.on_cleanup.insert(0, _stop_review_recording)
|
||||
|
||||
# Start monitoring
|
||||
_global_image_monitor.start_monitoring()
|
||||
_init_logger.info("Global image monitoring system started successfully")
|
||||
|
||||
+2
-5
@@ -637,8 +637,7 @@ class LibraryOperations:
|
||||
pairs (every pair's positive is eligible). ``images`` counts indexed
|
||||
``generated_images`` rows, not files on disk. An image has no rating
|
||||
of its own (it shows its linked prompt's), so ``rated_images`` counts
|
||||
the indexed images whose linked prompt is rated. ``review_pending`` is
|
||||
the Needs review queue's pending total.
|
||||
the indexed images whose linked prompt is rated.
|
||||
"""
|
||||
with self.library_snapshot() as (conn, _epoch, _ceiling):
|
||||
row = conn.execute(
|
||||
@@ -646,15 +645,13 @@ class LibraryOperations:
|
||||
"(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0),"
|
||||
"(SELECT COUNT(*) FROM generated_images),"
|
||||
"(SELECT COUNT(*) FROM generated_images gi JOIN prompts r "
|
||||
"ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL),"
|
||||
"(SELECT COUNT(*) FROM media_review_items WHERE state='pending')"
|
||||
"ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL)"
|
||||
).fetchone()
|
||||
return {
|
||||
"positive_prompts": row[0],
|
||||
"negative_variants": row[1],
|
||||
"images": row[2],
|
||||
"rated_images": row[3],
|
||||
"review_pending": row[4],
|
||||
}
|
||||
|
||||
def usage_by_category(self) -> list[dict]:
|
||||
|
||||
+3
-20
@@ -18,11 +18,6 @@ from typing import Optional, Set, Tuple
|
||||
from .library_schema import library_schema_error, migrate_library_state
|
||||
from .image_favorites import image_favorites_schema_error, migrate_image_favorites
|
||||
from .annotation_schema import annotation_schema_error, migrate_annotation_state
|
||||
from .review_schema import (
|
||||
migrate_review_state,
|
||||
review_schema_error,
|
||||
review_without_annotations_error,
|
||||
)
|
||||
from .saved_views import migrate_saved_views, saved_views_schema_error
|
||||
from .execution_history_schema import (
|
||||
history_schema_error,
|
||||
@@ -308,14 +303,7 @@ class PromptModel:
|
||||
migrate_library_state(conn)
|
||||
migrate_image_favorites(conn)
|
||||
migrate_saved_views(conn)
|
||||
# A review group without its annotation group is refused before the
|
||||
# annotation install could make it look valid (catalog only; the rows
|
||||
# are validated once, in migrate_review_state).
|
||||
review_error = review_without_annotations_error(conn)
|
||||
if review_error:
|
||||
raise sqlite3.DatabaseError(review_error)
|
||||
migrate_annotation_state(conn)
|
||||
migrate_review_state(conn)
|
||||
|
||||
def _migrate_prompt_pairs(self, conn: sqlite3.Connection) -> None:
|
||||
conn.execute("SAVEPOINT prompt_pair_schema")
|
||||
@@ -1035,9 +1023,9 @@ class PromptModel:
|
||||
Args:
|
||||
path: File to inspect
|
||||
require_library_state: Also require the tables every current
|
||||
database has (Library state, image favorites, saved views,
|
||||
image annotations and the review queue), as a migrated
|
||||
restore stage must; old backups may lack them.
|
||||
database has (Library state, image favorites, saved views
|
||||
and image annotations), as a migrated restore stage must;
|
||||
old backups may lack them.
|
||||
|
||||
Returns:
|
||||
(ok, reason): reason is "ok" when the file is acceptable, otherwise
|
||||
@@ -1107,11 +1095,6 @@ class PromptModel:
|
||||
)
|
||||
if annotation_error:
|
||||
return False, annotation_error
|
||||
review_error = review_schema_error(
|
||||
conn, allow_missing=not require_library_state
|
||||
)
|
||||
if review_error:
|
||||
return False, review_error
|
||||
return True, "ok"
|
||||
except sqlite3.Error as e:
|
||||
return False, f"Invalid SQLite database: {e}"
|
||||
|
||||
@@ -14,8 +14,6 @@ from .pairs import PairDependencyError, PairOperations
|
||||
from .library import LibraryOperations, titled
|
||||
from .image_favorites import ImageFavoriteOperations
|
||||
from .annotations import AnnotationOperations
|
||||
from .review import ReviewOperations
|
||||
from .review_decisions import ReviewDecisionOperations
|
||||
from .saved_views import SavedViewOperations
|
||||
from .library_commands import LibraryCommandOperations
|
||||
from .vocabulary import VocabularyOperations
|
||||
@@ -347,8 +345,6 @@ class PromptDatabase(
|
||||
HistoryReadOperations,
|
||||
ImageFavoriteOperations,
|
||||
AnnotationOperations,
|
||||
ReviewOperations,
|
||||
ReviewDecisionOperations,
|
||||
SavedViewOperations,
|
||||
HealthOperations,
|
||||
):
|
||||
|
||||
+107
-135
@@ -9,11 +9,9 @@ from .operations import _coerce_file_info
|
||||
try:
|
||||
from ..utils.hashing import generate_prompt_hash
|
||||
from ..utils.json_strict import dumps_strict
|
||||
from ..utils.prompt_pair_resolver import RESOLVER_VERSION
|
||||
except ImportError:
|
||||
from utils.hashing import generate_prompt_hash
|
||||
from utils.json_strict import dumps_strict
|
||||
from utils.prompt_pair_resolver import RESOLVER_VERSION
|
||||
|
||||
|
||||
def ingest_resolved_pair(
|
||||
@@ -22,143 +20,117 @@ def ingest_resolved_pair(
|
||||
state = resolution.get("state", "unresolved")
|
||||
if state != "resolved":
|
||||
return {"status": state, "pairs_added": 0, "links_added": 0}
|
||||
with db._pair_transaction() as conn:
|
||||
return _ingest_resolved_pair_in(conn, db, image_path, metadata, resolution)
|
||||
|
||||
|
||||
def _conflict() -> Dict[str, Any]:
|
||||
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
|
||||
|
||||
|
||||
def _ingest_resolved_pair_in(
|
||||
conn: Any,
|
||||
db: Any,
|
||||
image_path: str,
|
||||
metadata: Dict[str, Any],
|
||||
resolution: Dict[str, Any],
|
||||
*,
|
||||
provenance: Dict[str, Any] | None = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""``ingest_resolved_pair`` inside the caller's open ``_pair_transaction``.
|
||||
|
||||
The compatible-only rules are the public ingester's: an image with more
|
||||
than one exact row, another positive, or a prior pair with another
|
||||
negative is a ``conflict`` and nothing is written. The caller owns the
|
||||
transaction, so its own later failure rolls the link back too.
|
||||
``provenance`` replaces the link's default resolver provenance (Needs
|
||||
review records the user's selection there).
|
||||
"""
|
||||
positive, negative = resolution["positive"], resolution["negative"]
|
||||
positive_text = positive.get("storage_text", positive["text"])
|
||||
negative_text = negative.get("storage_text", negative["text"])
|
||||
normalized = normalize_image_path(image_path)
|
||||
if provenance is None:
|
||||
provenance = {
|
||||
"resolver_version": RESOLVER_VERSION,
|
||||
"positive": positive,
|
||||
"negative": negative,
|
||||
}
|
||||
provenance = {"resolver_version": 1, "positive": positive, "negative": negative}
|
||||
encoded = dumps_strict(provenance, sort_keys=True, separators=(",", ":"))
|
||||
images = conn.execute(
|
||||
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
|
||||
"JOIN prompts p ON p.id=gi.prompt_id "
|
||||
"WHERE gi.file_path=? OR gi.image_path=?",
|
||||
(normalized, image_path),
|
||||
).fetchall()
|
||||
if len(images) > 1 or (
|
||||
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
|
||||
):
|
||||
return _conflict()
|
||||
existing = images[0] if images else None
|
||||
if existing:
|
||||
prior = conn.execute(
|
||||
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
|
||||
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
|
||||
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
|
||||
(existing["id"],),
|
||||
with db._pair_transaction() as conn:
|
||||
images = conn.execute(
|
||||
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
|
||||
"JOIN prompts p ON p.id=gi.prompt_id "
|
||||
"WHERE gi.file_path=? OR gi.image_path=?",
|
||||
(normalized, image_path),
|
||||
).fetchall()
|
||||
expected = (
|
||||
generate_prompt_hash(negative_text) if negative_text.strip() else None
|
||||
)
|
||||
if any(row["negative_hash"] != expected for row in prior):
|
||||
return _conflict()
|
||||
prior_pair = conn.execute(
|
||||
"SELECT pp.id FROM prompt_pairs pp "
|
||||
"JOIN prompts p ON p.id=pp.positive_prompt_id "
|
||||
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
|
||||
"WHERE p.hash=? AND n.hash IS ?",
|
||||
(
|
||||
generate_prompt_hash(positive_text),
|
||||
generate_prompt_hash(negative_text) if negative_text.strip() else None,
|
||||
),
|
||||
).fetchone()
|
||||
pair_id = db._save_pair(conn, positive_text, negative_text)
|
||||
pair = conn.execute("SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)).fetchone()
|
||||
pairs_added = int(prior_pair is None)
|
||||
file_size, width, height, image_format = _coerce_file_info(metadata)
|
||||
# ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON.
|
||||
snapshots = {
|
||||
column: dumps_strict(metadata[key], sort_keys=True)
|
||||
for column, key in (
|
||||
("workflow_data", "workflow"),
|
||||
("prompt_metadata", "prompt"),
|
||||
("parameters", "parameters"),
|
||||
)
|
||||
if key in metadata
|
||||
}
|
||||
info = {
|
||||
"file_size": file_size,
|
||||
"width": width,
|
||||
"height": height,
|
||||
"format": image_format,
|
||||
}
|
||||
snapshots.update({key: value for key, value in info.items() if value is not None})
|
||||
changed = False
|
||||
if existing:
|
||||
image_id = existing["id"]
|
||||
updates = {
|
||||
key: value for key, value in snapshots.items() if existing[key] != value
|
||||
}
|
||||
if updates:
|
||||
conn.execute(
|
||||
"UPDATE generated_images SET "
|
||||
+ ",".join(f"{key}=?" for key in updates)
|
||||
+ " WHERE id=?",
|
||||
[*updates.values(), image_id],
|
||||
if len(images) > 1 or (
|
||||
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
|
||||
):
|
||||
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
|
||||
existing = images[0] if images else None
|
||||
if existing:
|
||||
prior = conn.execute(
|
||||
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
|
||||
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
|
||||
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
|
||||
(existing["id"],),
|
||||
).fetchall()
|
||||
expected = (
|
||||
generate_prompt_hash(negative_text) if negative_text.strip() else None
|
||||
)
|
||||
changed = True
|
||||
else:
|
||||
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
|
||||
values = [
|
||||
pair["positive_prompt_id"],
|
||||
image_path,
|
||||
os.path.basename(image_path),
|
||||
normalized,
|
||||
*snapshots.values(),
|
||||
]
|
||||
image_id = conn.execute(
|
||||
"INSERT INTO generated_images("
|
||||
+ ",".join(columns)
|
||||
+ ") VALUES("
|
||||
+ ",".join("?" for _ in columns)
|
||||
+ ")",
|
||||
values,
|
||||
).lastrowid
|
||||
linked = conn.execute(
|
||||
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
|
||||
"AND image_id=? AND provenance=?",
|
||||
(pair_id, image_id, encoded),
|
||||
).fetchone()
|
||||
db._link_image_to_pair(conn, pair_id, image_id, provenance)
|
||||
status = (
|
||||
"added"
|
||||
if existing is None
|
||||
else "enriched" if changed or not linked else "unchanged"
|
||||
)
|
||||
return {
|
||||
"status": status,
|
||||
"pairs_added": pairs_added,
|
||||
"links_added": int(not linked),
|
||||
"pair_id": pair_id,
|
||||
"image_id": image_id,
|
||||
}
|
||||
if any(row["negative_hash"] != expected for row in prior):
|
||||
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
|
||||
prior_pair = conn.execute(
|
||||
"SELECT pp.id FROM prompt_pairs pp "
|
||||
"JOIN prompts p ON p.id=pp.positive_prompt_id "
|
||||
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
|
||||
"WHERE p.hash=? AND n.hash IS ?",
|
||||
(
|
||||
generate_prompt_hash(positive_text),
|
||||
generate_prompt_hash(negative_text) if negative_text.strip() else None,
|
||||
),
|
||||
).fetchone()
|
||||
pair_id = db._save_pair(conn, positive_text, negative_text)
|
||||
pair = conn.execute(
|
||||
"SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)
|
||||
).fetchone()
|
||||
pairs_added = int(prior_pair is None)
|
||||
file_size, width, height, image_format = _coerce_file_info(metadata)
|
||||
# ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON.
|
||||
snapshots = {
|
||||
column: dumps_strict(metadata[key], sort_keys=True)
|
||||
for column, key in (
|
||||
("workflow_data", "workflow"),
|
||||
("prompt_metadata", "prompt"),
|
||||
("parameters", "parameters"),
|
||||
)
|
||||
if key in metadata
|
||||
}
|
||||
info = {
|
||||
"file_size": file_size,
|
||||
"width": width,
|
||||
"height": height,
|
||||
"format": image_format,
|
||||
}
|
||||
snapshots.update(
|
||||
{key: value for key, value in info.items() if value is not None}
|
||||
)
|
||||
changed = False
|
||||
if existing:
|
||||
image_id = existing["id"]
|
||||
updates = {
|
||||
key: value for key, value in snapshots.items() if existing[key] != value
|
||||
}
|
||||
if updates:
|
||||
conn.execute(
|
||||
"UPDATE generated_images SET "
|
||||
+ ",".join(f"{key}=?" for key in updates)
|
||||
+ " WHERE id=?",
|
||||
[*updates.values(), image_id],
|
||||
)
|
||||
changed = True
|
||||
else:
|
||||
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
|
||||
values = [
|
||||
pair["positive_prompt_id"],
|
||||
image_path,
|
||||
os.path.basename(image_path),
|
||||
normalized,
|
||||
*snapshots.values(),
|
||||
]
|
||||
image_id = conn.execute(
|
||||
"INSERT INTO generated_images("
|
||||
+ ",".join(columns)
|
||||
+ ") VALUES("
|
||||
+ ",".join("?" for _ in columns)
|
||||
+ ")",
|
||||
values,
|
||||
).lastrowid
|
||||
linked = conn.execute(
|
||||
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
|
||||
"AND image_id=? AND provenance=?",
|
||||
(pair_id, image_id, encoded),
|
||||
).fetchone()
|
||||
db._link_image_to_pair(conn, pair_id, image_id, provenance)
|
||||
status = (
|
||||
"added"
|
||||
if existing is None
|
||||
else "enriched" if changed or not linked else "unchanged"
|
||||
)
|
||||
return {
|
||||
"status": status,
|
||||
"pairs_added": pairs_added,
|
||||
"links_added": int(not linked),
|
||||
"pair_id": pair_id,
|
||||
"image_id": image_id,
|
||||
}
|
||||
|
||||
@@ -1,561 +0,0 @@
|
||||
"""Needs review items (#249): persist and read capture-bound ambiguous evidence.
|
||||
|
||||
An item is one ambiguous resolver result for one annotation byte record (the
|
||||
whole-file SHA-256 of one physical file at one locator), keyed by
|
||||
``(byte_id, resolver_version, evidence_hash)``. Recording is idempotent: the
|
||||
same evidence for the same bytes is a duplicate and never touches an existing
|
||||
item, its state, revision or decisions. Changed bytes are a new byte record
|
||||
and therefore a new pending item; the old one stays.
|
||||
|
||||
Callers pass verified facts (the capture's digest, size and ``mtime_ns``);
|
||||
nothing here reads the filesystem. Writes run in one ``_pair_transaction``
|
||||
fenced by the storage epoch the capture was taken under.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
from typing import Iterator
|
||||
|
||||
try:
|
||||
from ..utils.hashing import generate_prompt_hash
|
||||
except ImportError:
|
||||
from utils.hashing import generate_prompt_hash
|
||||
|
||||
from .annotation_schema import (
|
||||
BYTES_TABLE,
|
||||
FILES_TABLE,
|
||||
METADATA_TABLE as ANNOTATION_METADATA_TABLE,
|
||||
)
|
||||
from .annotations import _new_id, _validate_locator, _validate_target
|
||||
from .models import normalize_image_path, utc_now_iso
|
||||
from .review_changes import notify_review_change
|
||||
from .review_schema import (
|
||||
DECISIONS_TABLE,
|
||||
ITEMS_TABLE,
|
||||
MAX_CANDIDATES,
|
||||
MAX_SPELLED_PATH_BYTES,
|
||||
METADATA_TABLE,
|
||||
canonical_evidence,
|
||||
evidence_digest,
|
||||
evidence_offers_choice,
|
||||
)
|
||||
|
||||
# Only this module's table constants are interpolated into the SQL below.
|
||||
_INSERT_ITEM = (
|
||||
f"INSERT OR IGNORE INTO {ITEMS_TABLE}(id,byte_id,spelled_path," # nosec B608
|
||||
"resolver_version,evidence_hash,evidence,candidate_count,limited,state,"
|
||||
"revision,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?,'pending',0,?,?)"
|
||||
)
|
||||
_BUMP_REVIEW_INDEX = (
|
||||
f"UPDATE {METADATA_TABLE} SET " # nosec B608
|
||||
"review_index_revision=review_index_revision+1 WHERE singleton_id=1"
|
||||
)
|
||||
# A later spelling of the same locator (a case-only rename on a case-folding
|
||||
# filesystem) replaces the recorded one, so targets keep formatting.
|
||||
_REFRESH_SPELLING = (
|
||||
f"UPDATE {ITEMS_TABLE} SET spelled_path=? " # nosec B608
|
||||
f"WHERE spelled_path<>? AND byte_id IN (SELECT b.id FROM {BYTES_TABLE} b "
|
||||
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE f.locator=?)"
|
||||
)
|
||||
_REVIEW_INDEX = (
|
||||
f"SELECT review_index_revision FROM {METADATA_TABLE} " # nosec B608
|
||||
"WHERE singleton_id=1"
|
||||
)
|
||||
_BUMP_IMAGE_INDEX = (
|
||||
f"UPDATE {ANNOTATION_METADATA_TABLE} SET " # nosec B608
|
||||
"image_index_revision=image_index_revision+1 WHERE singleton_id=1"
|
||||
)
|
||||
_ITEMS_OF_LOCATOR = (
|
||||
"SELECT i.id,i.byte_id,i.resolver_version,i.evidence_hash," # nosec B608
|
||||
"i.evidence,i.candidate_count,i.limited,i.state,i.revision,i.created_at,"
|
||||
f"i.updated_at,b.sha256 FROM {FILES_TABLE} f "
|
||||
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
|
||||
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
|
||||
"WHERE f.locator=? ORDER BY i.created_at,i.id"
|
||||
)
|
||||
_QUEUED = (
|
||||
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
|
||||
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
|
||||
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
|
||||
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
|
||||
"LIMIT 1"
|
||||
)
|
||||
_PENDING_AT = (
|
||||
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
|
||||
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
|
||||
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
|
||||
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
|
||||
"AND i.state='pending' LIMIT 1"
|
||||
)
|
||||
# Pending items of one file: the candidates a link settles.
|
||||
_PENDING_OF_LOCATOR = (
|
||||
f"SELECT i.id FROM {FILES_TABLE} f " # nosec B608
|
||||
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
|
||||
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
|
||||
"WHERE f.locator=? AND i.state='pending'"
|
||||
)
|
||||
# Pending items of one file with their evidence: an untraced file's settle.
|
||||
_PENDING_EVIDENCE_OF_LOCATOR = (
|
||||
f"SELECT i.id,i.evidence FROM {FILES_TABLE} f " # nosec B608
|
||||
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
|
||||
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
|
||||
"WHERE f.locator=? AND i.state='pending'"
|
||||
)
|
||||
# Only an undecided item goes: a decision row (RESTRICT) is never deleted.
|
||||
_SETTLE_ITEM = (
|
||||
f"DELETE FROM {ITEMS_TABLE} WHERE id=? AND state='pending' " # nosec B608
|
||||
f"AND revision=0 AND NOT EXISTS (SELECT 1 FROM {DECISIONS_TABLE} d "
|
||||
f"WHERE d.item_id={ITEMS_TABLE}.id)"
|
||||
)
|
||||
_EPOCH = "SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id=1"
|
||||
_PENDING_COUNT = (
|
||||
f"SELECT COUNT(*) FROM {ITEMS_TABLE} WHERE state='pending'" # nosec B608
|
||||
)
|
||||
_QUEUE_COLUMNS = (
|
||||
"id",
|
||||
"state",
|
||||
"revision",
|
||||
"created_at",
|
||||
"candidate_count",
|
||||
"limited",
|
||||
"spelled_path",
|
||||
"locator",
|
||||
)
|
||||
_QUEUE_SELECT = (
|
||||
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
|
||||
f"i.limited,i.spelled_path,f.locator FROM {ITEMS_TABLE} i "
|
||||
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
|
||||
f"JOIN {FILES_TABLE} f ON f.id=b.file_id "
|
||||
)
|
||||
_QUEUE_FIRST = _QUEUE_SELECT + (
|
||||
"WHERE i.state='pending' ORDER BY i.created_at,i.id LIMIT ?"
|
||||
)
|
||||
_QUEUE_AFTER = _QUEUE_SELECT + (
|
||||
"WHERE i.state='pending' AND (i.created_at,i.id)>(?,?) "
|
||||
"ORDER BY i.created_at,i.id LIMIT ?"
|
||||
)
|
||||
_DETAIL_COLUMNS = _QUEUE_COLUMNS + (
|
||||
"byte_id",
|
||||
"resolver_version",
|
||||
"evidence_hash",
|
||||
"evidence",
|
||||
"updated_at",
|
||||
"sha256",
|
||||
)
|
||||
_DETAIL = (
|
||||
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
|
||||
"i.limited,i.spelled_path,f.locator,i.byte_id,i.resolver_version,"
|
||||
"i.evidence_hash,i.evidence,i.updated_at,b.sha256 "
|
||||
f"FROM {ITEMS_TABLE} i JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
|
||||
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE i.id=?"
|
||||
)
|
||||
# The legacy rows and pair links the ingester's compatibility rules read.
|
||||
_DEPENDENT_IMAGES = (
|
||||
"SELECT gi.id,p.hash,gi.image_path FROM generated_images gi "
|
||||
"JOIN prompts p ON p.id=gi.prompt_id "
|
||||
"WHERE gi.file_path=? OR gi.image_path=? ORDER BY gi.id"
|
||||
)
|
||||
_DEPENDENT_LINKS = (
|
||||
"SELECT DISTINCT ip.pair_id,n.hash FROM image_prompt_pairs ip "
|
||||
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
|
||||
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
|
||||
"WHERE ip.image_id=? ORDER BY ip.pair_id"
|
||||
)
|
||||
# Rows whose spelling ends with the file's name: the candidates for another
|
||||
# spelling of the same physical file (the link summaries' filename fallback).
|
||||
_SAME_NAME_ROWS = (
|
||||
"SELECT id,image_path FROM generated_images "
|
||||
"WHERE image_path LIKE ? ESCAPE '\\' ORDER BY id"
|
||||
)
|
||||
_TITLES = (
|
||||
"SELECT p.hash,s.title FROM prompts p "
|
||||
"JOIN prompt_library_state s ON s.prompt_id=p.id "
|
||||
"WHERE p.hash IN ({marks}) AND s.title IS NOT NULL ORDER BY p.id"
|
||||
)
|
||||
_ITEM_ID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
|
||||
_ITEM_COLUMNS = (
|
||||
"id",
|
||||
"byte_id",
|
||||
"resolver_version",
|
||||
"evidence_hash",
|
||||
"evidence",
|
||||
"candidate_count",
|
||||
"limited",
|
||||
"state",
|
||||
"revision",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
"sha256",
|
||||
)
|
||||
|
||||
|
||||
def _validate_spelling(spelled_path: object, locator: str) -> None:
|
||||
if (
|
||||
not isinstance(spelled_path, str)
|
||||
or not spelled_path
|
||||
or len(spelled_path.encode("utf-8", "surrogatepass")) > MAX_SPELLED_PATH_BYTES
|
||||
or os.path.normcase(spelled_path) != locator
|
||||
):
|
||||
raise ValueError("spelled_path must be the file's spelling of its locator")
|
||||
|
||||
|
||||
def dependent_images(conn: sqlite3.Connection, spellings) -> list:
|
||||
"""``(id, image_path, positive_hash)`` of every legacy row of the file.
|
||||
|
||||
``spellings`` are the paths the file is linked under (the Apply path and
|
||||
every alias of the same physical file); each is matched as the ingester
|
||||
matches one path. Rows are distinct and ordered by id.
|
||||
"""
|
||||
found = {}
|
||||
for spelled in dict.fromkeys(spellings):
|
||||
for row in conn.execute(
|
||||
_DEPENDENT_IMAGES, (normalize_image_path(spelled), spelled)
|
||||
):
|
||||
found[row[0]] = (row[0], row[2], row[1])
|
||||
return [found[key] for key in sorted(found)]
|
||||
|
||||
|
||||
def same_name_rows(conn: sqlite3.Connection, name: str) -> list:
|
||||
"""``(id, image_path)`` of rows whose spelling may end with ``name``."""
|
||||
# Imported here: operations composes this module's mixin.
|
||||
from .operations import escape_like
|
||||
|
||||
return [
|
||||
(row[0], row[1])
|
||||
for row in conn.execute(_SAME_NAME_ROWS, (f"%{escape_like(name)}",))
|
||||
]
|
||||
|
||||
|
||||
def dependency_fingerprint(
|
||||
conn: sqlite3.Connection, image_path: str, aliases=()
|
||||
) -> str:
|
||||
"""SHA-256 of what Apply's compatibility depends on for ``image_path``.
|
||||
|
||||
The exact legacy rows of the file under ``image_path`` and every alias
|
||||
spelling of it, with their positive hashes, and each row's pair links
|
||||
with the negative hash, read as the ingester reads them. Unrelated prompt
|
||||
changes (tags, titles) do not move it.
|
||||
"""
|
||||
rows = []
|
||||
for image_id, _spelled, positive_hash in dependent_images(
|
||||
conn, (image_path, *aliases)
|
||||
):
|
||||
links = conn.execute(_DEPENDENT_LINKS, (image_id,)).fetchall()
|
||||
rows.append([image_id, positive_hash, [list(link) for link in links]])
|
||||
text = canonical_evidence({"v": 1, "images": rows})
|
||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
class ReviewOperations:
|
||||
"""PromptDatabase mixin: insert and read Needs review items."""
|
||||
|
||||
def record_review_item(
|
||||
self,
|
||||
*,
|
||||
locator: str,
|
||||
spelled_path: str,
|
||||
sha256: str,
|
||||
size: int,
|
||||
mtime_ns: int,
|
||||
epoch: str,
|
||||
resolver_version: int,
|
||||
evidence: dict,
|
||||
candidate_count: int,
|
||||
limited: bool,
|
||||
) -> bool:
|
||||
"""Store one ambiguous result for these exact bytes; True when new.
|
||||
|
||||
The byte record is created (or its verified fingerprint refreshed)
|
||||
like an observe. ``spelled_path`` is the file's own spelling (it must
|
||||
fold to ``locator``); targets are formatted from it. Raises
|
||||
``LibraryCommandError("storage-changed")`` when the storage epoch is
|
||||
no longer ``epoch`` (a restore since the capture) and ``ValueError``
|
||||
for invalid facts or evidence.
|
||||
"""
|
||||
_validate_target(locator, sha256, size, mtime_ns)
|
||||
_validate_spelling(spelled_path, locator)
|
||||
if type(resolver_version) is not int or resolver_version < 1:
|
||||
raise ValueError("resolver_version must be a positive integer")
|
||||
if (
|
||||
type(candidate_count) is not int
|
||||
or not 0 <= candidate_count <= MAX_CANDIDATES
|
||||
):
|
||||
raise ValueError(f"candidate_count must be between 0 and {MAX_CANDIDATES}")
|
||||
if not isinstance(limited, bool) or (limited and candidate_count):
|
||||
raise ValueError("A limited item carries no candidates")
|
||||
text = canonical_evidence(evidence)
|
||||
evidence_hash = evidence_digest(text)
|
||||
with self._pair_transaction() as conn:
|
||||
self._library_epoch(conn, epoch)
|
||||
now = utc_now_iso()
|
||||
file_id = self._annotation_owner(conn, locator, now)
|
||||
byte_id, _revision, created, moved = self._annotation_byte(
|
||||
conn, file_id, sha256, size, mtime_ns, now
|
||||
)
|
||||
if created or moved:
|
||||
conn.execute(_BUMP_IMAGE_INDEX)
|
||||
cursor = conn.execute(
|
||||
_INSERT_ITEM,
|
||||
(
|
||||
_new_id(),
|
||||
byte_id,
|
||||
spelled_path,
|
||||
resolver_version,
|
||||
evidence_hash,
|
||||
text,
|
||||
candidate_count,
|
||||
int(limited),
|
||||
now,
|
||||
now,
|
||||
),
|
||||
)
|
||||
inserted = cursor.rowcount > 0
|
||||
refreshed = _refresh_spelling(conn, locator, spelled_path)
|
||||
if inserted or refreshed:
|
||||
conn.execute(_BUMP_REVIEW_INDEX)
|
||||
if inserted or refreshed:
|
||||
notify_review_change(self.model.db_path)
|
||||
return inserted
|
||||
|
||||
def refresh_review_spelling(self, locator: str, spelled_path: str) -> bool:
|
||||
"""Record ``spelled_path`` as the spelling of every item at ``locator``.
|
||||
|
||||
The rescan's skip calls it for a file it does not copy again: a
|
||||
case-only rename keeps size and ``mtime_ns``, and a target formatted
|
||||
from the old spelling would be refused. True when a spelling changed
|
||||
(the review index revision moves; item revisions do not, so an open
|
||||
decision stays valid).
|
||||
"""
|
||||
_validate_locator(locator)
|
||||
_validate_spelling(spelled_path, locator)
|
||||
with self._pair_transaction() as conn:
|
||||
refreshed = _refresh_spelling(conn, locator, spelled_path)
|
||||
if refreshed:
|
||||
conn.execute(_BUMP_REVIEW_INDEX)
|
||||
if refreshed:
|
||||
notify_review_change(self.model.db_path)
|
||||
return refreshed
|
||||
|
||||
def review_index_revision(self) -> int:
|
||||
"""The queue's revision (moved by every queue change)."""
|
||||
row = self.model.get_connection().execute(_REVIEW_INDEX).fetchone()
|
||||
if row is None:
|
||||
raise sqlite3.DatabaseError("Missing review queue metadata")
|
||||
return row[0]
|
||||
|
||||
def review_queue_version(self) -> dict:
|
||||
"""``{library_epoch, review_revision}`` of the queue, from one snapshot.
|
||||
|
||||
The pair a list read answers: equal pairs mean the same queue (a
|
||||
restore moves the epoch, every queue change moves the revision).
|
||||
"""
|
||||
with self._annotation_read() as conn:
|
||||
epoch = conn.execute(_EPOCH).fetchone()
|
||||
revision = conn.execute(_REVIEW_INDEX).fetchone()
|
||||
if epoch is None or revision is None:
|
||||
raise sqlite3.DatabaseError("Missing review queue metadata")
|
||||
return {"library_epoch": epoch[0], "review_revision": revision[0]}
|
||||
|
||||
def review_items_for(self, locator: str) -> list:
|
||||
"""Every item of the file at ``locator``, oldest first, evidence parsed."""
|
||||
_validate_locator(locator)
|
||||
return list(self._iter_review_items(locator))
|
||||
|
||||
def _iter_review_items(self, locator: str) -> Iterator[dict]:
|
||||
conn = self.model.get_connection()
|
||||
for row in conn.execute(_ITEMS_OF_LOCATOR, (locator,)):
|
||||
item = dict(zip(_ITEM_COLUMNS, tuple(row)))
|
||||
item["evidence"] = json.loads(item["evidence"])
|
||||
item["limited"] = bool(item["limited"])
|
||||
yield item
|
||||
|
||||
# -- queue reads (#249 E3) -------------------------------------------------
|
||||
|
||||
def review_fingerprint_queued(
|
||||
self, locator: str, size: int, mtime_ns: int, resolver_version: int
|
||||
) -> bool:
|
||||
"""True when an item cites a byte record verified at this exact stat.
|
||||
|
||||
A rescan uses it to skip the copy and hash of a file it already
|
||||
queued: the same ``(size, mtime_ns)`` the bytes were verified with.
|
||||
"""
|
||||
row = (
|
||||
self.model.get_connection()
|
||||
.execute(_QUEUED, (locator, size, mtime_ns, resolver_version))
|
||||
.fetchone()
|
||||
)
|
||||
return row is not None
|
||||
|
||||
def review_pending_at(
|
||||
self, locator: str, size: int, mtime_ns: int, resolver_version: int
|
||||
) -> bool:
|
||||
"""True when a *pending* item cites bytes verified at this exact stat.
|
||||
|
||||
``review_fingerprint_queued`` also matches decided items; a file whose
|
||||
only item is decided is not in the queue.
|
||||
"""
|
||||
row = (
|
||||
self.model.get_connection()
|
||||
.execute(_PENDING_AT, (locator, size, mtime_ns, resolver_version))
|
||||
.fetchone()
|
||||
)
|
||||
return row is not None
|
||||
|
||||
def review_file_linked(self, spellings) -> bool:
|
||||
"""True when a saved row links the file under any of ``spellings``."""
|
||||
with self._annotation_read() as conn:
|
||||
return bool(dependent_images(conn, spellings))
|
||||
|
||||
def settle_linked_review_items(self, files) -> int:
|
||||
"""Remove the pending, undecided items of files that are now linked.
|
||||
|
||||
``files`` holds ``(locator, spellings)`` per file. An item leaves the
|
||||
queue only when it is still ``pending`` at revision 0 with no decision
|
||||
row and a saved row links its file under one of ``spellings``; a
|
||||
decided item is never touched. Files without a pending item cost one
|
||||
indexed read and open no write transaction. Returns how many items
|
||||
were removed (the review index revision moves when any was).
|
||||
"""
|
||||
files = [(locator, tuple(spellings)) for locator, spellings in files]
|
||||
for locator, _spellings in files:
|
||||
_validate_locator(locator)
|
||||
conn = self.model.get_connection()
|
||||
candidates = [
|
||||
(locator, spellings)
|
||||
for locator, spellings in files
|
||||
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
|
||||
]
|
||||
if not candidates:
|
||||
return 0
|
||||
removed = 0
|
||||
with self._pair_transaction() as conn:
|
||||
for locator, spellings in candidates:
|
||||
if not dependent_images(conn, spellings):
|
||||
continue
|
||||
for (item_id,) in conn.execute(
|
||||
_PENDING_OF_LOCATOR, (locator,)
|
||||
).fetchall():
|
||||
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
|
||||
if removed:
|
||||
conn.execute(_BUMP_REVIEW_INDEX)
|
||||
if removed:
|
||||
notify_review_change(self.model.db_path)
|
||||
return removed
|
||||
|
||||
def settle_untraced_review_items(self, locators) -> int:
|
||||
"""Remove the pending, undecided items that offer nothing to choose.
|
||||
|
||||
``locators`` are files the caller found ambiguous without a choice
|
||||
(fewer than two candidates carry a prompt). Of their items, one leaves
|
||||
the queue only when its own stored evidence offers no choice either
|
||||
(``evidence_offers_choice``) and it is still ``pending`` at revision 0
|
||||
with no decision row, the same safe delete as
|
||||
``settle_linked_review_items``. Files without a pending item cost one
|
||||
indexed read and open no write transaction. Returns how many items
|
||||
were removed (the review index revision moves when any was).
|
||||
"""
|
||||
locators = list(locators)
|
||||
for locator in locators:
|
||||
_validate_locator(locator)
|
||||
conn = self.model.get_connection()
|
||||
pending = [
|
||||
locator
|
||||
for locator in locators
|
||||
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
|
||||
]
|
||||
if not pending:
|
||||
return 0
|
||||
removed = 0
|
||||
with self._pair_transaction() as conn:
|
||||
for locator in pending:
|
||||
rows = conn.execute(_PENDING_EVIDENCE_OF_LOCATOR, (locator,)).fetchall()
|
||||
for item_id, evidence in rows:
|
||||
if evidence_offers_choice(json.loads(evidence)):
|
||||
continue
|
||||
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
|
||||
if removed:
|
||||
conn.execute(_BUMP_REVIEW_INDEX)
|
||||
return removed
|
||||
|
||||
def review_queue_page(self, *, limit: int, after: tuple | None = None) -> dict:
|
||||
"""One page of pending items, oldest first, from one snapshot.
|
||||
|
||||
``after`` is the ``(created_at, id)`` keyset of the last item already
|
||||
shown. Returns ``{epoch, revision, total, items, more}``; ``total`` is
|
||||
the exact pending count and ``items`` carry their ``spelled_path`` and
|
||||
``locator`` (server-side only).
|
||||
"""
|
||||
if type(limit) is not int or limit < 1:
|
||||
raise ValueError("limit must be a positive integer")
|
||||
with self._annotation_read() as conn:
|
||||
epoch = conn.execute(_EPOCH).fetchone()
|
||||
revision = conn.execute(_REVIEW_INDEX).fetchone()
|
||||
if epoch is None or revision is None:
|
||||
raise sqlite3.DatabaseError("Missing review queue metadata")
|
||||
total = conn.execute(_PENDING_COUNT).fetchone()[0]
|
||||
if after is None:
|
||||
rows = conn.execute(_QUEUE_FIRST, (limit + 1,)).fetchall()
|
||||
else:
|
||||
rows = conn.execute(_QUEUE_AFTER, (*after, limit + 1)).fetchall()
|
||||
items = [_queue_row(row, _QUEUE_COLUMNS) for row in rows[:limit]]
|
||||
return {
|
||||
"epoch": epoch[0],
|
||||
"revision": revision[0],
|
||||
"total": total,
|
||||
"items": items,
|
||||
"more": len(rows) > limit,
|
||||
}
|
||||
|
||||
def review_item(self, item_id: str) -> dict | None:
|
||||
"""One item with its parsed evidence and the pending total, or None."""
|
||||
if not isinstance(item_id, str) or _ITEM_ID.fullmatch(item_id) is None:
|
||||
return None
|
||||
with self._annotation_read() as conn:
|
||||
epoch = conn.execute(_EPOCH).fetchone()
|
||||
if epoch is None:
|
||||
raise sqlite3.DatabaseError("Missing execution history metadata")
|
||||
row = conn.execute(_DETAIL, (item_id,)).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
item = _queue_row(row, _DETAIL_COLUMNS)
|
||||
item["evidence"] = json.loads(item["evidence"])
|
||||
item["epoch"] = epoch[0]
|
||||
item["total"] = conn.execute(_PENDING_COUNT).fetchone()[0]
|
||||
return item
|
||||
|
||||
def review_dependency_fingerprint(self, image_path: str, aliases=()) -> str:
|
||||
with self._annotation_read() as conn:
|
||||
return dependency_fingerprint(conn, image_path, aliases)
|
||||
|
||||
def review_same_name_rows(self, name: str) -> list:
|
||||
"""``(id, image_path)`` of legacy rows that may spell a file ``name``."""
|
||||
with self._annotation_read() as conn:
|
||||
return same_name_rows(conn, name)
|
||||
|
||||
def stored_prompt_titles(self, texts) -> dict:
|
||||
"""Library titles of saved prompts by text (dedup hash), when set."""
|
||||
hashes = {generate_prompt_hash(text): text for text in texts if text.strip()}
|
||||
if not hashes:
|
||||
return {}
|
||||
marks = ",".join("?" * len(hashes))
|
||||
found: dict[str, tuple] = {}
|
||||
conn = self.model.get_connection()
|
||||
for prompt_hash, title in conn.execute(
|
||||
_TITLES.format(marks=marks), list(hashes)
|
||||
):
|
||||
text = hashes[prompt_hash]
|
||||
found[text] = found.get(text, ()) + (title,)
|
||||
return found
|
||||
|
||||
|
||||
def _refresh_spelling(conn: sqlite3.Connection, locator: str, spelled: str) -> bool:
|
||||
return conn.execute(_REFRESH_SPELLING, (spelled, spelled, locator)).rowcount > 0
|
||||
|
||||
|
||||
def _queue_row(row, columns) -> dict:
|
||||
item = dict(zip(columns, tuple(row)))
|
||||
item["limited"] = bool(item["limited"])
|
||||
return item
|
||||
@@ -1,64 +0,0 @@
|
||||
"""Needs review queue change notices (#249 PR F2).
|
||||
|
||||
Every committed change to the pending queue (an item recorded by the scan or
|
||||
the monitor, a spelling refresh, a decision) calls ``notify_review_change``
|
||||
after its transaction, from whichever thread wrote it. The review event
|
||||
stream (``py/api/review_events.py``) subscribes, so an open Needs review page
|
||||
re-reads its queue when it changes instead of polling. Observers are keyed by
|
||||
the database file, like ``subscribe_history_restore``; an observer that fails
|
||||
is logged and never fails the write.
|
||||
"""
|
||||
|
||||
import os
|
||||
import threading
|
||||
from typing import Callable
|
||||
|
||||
try:
|
||||
from ..utils.logging_config import get_logger
|
||||
except ImportError:
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
_logger = get_logger("prompt_manager.database.review_changes")
|
||||
_lock = threading.Lock()
|
||||
_observers: dict[str, dict[object, Callable[[], None]]] = {}
|
||||
|
||||
|
||||
def _identity(db_path: str) -> str:
|
||||
return os.path.normcase(os.path.realpath(db_path))
|
||||
|
||||
|
||||
def subscribe_review_changes(
|
||||
db_path: str, callback: Callable[[], None]
|
||||
) -> Callable[[], None]:
|
||||
"""Call ``callback()`` after each committed queue change; return the unsubscribe."""
|
||||
identity = _identity(db_path)
|
||||
token = object()
|
||||
with _lock:
|
||||
_observers.setdefault(identity, {})[token] = callback
|
||||
|
||||
def unsubscribe() -> None:
|
||||
with _lock:
|
||||
observers = _observers.get(identity)
|
||||
if observers is not None:
|
||||
observers.pop(token, None)
|
||||
if not observers:
|
||||
_observers.pop(identity, None)
|
||||
|
||||
return unsubscribe
|
||||
|
||||
|
||||
def observer_count(db_path: str) -> int:
|
||||
"""How many observers watch ``db_path`` (tests and diagnostics)."""
|
||||
with _lock:
|
||||
return len(_observers.get(_identity(db_path), {}))
|
||||
|
||||
|
||||
def notify_review_change(db_path: str) -> None:
|
||||
"""Tell every observer of ``db_path``; call it after the commit."""
|
||||
with _lock:
|
||||
callbacks = tuple(_observers.get(_identity(db_path), {}).values())
|
||||
for callback in callbacks:
|
||||
try:
|
||||
callback()
|
||||
except Exception:
|
||||
_logger.warning("Needs review change observer failed", exc_info=True)
|
||||
@@ -1,291 +0,0 @@
|
||||
"""Needs review decisions (#249 E3): explicit, auditable and compatible-only.
|
||||
|
||||
A decision is one row of ``media_review_decisions`` for one item revision.
|
||||
``selected`` links the chosen candidate's pair to the file through the public
|
||||
ingester's compatible-only rules (``_ingest_resolved_pair_in``) in the same
|
||||
``_pair_transaction`` as the decision row, the item's new state and revision
|
||||
and the review index revision; any conflict or later failure rolls all of it
|
||||
back. ``unresolved`` records the decision and links nothing. No decision
|
||||
counts a run, touches usage or calls an execution writer.
|
||||
|
||||
Callers verify the file first (the capture must be the item's bytes and the
|
||||
current file must still hash to them); nothing here reads the filesystem.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sqlite3
|
||||
|
||||
from .annotation_schema import BYTES_TABLE
|
||||
from .annotations import _new_id
|
||||
from .library_commands import LibraryCommandError
|
||||
from .models import utc_now_iso
|
||||
from .review import (
|
||||
_BUMP_REVIEW_INDEX,
|
||||
_DETAIL,
|
||||
_DETAIL_COLUMNS,
|
||||
_PENDING_COUNT,
|
||||
_queue_row,
|
||||
dependency_fingerprint,
|
||||
dependent_images,
|
||||
same_name_rows,
|
||||
)
|
||||
from .review_changes import notify_review_change
|
||||
from .review_schema import DECISIONS_TABLE, ITEMS_TABLE, canonical_evidence
|
||||
|
||||
DECISION_KIND = "user-selection"
|
||||
_INSERT_DECISION = (
|
||||
f"INSERT INTO {DECISIONS_TABLE}(id,item_id,decision_revision," # nosec B608
|
||||
"created_at,outcome,candidate_hash,evidence,pair_id,image_id) "
|
||||
"VALUES(?,?,?,?,?,?,?,?,?)"
|
||||
)
|
||||
_ADVANCE_ITEM = (
|
||||
f"UPDATE {ITEMS_TABLE} SET state=?,revision=revision+1," # nosec B608
|
||||
"updated_at=? WHERE id=? AND revision=?"
|
||||
)
|
||||
_ITEM_BYTES = (
|
||||
f"SELECT b.sha256 FROM {ITEMS_TABLE} i " # nosec B608
|
||||
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id WHERE i.id=?"
|
||||
)
|
||||
|
||||
|
||||
class ReviewCommandError(LibraryCommandError):
|
||||
"""A decision lost a race or would rewrite an association.
|
||||
|
||||
``code`` is ``review-conflict`` (the item was decided or changed since it
|
||||
was read), ``association-conflict`` (the choice is not compatible with
|
||||
the file's saved links, or those links changed), ``file-changed`` (the
|
||||
item is not about these bytes) or ``review-missing``. ``item`` is the
|
||||
current item read inside the transaction, ``total`` the pending count and
|
||||
``fingerprint`` the current dependency fingerprint, when known;
|
||||
``changed`` marks an association conflict caused by links that changed
|
||||
since the reviewer read them (rather than an incompatible choice).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, code, epoch, *, item=None, total=None, fingerprint=None, changed=False
|
||||
):
|
||||
super().__init__(code, epoch)
|
||||
self.item = item
|
||||
self.total = total
|
||||
self.fingerprint = fingerprint
|
||||
self.changed = changed
|
||||
# The public view of ``item``, set by the API when it formats one.
|
||||
self.view = None
|
||||
|
||||
|
||||
class _Rollback(Exception):
|
||||
"""Raised inside the transaction so the context manager rolls back."""
|
||||
|
||||
def __init__(self, error: ReviewCommandError):
|
||||
super().__init__(error.code)
|
||||
self.error = error
|
||||
|
||||
|
||||
class ReviewDecisionOperations:
|
||||
"""PromptDatabase mixin: record one decision on one Needs review item."""
|
||||
|
||||
def decide_review_item(
|
||||
self,
|
||||
*,
|
||||
item_id: str,
|
||||
expected_epoch: str,
|
||||
expected_revision: int,
|
||||
sha256: str,
|
||||
outcome: str,
|
||||
candidate: dict | None = None,
|
||||
image_path: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
expected_fingerprint: str | None = None,
|
||||
aliases: tuple = (),
|
||||
known_rows: frozenset | None = None,
|
||||
) -> dict:
|
||||
"""Record the decision; return ``{item, total, pair}``.
|
||||
|
||||
``selected`` needs the evidence ``candidate`` (with its ``hash``), the
|
||||
``image_path`` the scan and the monitor link under, the captured
|
||||
``metadata`` and the ``expected_fingerprint`` the reviewer saw.
|
||||
``aliases`` are the other spellings legacy rows of the same physical
|
||||
file use (a symlinked, junctioned or mapped root) and ``known_rows``
|
||||
the ids of every same-name row the caller classified; a same-name row
|
||||
it did not see is a changed association.
|
||||
Raises ``LibraryCommandError("storage-changed")`` after a restore and
|
||||
:class:`ReviewCommandError` for every refused decision, with zero
|
||||
writes either way.
|
||||
"""
|
||||
if outcome not in ("selected", "unresolved"):
|
||||
raise ValueError("outcome must be selected or unresolved")
|
||||
if outcome == "selected" and (
|
||||
not isinstance(candidate, dict) or not image_path
|
||||
):
|
||||
raise ValueError("A selected decision needs a candidate and a file")
|
||||
try:
|
||||
with self._pair_transaction() as conn:
|
||||
decided = self._decide_in(
|
||||
conn,
|
||||
item_id=item_id,
|
||||
expected_epoch=expected_epoch,
|
||||
expected_revision=expected_revision,
|
||||
sha256=sha256,
|
||||
outcome=outcome,
|
||||
candidate=candidate,
|
||||
image_path=image_path,
|
||||
metadata=metadata or {},
|
||||
expected_fingerprint=expected_fingerprint,
|
||||
aliases=tuple(aliases),
|
||||
known_rows=known_rows,
|
||||
)
|
||||
except _Rollback as rollback:
|
||||
raise rollback.error from None
|
||||
# After the commit: open Needs review pages re-read their queue.
|
||||
notify_review_change(self.model.db_path)
|
||||
return decided
|
||||
|
||||
def _decide_in(self, conn: sqlite3.Connection, **command) -> dict:
|
||||
epoch = self._library_epoch(conn, command["expected_epoch"])
|
||||
item_id = command["item_id"]
|
||||
current = _current_item(conn, item_id)
|
||||
if current is None:
|
||||
raise _Rollback(ReviewCommandError("review-missing", epoch))
|
||||
total = conn.execute(_PENDING_COUNT).fetchone()[0]
|
||||
if (
|
||||
current["state"] != "pending"
|
||||
or current["revision"] != command["expected_revision"]
|
||||
):
|
||||
raise _Rollback(
|
||||
ReviewCommandError("review-conflict", epoch, item=current, total=total)
|
||||
)
|
||||
if conn.execute(_ITEM_BYTES, (item_id,)).fetchone()[0] != command["sha256"]:
|
||||
raise _Rollback(ReviewCommandError("file-changed", epoch))
|
||||
decision_id = _new_id()
|
||||
pair = None
|
||||
candidate = command["candidate"] if command["outcome"] == "selected" else None
|
||||
if candidate is not None:
|
||||
pair = self._apply_candidate(
|
||||
conn, epoch, current, decision_id, total, **command
|
||||
)
|
||||
evidence = {
|
||||
"kind": DECISION_KIND,
|
||||
"decision_id": decision_id,
|
||||
"outcome": command["outcome"],
|
||||
"reviewed_digest": command["sha256"],
|
||||
"item_evidence_hash": current["evidence_hash"],
|
||||
"resolver_version": current["resolver_version"],
|
||||
"candidate": candidate,
|
||||
"dependency_fingerprint": command["expected_fingerprint"],
|
||||
"pair": pair,
|
||||
}
|
||||
now = utc_now_iso()
|
||||
self._insert_review_decision(
|
||||
conn,
|
||||
(
|
||||
decision_id,
|
||||
item_id,
|
||||
current["revision"] + 1,
|
||||
now,
|
||||
command["outcome"],
|
||||
candidate["hash"] if candidate is not None else None,
|
||||
canonical_evidence(evidence),
|
||||
pair["id"] if pair else None,
|
||||
pair["image_id"] if pair else None,
|
||||
),
|
||||
)
|
||||
conn.execute(
|
||||
_ADVANCE_ITEM, (command["outcome"], now, item_id, current["revision"])
|
||||
)
|
||||
conn.execute(_BUMP_REVIEW_INDEX)
|
||||
return {
|
||||
"item": _current_item(conn, item_id),
|
||||
"total": conn.execute(_PENDING_COUNT).fetchone()[0],
|
||||
"pair": pair,
|
||||
"decision_id": decision_id,
|
||||
}
|
||||
|
||||
def _apply_candidate(
|
||||
self, conn, epoch, current, decision_id, total, **command
|
||||
) -> dict:
|
||||
"""Link the candidate's pair, compatible-only; raise on any conflict."""
|
||||
# Imported here: pair_ingestion imports operations, which composes this.
|
||||
from .pair_ingestion import _ingest_resolved_pair_in
|
||||
|
||||
image_path = command["image_path"]
|
||||
aliases = command["aliases"]
|
||||
fingerprint = dependency_fingerprint(conn, image_path, aliases)
|
||||
known = command["known_rows"]
|
||||
unseen = known is not None and any(
|
||||
row_id not in known
|
||||
for row_id, _path in same_name_rows(conn, os.path.basename(image_path))
|
||||
)
|
||||
if unseen or fingerprint != command["expected_fingerprint"]:
|
||||
raise _Rollback(
|
||||
ReviewCommandError(
|
||||
"association-conflict",
|
||||
epoch,
|
||||
item=current,
|
||||
total=total,
|
||||
fingerprint=fingerprint,
|
||||
changed=True,
|
||||
)
|
||||
)
|
||||
rows = dependent_images(conn, (image_path, *aliases))
|
||||
if len(rows) > 1:
|
||||
# One file linked under several spellings: no single row to enrich.
|
||||
raise _Rollback(
|
||||
ReviewCommandError(
|
||||
"association-conflict",
|
||||
epoch,
|
||||
item=current,
|
||||
total=total,
|
||||
fingerprint=fingerprint,
|
||||
)
|
||||
)
|
||||
if rows:
|
||||
# Enrich the one row of this file under the spelling it was
|
||||
# linked with, so the ingester's compatibility rules see it.
|
||||
image_path = rows[0][1]
|
||||
candidate = command["candidate"]
|
||||
positive, negative = candidate["positive"], candidate["negative"]
|
||||
provenance = {
|
||||
"kind": DECISION_KIND,
|
||||
"decision_id": decision_id,
|
||||
"reviewed_digest": command["sha256"],
|
||||
"candidate_hash": candidate["hash"],
|
||||
"resolver_version": current["resolver_version"],
|
||||
"positive": positive,
|
||||
"negative": negative,
|
||||
}
|
||||
result = _ingest_resolved_pair_in(
|
||||
conn,
|
||||
self,
|
||||
image_path,
|
||||
command["metadata"],
|
||||
{"state": "resolved", "positive": positive, "negative": negative},
|
||||
provenance=provenance,
|
||||
)
|
||||
if result["status"] == "conflict":
|
||||
raise _Rollback(
|
||||
ReviewCommandError(
|
||||
"association-conflict",
|
||||
epoch,
|
||||
item=current,
|
||||
total=total,
|
||||
fingerprint=fingerprint,
|
||||
)
|
||||
)
|
||||
return {
|
||||
"id": result["pair_id"],
|
||||
"image_id": result["image_id"],
|
||||
"status": result["status"],
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _insert_review_decision(conn: sqlite3.Connection, values: tuple) -> None:
|
||||
conn.execute(_INSERT_DECISION, values)
|
||||
|
||||
|
||||
def _current_item(conn: sqlite3.Connection, item_id: str) -> dict | None:
|
||||
row = conn.execute(_DETAIL, (item_id,)).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
item = _queue_row(row, _DETAIL_COLUMNS)
|
||||
item.pop("evidence")
|
||||
return item
|
||||
@@ -1,507 +0,0 @@
|
||||
"""Needs review schema group (#249): capture-bound ambiguous items and decisions.
|
||||
|
||||
The group installs like ``image_annotations``: optional in old backups,
|
||||
required once a database is current, and valid only beside a valid image
|
||||
annotation group, because every item cites one annotation byte record. It owns
|
||||
|
||||
- ``media_review_metadata``: one row, the group's schema version and the
|
||||
review index revision (moved by every queue change);
|
||||
- ``media_review_items``: one ambiguous resolver result for one byte record,
|
||||
keyed by ``(byte_id, resolver_version, evidence_hash)``, with the file's
|
||||
spelling when it was recorded (the locator is case-folded on Windows, a
|
||||
target is not), its canonical evidence, candidate count, ``limited`` flag,
|
||||
state and revision;
|
||||
- ``media_review_decisions``: the auditable decision history of an item, one
|
||||
row per decision revision, with optional navigation to the pair and image
|
||||
the decision linked (nulled, never cascaded, when those are deleted).
|
||||
|
||||
Evidence is canonical JSON (:func:`canonical_evidence`): one object, sorted
|
||||
keys, no whitespace, ASCII only (so characters, bytes and the 8 MiB bound
|
||||
agree and lone surrogates survive), no NaN or infinity. ``evidence_hash`` is
|
||||
the SHA-256 of the item's evidence text.
|
||||
|
||||
Installation never backfills anything and never touches the filesystem.
|
||||
Validation reads only: catalog objects compared after whitespace compaction,
|
||||
columns and foreign keys through the PRAGMAs, persisted values in single
|
||||
bounded SQL passes, then one streamed pass that re-canonicalizes each evidence
|
||||
document (one row in memory at a time).
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
from typing import Any, Optional
|
||||
|
||||
from .annotation_schema import BYTES_TABLE
|
||||
from .annotation_schema import TABLES as ANNOTATION_TABLES
|
||||
from .annotation_schema import _catalog as _annotation_catalog
|
||||
from .annotation_schema import annotation_schema_error
|
||||
|
||||
try:
|
||||
from ..utils.json_strict import dumps_strict
|
||||
except ImportError:
|
||||
from utils.json_strict import dumps_strict
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
METADATA_TABLE = "media_review_metadata"
|
||||
ITEMS_TABLE = "media_review_items"
|
||||
DECISIONS_TABLE = "media_review_decisions"
|
||||
TABLES = (METADATA_TABLE, ITEMS_TABLE, DECISIONS_TABLE)
|
||||
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
|
||||
MAX_CANDIDATES = 16
|
||||
# The file's own spelling when it was recorded (bytes of UTF-8 text). Locators
|
||||
# are case-folded on Windows, so a target is formatted from this instead.
|
||||
MAX_SPELLED_PATH_BYTES = 16384
|
||||
ITEM_STATES = ("pending", "selected", "unresolved")
|
||||
DECISION_OUTCOMES = ("selected", "unresolved")
|
||||
_TIMESTAMP_BYTES = 64
|
||||
_HEX64 = "length({c})=64 AND {c} NOT GLOB '*[^0-9a-f]*'"
|
||||
_EVIDENCE_CHECK = f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES})"
|
||||
|
||||
_TABLE_DDL = {
|
||||
METADATA_TABLE: (
|
||||
f"CREATE TABLE {METADATA_TABLE} (singleton_id INTEGER PRIMARY KEY "
|
||||
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
|
||||
f"CHECK(schema_version={SCHEMA_VERSION}), "
|
||||
"review_index_revision INTEGER NOT NULL DEFAULT 0 "
|
||||
"CHECK(review_index_revision>=0))"
|
||||
),
|
||||
ITEMS_TABLE: (
|
||||
f"CREATE TABLE {ITEMS_TABLE} (id TEXT PRIMARY KEY CHECK(length(id)=36), "
|
||||
f"byte_id TEXT NOT NULL REFERENCES {BYTES_TABLE}(id) ON DELETE RESTRICT, "
|
||||
"spelled_path TEXT NOT NULL CHECK(length(CAST(spelled_path AS BLOB)) "
|
||||
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}), "
|
||||
"resolver_version INTEGER NOT NULL, evidence_hash TEXT NOT NULL "
|
||||
f"CHECK({_HEX64.format(c='evidence_hash')}), "
|
||||
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
|
||||
"candidate_count INTEGER NOT NULL "
|
||||
f"CHECK(candidate_count BETWEEN 0 AND {MAX_CANDIDATES}), "
|
||||
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), "
|
||||
"state TEXT NOT NULL CHECK(state IN ('pending','selected','unresolved')), "
|
||||
"revision INTEGER NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, "
|
||||
"updated_at TEXT NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
|
||||
),
|
||||
DECISIONS_TABLE: (
|
||||
f"CREATE TABLE {DECISIONS_TABLE} (id TEXT PRIMARY KEY "
|
||||
"CHECK(length(id)=36), item_id TEXT NOT NULL "
|
||||
f"REFERENCES {ITEMS_TABLE}(id) ON DELETE RESTRICT, "
|
||||
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
|
||||
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
|
||||
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
|
||||
f"({_HEX64.format(c='candidate_hash')})), "
|
||||
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
|
||||
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
|
||||
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
|
||||
"UNIQUE(item_id, decision_revision), "
|
||||
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
|
||||
),
|
||||
}
|
||||
_INDEX_DDL = {
|
||||
# The queue reads pending items oldest first with a stable tiebreak.
|
||||
"idx_media_review_items_queue": (
|
||||
"CREATE INDEX idx_media_review_items_queue ON "
|
||||
f"{ITEMS_TABLE}(state, created_at, id)"
|
||||
),
|
||||
# Deleting a pair or an image nulls decision navigation by index, not scan.
|
||||
"idx_media_review_decisions_pair": (
|
||||
"CREATE INDEX idx_media_review_decisions_pair ON " f"{DECISIONS_TABLE}(pair_id)"
|
||||
),
|
||||
"idx_media_review_decisions_image": (
|
||||
"CREATE INDEX idx_media_review_decisions_image ON "
|
||||
f"{DECISIONS_TABLE}(image_id)"
|
||||
),
|
||||
}
|
||||
# ``sqlite_autoindex_<table>_<n>``: the TEXT primary key and the UNIQUE key.
|
||||
_AUTOINDEXES = {ITEMS_TABLE: 2, DECISIONS_TABLE: 2}
|
||||
# (name, declared type, not null, default, primary key position)
|
||||
_COLUMNS = {
|
||||
METADATA_TABLE: (
|
||||
("singleton_id", "INTEGER", 0, None, 1),
|
||||
("schema_version", "INTEGER", 1, None, 0),
|
||||
("review_index_revision", "INTEGER", 1, "0", 0),
|
||||
),
|
||||
ITEMS_TABLE: (
|
||||
("id", "TEXT", 0, None, 1),
|
||||
("byte_id", "TEXT", 1, None, 0),
|
||||
("spelled_path", "TEXT", 1, None, 0),
|
||||
("resolver_version", "INTEGER", 1, None, 0),
|
||||
("evidence_hash", "TEXT", 1, None, 0),
|
||||
("evidence", "TEXT", 1, None, 0),
|
||||
("candidate_count", "INTEGER", 1, None, 0),
|
||||
("limited", "INTEGER", 1, None, 0),
|
||||
("state", "TEXT", 1, None, 0),
|
||||
("revision", "INTEGER", 1, None, 0),
|
||||
("created_at", "TEXT", 1, None, 0),
|
||||
("updated_at", "TEXT", 1, None, 0),
|
||||
),
|
||||
DECISIONS_TABLE: (
|
||||
("id", "TEXT", 0, None, 1),
|
||||
("item_id", "TEXT", 1, None, 0),
|
||||
("decision_revision", "INTEGER", 1, None, 0),
|
||||
("created_at", "TEXT", 1, None, 0),
|
||||
("outcome", "TEXT", 1, None, 0),
|
||||
("candidate_hash", "TEXT", 0, None, 0),
|
||||
("evidence", "TEXT", 1, None, 0),
|
||||
("pair_id", "INTEGER", 0, None, 0),
|
||||
("image_id", "INTEGER", 0, None, 0),
|
||||
),
|
||||
}
|
||||
# (column, parent table, parent column, on delete)
|
||||
_FOREIGN_KEYS = {
|
||||
METADATA_TABLE: set(),
|
||||
ITEMS_TABLE: {("byte_id", BYTES_TABLE, "id", "RESTRICT")},
|
||||
DECISIONS_TABLE: {
|
||||
("item_id", ITEMS_TABLE, "id", "RESTRICT"),
|
||||
("pair_id", "prompt_pairs", "id", "SET NULL"),
|
||||
("image_id", "generated_images", "id", "SET NULL"),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class _NonCanonical(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def _reject_constant(name: str) -> Any:
|
||||
raise _NonCanonical(f"JSON constant {name} is not allowed")
|
||||
|
||||
|
||||
# Built once: validation re-encodes every stored document at startup.
|
||||
_ENCODER = json.JSONEncoder(
|
||||
sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False
|
||||
)
|
||||
_DECODER = json.JSONDecoder(parse_constant=_reject_constant)
|
||||
|
||||
|
||||
def canonical_evidence(value: Any) -> str:
|
||||
"""Serialize one evidence object canonically; ``ValueError`` when invalid.
|
||||
|
||||
The text is a JSON object with sorted keys, no whitespace, ASCII only and
|
||||
no NaN/infinity, at most ``MAX_EVIDENCE_BYTES`` bytes. A non-finite number
|
||||
(ComfyUI stamps ``NaN`` into graphs) is written as ``null``, the store's
|
||||
policy everywhere (``utils.json_strict.dumps_strict``).
|
||||
"""
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("Review evidence must be a JSON object")
|
||||
try:
|
||||
try:
|
||||
text = _ENCODER.encode(value)
|
||||
except ValueError as error:
|
||||
if "Out of range float" not in str(error):
|
||||
raise
|
||||
text = dumps_strict(
|
||||
value, sort_keys=True, separators=(",", ":"), ensure_ascii=True
|
||||
)
|
||||
except (TypeError, RecursionError) as error:
|
||||
raise ValueError(f"Review evidence is not serializable: {error}") from None
|
||||
if len(text) > MAX_EVIDENCE_BYTES:
|
||||
raise ValueError("Review evidence is larger than 8 MiB")
|
||||
return text
|
||||
|
||||
|
||||
def choosable(candidate: Any) -> bool:
|
||||
"""True when a candidate offers a prompt to choose: it is ``resolved`` and
|
||||
its positive side carries non-blank text. The page disables any other
|
||||
candidate, so it is never something to choose.
|
||||
"""
|
||||
if not isinstance(candidate, dict) or candidate.get("state") != "resolved":
|
||||
return False
|
||||
positive = candidate.get("positive")
|
||||
if not isinstance(positive, dict):
|
||||
return False
|
||||
text = positive.get("text")
|
||||
return isinstance(text, str) and bool(text.strip())
|
||||
|
||||
|
||||
# An item is worth a decision only when at least this many candidates offer
|
||||
# a prompt; with fewer there is nothing to choose between (maintainer test of
|
||||
# #284/#288: 39 items whose candidates were all "unresolved").
|
||||
MIN_CHOOSABLE = 2
|
||||
|
||||
|
||||
def evidence_offers_choice(evidence: Any) -> bool:
|
||||
"""False when stored evidence provably offers nothing to choose.
|
||||
|
||||
Unlimited evidence must list ``MIN_CHOOSABLE`` choosable candidates. A
|
||||
limited item stores none, only ``candidate_total``; it may offer a choice
|
||||
(more than ``MAX_CANDIDATES`` prompts) unless that total is too small.
|
||||
"""
|
||||
if not isinstance(evidence, dict):
|
||||
return False
|
||||
if evidence.get("limited"):
|
||||
total = evidence.get("candidate_total")
|
||||
return type(total) is int and total >= MIN_CHOOSABLE
|
||||
candidates = evidence.get("candidates")
|
||||
if not isinstance(candidates, list):
|
||||
return False
|
||||
return sum(1 for c in candidates if choosable(c)) >= MIN_CHOOSABLE
|
||||
|
||||
|
||||
def evidence_digest(text: str) -> str:
|
||||
"""SHA-256 of canonical evidence text (ASCII, hence UTF-8 identical)."""
|
||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _is_canonical(text: Any) -> bool:
|
||||
if not isinstance(text, str):
|
||||
return False
|
||||
try:
|
||||
value = _DECODER.decode(text)
|
||||
return canonical_evidence(value) == text
|
||||
except (ValueError, RecursionError):
|
||||
return False
|
||||
|
||||
|
||||
def _compact(sql: str) -> str:
|
||||
return re.sub(r"\s+", "", sql).lower()
|
||||
|
||||
|
||||
def _uuid(column: str) -> str:
|
||||
"""SQL: *column* is a lowercase hyphenated 36-character UUID string."""
|
||||
return (
|
||||
f"typeof({column}) = 'text' AND length({column}) = 36 "
|
||||
f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' "
|
||||
f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' "
|
||||
f"AND length(replace({column}, '-', '')) = 32 "
|
||||
f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'"
|
||||
)
|
||||
|
||||
|
||||
def _timestamp(column: str) -> str:
|
||||
return (
|
||||
f"typeof({column}) = 'text' "
|
||||
f"AND length(CAST({column} AS BLOB)) <= {_TIMESTAMP_BYTES}"
|
||||
)
|
||||
|
||||
|
||||
def _non_negative(column: str) -> str:
|
||||
return f"typeof({column}) = 'integer' AND {column} >= 0"
|
||||
|
||||
|
||||
def _hex64(column: str) -> str:
|
||||
return f"typeof({column}) = 'text' AND " + _HEX64.format(c=column)
|
||||
|
||||
|
||||
def _evidence(column: str) -> str:
|
||||
# Type and size only: the streamed pass below proves the text canonical
|
||||
# (which implies printable ASCII), so a pattern scan here would read every
|
||||
# document a second time.
|
||||
return (
|
||||
f"typeof({column}) = 'text' "
|
||||
f"AND length(CAST({column} AS BLOB)) BETWEEN 2 AND {MAX_EVIDENCE_BYTES}"
|
||||
)
|
||||
|
||||
|
||||
def _nullable_integer(column: str) -> str:
|
||||
return f"({column} IS NULL OR typeof({column}) = 'integer')"
|
||||
|
||||
|
||||
def _in(column: str, values: tuple) -> str:
|
||||
return f"typeof({column}) = 'text' AND {column} IN ({repr(values)[1:-1]})"
|
||||
|
||||
|
||||
# Every row of a table must satisfy its predicate; one pass each, LIMIT 1.
|
||||
_ROW_PREDICATES = {
|
||||
ITEMS_TABLE: " AND ".join(
|
||||
(
|
||||
_uuid("id"),
|
||||
_uuid("byte_id"),
|
||||
"typeof(spelled_path) = 'text' AND length(CAST(spelled_path AS BLOB)) "
|
||||
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}",
|
||||
"typeof(resolver_version) = 'integer'",
|
||||
_hex64("evidence_hash"),
|
||||
_evidence("evidence"),
|
||||
_non_negative("candidate_count"),
|
||||
f"candidate_count <= {MAX_CANDIDATES}",
|
||||
"typeof(limited) = 'integer' AND limited IN (0, 1)",
|
||||
_in("state", ITEM_STATES),
|
||||
_non_negative("revision"),
|
||||
_timestamp("created_at"),
|
||||
_timestamp("updated_at"),
|
||||
)
|
||||
),
|
||||
DECISIONS_TABLE: " AND ".join(
|
||||
(
|
||||
_uuid("id"),
|
||||
_uuid("item_id"),
|
||||
_non_negative("decision_revision"),
|
||||
_timestamp("created_at"),
|
||||
_in("outcome", DECISION_OUTCOMES),
|
||||
"(CASE WHEN outcome = 'selected' THEN "
|
||||
f"{_hex64('candidate_hash')} ELSE candidate_hash IS NULL END)",
|
||||
_evidence("evidence"),
|
||||
_nullable_integer("pair_id"),
|
||||
_nullable_integer("image_id"),
|
||||
)
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _expected_catalog() -> set:
|
||||
objects = {("table", name, _compact(sql)) for name, sql in _TABLE_DDL.items()}
|
||||
objects |= {("index", name, _compact(sql)) for name, sql in _INDEX_DDL.items()}
|
||||
objects |= {
|
||||
("index", f"sqlite_autoindex_{table}_{n}", "")
|
||||
for table, count in _AUTOINDEXES.items()
|
||||
for n in range(1, count + 1)
|
||||
}
|
||||
return objects
|
||||
|
||||
|
||||
def _catalog(conn: sqlite3.Connection) -> list:
|
||||
names = TABLES + tuple(_INDEX_DDL)
|
||||
marks = ",".join("?" * len(names))
|
||||
table_marks = ",".join("?" * len(TABLES))
|
||||
return conn.execute( # only "?" placeholders are interpolated
|
||||
"SELECT type,name,sql FROM sqlite_master "
|
||||
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
|
||||
names + TABLES,
|
||||
).fetchall()
|
||||
|
||||
|
||||
def _structure_error(conn: sqlite3.Connection) -> Optional[str]:
|
||||
for table, expected in _COLUMNS.items():
|
||||
columns = tuple(
|
||||
(row[1], row[2].upper(), row[3], row[4], row[5])
|
||||
for row in conn.execute(f"PRAGMA table_info({table})")
|
||||
)
|
||||
if columns != expected:
|
||||
return f"Review queue table {table} has invalid columns"
|
||||
foreign_keys = {
|
||||
(row[3], row[2], row[4], row[6])
|
||||
for row in conn.execute(f"PRAGMA foreign_key_list({table})")
|
||||
}
|
||||
if foreign_keys != _FOREIGN_KEYS[table]:
|
||||
return f"Review queue table {table} has invalid foreign keys"
|
||||
if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
|
||||
return f"Review queue table {table} contains foreign key violations"
|
||||
return None
|
||||
|
||||
|
||||
def _evidence_error(conn: sqlite3.Connection) -> Optional[str]:
|
||||
"""Stream every evidence document once; one row is held at a time."""
|
||||
for evidence_hash, text in conn.execute(
|
||||
f"SELECT evidence_hash, evidence FROM {ITEMS_TABLE}" # nosec B608
|
||||
):
|
||||
if not _is_canonical(text) or evidence_digest(text) != evidence_hash:
|
||||
return f"Review queue table {ITEMS_TABLE} contains invalid evidence"
|
||||
for (text,) in conn.execute(
|
||||
f"SELECT evidence FROM {DECISIONS_TABLE}" # nosec B608
|
||||
):
|
||||
if not _is_canonical(text):
|
||||
return f"Review queue table {DECISIONS_TABLE} contains invalid evidence"
|
||||
return None
|
||||
|
||||
|
||||
def _value_error(conn: sqlite3.Connection) -> Optional[str]:
|
||||
# Only this module's constants are interpolated into the SQL below.
|
||||
rows = conn.execute(
|
||||
"SELECT COUNT(*), MIN(typeof(singleton_id) = 'integer' " # nosec B608
|
||||
"AND singleton_id = 1 AND typeof(schema_version) = 'integer' "
|
||||
f"AND schema_version = {SCHEMA_VERSION} "
|
||||
f"AND {_non_negative('review_index_revision')}) FROM {METADATA_TABLE}"
|
||||
).fetchone()
|
||||
if rows[0] != 1 or rows[1] != 1:
|
||||
return "Review queue metadata is invalid"
|
||||
for table, predicate in _ROW_PREDICATES.items():
|
||||
if conn.execute(
|
||||
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1" # nosec B608
|
||||
).fetchone():
|
||||
return f"Review queue table {table} contains invalid values"
|
||||
return _evidence_error(conn)
|
||||
|
||||
|
||||
def review_without_annotations_error(conn: sqlite3.Connection) -> Optional[str]:
|
||||
"""Catalog only: a review group present without its annotation tables.
|
||||
|
||||
Startup asks this before the annotation group installs, which would make
|
||||
a review group copied without it look valid. It reads no rows; the full
|
||||
validation runs once in :func:`migrate_review_state`.
|
||||
"""
|
||||
if not _catalog(conn):
|
||||
return None
|
||||
present = {
|
||||
name for kind, name, _sql in _annotation_catalog(conn) if kind == "table"
|
||||
}
|
||||
if present != set(ANNOTATION_TABLES):
|
||||
return (
|
||||
"Review queue schema requires a valid image annotation schema: "
|
||||
"Missing image annotation schema"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def review_schema_error(
|
||||
conn: sqlite3.Connection,
|
||||
*,
|
||||
allow_missing: bool = True,
|
||||
annotations_valid: bool = False,
|
||||
) -> Optional[str]:
|
||||
"""Reject partial or unsupported review schemas and invalid rows.
|
||||
|
||||
Reads only: nothing is ever repaired. A database without any of the
|
||||
group's objects is acceptable for backups that predate it unless
|
||||
``allow_missing`` is false. A present group requires a present, valid
|
||||
image annotation group; ``annotations_valid`` says the caller has just
|
||||
validated it (startup's annotation install), so only its presence is
|
||||
checked again.
|
||||
"""
|
||||
objects = _catalog(conn)
|
||||
if not objects:
|
||||
return None if allow_missing else "Missing review queue schema"
|
||||
if annotations_valid:
|
||||
annotation_error = review_without_annotations_error(conn)
|
||||
else:
|
||||
annotation_error = annotation_schema_error(conn, allow_missing=False)
|
||||
if annotation_error:
|
||||
annotation_error = (
|
||||
"Review queue schema requires a valid image annotation schema: "
|
||||
+ annotation_error
|
||||
)
|
||||
if annotation_error:
|
||||
return annotation_error
|
||||
present = {name for kind, name, _sql in objects if kind == "table"}
|
||||
if present != set(TABLES):
|
||||
return "Incomplete review queue schema"
|
||||
actual = {(kind, name, _compact(sql or "")) for kind, name, sql in objects}
|
||||
if actual != _expected_catalog():
|
||||
return "Review queue has an invalid schema definition"
|
||||
return _structure_error(conn) or _value_error(conn)
|
||||
|
||||
|
||||
def migrate_review_state(conn: sqlite3.Connection) -> None:
|
||||
"""Install the group atomically; validate before and after creating it.
|
||||
|
||||
Runs after ``migrate_annotation_state`` (which has just validated the
|
||||
annotation group): the group is only valid beside the annotation group it
|
||||
cites. A present group is validated once, evidence included.
|
||||
"""
|
||||
conn.execute("SAVEPOINT media_review_schema")
|
||||
try:
|
||||
installed = bool(_catalog(conn))
|
||||
error = review_schema_error(conn, annotations_valid=True)
|
||||
if error:
|
||||
raise sqlite3.DatabaseError(error)
|
||||
if installed:
|
||||
# Already complete and valid: nothing to create, and the evidence
|
||||
# pass is the expensive part of validation, so it runs once.
|
||||
conn.execute("RELEASE media_review_schema")
|
||||
return
|
||||
for sql in _TABLE_DDL.values():
|
||||
conn.execute(sql.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
|
||||
for sql in _INDEX_DDL.values():
|
||||
conn.execute(sql.replace("INDEX", "INDEX IF NOT EXISTS", 1))
|
||||
conn.execute(
|
||||
f"INSERT OR IGNORE INTO {METADATA_TABLE} "
|
||||
"(singleton_id, schema_version, review_index_revision) VALUES (1, ?, 0)",
|
||||
(SCHEMA_VERSION,),
|
||||
)
|
||||
error = review_schema_error(conn, allow_missing=False, annotations_valid=True)
|
||||
if error:
|
||||
raise sqlite3.DatabaseError(error)
|
||||
conn.execute("RELEASE media_review_schema")
|
||||
except BaseException:
|
||||
conn.execute("ROLLBACK TO media_review_schema")
|
||||
conn.execute("RELEASE media_review_schema")
|
||||
raise
|
||||
@@ -8,10 +8,6 @@ Split into domain-specific mixins for maintainability:
|
||||
- AutotagRoutesMixin: auto-tagging model management and tagging
|
||||
- JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py)
|
||||
- ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs
|
||||
- ThumbnailJobsMixin: thumbnail plan, thumbnail job and the gallery stream on it
|
||||
- ReviewRoutesMixin: the Needs review queue, item detail and decisions
|
||||
- ReviewRecordingMixin: Needs review recording for the scan and the monitor
|
||||
- ReviewEventsMixin: the Needs review change stream an open page re-reads on
|
||||
- SystemRoutesMixin: System health check, Overview and usage statistics
|
||||
"""
|
||||
|
||||
@@ -34,9 +30,6 @@ from .pairs import PairRoutesMixin
|
||||
from .library import LibraryRoutesMixin
|
||||
from .vocabulary import VocabularyRoutesMixin
|
||||
from .annotations import AnnotationRoutesMixin
|
||||
from .review import ReviewRoutesMixin
|
||||
from .review_recording import ReviewRecordingMixin
|
||||
from .review_events import ReviewEventsMixin
|
||||
from .saved_views import SavedViewRoutesMixin
|
||||
from .history import HistoryRoutesMixin
|
||||
from .images import ImageRoutesMixin
|
||||
@@ -44,7 +37,6 @@ from .admin import AdminRoutesMixin
|
||||
from .settings_routes import SettingsRoutesMixin
|
||||
from .job_routes import JobRoutesMixin
|
||||
from .image_tag_jobs import ImageTagJobsMixin
|
||||
from .thumbnail_jobs import ThumbnailJobsMixin
|
||||
from .system import SystemRoutesMixin
|
||||
from .jobs import JobRegistry
|
||||
from .logging_routes import LoggingRoutesMixin
|
||||
@@ -272,9 +264,6 @@ class PromptManagerAPI(
|
||||
LibraryRoutesMixin,
|
||||
VocabularyRoutesMixin,
|
||||
AnnotationRoutesMixin,
|
||||
ReviewRoutesMixin,
|
||||
ReviewRecordingMixin,
|
||||
ReviewEventsMixin,
|
||||
SavedViewRoutesMixin,
|
||||
HistoryRoutesMixin,
|
||||
PromptRoutesMixin,
|
||||
@@ -283,7 +272,6 @@ class PromptManagerAPI(
|
||||
SettingsRoutesMixin,
|
||||
JobRoutesMixin,
|
||||
ImageTagJobsMixin,
|
||||
ThumbnailJobsMixin,
|
||||
SystemRoutesMixin,
|
||||
LoggingRoutesMixin,
|
||||
AutotagRoutesMixin,
|
||||
@@ -316,8 +304,6 @@ class PromptManagerAPI(
|
||||
# created on first use with their restore subscription.
|
||||
self._captures = None
|
||||
self._captures_unsubscribe = None
|
||||
# Set by close_annotation_captures (shutdown): no store is recreated.
|
||||
self._captures_closed = False
|
||||
self._cached_output_dir = None # Lazy-cached by _find_comfyui_output_dir()
|
||||
self._html_cache = {} # Cached HTML file contents keyed by path
|
||||
self._gallery_cache = {} # dict: path_str -> (files, timestamp)
|
||||
@@ -498,8 +484,6 @@ class PromptManagerAPI(
|
||||
self._register_library_routes(routes)
|
||||
self._register_vocabulary_routes(routes)
|
||||
self._register_annotation_routes(routes)
|
||||
self._register_review_routes(routes)
|
||||
self._register_review_event_routes(routes)
|
||||
self._register_saved_view_routes(routes)
|
||||
self._register_history_routes(routes)
|
||||
self._register_prompt_routes(routes)
|
||||
|
||||
+10
-87
@@ -45,7 +45,6 @@ try:
|
||||
parse_page_json,
|
||||
)
|
||||
from ...utils.prompt_pair_resolver import resolve_prompt_pair
|
||||
from ...utils.review_capture import reviewable
|
||||
except ImportError:
|
||||
from database.operations import PromptDatabase
|
||||
from database.pair_ingestion import ingest_resolved_pair
|
||||
@@ -56,7 +55,6 @@ except ImportError:
|
||||
parse_page_json,
|
||||
)
|
||||
from utils.prompt_pair_resolver import resolve_prompt_pair
|
||||
from utils.review_capture import reviewable
|
||||
|
||||
|
||||
try:
|
||||
@@ -214,30 +212,6 @@ def _time_left(done, total, batches, elapsed):
|
||||
return f" · about {minutes} min left"
|
||||
|
||||
|
||||
def new_review_counts():
|
||||
"""The scan's Needs review tally: ``added`` new items, ``queued`` files
|
||||
in the queue after this scan (new or already there), ``settled`` items
|
||||
removed because their file is linked now, ``untraced`` items removed
|
||||
because they offer nothing to choose, and ``errors``."""
|
||||
return {"added": 0, "queued": 0, "settled": 0, "untraced": 0, "errors": 0}
|
||||
|
||||
|
||||
def _review_scope(review):
|
||||
"""The scan job's ``scope`` keys for its Needs review tally.
|
||||
|
||||
``review_items`` is the number of this scan's files that are in the
|
||||
pending queue after it: the review group's ambiguous files, less those a
|
||||
capture could not read or a decision already settled.
|
||||
"""
|
||||
return {
|
||||
"review_items": review["queued"],
|
||||
"review_added": review["added"],
|
||||
"review_settled": review["settled"],
|
||||
"review_untraced": review["untraced"],
|
||||
"review_errors": review["errors"],
|
||||
}
|
||||
|
||||
|
||||
def _scan_status_line(done, total, batches, elapsed):
|
||||
""" "Processing file N of M", with an estimate once enough batches ran."""
|
||||
line = f"Processing file {done:,} of {total:,}"
|
||||
@@ -1487,13 +1461,10 @@ class AdminRoutesMixin:
|
||||
known = set()
|
||||
if media_files and job.scope.get("files") != "all":
|
||||
known = await self._known_scan_files(job, media_files)
|
||||
review = new_review_counts()
|
||||
job.scope.update(_review_scope(review))
|
||||
async for event in self._scan_batches(
|
||||
media_files, counts, job, roots, known, review
|
||||
media_files, counts, job, roots, known
|
||||
):
|
||||
yield event
|
||||
self._log_review_items(job, review)
|
||||
|
||||
self.logger.info(
|
||||
f"Scan completed: processed={counts['processed']}, "
|
||||
@@ -1530,16 +1501,12 @@ class AdminRoutesMixin:
|
||||
sizes = {str(media_file): _file_size(media_file) for media_file in media_files}
|
||||
return self.db.get_known_resolved_paths(sizes)
|
||||
|
||||
async def _scan_batches(
|
||||
self, media_files, counts, job, roots=(), known=(), review=None
|
||||
):
|
||||
async def _scan_batches(self, media_files, counts, job, roots=(), known=()):
|
||||
"""Process ``media_files`` in executor batches, updating ``counts`` in place.
|
||||
|
||||
Records each file's outcome in ``job`` and yields one progress event
|
||||
per batch. Files in ``known`` (path strings) are not read: they count
|
||||
as processed and unchanged and are recorded as skipped. ``review``
|
||||
counts the Needs review items the batches record (``job.scope``
|
||||
``review_items`` / ``review_errors`` follow it after every batch).
|
||||
as processed and unchanged and are recorded as skipped.
|
||||
"""
|
||||
total = len(media_files)
|
||||
started = time.monotonic()
|
||||
@@ -1562,16 +1529,9 @@ class AdminRoutesMixin:
|
||||
batch_results,
|
||||
roots=roots,
|
||||
outcomes=outcomes,
|
||||
review=review,
|
||||
)
|
||||
for key in counts:
|
||||
counts[key] += batch_counts[key]
|
||||
if review is not None:
|
||||
# After the batch's links: a file linked now leaves the queue.
|
||||
await self._run_in_executor(
|
||||
self._settle_scanned_reviews, batch, roots, review
|
||||
)
|
||||
job.scope.update(_review_scope(review))
|
||||
for group, row, log_line in outcomes:
|
||||
job.record(group, row, log_line)
|
||||
|
||||
@@ -1621,20 +1581,12 @@ class AdminRoutesMixin:
|
||||
|
||||
return map_parallel(extract, file_batch, PromptManagerConfig.WORKER_THREADS)
|
||||
|
||||
def _ingest_scan_batch_sync(
|
||||
self, batch_results, roots=(), outcomes=None, review=None
|
||||
):
|
||||
def _ingest_scan_batch_sync(self, batch_results, roots=(), outcomes=None):
|
||||
"""Store the prompts found in a metadata batch (blocking DB work).
|
||||
|
||||
Returns the legacy counts. When ``outcomes`` is a list, every processed
|
||||
file appends its ``(group, row, log_line)`` for the scan job, with the
|
||||
path relative to its root in ``roots`` (see ``_scan_roots``). A file
|
||||
whose final outcome is ambiguous and unlinked (the resolver found it
|
||||
ambiguous and the fallback did not link it) is recorded for Needs
|
||||
review (best effort, counted in ``review`` when given); one a saved
|
||||
row already links is reported ``unchanged`` and never queued. An
|
||||
ambiguous file with nothing to choose is reported ``unresolved`` and
|
||||
its pending items without a choice leave the queue.
|
||||
path relative to its root in ``roots`` (see ``_scan_roots``).
|
||||
"""
|
||||
counts = dict.fromkeys(SCAN_COUNT_KEYS, 0)
|
||||
|
||||
@@ -1646,27 +1598,13 @@ class AdminRoutesMixin:
|
||||
if self._scan_job and self._scan_job.cancelled:
|
||||
break
|
||||
counts["processed"] += 1
|
||||
ambiguous, untraced = [], []
|
||||
try:
|
||||
outcome = self._ingest_scanned_file(
|
||||
media_file,
|
||||
metadata,
|
||||
detailed=True,
|
||||
ambiguous=ambiguous,
|
||||
untraced=untraced,
|
||||
)
|
||||
outcome = self._ingest_scanned_file(media_file, metadata, detailed=True)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error processing {media_file.name}: {e}")
|
||||
counts["failed"] += 1
|
||||
note(media_file, "error")
|
||||
continue
|
||||
if ambiguous and outcome == "ambiguous":
|
||||
if self._review_file_linked(media_file, roots):
|
||||
outcome = "unchanged" # already linked: not a review case
|
||||
else:
|
||||
self._record_scanned_review(media_file, roots, review)
|
||||
elif untraced and outcome == "unresolved":
|
||||
self._settle_untraced_reviews(untraced, roots, review)
|
||||
if outcome is None:
|
||||
note(media_file, None)
|
||||
continue
|
||||
@@ -1700,21 +1638,13 @@ class AdminRoutesMixin:
|
||||
return None
|
||||
return positive.strip() or None
|
||||
|
||||
def _ingest_scanned_file(
|
||||
self, media_file, metadata, detailed=False, ambiguous=None, untraced=None
|
||||
):
|
||||
def _ingest_scanned_file(self, media_file, metadata, detailed=False):
|
||||
"""Enrich exact pairs; retain legacy positive-only import for absent pairs.
|
||||
|
||||
Returns the file's scan status, or None without prompt metadata. With
|
||||
``detailed``, a stored or linked file returns a dict instead: its
|
||||
``status``, the prompt's display ``title`` and, for a resolved pair,
|
||||
the pair counts. When the resolver finds the file ambiguous (whether
|
||||
or not the browser parser then links it) and ``ambiguous`` is a list,
|
||||
``media_file`` is appended to it; only a file that then returns
|
||||
``"ambiguous"`` (nothing linked it) is a Needs review case. An
|
||||
ambiguous result with nothing to choose (``reviewable`` is False: fewer
|
||||
than two candidates carry a prompt) that nothing links returns
|
||||
``"unresolved"`` instead, and ``media_file`` goes to ``untraced``.
|
||||
the pair counts.
|
||||
"""
|
||||
if not metadata:
|
||||
return None
|
||||
@@ -1730,13 +1660,6 @@ class AdminRoutesMixin:
|
||||
return None
|
||||
_with_file_size(canonical, media_file)
|
||||
resolution = resolve_prompt_pair(canonical)
|
||||
# Ambiguous without a choice: unresolved unless something links it.
|
||||
unchoosable = resolution["state"] == "ambiguous" and not reviewable(resolution)
|
||||
if unchoosable:
|
||||
if untraced is not None:
|
||||
untraced.append(media_file)
|
||||
elif resolution["state"] == "ambiguous" and ambiguous is not None:
|
||||
ambiguous.append(media_file)
|
||||
if resolution["state"] == "resolved":
|
||||
result = ingest_resolved_pair(
|
||||
self.db, str(media_file), canonical, resolution
|
||||
@@ -1756,9 +1679,9 @@ class AdminRoutesMixin:
|
||||
# Graphs without a recognized sampler keep the legacy extractor below.
|
||||
prompt_text = self._graph_prompt_text(metadata)
|
||||
if not prompt_text:
|
||||
return "unresolved" if unchoosable else resolution["state"]
|
||||
return resolution["state"]
|
||||
elif resolution["state"] == "ambiguous":
|
||||
return "unresolved" if unchoosable else "ambiguous"
|
||||
return "ambiguous"
|
||||
elif resolution["state"] == "unresolved" and not isinstance(
|
||||
canonical.get("parameters"), str
|
||||
):
|
||||
|
||||
+13
-35
@@ -13,7 +13,6 @@ import datetime
|
||||
import functools
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
from aiohttp import web
|
||||
@@ -74,9 +73,6 @@ _HEX64 = re.compile(r"[0-9a-f]{64}")
|
||||
_UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
|
||||
# secrets.token_urlsafe(32) is 43 characters; the bound only rejects junk.
|
||||
_TOKEN = re.compile(r"[A-Za-z0-9_-]{1,128}")
|
||||
# The store is created on first use by the event loop, the scan's executor or
|
||||
# the monitor's worker (Needs review), so its creation is serialized.
|
||||
_CAPTURE_STORE_LOCK = threading.Lock()
|
||||
|
||||
_PREVIEW_TYPES = {
|
||||
".png": "image/png",
|
||||
@@ -263,46 +259,31 @@ class AnnotationRoutesMixin:
|
||||
Created lazily so an API object that never observes a file (most tests,
|
||||
a server that never opens the Tag dialog) owns no temp directory. A
|
||||
restore of the database dooms every capture (``invalidate_all``).
|
||||
Thread-safe: Needs review recording calls it off the event loop.
|
||||
Raises ``CaptureError`` after ``close_annotation_captures``.
|
||||
"""
|
||||
with _CAPTURE_STORE_LOCK:
|
||||
if getattr(self, "_captures_closed", False):
|
||||
# A recording still in flight at shutdown must not create a
|
||||
# new temp directory after the cleanup removed the old one.
|
||||
raise CaptureError("The capture store is closed")
|
||||
if self._captures is None:
|
||||
self._captures = CaptureStore()
|
||||
if self._captures_unsubscribe is None:
|
||||
self._captures_unsubscribe = subscribe_history_restore(
|
||||
self.db.model.db_path, self._invalidate_captures
|
||||
)
|
||||
return self._captures
|
||||
if self._captures is None:
|
||||
self._captures = CaptureStore()
|
||||
if self._captures_unsubscribe is None:
|
||||
self._captures_unsubscribe = subscribe_history_restore(
|
||||
self.db.model.db_path, self._invalidate_captures
|
||||
)
|
||||
return self._captures
|
||||
|
||||
def _invalidate_captures(self) -> None:
|
||||
store = self._captures
|
||||
if store is not None:
|
||||
store.invalidate_all()
|
||||
|
||||
def close_annotation_captures(self, *, final: bool = False) -> None:
|
||||
"""Remove every capture file and the owned directory. Idempotent.
|
||||
|
||||
Taken under the store lock. ``final`` (the shutdown cleanup) also makes
|
||||
``_capture_store`` refuse to create another store, so a recording
|
||||
still in flight cannot leak a new temp directory after it.
|
||||
"""
|
||||
with _CAPTURE_STORE_LOCK:
|
||||
if final:
|
||||
self._captures_closed = True
|
||||
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
|
||||
store, self._captures = self._captures, None
|
||||
def close_annotation_captures(self) -> None:
|
||||
"""Remove every capture file and the owned directory. Idempotent."""
|
||||
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
|
||||
if unsubscribe is not None:
|
||||
unsubscribe()
|
||||
store, self._captures = self._captures, None
|
||||
if store is not None:
|
||||
store.close()
|
||||
|
||||
async def cleanup_annotation_captures(self, app: web.Application) -> None:
|
||||
self.close_annotation_captures(final=True)
|
||||
self.close_annotation_captures()
|
||||
|
||||
# -- shared request steps --------------------------------------------------
|
||||
|
||||
@@ -348,10 +329,7 @@ class AnnotationRoutesMixin:
|
||||
root, error = self._scoped_root(index, scope)
|
||||
if error is not None:
|
||||
return error
|
||||
try:
|
||||
store = self._capture_store()
|
||||
except CaptureError as exc: # the server is shutting down
|
||||
return _capture_error(exc)
|
||||
store = self._capture_store()
|
||||
try:
|
||||
epoch = await self._run_in_executor(self.db.get_execution_epoch)
|
||||
capture = await _settled(
|
||||
|
||||
+183
-192
@@ -20,7 +20,7 @@ from typing import Callable, NamedTuple
|
||||
from aiohttp import web
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
from .jobs import MAX_OUTCOME_ROWS, THUMBNAILS_JOB_KIND
|
||||
from .jobs import THUMBNAILS_JOB_KIND
|
||||
from .prompts import bad_request, parse_page_params, publish_image_paths
|
||||
from .prompts import thumbnail_url_for
|
||||
from .prompts import safe_error_message as _safe_error
|
||||
@@ -235,8 +235,6 @@ MEDIA_EXTENSIONS = GALLERY_IMAGE_EXTENSIONS | VIDEO_EXTENSIONS
|
||||
# Caps on request-triggered filesystem work.
|
||||
MAX_SCAN_DEPTH = 12
|
||||
MAX_THUMBNAIL_FILES = 10_000
|
||||
# Already-fresh files a thumbnail job lists under Skipped (its counter stays exact).
|
||||
MAX_FRESH_LISTED = MAX_OUTCOME_ROWS
|
||||
|
||||
|
||||
def _worker_threads():
|
||||
@@ -273,52 +271,15 @@ def _atomic_save(img, dst, fmt=None, **save_kwargs):
|
||||
tmp.unlink()
|
||||
|
||||
|
||||
# Encoder settings per thumbnail quality (the System thumbnail dialog's
|
||||
# "Medium · optimized", "High · original", "Low · fastest"). The legacy
|
||||
# routes always encode as "medium", which is what they always wrote.
|
||||
THUMBNAIL_ENCODINGS = {
|
||||
"low": {"quality": 70, "optimize": False},
|
||||
"medium": {"quality": 85, "optimize": True},
|
||||
"high": {"quality": 95, "optimize": True},
|
||||
}
|
||||
DEFAULT_THUMBNAIL_ENCODING = THUMBNAIL_ENCODINGS["medium"]
|
||||
# ffmpeg's JPEG scale (-q:v, 2 best to 31 worst) for each encoding's quality.
|
||||
_FFMPEG_QSCALE = {95: 2, 85: 4, 70: 8}
|
||||
|
||||
# A preview that could not be replaced (#249 S5 M6). Windows refuses to rename
|
||||
# over a file another process has open (WinError 5 or 32), such as a preview
|
||||
# the gallery is serving; the old preview stays and the file counts as an error.
|
||||
PREVIEW_IN_USE = "Preview in use by another program; the old one was kept"
|
||||
PREVIEW_NOT_WRITABLE = "Permission denied writing the preview"
|
||||
_WINDOWS_LOCKED = (5, 32) # ERROR_ACCESS_DENIED, ERROR_SHARING_VIOLATION
|
||||
|
||||
|
||||
def _preview_write_error(exc):
|
||||
"""The readable error for a PermissionError while writing a preview."""
|
||||
if getattr(exc, "winerror", None) in _WINDOWS_LOCKED:
|
||||
return PREVIEW_IN_USE
|
||||
return PREVIEW_NOT_WRITABLE
|
||||
|
||||
|
||||
def _ffmpeg_qscale(encoding):
|
||||
"""ffmpeg's ``-q:v`` for a THUMBNAIL_ENCODINGS entry."""
|
||||
return str(_FFMPEG_QSCALE.get(encoding["quality"], 4))
|
||||
|
||||
|
||||
def _write_image_thumbnail(src, dst, thumbnail_size, encoding=None):
|
||||
"""Resize *src* into *dst* (blocking PIL work), atomically.
|
||||
|
||||
``encoding`` is one of THUMBNAIL_ENCODINGS (medium when None); a PNG
|
||||
preview keeps only its ``optimize`` flag.
|
||||
"""
|
||||
encoding = encoding or DEFAULT_THUMBNAIL_ENCODING
|
||||
def _write_image_thumbnail(src, dst, thumbnail_size):
|
||||
"""Resize *src* into *dst* (blocking PIL work), atomically."""
|
||||
with Image.open(src) as img:
|
||||
if img.mode in ("RGBA", "LA", "P"):
|
||||
img = img.convert("RGB")
|
||||
img.thumbnail(thumbnail_size, Image.Resampling.LANCZOS)
|
||||
save_kwargs = dict(encoding)
|
||||
save_kwargs = {"quality": 85, "optimize": True}
|
||||
if dst.suffix.lower() == ".png":
|
||||
save_kwargs = {"optimize": encoding["optimize"]}
|
||||
save_kwargs = {"optimize": True}
|
||||
_atomic_save(img, dst, **save_kwargs)
|
||||
|
||||
|
||||
@@ -710,25 +671,6 @@ def _iter_media_files(output_path, max_files):
|
||||
yield root_path / name
|
||||
|
||||
|
||||
def media_thumbnail_url(output_path, rel_path, root_index, *, root_scope=None):
|
||||
"""The Images listing's thumbnail URL for *rel_path* under *output_path*,
|
||||
or None when no thumbnail exists (one blocking stat). The Needs review
|
||||
queue cards use the same URL (#249 F, M10)."""
|
||||
rel_path = Path(rel_path)
|
||||
extension = rel_path.suffix.lower()
|
||||
thumb_rel = (
|
||||
f"thumbnails/{rel_path.with_suffix('').as_posix()}_thumb"
|
||||
f"{'.jpg' if extension in VIDEO_EXTENSIONS else extension}"
|
||||
)
|
||||
thumbnail_url = thumbnail_url_for(thumb_rel, Path(output_path) / thumb_rel)
|
||||
if thumbnail_url is None:
|
||||
return None
|
||||
thumbnail_url += f"&root={root_index}"
|
||||
if root_scope is not None:
|
||||
thumbnail_url += f"&root_scope={root_scope}"
|
||||
return thumbnail_url
|
||||
|
||||
|
||||
def _output_image_entry(
|
||||
media_path, output_path, root_index, public_root=None, *, root_scope=None
|
||||
):
|
||||
@@ -744,13 +686,19 @@ def _output_image_entry(
|
||||
extension = media_path.suffix.lower()
|
||||
is_video = extension in VIDEO_EXTENSIONS
|
||||
|
||||
thumbnail_url = media_thumbnail_url(
|
||||
output_path, rel_path, root_index, root_scope=root_scope
|
||||
thumb_rel = (
|
||||
f"thumbnails/{rel_path.with_suffix('').as_posix()}_thumb"
|
||||
f"{'.jpg' if is_video else extension}"
|
||||
)
|
||||
thumbnail_url = thumbnail_url_for(thumb_rel, output_path / thumb_rel)
|
||||
if thumbnail_url is not None:
|
||||
thumbnail_url += f"&root={root_index}"
|
||||
encoded_path = urllib.parse.quote(rel_path.as_posix(), safe="/")
|
||||
url = f"/prompt_manager/images/serve/{encoded_path}?root={root_index}"
|
||||
if root_scope is not None:
|
||||
url += f"&root_scope={root_scope}"
|
||||
if thumbnail_url is not None:
|
||||
thumbnail_url += f"&root_scope={root_scope}"
|
||||
|
||||
entry = {
|
||||
"id": hashlib.sha1(
|
||||
@@ -984,10 +932,6 @@ class ImageRoutesMixin:
|
||||
async def thumbnail_status_route(request):
|
||||
return await self.get_thumbnail_status(request)
|
||||
|
||||
@routes.get("/prompt_manager/images/thumbnails/plan")
|
||||
async def thumbnail_plan_route(request):
|
||||
return await self.get_thumbnail_plan(request)
|
||||
|
||||
@routes.post("/prompt_manager/images/generate-thumbnails")
|
||||
async def generate_thumbnails_route(request):
|
||||
return await self.generate_thumbnails(request)
|
||||
@@ -1860,35 +1804,15 @@ class ImageRoutesMixin:
|
||||
self.logger.info(payload["message"])
|
||||
return payload
|
||||
|
||||
def _plan_gallery_thumbnails(
|
||||
self, output_dirs: list[Path], rebuild: bool = False, stop=None
|
||||
) -> dict:
|
||||
"""Plan every gallery root with one shared per-run generation cap.
|
||||
|
||||
See ``_plan_thumbnails``; ``fresh_files`` keeps at most
|
||||
MAX_FRESH_LISTED files across the roots. ``stop()`` true ends the walk
|
||||
at the next file (a cancelled job); the partial plan is returned.
|
||||
"""
|
||||
combined = {
|
||||
"targets": [],
|
||||
"skipped": 0,
|
||||
"remaining": 0,
|
||||
"fresh": 0,
|
||||
"fresh_files": [],
|
||||
}
|
||||
def _plan_gallery_thumbnails(self, output_dirs: list[Path]) -> dict:
|
||||
"""Plan every gallery root with one shared per-run generation cap."""
|
||||
combined = {"targets": [], "skipped": 0, "remaining": 0}
|
||||
for root in output_dirs:
|
||||
if stop is not None and stop():
|
||||
break
|
||||
root = Path(root)
|
||||
plan = self._plan_thumbnails(
|
||||
root, root / "thumbnails", rebuild=rebuild, stop=stop
|
||||
)
|
||||
plan = self._plan_thumbnails(root, root / "thumbnails")
|
||||
available = max(0, MAX_THUMBNAIL_FILES - len(combined["targets"]))
|
||||
combined["targets"].extend(plan["targets"][:available])
|
||||
combined["skipped"] += plan["skipped"]
|
||||
combined["fresh"] += plan["fresh"]
|
||||
room = max(0, MAX_FRESH_LISTED - len(combined["fresh_files"]))
|
||||
combined["fresh_files"].extend(plan["fresh_files"][:room])
|
||||
combined["remaining"] += plan["remaining"] + max(
|
||||
0, len(plan["targets"]) - available
|
||||
)
|
||||
@@ -1921,30 +1845,22 @@ class ImageRoutesMixin:
|
||||
{"success": False, "error": _safe_error(exc)}, status=500
|
||||
)
|
||||
|
||||
def _plan_thumbnails(self, output_path, thumbnails_dir, rebuild=False, stop=None):
|
||||
def _plan_thumbnails(self, output_path, thumbnails_dir):
|
||||
"""Decide which media files need a thumbnail (blocking walk).
|
||||
|
||||
Returns ``{"targets": [(src, dst, is_video)], "skipped": n,
|
||||
"remaining": n, "fresh": n, "fresh_files": [src]}``. Files whose
|
||||
thumbnail is already newer than the source are ``fresh``. Without
|
||||
``rebuild`` they are counted as skipped and never queued, so a large
|
||||
library that is mostly done only pays for the files that changed;
|
||||
``fresh_files`` lists the first MAX_FRESH_LISTED of them. With
|
||||
``rebuild`` every file is a target (nothing is skipped). The walk is
|
||||
bounded like the gallery listing (MAX_SCAN_DEPTH, MAX_GALLERY_FILES,
|
||||
no symlinks) and at most MAX_THUMBNAIL_FILES targets are queued per
|
||||
run; ``remaining`` counts the targets beyond that cap. ``stop()`` true
|
||||
ends the walk before the next file.
|
||||
Returns ``{"targets": [(src, dst, is_video)], "skipped": n, "remaining": n}``.
|
||||
Files whose thumbnail is already newer than the source are counted as
|
||||
skipped and never queued, so a large library that is mostly done only
|
||||
pays for the files that changed. The walk is bounded like the gallery
|
||||
listing (MAX_SCAN_DEPTH, MAX_GALLERY_FILES, no symlinks) and at most
|
||||
MAX_THUMBNAIL_FILES pending files are queued per run; ``remaining``
|
||||
counts the pending files beyond that cap.
|
||||
"""
|
||||
output_path = Path(output_path)
|
||||
targets = []
|
||||
skipped = 0
|
||||
remaining = 0
|
||||
fresh = 0
|
||||
fresh_files = []
|
||||
for src in _iter_media_files(output_path, MAX_GALLERY_FILES):
|
||||
if stop is not None and stop():
|
||||
break
|
||||
suffix = src.suffix.lower()
|
||||
is_video = suffix in VIDEO_EXTENSIONS
|
||||
rel_no_ext = src.relative_to(output_path).with_suffix("")
|
||||
@@ -1954,12 +1870,8 @@ class ImageRoutesMixin:
|
||||
self.logger.warning(f"Skipping thumbnail outside safe dir: {src.name}")
|
||||
continue
|
||||
if _thumbnail_is_fresh(src, dst):
|
||||
fresh += 1
|
||||
if not rebuild:
|
||||
skipped += 1
|
||||
if len(fresh_files) < MAX_FRESH_LISTED:
|
||||
fresh_files.append(src)
|
||||
continue
|
||||
skipped += 1
|
||||
continue
|
||||
if len(targets) >= MAX_THUMBNAIL_FILES:
|
||||
remaining += 1
|
||||
continue
|
||||
@@ -1969,81 +1881,190 @@ class ImageRoutesMixin:
|
||||
f"Thumbnail run capped at {MAX_THUMBNAIL_FILES} new files; "
|
||||
f"{remaining} more will be generated on the next run"
|
||||
)
|
||||
return {
|
||||
"targets": targets,
|
||||
"skipped": skipped,
|
||||
"remaining": remaining,
|
||||
"fresh": fresh,
|
||||
"fresh_files": fresh_files,
|
||||
}
|
||||
return {"targets": targets, "skipped": skipped, "remaining": remaining}
|
||||
|
||||
def _thumbnail_targets(self, output_path, thumbnails_dir):
|
||||
"""Pending (src, dst, is_video) triples; see _plan_thumbnails."""
|
||||
return self._plan_thumbnails(output_path, thumbnails_dir)["targets"]
|
||||
|
||||
def _generate_batch(self, targets, thumbnail_size, **options):
|
||||
def _generate_batch(self, targets, thumbnail_size):
|
||||
"""Create thumbnails for ``targets`` on the configured worker threads.
|
||||
|
||||
Blocking; returns one result dict per target, in order. ``options``
|
||||
go to ``_generate_one`` (``encoding``, ``rebuild``, ``stop``).
|
||||
Blocking; returns one result dict per target, in order.
|
||||
"""
|
||||
|
||||
def one(target):
|
||||
src, dst, is_video = target
|
||||
return self._generate_one(src, dst, thumbnail_size, is_video, **options)
|
||||
return self._generate_one(src, dst, thumbnail_size, is_video)
|
||||
|
||||
return map_parallel(one, targets, _worker_threads())
|
||||
|
||||
def _generate_one(
|
||||
self,
|
||||
src,
|
||||
dst,
|
||||
thumbnail_size,
|
||||
is_video,
|
||||
encoding=None,
|
||||
rebuild=False,
|
||||
stop=None,
|
||||
):
|
||||
"""Create one thumbnail (blocking). Never raises; returns a result dict.
|
||||
|
||||
``stop()`` true before the file starts gives ``action: "cancelled"``
|
||||
and touches nothing (a job's file boundary). ``rebuild`` writes a
|
||||
fresh preview again and never replaces an existing video preview with
|
||||
the placeholder. Every write is atomic, so a failure keeps the old
|
||||
preview.
|
||||
"""
|
||||
def _generate_one(self, src, dst, thumbnail_size, is_video):
|
||||
"""Create one thumbnail (blocking). Never raises; returns a result dict."""
|
||||
result = {
|
||||
"file": src.name,
|
||||
"dir": src.parent.name,
|
||||
"type": "video" if is_video else "image",
|
||||
}
|
||||
try:
|
||||
if stop is not None and stop():
|
||||
return {**result, "action": "cancelled"}
|
||||
if not src.is_file():
|
||||
return {**result, "action": "error", "error": "File no longer exists"}
|
||||
if not rebuild and _thumbnail_is_fresh(src, dst):
|
||||
if _thumbnail_is_fresh(src, dst):
|
||||
return {**result, "action": "skipped"}
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
if is_video:
|
||||
made = self._generate_video_thumbnail(
|
||||
src, dst, thumbnail_size, encoding, keep_existing=rebuild
|
||||
)
|
||||
if not made:
|
||||
if not self._generate_video_thumbnail(src, dst, thumbnail_size):
|
||||
return {**result, "action": "error", "error": "Video decode failed"}
|
||||
else:
|
||||
_write_image_thumbnail(src, dst, thumbnail_size, encoding)
|
||||
_write_image_thumbnail(src, dst, thumbnail_size)
|
||||
return {**result, "action": "generated"}
|
||||
except UnidentifiedImageError:
|
||||
return {**result, "action": "error", "error": "Not a valid image file"}
|
||||
except PermissionError as e:
|
||||
message = _preview_write_error(e)
|
||||
self.logger.warning(f"Thumbnail failed for {src.name}: {message}")
|
||||
return {**result, "action": "error", "error": message}
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Thumbnail failed for {src.name}", exc_info=True)
|
||||
return {**result, "action": "error", "error": _safe_error(e)}
|
||||
|
||||
async def generate_thumbnails_with_progress(self, request):
|
||||
"""Generate thumbnails with Server-Sent Events progress updates (POST).
|
||||
|
||||
``quality`` is read from the query string so a POST-capable SSE
|
||||
client can keep using the same URL shape. Every blocking step (scan,
|
||||
PIL/ffmpeg) runs in the executor; the event loop only writes SSE
|
||||
frames between files.
|
||||
"""
|
||||
thumbnail_size = self._thumbnail_size(request.query.get("quality", "medium"))
|
||||
response = web.StreamResponse(
|
||||
status=200,
|
||||
headers={
|
||||
"Content-Type": "text/event-stream",
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
},
|
||||
)
|
||||
busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded")
|
||||
if busy is not None:
|
||||
# The gallery reads this stream: refuse with its error event.
|
||||
await response.prepare(request)
|
||||
message = f"{busy.title} already running"
|
||||
await self._emit_sse(
|
||||
response, "error", {"error": message, "message": message}
|
||||
)
|
||||
return response
|
||||
async with self._jobs.hold(
|
||||
THUMBNAILS_JOB_KIND, THUMBNAIL_GENERATION_TITLE, admission="guarded"
|
||||
) as job:
|
||||
await response.prepare(request)
|
||||
if not await self._thumbnail_progress_stream(response, thumbnail_size):
|
||||
job.finish("failed") # an error event ended the stream
|
||||
return response
|
||||
|
||||
async def _thumbnail_progress_stream(self, response, thumbnail_size) -> bool:
|
||||
"""Stream one generation run; False when it ended with an error event."""
|
||||
try:
|
||||
output_dirs = self._get_all_output_dirs()
|
||||
if not output_dirs:
|
||||
await self._emit_sse(
|
||||
response, "error", {"error": "ComfyUI output directory not found"}
|
||||
)
|
||||
return False
|
||||
|
||||
await self._emit_sse(
|
||||
response,
|
||||
"status",
|
||||
{"phase": "scanning", "message": "Scanning for images and videos..."},
|
||||
)
|
||||
plan = await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails, output_dirs
|
||||
)
|
||||
await self._stream_thumbnails(
|
||||
response,
|
||||
plan["targets"],
|
||||
thumbnail_size,
|
||||
skipped=plan["skipped"],
|
||||
remaining=plan["remaining"],
|
||||
)
|
||||
self.invalidate_gallery_cache()
|
||||
return True
|
||||
except Exception:
|
||||
self.logger.exception("Thumbnail generation failed")
|
||||
await self._emit_sse(
|
||||
response,
|
||||
"error",
|
||||
{
|
||||
"error": "An internal error occurred",
|
||||
"message": "Thumbnail generation failed. Check server logs.",
|
||||
},
|
||||
)
|
||||
return False
|
||||
|
||||
async def _stream_thumbnails(
|
||||
self, response, targets, thumbnail_size, skipped=0, remaining=0
|
||||
):
|
||||
"""Process pending *targets* in worker-thread batches, emitting SSE per file.
|
||||
|
||||
``skipped`` files already had fresh thumbnails and ``remaining`` were
|
||||
left for the next run; both only feed the start/complete payloads.
|
||||
"""
|
||||
total = len(targets)
|
||||
video_count = sum(1 for _, _, is_video in targets if is_video)
|
||||
message = (
|
||||
f"Found {total - video_count} images and {video_count} videos to process"
|
||||
)
|
||||
if skipped:
|
||||
message += f" ({skipped} already have thumbnails)"
|
||||
connected = await self._emit_sse(
|
||||
response,
|
||||
"start",
|
||||
{
|
||||
"total_images": total,
|
||||
"skipped": skipped,
|
||||
"remaining": remaining,
|
||||
"phase": "processing",
|
||||
"image_count": total - video_count,
|
||||
"video_count": video_count,
|
||||
"message": message,
|
||||
},
|
||||
)
|
||||
if not connected:
|
||||
return
|
||||
stats = _new_thumbnail_stats()
|
||||
stats["skipped"] = skipped
|
||||
start = _time.monotonic()
|
||||
batch_size = _thumbnail_batch_size(_worker_threads())
|
||||
index = 0
|
||||
for batch_start in range(0, total, batch_size):
|
||||
batch = targets[batch_start : batch_start + batch_size]
|
||||
results = await self._run_in_executor(
|
||||
self._generate_batch, batch, thumbnail_size
|
||||
)
|
||||
for result in results:
|
||||
index += 1
|
||||
_record_thumbnail_result(stats, result)
|
||||
if result["action"] == "error":
|
||||
await self._emit_sse(
|
||||
response,
|
||||
"file_error",
|
||||
{"file": result["file"], "error": result["error"]},
|
||||
)
|
||||
connected = await self._emit_sse(
|
||||
response,
|
||||
"progress",
|
||||
_thumbnail_progress_payload(index, total, stats, result, start),
|
||||
)
|
||||
if not connected:
|
||||
self.logger.info("Thumbnail client disconnected; stopping early")
|
||||
return
|
||||
await asyncio.sleep(0) # let other requests run between batches
|
||||
payload = _thumbnail_complete_payload(
|
||||
total + skipped + remaining,
|
||||
stats,
|
||||
start,
|
||||
remaining=remaining,
|
||||
processed=total,
|
||||
)
|
||||
await self._emit_sse(response, "complete", payload)
|
||||
self.logger.info(payload["message"])
|
||||
|
||||
async def _emit_sse(self, response, event, data) -> bool:
|
||||
"""Write one SSE frame. Returns False once the client has gone away."""
|
||||
try:
|
||||
@@ -2054,25 +2075,8 @@ class ImageRoutesMixin:
|
||||
self.logger.warning(f"Failed to send SSE message: {e}")
|
||||
return False
|
||||
|
||||
def _generate_video_thumbnail(
|
||||
self,
|
||||
video_path,
|
||||
thumbnail_path,
|
||||
thumbnail_size,
|
||||
encoding=None,
|
||||
keep_existing=False,
|
||||
):
|
||||
"""Generate thumbnail from video file. Returns True if successful.
|
||||
|
||||
``encoding`` is one of THUMBNAIL_ENCODINGS (medium when None); it
|
||||
encodes the OpenCV frame, ffmpeg's frame (as ``-q:v``) and the
|
||||
placeholder alike. Both decoders fit the frame inside
|
||||
``thumbnail_size`` keeping its aspect ratio. With ``keep_existing`` an
|
||||
existing preview is kept, and the run fails, when no frame could be
|
||||
decoded, rather than replaced by the placeholder. A PermissionError
|
||||
while replacing the preview propagates (``_generate_one`` names it).
|
||||
"""
|
||||
encoding = encoding or DEFAULT_THUMBNAIL_ENCODING
|
||||
def _generate_video_thumbnail(self, video_path, thumbnail_path, thumbnail_size):
|
||||
"""Generate thumbnail from video file. Returns True if successful."""
|
||||
try:
|
||||
# Try using OpenCV first (most reliable)
|
||||
try:
|
||||
@@ -2096,7 +2100,7 @@ class ImageRoutesMixin:
|
||||
|
||||
img = Image.fromarray(frame_rgb)
|
||||
img.thumbnail(thumbnail_size, Image.Resampling.LANCZOS)
|
||||
_atomic_save(img, thumbnail_path, "JPEG", **encoding)
|
||||
_atomic_save(img, thumbnail_path, "JPEG", quality=85, optimize=True)
|
||||
self.logger.debug(
|
||||
f"Generated video thumbnail using OpenCV: {thumbnail_path}"
|
||||
)
|
||||
@@ -2118,11 +2122,8 @@ class ImageRoutesMixin:
|
||||
"00:00:01",
|
||||
"-vframes",
|
||||
"1",
|
||||
"-vf",
|
||||
f"scale={thumbnail_size[0]}:{thumbnail_size[1]}"
|
||||
":force_original_aspect_ratio=decrease",
|
||||
"-q:v",
|
||||
_ffmpeg_qscale(encoding),
|
||||
"-s",
|
||||
f"{thumbnail_size[0]}x{thumbnail_size[1]}",
|
||||
"-y",
|
||||
str(tmp_path),
|
||||
]
|
||||
@@ -2147,12 +2148,6 @@ class ImageRoutesMixin:
|
||||
except (NameError, OSError):
|
||||
pass
|
||||
|
||||
if keep_existing and thumbnail_path.exists():
|
||||
self.logger.warning(
|
||||
f"No frame decoded for {video_path.name}; keeping its preview"
|
||||
)
|
||||
return False
|
||||
|
||||
# Last resort: create a placeholder thumbnail
|
||||
try:
|
||||
from PIL import ImageDraw, ImageFont
|
||||
@@ -2187,22 +2182,18 @@ class ImageRoutesMixin:
|
||||
except (OSError, AttributeError):
|
||||
pass
|
||||
|
||||
_atomic_save(img, thumbnail_path, "JPEG", **encoding)
|
||||
_atomic_save(img, thumbnail_path, "JPEG", quality=85)
|
||||
self.logger.debug(
|
||||
f"Generated placeholder video thumbnail: {thumbnail_path}"
|
||||
)
|
||||
return True
|
||||
|
||||
except PermissionError:
|
||||
raise
|
||||
except Exception as e:
|
||||
self.logger.warning(
|
||||
f"Failed to create placeholder thumbnail for {video_path}: {e}"
|
||||
)
|
||||
return False
|
||||
|
||||
except PermissionError:
|
||||
raise
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f"Video thumbnail generation failed for {video_path}: {e}"
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
"""Session job routes (Activity): list, inspect, follow and cancel jobs.
|
||||
|
||||
The registry itself is ``py/api/jobs.py``. The scan job's runner is in
|
||||
``py/api/admin.py``, the image tagging job's in ``py/api/image_tag_jobs.py``
|
||||
and the thumbnail job's in ``py/api/thumbnail_jobs.py``.
|
||||
``py/api/admin.py``, the image tagging job's in ``py/api/image_tag_jobs.py``.
|
||||
Every response leaves out absolute server paths: job rows already carry
|
||||
root-relative paths and public root labels.
|
||||
"""
|
||||
@@ -130,10 +129,6 @@ class JobRoutesMixin:
|
||||
async def start_image_tag_job_route(request):
|
||||
return await self.start_image_tag_job(request)
|
||||
|
||||
@routes.post("/prompt_manager/jobs/thumbnails")
|
||||
async def start_thumbnail_job_route(request):
|
||||
return await self.start_thumbnail_job(request)
|
||||
|
||||
@routes.get("/prompt_manager/jobs/{job_id}")
|
||||
async def job_detail_route(request):
|
||||
return await self.job_detail(request)
|
||||
|
||||
+10
-49
@@ -72,9 +72,6 @@ SCAN_GROUPS = {
|
||||
IMAGE_TAG_JOB_KIND = "image-tags"
|
||||
IMAGE_TAG_GROUP_NAMES = ("tagged", "unchanged", "skipped", "errors")
|
||||
|
||||
# Thumbnail generation (#249 System S5): previews written, already fresh, failed.
|
||||
THUMBNAIL_GROUP_NAMES = ("generated", "skipped", "errors")
|
||||
|
||||
ADMISSIONS = ("read", "guarded", "records")
|
||||
# What each admission class is refused beside (besides its own kind).
|
||||
_REFUSED_BESIDE = {
|
||||
@@ -250,32 +247,6 @@ class Job:
|
||||
)
|
||||
self.log_line(log_line)
|
||||
|
||||
def record_counted(self, group, count, rows=()):
|
||||
"""Count ``count`` outcomes in ``group`` at once, listing ``rows``.
|
||||
|
||||
For outcomes that need no log line each (a thumbnail job's already
|
||||
fresh previews): the counter is exact, ``rows`` (``record``'s row
|
||||
shape) are listed up to MAX_OUTCOME_ROWS, and nothing is logged.
|
||||
"""
|
||||
if group not in self.counters:
|
||||
raise ValueError(f"Unknown outcome group {group!r} for {self.kind}")
|
||||
self.counters[group] += max(0, int(count))
|
||||
listed = self.outcomes[group]
|
||||
for row in rows:
|
||||
if len(listed) >= MAX_OUTCOME_ROWS:
|
||||
break
|
||||
index = row.get("root_index")
|
||||
if isinstance(index, bool) or not isinstance(index, int) or index < 0:
|
||||
index = None
|
||||
listed.append(
|
||||
{
|
||||
"path": _relative_path(row.get("path")),
|
||||
"root": str(row.get("root") or ""),
|
||||
"root_index": index,
|
||||
"detail": str(row.get("detail") or ""),
|
||||
}
|
||||
)
|
||||
|
||||
def finish(self, state="finished"):
|
||||
"""End the job in ``state`` and release every subscriber.
|
||||
|
||||
@@ -292,24 +263,18 @@ class Job:
|
||||
for queue in list(self._subscribers):
|
||||
queue.put_nowait(None)
|
||||
|
||||
async def events(self, lead=None):
|
||||
async def events(self):
|
||||
"""Yield the latest event, then each later one, until the job finishes.
|
||||
|
||||
``lead``, when given, is yielded first, after the subscription is in
|
||||
place, so nothing published while the consumer sends it is missed.
|
||||
Consume it under ``contextlib.aclosing`` so a consumer that stops early
|
||||
(a closed socket) drops its queue at once instead of at garbage collection.
|
||||
"""
|
||||
queue = asyncio.Queue()
|
||||
self._subscribers.add(queue)
|
||||
# As of the subscription: a later event or the end reaches the queue.
|
||||
snapshot, ended = self.last_event, self.finished
|
||||
try:
|
||||
if lead is not None:
|
||||
yield lead
|
||||
if snapshot is not None:
|
||||
yield snapshot
|
||||
if ended:
|
||||
if self.last_event is not None:
|
||||
yield self.last_event
|
||||
if self.finished:
|
||||
return
|
||||
while True:
|
||||
payload = await queue.get()
|
||||
@@ -460,24 +425,20 @@ class JobRegistry:
|
||||
self._jobs[job.id] = job
|
||||
return job
|
||||
|
||||
def blocker(self, kind, *, admission=None, record_changing=None, compatible=()):
|
||||
def blocker(self, kind, *, admission=None, record_changing=None):
|
||||
"""The running job a new ``kind`` job would be refused for, or None.
|
||||
|
||||
The same kind refuses first; then the oldest running job whose class
|
||||
the new job's admission class may not run beside. Running jobs of a
|
||||
kind in ``compatible`` never refuse it: the caller knows that work
|
||||
cannot conflict with its own (the monitor's Needs review recording
|
||||
beside thumbnail work, which writes previews and no records).
|
||||
the new job's admission class may not run beside.
|
||||
"""
|
||||
same_kind = self.active(kind)
|
||||
if same_kind is not None:
|
||||
return same_kind
|
||||
refused = _REFUSED_BESIDE[resolve_admission(admission, record_changing)]
|
||||
# A snapshot: the monitor's worker thread asks too (Needs review).
|
||||
running = [job for job in tuple(self._jobs.values()) if job.running]
|
||||
running = [job for job in self._jobs.values() if job.running]
|
||||
running.sort(key=lambda job: job._started_seq)
|
||||
for job in running:
|
||||
if job.admission in refused and job.kind not in compatible:
|
||||
if job.admission in refused:
|
||||
return job
|
||||
return None
|
||||
|
||||
@@ -505,8 +466,8 @@ class JobRegistry:
|
||||
return self._jobs.get(job_id)
|
||||
|
||||
def active(self, kind):
|
||||
"""The running job of ``kind``, or None (safe off the event loop)."""
|
||||
for job in tuple(self._jobs.values()):
|
||||
"""The running job of ``kind``, or None."""
|
||||
for job in self._jobs.values():
|
||||
if job.kind == kind and job.running:
|
||||
return job
|
||||
return None
|
||||
|
||||
@@ -1,706 +0,0 @@
|
||||
"""Needs review routes (#249 E3): the pending queue, item detail, decisions.
|
||||
|
||||
``GET /prompt_manager/review/items`` lists pending items oldest first behind
|
||||
an HMAC keyset cursor bound to the storage epoch only (a restore makes an
|
||||
old cursor stale; a decision or a new item does not, because the
|
||||
``(created_at, id)`` keyset stays ordered when items leave or are appended);
|
||||
``total`` is the exact pending count. ``GET .../items/{id}`` adds the
|
||||
candidates and the dependency fingerprint. ``POST .../items/{id}/decision``
|
||||
records one explicit decision: Apply links the chosen sampler's pair only
|
||||
when the capture is the item's bytes, the current file still hashes to them,
|
||||
the evidence they yield is unchanged and the ingester's compatible-only rules
|
||||
allow it; Leave unresolved links nothing.
|
||||
|
||||
Targets are formatted from the recorded spelling against the current roots
|
||||
(no file access beyond the roots' real paths); a file under no current root
|
||||
is ``unavailable``. Responses carry no server path, no locator and no
|
||||
exception text.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import urllib.parse
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
from .annotations import _HEX64, _TOKEN, _UUID, _capture_error, _error
|
||||
from .images import media_thumbnail_url
|
||||
from .inspection_service import VIDEO_EXTENSIONS
|
||||
from .prompts import bad_request
|
||||
|
||||
try:
|
||||
from ...database.library import display_title
|
||||
from ...database.library_commands import LibraryCommandError
|
||||
from ...database.review_decisions import ReviewCommandError
|
||||
from ...utils.media_capture import CaptureError
|
||||
from ...utils.review_capture import captured_review, review_evidence
|
||||
except ImportError:
|
||||
from database.library import display_title
|
||||
from database.library_commands import LibraryCommandError
|
||||
from database.review_decisions import ReviewCommandError
|
||||
from utils.media_capture import CaptureError
|
||||
from utils.review_capture import captured_review, review_evidence
|
||||
|
||||
ITEMS_URL = "/prompt_manager/review/items"
|
||||
DEFAULT_LIMIT = 40
|
||||
MAX_LIMIT = 40
|
||||
MAX_CURSOR = 4096
|
||||
MAX_BODY_BYTES = 64 * 1024
|
||||
# The serialized detail stays under this; past it the detail is ``limited``.
|
||||
MAX_DETAIL_BYTES = 8 * 1024 * 1024
|
||||
UNRESOLVED = "unresolved"
|
||||
_DECISION_FIELDS = frozenset(
|
||||
{
|
||||
"expected_epoch",
|
||||
"expected_revision",
|
||||
"capture_token",
|
||||
"sha256",
|
||||
"dependency_fingerprint",
|
||||
"choice",
|
||||
}
|
||||
)
|
||||
_CURSOR_SECRET = os.urandom(32)
|
||||
# The real path of a spelling; module-level so a test can fake a junction.
|
||||
_realpath = os.path.realpath
|
||||
_LIMIT = re.compile(r"[0-9]{1,3}")
|
||||
_NO_STORE = {"Cache-Control": "no-store"}
|
||||
|
||||
_STATUS = {
|
||||
"review-missing": 404,
|
||||
"review-conflict": 409,
|
||||
"association-conflict": 409,
|
||||
"file-changed": 409,
|
||||
"storage-changed": 409,
|
||||
"source-changed": 409,
|
||||
"capture-expired": 409,
|
||||
}
|
||||
_MESSAGES = {
|
||||
"review-missing": "This review item no longer exists. Reload the queue.",
|
||||
"review-conflict": (
|
||||
"This item was decided or changed since it was opened. Reload the queue."
|
||||
),
|
||||
"association-conflict": (
|
||||
"This image is already linked to another prompt or negative, so this "
|
||||
"sampler path cannot be attached safely. The existing links were "
|
||||
"preserved."
|
||||
),
|
||||
"dependencies-changed": (
|
||||
"The image's saved links changed since this item was opened. Reopen it "
|
||||
"to see the current association."
|
||||
),
|
||||
"file-changed": (
|
||||
"The file changed since it was reviewed. A metadata scan adds its new "
|
||||
"bytes as a new item."
|
||||
),
|
||||
"storage-changed": "Prompt storage was restored; reload the review queue.",
|
||||
"source-changed": (
|
||||
"The source changed. Refresh the image collection before reviewing this "
|
||||
"file."
|
||||
),
|
||||
"capture-expired": "The preview expired. Open the item again.",
|
||||
"other-file": "The preview belongs to another file. Open the item again.",
|
||||
"evidence-changed": (
|
||||
"This file's metadata no longer gives the evidence that was reviewed. "
|
||||
"Reload the queue."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
_DEFAULT_MESSAGE = "The decision could not be saved. Reload the queue."
|
||||
|
||||
|
||||
def _coded(code: str, message_key: str | None = None, **extra) -> web.Response:
|
||||
message = _MESSAGES.get(message_key or code, _DEFAULT_MESSAGE)
|
||||
return _error(_STATUS.get(code, 409), code, message, **extra)
|
||||
|
||||
|
||||
def _failed() -> web.Response:
|
||||
return _error(500, "review-failed", "The review queue could not be read.")
|
||||
|
||||
|
||||
def format_review_target(spelled_path, locator, real_roots, *, pathmod=os.path):
|
||||
"""``(root_index, relative)`` of the first root that lists the file, or None.
|
||||
|
||||
``spelled_path`` is the file's recorded spelling, ``locator`` its folded
|
||||
identity and ``real_roots`` ``(index, realpath)`` pairs. The relative
|
||||
path keeps the recorded spelling (a Windows locator is case-folded, a
|
||||
listing target is not) and must fold back to the locator.
|
||||
"""
|
||||
sep = pathmod.sep
|
||||
for index, real_root in real_roots:
|
||||
head = spelled_path[: len(real_root)]
|
||||
if pathmod.normcase(head) != pathmod.normcase(real_root):
|
||||
continue
|
||||
rest = spelled_path[len(real_root) :]
|
||||
if not real_root.endswith(sep):
|
||||
if not rest.startswith(sep):
|
||||
continue
|
||||
rest = rest[1:]
|
||||
parts = rest.split(sep)
|
||||
if any(
|
||||
part in ("", ".", "..") or "/" in part or "\\" in part or ":" in part
|
||||
for part in parts
|
||||
):
|
||||
continue
|
||||
if pathmod.normcase(pathmod.join(real_root, *parts)) != locator:
|
||||
continue
|
||||
return index, "/".join(parts)
|
||||
return None
|
||||
|
||||
|
||||
class _StaleCursor(ValueError):
|
||||
"""A well-formed cursor this process cannot verify (e.g. after a restart)."""
|
||||
|
||||
|
||||
def _encode_cursor(epoch: str, key: tuple) -> str:
|
||||
payload = json.dumps(
|
||||
{"v": 2, "e": epoch, "k": list(key)},
|
||||
separators=(",", ":"),
|
||||
ensure_ascii=True,
|
||||
).encode()
|
||||
signature = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
|
||||
return base64.urlsafe_b64encode(signature + payload).decode().rstrip("=")
|
||||
|
||||
|
||||
def _decode_cursor(token: str) -> tuple:
|
||||
"""``(epoch, (created_at, id))`` of a cursor.
|
||||
|
||||
``ValueError`` when malformed; :class:`_StaleCursor` when well-formed but
|
||||
signed by another secret (the per-process secret changes on every start,
|
||||
so an open page's cursor is stale after a restart, not forged).
|
||||
"""
|
||||
try:
|
||||
if len(token) > MAX_CURSOR:
|
||||
raise ValueError()
|
||||
raw = base64.b64decode(
|
||||
token + "=" * (-len(token) % 4), altchars=b"-_", validate=True
|
||||
)
|
||||
signature, payload = raw[:32], raw[32:]
|
||||
if len(signature) != 32 or not payload:
|
||||
raise ValueError()
|
||||
expected = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
|
||||
if not hmac.compare_digest(signature, expected):
|
||||
raise _StaleCursor("This page of the queue is out of date")
|
||||
data = json.loads(payload)
|
||||
key = data["k"]
|
||||
if (
|
||||
data["v"] != 2
|
||||
or not isinstance(data["e"], str)
|
||||
or not isinstance(key, list)
|
||||
or len(key) != 2
|
||||
or not all(isinstance(part, str) for part in key)
|
||||
):
|
||||
raise ValueError()
|
||||
return data["e"], tuple(key)
|
||||
except _StaleCursor:
|
||||
raise
|
||||
except (ValueError, KeyError, TypeError, UnicodeError, binascii.Error):
|
||||
raise ValueError("Invalid review cursor; reload the queue") from None
|
||||
|
||||
|
||||
def _limit(value):
|
||||
if value is None:
|
||||
return DEFAULT_LIMIT
|
||||
if _LIMIT.fullmatch(value) is None or not 1 <= int(value) <= MAX_LIMIT:
|
||||
return None
|
||||
return int(value)
|
||||
|
||||
|
||||
def _side_text(side) -> str | None:
|
||||
if isinstance(side, dict) and isinstance(side.get("text"), str):
|
||||
return side["text"]
|
||||
return None
|
||||
|
||||
|
||||
def _workable(candidate: dict) -> bool:
|
||||
return (
|
||||
candidate.get("state") == "resolved"
|
||||
and _side_text(candidate.get("positive")) is not None
|
||||
and _side_text(candidate.get("negative")) is not None
|
||||
)
|
||||
|
||||
|
||||
def _side_label(titles: dict, side, empty: str) -> str:
|
||||
"""The Library title of the side's saved prompt, else its first line."""
|
||||
text = _side_text(side)
|
||||
if text is None:
|
||||
return empty
|
||||
stored = side.get("storage_text")
|
||||
stored = stored if isinstance(stored, str) else text
|
||||
return display_title(titles.get(stored, ()), text) or empty
|
||||
|
||||
|
||||
class ReviewRoutesMixin:
|
||||
"""The Needs review queue over E1's schema and E2's recorded items."""
|
||||
|
||||
def _register_review_routes(self, routes: web.RouteTableDef) -> None:
|
||||
routes.get(ITEMS_URL)(self.list_review_items)
|
||||
routes.get(ITEMS_URL + "/{item_id}")(self.get_review_item)
|
||||
routes.post(ITEMS_URL + "/{item_id}/decision")(self.decide_review_item)
|
||||
|
||||
# -- shared ----------------------------------------------------------------
|
||||
|
||||
def _review_roots(self):
|
||||
"""``(roots, real_roots)``: the current gallery roots and their realpaths.
|
||||
|
||||
Blocking (it lists the roots and resolves them): call it on a worker.
|
||||
"""
|
||||
roots = list(self._get_all_output_dirs())
|
||||
return roots, [(i, _realpath(str(root))) for i, root in enumerate(roots)]
|
||||
|
||||
def _review_view(self, row: dict, roots, real_roots) -> dict:
|
||||
"""The public ReviewItem of one stored row (no path, no locator)."""
|
||||
found = format_review_target(row["spelled_path"], row["locator"], real_roots)
|
||||
target = url = thumbnail_url = None
|
||||
if found is not None:
|
||||
index, relative = found
|
||||
target = {
|
||||
"root_index": index,
|
||||
"root_scope": self._gallery_root_scope(roots[index], index),
|
||||
"path": relative,
|
||||
}
|
||||
# The Images listing's media URL for the same file (formatting only).
|
||||
url = (
|
||||
"/prompt_manager/images/serve/"
|
||||
f"{urllib.parse.quote(relative, safe='/')}?root={index}"
|
||||
f"&root_scope={target['root_scope']}"
|
||||
)
|
||||
# The Images listing's thumbnail, when one exists (one stat).
|
||||
thumbnail_url = media_thumbnail_url(
|
||||
roots[index], relative, index, root_scope=target["root_scope"]
|
||||
)
|
||||
filename = os.path.basename(row["spelled_path"])
|
||||
return {
|
||||
"id": row["id"],
|
||||
"state": row["state"],
|
||||
"revision": row["revision"],
|
||||
"created_at": row["created_at"],
|
||||
"filename": filename,
|
||||
"target": target,
|
||||
"unavailable": target is None,
|
||||
"candidate_count": row["candidate_count"],
|
||||
"limited": row["limited"],
|
||||
"url": url,
|
||||
"thumbnail_url": thumbnail_url,
|
||||
"is_video": os.path.splitext(filename)[1].lower() in VIDEO_EXTENSIONS,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _review_image_path(row: dict, roots, target) -> str:
|
||||
"""The path the scan and the monitor link this file under."""
|
||||
if target is None:
|
||||
return row["spelled_path"]
|
||||
root = roots[target["root_index"]]
|
||||
return os.path.join(str(root), *target["path"].split("/"))
|
||||
|
||||
# -- queue -----------------------------------------------------------------
|
||||
|
||||
async def list_review_items(self, request: web.Request) -> web.Response:
|
||||
"""GET ``?limit=1..40&cursor=``: one page of pending items."""
|
||||
limit = _limit(request.query.get("limit"))
|
||||
if limit is None:
|
||||
return bad_request(f"limit must be an integer from 1 to {MAX_LIMIT}")
|
||||
bound = None
|
||||
cursor = request.query.get("cursor")
|
||||
if cursor:
|
||||
try:
|
||||
bound = _decode_cursor(cursor)
|
||||
except _StaleCursor:
|
||||
return await self._stale_cursor()
|
||||
except ValueError as exc:
|
||||
return bad_request(str(exc))
|
||||
try:
|
||||
page, views = await self._run_in_executor(
|
||||
self._review_page, limit, bound[1] if bound else None
|
||||
)
|
||||
except Exception:
|
||||
self.logger.exception("Review queue read failed")
|
||||
return _failed()
|
||||
if bound is not None and bound[0] != page["epoch"]:
|
||||
return _coded("storage-changed", library_epoch=page["epoch"])
|
||||
items = page["items"]
|
||||
next_cursor = None
|
||||
if page["more"] and items:
|
||||
last = items[-1]
|
||||
next_cursor = _encode_cursor(
|
||||
page["epoch"], (last["created_at"], last["id"])
|
||||
)
|
||||
return web.json_response(
|
||||
{
|
||||
"success": True,
|
||||
"library_epoch": page["epoch"],
|
||||
# F3: the queue version the change stream's ``ready`` names.
|
||||
"review_revision": page["revision"],
|
||||
"total": page["total"],
|
||||
"items": views,
|
||||
"next_cursor": next_cursor,
|
||||
},
|
||||
headers=_NO_STORE,
|
||||
)
|
||||
|
||||
async def _stale_cursor(self) -> web.Response:
|
||||
"""409 ``review-conflict`` with the current total: reload the queue."""
|
||||
try:
|
||||
page = await self._run_in_executor(self.db.review_queue_page, limit=1)
|
||||
except Exception:
|
||||
self.logger.exception("Review queue read failed")
|
||||
return _failed()
|
||||
return _coded(
|
||||
"review-conflict", total=page["total"], library_epoch=page["epoch"]
|
||||
)
|
||||
|
||||
def _review_page(self, limit: int, after):
|
||||
"""Worker thread: one queue page and the public views of its items."""
|
||||
page = self.db.review_queue_page(limit=limit, after=after)
|
||||
roots, real_roots = self._review_roots()
|
||||
return page, [
|
||||
self._review_view(row, roots, real_roots) for row in page["items"]
|
||||
]
|
||||
|
||||
# -- detail ----------------------------------------------------------------
|
||||
|
||||
async def get_review_item(self, request: web.Request) -> web.Response:
|
||||
"""GET one item with its candidates and dependency fingerprint."""
|
||||
try:
|
||||
read = await self._run_in_executor(
|
||||
self._review_detail, request.match_info["item_id"]
|
||||
)
|
||||
except Exception:
|
||||
self.logger.exception("Review item read failed")
|
||||
return _failed()
|
||||
if read is None:
|
||||
return _coded("review-missing")
|
||||
item, view, fingerprint, candidates = read
|
||||
detail = {
|
||||
"success": True,
|
||||
"library_epoch": item["epoch"],
|
||||
**view,
|
||||
"candidates": candidates,
|
||||
"dependency_fingerprint": fingerprint,
|
||||
"sha256": item["sha256"],
|
||||
"total": item["total"],
|
||||
}
|
||||
if len(json.dumps(detail).encode("utf-8")) > MAX_DETAIL_BYTES:
|
||||
detail = {**detail, "candidates": [], "limited": True}
|
||||
return web.json_response(detail, headers=_NO_STORE)
|
||||
|
||||
def _review_detail(self, item_id: str):
|
||||
"""Worker thread: ``(item, view, fingerprint, candidates)`` or None."""
|
||||
item = self.db.review_item(item_id)
|
||||
if item is None:
|
||||
return None
|
||||
roots, real_roots = self._review_roots()
|
||||
view = self._review_view(item, roots, real_roots)
|
||||
image_path = self._review_image_path(item, roots, view["target"])
|
||||
aliases, _known = physical_spellings(self.db, image_path, item["locator"])
|
||||
fingerprint = self.db.review_dependency_fingerprint(image_path, aliases)
|
||||
return item, view, fingerprint, self._review_candidates(item)
|
||||
|
||||
def _review_candidates(self, item: dict) -> list:
|
||||
"""The detail's candidates: role texts, a label and selectability."""
|
||||
if item["limited"]:
|
||||
return []
|
||||
candidates = [
|
||||
c for c in item["evidence"].get("candidates", []) if isinstance(c, dict)
|
||||
]
|
||||
texts = set()
|
||||
for candidate in candidates:
|
||||
for role in ("positive", "negative"):
|
||||
side = candidate.get(role)
|
||||
if isinstance(side, dict):
|
||||
for key in ("text", "storage_text"):
|
||||
if isinstance(side.get(key), str):
|
||||
texts.add(side[key])
|
||||
titles = self.db.stored_prompt_titles(texts)
|
||||
rows = []
|
||||
for candidate in candidates:
|
||||
positive = _side_text(candidate.get("positive"))
|
||||
negative = _side_text(candidate.get("negative"))
|
||||
label = (
|
||||
_side_label(titles, candidate.get("positive"), "No positive")
|
||||
+ " / "
|
||||
+ _side_label(titles, candidate.get("negative"), "No negative")
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"hash": candidate.get("hash"),
|
||||
"sampler_id": candidate.get("sampler_id"),
|
||||
"state": "resolved" if _workable(candidate) else "unresolved",
|
||||
"reason": candidate.get("reason") or "",
|
||||
"label": label,
|
||||
"positive": {"text": positive} if positive is not None else None,
|
||||
"negative": {"text": negative} if negative is not None else None,
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
# -- decision --------------------------------------------------------------
|
||||
|
||||
async def decide_review_item(self, request: web.Request) -> web.Response:
|
||||
"""POST one explicit decision on one pending item."""
|
||||
data, error = await self._read_json_body(request, max_bytes=MAX_BODY_BYTES)
|
||||
if error is not None:
|
||||
return error
|
||||
guards, error = _decision_guards(data)
|
||||
if error is not None:
|
||||
return error
|
||||
try:
|
||||
item = await self._run_in_executor(
|
||||
self.db.review_item, request.match_info["item_id"]
|
||||
)
|
||||
except Exception:
|
||||
self.logger.exception("Review item read failed")
|
||||
return _failed()
|
||||
if item is None:
|
||||
return _coded("review-missing")
|
||||
choice = guards["choice"]
|
||||
candidate = None
|
||||
if choice != UNRESOLVED:
|
||||
candidate = next(
|
||||
(
|
||||
c
|
||||
for c in item["evidence"].get("candidates", [])
|
||||
if isinstance(c, dict) and c.get("hash") == choice
|
||||
),
|
||||
None,
|
||||
)
|
||||
if candidate is None or item["limited"]:
|
||||
return bad_request("choice is not one of this item's candidates")
|
||||
if not _workable(candidate):
|
||||
return bad_request("This sampler path cannot be applied")
|
||||
if guards["capture_token"] is None:
|
||||
return bad_request("Apply needs a capture of the file")
|
||||
verified = None
|
||||
if guards["capture_token"] is not None:
|
||||
verified, error = await self._verified_review_capture(
|
||||
item, guards, need_metadata=candidate is not None
|
||||
)
|
||||
if error is not None:
|
||||
return error
|
||||
elif guards["sha256"] != item["sha256"]:
|
||||
return bad_request("sha256 does not match the item")
|
||||
return await self._record_decision(item, guards, candidate, verified)
|
||||
|
||||
async def _verified_review_capture(self, item, guards, *, need_metadata):
|
||||
"""``((capture, facts, image_path, metadata), None)`` or ``(None, error)``.
|
||||
|
||||
The capture must be the item's file and bytes, under the expected
|
||||
epoch and a current root; the current file must re-hash to the same
|
||||
bytes; for Apply, the capture's metadata must still yield exactly the
|
||||
reviewed evidence.
|
||||
"""
|
||||
store = self._captures
|
||||
capture = store.get(guards["capture_token"]) if store is not None else None
|
||||
if capture is None:
|
||||
return None, _coded("capture-expired")
|
||||
if capture.locator != item["locator"]:
|
||||
return None, _coded("capture-expired", "other-file")
|
||||
if guards["sha256"] != capture.sha256:
|
||||
return None, bad_request("sha256 does not match the preview")
|
||||
if capture.sha256 != item["sha256"]:
|
||||
return None, _coded("file-changed")
|
||||
if guards["expected_epoch"] != capture.epoch:
|
||||
current = await self._run_in_executor(self.db.get_execution_epoch)
|
||||
return None, _coded("storage-changed", library_epoch=current)
|
||||
try:
|
||||
root, error = await self._run_in_executor(
|
||||
self._scoped_root, capture.root_index, capture.scope
|
||||
)
|
||||
if error is not None:
|
||||
return None, error
|
||||
read = await self._run_in_executor(
|
||||
_verify_and_read, store, capture, item, root, need_metadata
|
||||
)
|
||||
except CaptureError as exc:
|
||||
return None, _capture_error(exc)
|
||||
if read is None:
|
||||
return None, _coded("source-changed")
|
||||
relative, facts, captured = read
|
||||
if facts.locator != item["locator"]:
|
||||
return None, _coded("capture-expired", "other-file")
|
||||
if facts.sha256 != item["sha256"]:
|
||||
return None, _coded("file-changed")
|
||||
metadata = None
|
||||
if need_metadata:
|
||||
if captured is None or not _candidate_still_offered(
|
||||
captured[1], guards["choice"]
|
||||
):
|
||||
return None, _coded("review-conflict", "evidence-changed")
|
||||
metadata = captured[0]
|
||||
image_path = os.path.join(str(root), *relative.split("/"))
|
||||
return (capture, facts, image_path, metadata), None
|
||||
|
||||
def _decide_sync(self, item, guards, candidate, verified):
|
||||
"""Worker thread: resolve the file's spellings, decide, build the view.
|
||||
|
||||
Returns ``(result, view)``; a refusal raises with ``exc.view`` set.
|
||||
"""
|
||||
_capture, facts, image_path, metadata = verified or (None, None, None, None)
|
||||
aliases, known = (), None
|
||||
if candidate is not None:
|
||||
aliases, known = physical_spellings(self.db, image_path, item["locator"])
|
||||
try:
|
||||
result = self.db.decide_review_item(
|
||||
item_id=item["id"],
|
||||
expected_epoch=guards["expected_epoch"],
|
||||
expected_revision=guards["expected_revision"],
|
||||
# The bytes the re-hash proved, or the body's for Leave unresolved.
|
||||
sha256=facts.sha256 if facts is not None else guards["sha256"],
|
||||
outcome="selected" if candidate is not None else UNRESOLVED,
|
||||
candidate=candidate,
|
||||
image_path=image_path,
|
||||
metadata=metadata,
|
||||
expected_fingerprint=guards["dependency_fingerprint"],
|
||||
aliases=aliases,
|
||||
known_rows=known,
|
||||
)
|
||||
except ReviewCommandError as exc:
|
||||
if exc.item is not None:
|
||||
roots, real_roots = self._review_roots()
|
||||
exc.view = self._review_view(exc.item, roots, real_roots)
|
||||
raise
|
||||
roots, real_roots = self._review_roots()
|
||||
return result, self._review_view(result["item"], roots, real_roots)
|
||||
|
||||
async def _record_decision(self, item, guards, candidate, verified):
|
||||
capture = verified[0] if verified else None
|
||||
try:
|
||||
result, view = await self._run_in_executor(
|
||||
self._decide_sync, item, guards, candidate, verified
|
||||
)
|
||||
except ReviewCommandError as exc:
|
||||
return self._decision_refused(exc)
|
||||
except LibraryCommandError as exc:
|
||||
return _coded(exc.code, library_epoch=exc.epoch)
|
||||
except Exception:
|
||||
self.logger.exception("Review decision failed")
|
||||
return _error(500, "decision-failed", "The decision could not be saved.")
|
||||
if capture is not None:
|
||||
# The decision used the capture; the page returns to Images.
|
||||
self._captures.release(capture.token)
|
||||
body = {
|
||||
"success": True,
|
||||
"library_epoch": guards["expected_epoch"],
|
||||
"item": view,
|
||||
"total": result["total"],
|
||||
}
|
||||
if result["pair"] is not None:
|
||||
body["pair"] = result["pair"]
|
||||
return web.json_response(body, headers=_NO_STORE)
|
||||
|
||||
def _decision_refused(self, exc: ReviewCommandError) -> web.Response:
|
||||
extra = {"library_epoch": exc.epoch}
|
||||
if exc.total is not None:
|
||||
extra["total"] = exc.total
|
||||
if exc.view is not None:
|
||||
extra["item"] = exc.view
|
||||
message = None
|
||||
if exc.fingerprint is not None:
|
||||
extra["dependency_fingerprint"] = exc.fingerprint
|
||||
if exc.code == "association-conflict" and exc.changed:
|
||||
message = "dependencies-changed"
|
||||
return _coded(exc.code, message, **extra)
|
||||
|
||||
|
||||
def _decision_guards(data: dict):
|
||||
"""The validated decision fields, or ``(None, error)``."""
|
||||
if set(data) != _DECISION_FIELDS:
|
||||
return None, bad_request(
|
||||
"Request must contain exactly " + ", ".join(sorted(_DECISION_FIELDS))
|
||||
)
|
||||
epoch, revision = data["expected_epoch"], data["expected_revision"]
|
||||
if not isinstance(epoch, str) or _UUID.fullmatch(epoch) is None:
|
||||
return None, bad_request("expected_epoch must be a canonical UUID")
|
||||
if type(revision) is not int or revision < 0:
|
||||
return None, bad_request("expected_revision must be a non-negative integer")
|
||||
token = data["capture_token"]
|
||||
if token is not None and (
|
||||
not isinstance(token, str) or _TOKEN.fullmatch(token) is None
|
||||
):
|
||||
return None, bad_request("Invalid capture token")
|
||||
for name in ("sha256", "dependency_fingerprint"):
|
||||
value = data[name]
|
||||
if not isinstance(value, str) or _HEX64.fullmatch(value) is None:
|
||||
return None, bad_request(
|
||||
f"{name} must be 64 lowercase hexadecimal characters"
|
||||
)
|
||||
choice = data["choice"]
|
||||
if not isinstance(choice, str) or (
|
||||
choice != UNRESOLVED and _HEX64.fullmatch(choice) is None
|
||||
):
|
||||
return None, bad_request("choice must be a candidate hash or unresolved")
|
||||
return {**data}, None
|
||||
|
||||
|
||||
def _verify_and_read(store, capture, item, root, need_metadata: bool):
|
||||
"""Worker thread: ``(relative, facts, captured)``, or None off the root.
|
||||
|
||||
Formats the item's target under the capture's root (resolving the root's
|
||||
real path), re-hashes the current file, then reads the capture.
|
||||
"""
|
||||
found = format_review_target(
|
||||
item["spelled_path"],
|
||||
item["locator"],
|
||||
[(capture.root_index, _realpath(str(root)))],
|
||||
)
|
||||
if found is None:
|
||||
return None
|
||||
relative = found[1]
|
||||
facts = store.verify(root, relative)
|
||||
captured = captured_review(store, capture) if need_metadata else None
|
||||
return relative, facts, captured
|
||||
|
||||
|
||||
def _same_file(spelled: str, image_path: str, locator: str) -> bool:
|
||||
"""True when ``spelled`` reaches the physical file at ``locator``."""
|
||||
try:
|
||||
if os.path.normcase(_realpath(spelled)) == locator:
|
||||
return True
|
||||
return os.path.samefile(spelled, image_path)
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def physical_spellings(db, image_path: str, locator: str):
|
||||
"""``(aliases, known_rows)`` for the legacy rows of one physical file.
|
||||
|
||||
The roots are real paths, while the monitor links a file under the
|
||||
directory it watches, so a symlinked or junctioned root, a ``subst`` or a
|
||||
mapped drive spells the same file otherwise. Every row whose spelling
|
||||
ends with the file's name (the link summaries' fallback lookup) is a
|
||||
candidate; ``aliases`` are the candidates that resolve to ``locator``
|
||||
(``normcase(realpath)``, as the capture forms it) or are the same file,
|
||||
and ``known_rows`` every candidate id, so a decision can refuse a
|
||||
same-name row it did not classify. Blocking: call it on a worker.
|
||||
"""
|
||||
rows = db.review_same_name_rows(os.path.basename(image_path))
|
||||
aliases = tuple(
|
||||
spelled
|
||||
for _row_id, spelled in rows
|
||||
if isinstance(spelled, str)
|
||||
and spelled != image_path
|
||||
and _same_file(spelled, image_path, locator)
|
||||
)
|
||||
return tuple(dict.fromkeys(aliases)), frozenset(row_id for row_id, _ in rows)
|
||||
|
||||
|
||||
def _candidate_still_offered(resolution: dict, choice: str) -> bool:
|
||||
"""True when the captured bytes still offer the chosen candidate.
|
||||
|
||||
Only what the decision depends on is re-checked: the bytes are still
|
||||
``ambiguous`` and the chosen candidate (its hash covers its sampler,
|
||||
sides and provenance) is still among their candidates. A resolver change
|
||||
that rewords the top-level reason or reorders candidates leaves a
|
||||
pending item applicable.
|
||||
"""
|
||||
if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous":
|
||||
return False
|
||||
evidence, _count, limited = review_evidence(resolution)
|
||||
if limited:
|
||||
return False
|
||||
return any(c.get("hash") == choice for c in evidence.get("candidates", ()))
|
||||
@@ -1,171 +0,0 @@
|
||||
"""Needs review change stream (#249 PR F2).
|
||||
|
||||
``GET /prompt_manager/review/events`` is a server-sent event stream for an
|
||||
open Needs review page: ``event: ready`` when it opens, then
|
||||
``event: change`` after each committed queue change (an item the scan or the
|
||||
monitor recorded, a decision from any tab, a spelling refresh) and after a
|
||||
restore. The first notice is sent at once; notices in the next
|
||||
``COALESCE_SECONDS`` collapse into one trailing ``change`` (M9: a busy scan
|
||||
records file after file, and each event costs every open page a re-read).
|
||||
The page re-reads its queue on each ``change``; nothing polls. ``change``
|
||||
carries no data (``{}``). ``ready`` carries the queue's version
|
||||
``{library_epoch, review_revision}``, read after the stream subscribed, the
|
||||
same pair a list read answers: a page whose first read already answered that
|
||||
version skips the re-read (F3, one read per visit), and any later change is a
|
||||
``change`` event. Nothing else about the queue is pushed.
|
||||
|
||||
A ``: keepalive`` comment every ``HEARTBEAT_SECONDS`` lets the server notice
|
||||
a closed tab (its write fails) and drop the subscription; clients ignore
|
||||
comments. Shutdown (``close_review_events``, an ``on_shutdown`` hook) ends
|
||||
every stream, and a stream asked for afterwards is a 503.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
try:
|
||||
from ...database.execution_history import subscribe_history_restore
|
||||
from ...database.review_changes import subscribe_review_changes
|
||||
except ImportError:
|
||||
from database.execution_history import subscribe_history_restore
|
||||
from database.review_changes import subscribe_review_changes
|
||||
|
||||
EVENTS_URL = "/prompt_manager/review/events"
|
||||
HEARTBEAT_SECONDS = 25
|
||||
# After a ``change``, the next one waits this long, collapsing the notices in between.
|
||||
COALESCE_SECONDS = 0.4
|
||||
_CHANGE = b"event: change\ndata: {}\n\n"
|
||||
_KEEPALIVE = b": keepalive\n\n"
|
||||
|
||||
|
||||
def _ready(version: dict) -> bytes:
|
||||
data = json.dumps(version, separators=(",", ":"), sort_keys=True)
|
||||
return f"event: ready\ndata: {data}\n\n".encode("utf-8")
|
||||
|
||||
|
||||
class _Stream:
|
||||
"""One open stream: a change flag set from any thread, and a close flag."""
|
||||
|
||||
def __init__(self, loop: asyncio.AbstractEventLoop) -> None:
|
||||
self._loop = loop
|
||||
self.changed = asyncio.Event()
|
||||
self.closed = asyncio.Event()
|
||||
self.ready = False
|
||||
|
||||
def wake(self) -> None:
|
||||
"""Thread-safe: mark a change (the loop may already be gone)."""
|
||||
try:
|
||||
self._loop.call_soon_threadsafe(self.changed.set)
|
||||
except RuntimeError: # the loop closed
|
||||
pass
|
||||
|
||||
async def next(self, timeout: float) -> str:
|
||||
"""``change``, ``closed`` or ``idle`` (nothing within ``timeout``)."""
|
||||
waits = {
|
||||
asyncio.ensure_future(self.changed.wait()): "change",
|
||||
asyncio.ensure_future(self.closed.wait()): "closed",
|
||||
}
|
||||
try:
|
||||
done, _ = await asyncio.wait(
|
||||
waits, timeout=timeout, return_when=asyncio.FIRST_COMPLETED
|
||||
)
|
||||
finally:
|
||||
for waiter in waits:
|
||||
waiter.cancel()
|
||||
if self.closed.is_set():
|
||||
return "closed"
|
||||
if done:
|
||||
self.changed.clear()
|
||||
return "change"
|
||||
return "idle"
|
||||
|
||||
async def closed_within(self, timeout: float) -> bool:
|
||||
"""Wait up to ``timeout``; true when the stream closed meanwhile."""
|
||||
try:
|
||||
await asyncio.wait_for(self.closed.wait(), timeout)
|
||||
except asyncio.TimeoutError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
class ReviewEventsMixin:
|
||||
"""The Needs review change stream."""
|
||||
|
||||
def _register_review_event_routes(self, routes: web.RouteTableDef) -> None:
|
||||
routes.get(EVENTS_URL)(self.review_events)
|
||||
|
||||
def _review_streams(self) -> set:
|
||||
streams = getattr(self, "_open_review_streams", None)
|
||||
if streams is None:
|
||||
streams = self._open_review_streams = set()
|
||||
return streams
|
||||
|
||||
def review_stream_count(self) -> int:
|
||||
"""How many change streams have sent ``ready`` and are open (tests and
|
||||
diagnostics): a queue change after this count moved is a ``change``
|
||||
event on them, never part of the version ``ready`` named."""
|
||||
return sum(1 for stream in self._review_streams() if stream.ready)
|
||||
|
||||
async def close_review_events(self, app) -> None:
|
||||
"""End every open stream; later requests get 503 (shutdown hook)."""
|
||||
self._review_events_closed = True
|
||||
for stream in list(self._review_streams()):
|
||||
stream.closed.set()
|
||||
|
||||
async def _queue_version(self) -> dict:
|
||||
"""The queue version for ``ready``, read after subscribing (so a change
|
||||
after this read is an event); ``{}`` when it cannot be read, and the
|
||||
page then re-reads."""
|
||||
try:
|
||||
return await self._run_in_executor(self.db.review_queue_version)
|
||||
except Exception:
|
||||
self.logger.warning("Needs review version read failed", exc_info=True)
|
||||
return {}
|
||||
|
||||
async def review_events(self, request: web.Request) -> web.StreamResponse:
|
||||
if getattr(self, "_review_events_closed", False):
|
||||
return web.json_response(
|
||||
{"success": False, "error": "The server is shutting down."},
|
||||
status=503,
|
||||
)
|
||||
stream = _Stream(asyncio.get_running_loop())
|
||||
db_path = self.db.model.db_path
|
||||
unsubscribers = (
|
||||
subscribe_review_changes(db_path, stream.wake),
|
||||
subscribe_history_restore(db_path, stream.wake),
|
||||
)
|
||||
streams = self._review_streams()
|
||||
streams.add(stream)
|
||||
response = web.StreamResponse(
|
||||
status=200,
|
||||
headers={
|
||||
"Content-Type": "text/event-stream",
|
||||
"Cache-Control": "no-store",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
try:
|
||||
version = await self._queue_version()
|
||||
await response.prepare(request)
|
||||
await response.write(_ready(version))
|
||||
stream.ready = True
|
||||
while True:
|
||||
outcome = await stream.next(HEARTBEAT_SECONDS)
|
||||
if outcome == "closed":
|
||||
break
|
||||
if outcome == "idle":
|
||||
await response.write(_KEEPALIVE)
|
||||
continue
|
||||
await response.write(_CHANGE)
|
||||
if await stream.closed_within(COALESCE_SECONDS):
|
||||
break
|
||||
await response.write_eof()
|
||||
except (ConnectionError, RuntimeError) as exc: # the tab went away
|
||||
self.logger.debug(f"Needs review stream closed ({exc})")
|
||||
finally:
|
||||
streams.discard(stream)
|
||||
for unsubscribe in unsubscribers:
|
||||
unsubscribe()
|
||||
return response
|
||||
@@ -1,251 +0,0 @@
|
||||
"""Needs review recording for the output scan and the live monitor (#249).
|
||||
|
||||
The rule (maintainer test of #284/#288): a file is queued only when its
|
||||
final outcome is ambiguous and unlinked. The strict resolver must find it
|
||||
``ambiguous``, the scan's legacy fallback (the browser parser,
|
||||
``_graph_prompt_text``) must not link it, and no saved row may link it
|
||||
already. A file the fallback links, an exact pair the ingester stored and an
|
||||
already linked file are never queued. Neither is an ambiguous file with
|
||||
nothing to choose: fewer than two of its candidates carry a prompt
|
||||
(``utils.review_capture.reviewable``); the scan reports it unresolved ("could
|
||||
not be traced").
|
||||
|
||||
The queue heals itself: a pending item whose file is now linked (by the
|
||||
fallback, a later resolution or the monitor) is removed by the next scan
|
||||
that lists the file or by the monitor's next pass over it, and so is a
|
||||
pending item without a choice whose file is still ambiguous without one
|
||||
(``settle_untraced_review_items``). Only an undecided item goes
|
||||
(``pending``, revision 0, no decision row); a decision is never deleted or
|
||||
changed (``settle_linked_review_items``).
|
||||
|
||||
The scan records through ``_record_scanned_review`` (best effort: a failure
|
||||
is logged and counted); the monitor calls ``record_review_file`` from its
|
||||
worker thread after its own link. Both go through
|
||||
``utils.review_capture.record_ambiguous_file`` on the API's capture store.
|
||||
Nothing here links a file.
|
||||
|
||||
Recording changes records, so it follows the registry's ``records``
|
||||
admission (#289). The scan records under its own ``records`` job. The
|
||||
monitor's pass lasts one file and is not a job: it is refused while a
|
||||
``records`` or ``guarded`` job runs (``deferred``), and the next scan reads
|
||||
the file again. Thumbnail work (the job and the gallery's generate and clear
|
||||
holds, kind ``thumbnails``) is ``guarded`` but only writes and removes preview
|
||||
files under ``thumbnails/`` and never touches the database, so it does not
|
||||
defer the recording (#249 S5 M7): a long rebuild no longer holds Needs review
|
||||
back for its whole run.
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from .admin import SCAN_LOG_PREFIXES, _plural, _scan_location, _scan_roots
|
||||
from .jobs import THUMBNAILS_JOB_KIND
|
||||
|
||||
# The kind the monitor's recording checks admission under (never registered).
|
||||
REVIEW_RECORDING_KIND = "review_recording"
|
||||
# Guarded kinds that change no records, so the recording may run beside them.
|
||||
PREVIEW_ONLY_KINDS = (THUMBNAILS_JOB_KIND,)
|
||||
# record_ambiguous_file outcomes that leave the file in the pending queue.
|
||||
QUEUED_OUTCOMES = ("recorded", "limited", "duplicate")
|
||||
NEW_OUTCOMES = ("recorded", "limited")
|
||||
|
||||
try:
|
||||
from ...utils.review_capture import record_ambiguous_file
|
||||
except ImportError:
|
||||
from utils.review_capture import record_ambiguous_file
|
||||
|
||||
# The scan log's reason for items that leave the queue with nothing to choose.
|
||||
NOTHING_TO_CHOOSE = "nothing to choose"
|
||||
|
||||
|
||||
class ReviewRecordingMixin:
|
||||
"""Record the scan's and the monitor's ambiguous files for Needs review."""
|
||||
|
||||
def _record_scanned_review(self, media_file, roots, review=None):
|
||||
"""Record one ambiguous, unlinked scanned file (logged and counted)."""
|
||||
try:
|
||||
outcome = self._record_review_sync(media_file, roots, review)
|
||||
except Exception:
|
||||
self.logger.warning(
|
||||
f"{media_file.name} could not be added to Needs review",
|
||||
exc_info=True,
|
||||
)
|
||||
if review is not None:
|
||||
review["errors"] += 1
|
||||
return
|
||||
if review is None:
|
||||
return
|
||||
if outcome in NEW_OUTCOMES:
|
||||
review["added"] += 1
|
||||
if outcome in QUEUED_OUTCOMES:
|
||||
review["queued"] += 1
|
||||
|
||||
def _record_review_sync(self, media_file, roots, review=None):
|
||||
"""``record_ambiguous_file`` for a file under one of ``roots`` (blocking).
|
||||
|
||||
An ``untraced`` file (its captured bytes offer nothing to choose) also
|
||||
loses its pending items that offer nothing either.
|
||||
"""
|
||||
located = self._review_location(media_file, roots)
|
||||
if located is None:
|
||||
return "skipped"
|
||||
root, index, relative, _spelled = located
|
||||
outcome = record_ambiguous_file(
|
||||
self.db, self._capture_store(), Path(root), index, relative
|
||||
)
|
||||
if outcome == "untraced":
|
||||
self._settle_untraced_reviews([media_file], roots, review)
|
||||
return outcome
|
||||
|
||||
@staticmethod
|
||||
def _review_location(media_file, roots, real_roots=None):
|
||||
"""``(root, root_index, relative, spelled)`` of a listed file, or None.
|
||||
|
||||
``spelled`` is the capture's spelling of the file (the real root joined
|
||||
with the relative path); ``os.path.normcase`` of it is the locator
|
||||
review items are stored under. ``real_roots`` caches each root's
|
||||
``realpath`` across a batch.
|
||||
"""
|
||||
relative, _label, index = _scan_location(media_file, roots)
|
||||
root = next((r for r, _l, i in roots if i == index), None)
|
||||
if index is None or root is None:
|
||||
return None
|
||||
if real_roots is None:
|
||||
real_roots = {}
|
||||
real = real_roots.get(index)
|
||||
if real is None:
|
||||
real = real_roots[index] = os.path.realpath(os.fspath(root))
|
||||
spelled = os.path.join(real, *relative.split("/"))
|
||||
return root, index, relative, spelled
|
||||
|
||||
def _review_file_linked(self, media_file, roots):
|
||||
"""True when a saved row links the scanned file (any spelling)."""
|
||||
located = self._review_location(media_file, roots)
|
||||
spellings = (str(media_file),) + ((located[3],) if located else ())
|
||||
return self.db.review_file_linked(spellings)
|
||||
|
||||
def _settle_scanned_reviews(self, media_files, roots, review=None):
|
||||
"""Remove the pending items of ``media_files`` that are now linked.
|
||||
|
||||
Blocking; best effort like recording (a failure is logged and
|
||||
counted in ``review["errors"]``). Returns how many items left.
|
||||
"""
|
||||
real_roots = {}
|
||||
files = []
|
||||
for media_file in media_files:
|
||||
located = self._review_location(media_file, roots, real_roots)
|
||||
if located is not None:
|
||||
spelled = located[3]
|
||||
files.append((os.path.normcase(spelled), (str(media_file), spelled)))
|
||||
try:
|
||||
settled = self.db.settle_linked_review_items(files) if files else 0
|
||||
except Exception:
|
||||
self.logger.warning(
|
||||
"Needs review items of linked files could not be removed",
|
||||
exc_info=True,
|
||||
)
|
||||
if review is not None:
|
||||
review["errors"] += 1
|
||||
return 0
|
||||
if review is not None:
|
||||
review["settled"] += settled
|
||||
return settled
|
||||
|
||||
def _settle_untraced_reviews(self, media_files, roots, review=None):
|
||||
"""Remove the pending items without a choice of untraced ``media_files``.
|
||||
|
||||
``media_files`` are files found ambiguous with fewer than two
|
||||
candidates that carry a prompt. Blocking; best effort like the linked
|
||||
settle (a failure is logged and counted in ``review["errors"]``).
|
||||
Returns how many items left.
|
||||
"""
|
||||
locators = []
|
||||
for media_file in media_files:
|
||||
located = self._review_location(media_file, roots)
|
||||
if located is not None:
|
||||
locators.append(os.path.normcase(located[3]))
|
||||
try:
|
||||
removed = self.db.settle_untraced_review_items(locators) if locators else 0
|
||||
except Exception:
|
||||
self.logger.warning(
|
||||
"Needs review items without a choice could not be removed",
|
||||
exc_info=True,
|
||||
)
|
||||
if review is not None:
|
||||
review["errors"] += 1
|
||||
return 0
|
||||
if review is not None:
|
||||
review["untraced"] += removed
|
||||
return removed
|
||||
|
||||
def _fallback_links(self, metadata):
|
||||
"""True when the scan's legacy fallback would link these records."""
|
||||
return bool(metadata.get("prompt")) and bool(self._graph_prompt_text(metadata))
|
||||
|
||||
def record_review_file(self, image_path):
|
||||
"""Apply the Needs review rule to one monitor output (its worker thread).
|
||||
|
||||
Maps ``image_path`` to the gallery root that lists it; a file under no
|
||||
current root is ``skipped``. A linked file is never queued: its
|
||||
pending items are removed and the answer is ``linked``. An unlinked
|
||||
file the scan's fallback would link is ``skipped`` (the next scan
|
||||
links it); any other file is recorded when it is ambiguous with a
|
||||
choice, and is ``untraced`` (its pending items without a choice
|
||||
removed) when it is ambiguous without one. While a
|
||||
job runs that a ``records`` job may not run beside, other than
|
||||
thumbnail work (``PREVIEW_ONLY_KINDS``), nothing is written and the
|
||||
answer is ``deferred``. Errors propagate: the monitor logs
|
||||
them and keeps its link.
|
||||
"""
|
||||
busy = self._jobs.blocker(
|
||||
REVIEW_RECORDING_KIND, admission="records", compatible=PREVIEW_ONLY_KINDS
|
||||
)
|
||||
if busy is not None:
|
||||
self.logger.info(
|
||||
f"Needs review was not updated for "
|
||||
f"{os.path.basename(str(image_path))} while {busy.title} runs; "
|
||||
"the next scan reads it again"
|
||||
)
|
||||
return "deferred"
|
||||
full = os.path.abspath(str(image_path))
|
||||
located = Path(os.path.realpath(os.path.dirname(full))) / os.path.basename(full)
|
||||
roots = _scan_roots(self._get_all_output_dirs())
|
||||
if self._review_location(located, roots) is None:
|
||||
return "skipped"
|
||||
if self._review_file_linked(located, roots):
|
||||
self._settle_scanned_reviews([located], roots)
|
||||
return "linked"
|
||||
if self._fallback_links(self._extract_comfyui_metadata(str(located))):
|
||||
return "skipped"
|
||||
return self._record_review_sync(located, roots)
|
||||
|
||||
@staticmethod
|
||||
def _log_review_items(job, review):
|
||||
"""The scan log's Needs review summary lines (none when nothing moved)."""
|
||||
prefix = SCAN_LOG_PREFIXES["review"]
|
||||
if review["added"]:
|
||||
job.log_line(
|
||||
f"{prefix} {_plural(review['added'], 'image')} added to Needs review"
|
||||
)
|
||||
already = review["queued"] - review["added"]
|
||||
if already:
|
||||
job.log_line(
|
||||
f"{prefix} {_plural(already, 'image')} already in Needs review"
|
||||
)
|
||||
if review["settled"]:
|
||||
job.log_line(
|
||||
f"{SCAN_LOG_PREFIXES['known']} "
|
||||
f"{_plural(review['settled'], 'image')} left Needs review · now linked"
|
||||
)
|
||||
if review["untraced"]:
|
||||
job.log_line(
|
||||
f"{SCAN_LOG_PREFIXES['known']} "
|
||||
f"{_plural(review['untraced'], 'image')} left Needs review · "
|
||||
f"{NOTHING_TO_CHOOSE}"
|
||||
)
|
||||
if review["errors"]:
|
||||
job.log_line(
|
||||
f"{SCAN_LOG_PREFIXES['errors']} "
|
||||
f"{_plural(review['errors'], 'image')} could not be added to "
|
||||
"Needs review"
|
||||
)
|
||||
+4
-5
@@ -9,10 +9,9 @@
|
||||
kept.
|
||||
- ``GET /prompt_manager/system/overview`` is the System › Overview aggregate.
|
||||
Every value is real or null: there is no backup status until backups report
|
||||
one and no health result until a check ran; ``needs_review.pending`` is the
|
||||
Needs review queue's pending total. The missing-thumbnail count is not part
|
||||
of it: the page reads ``/images/thumbnails/status``, which walks every
|
||||
source.
|
||||
one, no health result until a check ran, and no Needs-review count until
|
||||
the review queue exists. The missing-thumbnail count is not part of it:
|
||||
the page reads ``/images/thumbnails/status``, which walks every source.
|
||||
- ``GET /prompt_manager/system/statistics`` lists runs and ratings per
|
||||
category, with the legacy ``/stats`` totals and the Overview's count of
|
||||
rated images (images whose linked prompt is rated).
|
||||
@@ -188,7 +187,7 @@ class SystemRoutesMixin:
|
||||
},
|
||||
"health": health_summary(self._current_health_report()),
|
||||
"backup": None, # no verified backup status exists yet (S8)
|
||||
"needs_review": {"pending": counts["review_pending"]},
|
||||
"needs_review": {"pending": None}, # no review queue yet (Q6)
|
||||
"sources": {
|
||||
"configured": len(GalleryConfig.MONITORING_DIRECTORIES),
|
||||
"effective": len(output_dirs),
|
||||
|
||||
@@ -1,494 +0,0 @@
|
||||
"""Thumbnail generation as an Activity job (#249 System S5, Task 12).
|
||||
|
||||
- ``GET /prompt_manager/images/thumbnails/plan?mode=&size=&quality=`` counts
|
||||
what a run would do over every gallery root and writes nothing:
|
||||
``{success, targets, fresh, remaining, roots, mode, size, quality}``.
|
||||
- ``POST /prompt_manager/jobs/thumbnails {mode, size, quality}`` starts the
|
||||
``thumbnails`` job (202 ``{job}``). Its admission class is ``guarded``: it
|
||||
never runs beside a record-changing job, and one thumbnail job runs at a
|
||||
time. It stops at a file boundary when cancelled, keeping the previews it
|
||||
wrote. Its outcome groups are ``generated``, ``skipped`` (already fresh)
|
||||
and ``errors``.
|
||||
|
||||
``mode`` is ``missing`` (previews that are absent or older than their file)
|
||||
or ``rebuild`` (every preview again, fresh ones too). ``size`` is the
|
||||
preview's longest edge, 128 to 2048 pixels. ``quality`` picks the encoder
|
||||
settings (``THUMBNAIL_ENCODINGS``). Every preview is written next to the old
|
||||
one and renamed over it, so nothing is cleared first and a failure keeps the
|
||||
old preview; originals are only read.
|
||||
|
||||
The legacy gallery stream ``POST /images/generate-thumbnails/progress``
|
||||
starts this job (its quality preset still sizes the previews) or attaches to
|
||||
the running one. Its first frame, ``job {id, started}``, names that job and
|
||||
whether this stream started it (the gallery cancels by that id, and only a
|
||||
job it started); then come the job's legacy ``status``/``start``/
|
||||
``progress``/``file_error``/``complete`` frames. A client that goes away no
|
||||
longer stops the work: the job outlives the request, as a scan does.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import json
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
from .admin import _plural, _scan_location, _scan_roots, _scan_status_line
|
||||
from .images import (
|
||||
THUMBNAIL_ENCODINGS,
|
||||
_BodyTooLarge,
|
||||
_new_thumbnail_stats,
|
||||
_read_json_body,
|
||||
_record_thumbnail_result,
|
||||
_thumbnail_batch_size,
|
||||
_thumbnail_complete_payload,
|
||||
_thumbnail_progress_payload,
|
||||
_worker_threads,
|
||||
)
|
||||
from .job_routes import busy_response
|
||||
from .jobs import THUMBNAIL_GROUP_NAMES, THUMBNAILS_JOB_KIND, JobBusy
|
||||
|
||||
THUMBNAIL_JOB_TITLE = "Thumbnail generation"
|
||||
THUMBNAIL_MODES = ("missing", "rebuild")
|
||||
THUMBNAIL_QUALITIES = tuple(THUMBNAIL_ENCODINGS)
|
||||
MIN_THUMBNAIL_EDGE = 128
|
||||
MAX_THUMBNAIL_EDGE = 2048
|
||||
# The concept's "Preview size limit" default.
|
||||
DEFAULT_THUMBNAIL_EDGE = 512
|
||||
DEFAULT_THUMBNAIL_OPTIONS = {
|
||||
"mode": "missing",
|
||||
"size": DEFAULT_THUMBNAIL_EDGE,
|
||||
"quality": "medium",
|
||||
}
|
||||
_OPTIONS_ERROR = (
|
||||
'mode must be "missing" or "rebuild", quality "low", "medium" or "high", '
|
||||
f"and size a whole number of pixels from {MIN_THUMBNAIL_EDGE} "
|
||||
f"to {MAX_THUMBNAIL_EDGE}"
|
||||
)
|
||||
CANCELLED_ERROR = "Thumbnail generation was cancelled"
|
||||
CANCELLED_BEFORE_WRITING = "Cancelled before any preview was written"
|
||||
NOTHING_TO_PREVIEW = "No images or videos to preview"
|
||||
_MODE_LOG = {"missing": "missing previews", "rebuild": "every preview"}
|
||||
# Outcome rows: what happened to each file's preview.
|
||||
_DETAILS = {
|
||||
"generated": "Preview written",
|
||||
"skipped": "Preview already up to date",
|
||||
}
|
||||
_LOG_PREFIXES = {"generated": "+", "errors": "!"}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ThumbnailOptions:
|
||||
"""One run's choices: the mode, the preview's longest edge and quality."""
|
||||
|
||||
mode: str
|
||||
edge: int
|
||||
quality: str
|
||||
|
||||
@property
|
||||
def size(self):
|
||||
return (self.edge, self.edge)
|
||||
|
||||
@property
|
||||
def encoding(self):
|
||||
return THUMBNAIL_ENCODINGS[self.quality]
|
||||
|
||||
@property
|
||||
def rebuild(self):
|
||||
return self.mode == "rebuild"
|
||||
|
||||
def scope(self):
|
||||
return {"mode": self.mode, "size": self.edge, "quality": self.quality}
|
||||
|
||||
|
||||
def _edge(value, *, from_query):
|
||||
"""The preview edge as an int, or None when it is not one in range."""
|
||||
if from_query and isinstance(value, str):
|
||||
if not value.isascii() or not value.isdigit() or len(value) > 5:
|
||||
return None
|
||||
value = int(value)
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
return None
|
||||
return value if MIN_THUMBNAIL_EDGE <= value <= MAX_THUMBNAIL_EDGE else None
|
||||
|
||||
|
||||
def thumbnail_options(raw, *, from_query=False):
|
||||
"""``(ThumbnailOptions, None)`` from a body or query, or ``(None, message)``.
|
||||
|
||||
Absent fields take ``DEFAULT_THUMBNAIL_OPTIONS``. Query values are
|
||||
strings, so a query ``size`` may be decimal digits; a JSON ``size`` must
|
||||
be an int.
|
||||
"""
|
||||
if not isinstance(raw, dict) and not hasattr(raw, "get"):
|
||||
return None, _OPTIONS_ERROR
|
||||
mode = raw.get("mode", DEFAULT_THUMBNAIL_OPTIONS["mode"])
|
||||
quality = raw.get("quality", DEFAULT_THUMBNAIL_OPTIONS["quality"])
|
||||
edge = _edge(raw.get("size", DEFAULT_THUMBNAIL_EDGE), from_query=from_query)
|
||||
if mode not in THUMBNAIL_MODES or quality not in THUMBNAIL_QUALITIES:
|
||||
return None, _OPTIONS_ERROR
|
||||
if edge is None:
|
||||
return None, _OPTIONS_ERROR
|
||||
return ThumbnailOptions(mode, edge, quality), None
|
||||
|
||||
|
||||
def _invalid(message):
|
||||
return web.json_response(
|
||||
{"success": False, "code": "invalid-request", "error": message}, status=400
|
||||
)
|
||||
|
||||
|
||||
def _frame(event, data):
|
||||
"""A legacy stream frame as the job publishes it."""
|
||||
return {"event": event, "data": data}
|
||||
|
||||
|
||||
def _finished_line(plan, stats):
|
||||
"""A finished job's status line: what it wrote, or that it had nothing to do."""
|
||||
if not plan["targets"]:
|
||||
if plan["skipped"]:
|
||||
return f"No previews to generate · {plan['skipped']:,} already up to date"
|
||||
return NOTHING_TO_PREVIEW
|
||||
parts = [f"{_plural(stats['generated'], 'preview')} written"]
|
||||
if stats["errors"]:
|
||||
parts.append(_plural(len(stats["errors"]), "error"))
|
||||
if plan["remaining"]:
|
||||
parts.append(f"{plan['remaining']:,} more on the next run")
|
||||
return " · ".join(parts)
|
||||
|
||||
|
||||
class ThumbnailJobsMixin:
|
||||
"""The thumbnail plan, the thumbnail job and the legacy stream on it."""
|
||||
|
||||
async def get_thumbnail_plan(self, request):
|
||||
"""What a thumbnail run would do (read-only; capped like a run)."""
|
||||
options, error = thumbnail_options(request.query, from_query=True)
|
||||
if error:
|
||||
return _invalid(error)
|
||||
try:
|
||||
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
|
||||
plan = await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails, output_dirs, rebuild=options.rebuild
|
||||
)
|
||||
except Exception:
|
||||
self.logger.exception("Thumbnail plan failed")
|
||||
return web.json_response(
|
||||
{
|
||||
"success": False,
|
||||
"code": None,
|
||||
"error": "The thumbnail plan could not be read",
|
||||
},
|
||||
status=500,
|
||||
)
|
||||
return web.json_response(
|
||||
{
|
||||
"success": True,
|
||||
"targets": len(plan["targets"]),
|
||||
"fresh": plan["fresh"],
|
||||
"remaining": plan["remaining"],
|
||||
"roots": [self._public_path(root) for root in output_dirs],
|
||||
**options.scope(),
|
||||
},
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
|
||||
async def start_thumbnail_job(self, request):
|
||||
"""Start the thumbnail job (202), or 409 naming the job in the way."""
|
||||
body = {}
|
||||
if request.can_read_body:
|
||||
try:
|
||||
body = await _read_json_body(request)
|
||||
except _BodyTooLarge:
|
||||
return web.json_response(
|
||||
{"success": False, "code": "too-large", "error": "Body too large"},
|
||||
status=413,
|
||||
)
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
return _invalid(_OPTIONS_ERROR)
|
||||
options, error = thumbnail_options(body)
|
||||
if error:
|
||||
return _invalid(error)
|
||||
try:
|
||||
job = self._start_thumbnail_job(options)
|
||||
except JobBusy as busy:
|
||||
return busy_response(busy)
|
||||
return web.json_response({"success": True, "job": job.summary()}, status=202)
|
||||
|
||||
def _start_thumbnail_job(self, options):
|
||||
"""Register the thumbnail job synchronously; raises JobBusy."""
|
||||
return self._jobs.start(
|
||||
THUMBNAILS_JOB_KIND,
|
||||
THUMBNAIL_JOB_TITLE,
|
||||
admission="guarded",
|
||||
cancellable=True,
|
||||
scope=options.scope(),
|
||||
runner=functools.partial(self._run_thumbnail_job, options=options),
|
||||
groups=THUMBNAIL_GROUP_NAMES,
|
||||
)
|
||||
|
||||
# ── the job ───────────────────────────────────────────────────────
|
||||
|
||||
async def _run_thumbnail_job(self, job, options):
|
||||
try:
|
||||
state = await self._thumbnail_job_body(job, options)
|
||||
except Exception:
|
||||
self.logger.exception("Thumbnail generation failed")
|
||||
job.update(status_line="Thumbnail generation stopped on an internal error.")
|
||||
job.log_line("Thumbnails failed · internal error")
|
||||
job.publish(
|
||||
_frame(
|
||||
"error",
|
||||
{
|
||||
"error": "An internal error occurred",
|
||||
"message": "Thumbnail generation failed. Check server logs.",
|
||||
},
|
||||
)
|
||||
)
|
||||
state = "failed"
|
||||
job.finish(state)
|
||||
|
||||
async def _thumbnail_job_body(self, job, options):
|
||||
"""Plan, then generate in worker batches; returns the final state.
|
||||
|
||||
Every exit sets the job's final status line (I3): what was written,
|
||||
that there was nothing to do, or what a cancel kept.
|
||||
"""
|
||||
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
|
||||
if not output_dirs:
|
||||
message = "ComfyUI output directory not found"
|
||||
job.update(status_line=message)
|
||||
job.log_line(f"Thumbnails failed · {message}")
|
||||
job.publish(_frame("error", {"error": message}))
|
||||
return "failed"
|
||||
folders = _plural(len(output_dirs), "output folder")
|
||||
job.scope["output_folders"] = len(output_dirs)
|
||||
job.update(status_line=f"Looking for media files in {folders}")
|
||||
job.log_line(
|
||||
f"Thumbnails started · {folders} · {_MODE_LOG[options.mode]} · "
|
||||
f"{options.edge} px · {options.quality}"
|
||||
)
|
||||
job.publish(
|
||||
_frame(
|
||||
"status",
|
||||
{"phase": "scanning", "message": "Scanning for images and videos..."},
|
||||
)
|
||||
)
|
||||
plan = await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails,
|
||||
output_dirs,
|
||||
rebuild=options.rebuild,
|
||||
stop=lambda: job.cancelled,
|
||||
)
|
||||
if job.cancelled: # M2: stopped during the walk, before any start frame
|
||||
return self._end_cancelled(job, 0, CANCELLED_BEFORE_WRITING)
|
||||
roots = _scan_roots(output_dirs)
|
||||
self._record_fresh(job, plan, roots)
|
||||
stats, start = await self._generate_thumbnail_targets(job, options, plan, roots)
|
||||
self.invalidate_gallery_cache()
|
||||
generated = _plural(stats["generated"], "preview")
|
||||
if job.cancelled:
|
||||
return self._end_cancelled(
|
||||
job, stats["generated"], f"Cancelled · {generated} written"
|
||||
)
|
||||
payload = _thumbnail_complete_payload(
|
||||
len(plan["targets"]) + plan["skipped"] + plan["remaining"],
|
||||
stats,
|
||||
start,
|
||||
remaining=plan["remaining"],
|
||||
processed=len(plan["targets"]),
|
||||
)
|
||||
job.update(progress=100, status_line=_finished_line(plan, stats))
|
||||
if plan["remaining"]:
|
||||
job.log_line(
|
||||
f"{plan['remaining']:,} more previews are left for the next run"
|
||||
)
|
||||
errors = _plural(len(stats["errors"]), "error")
|
||||
job.log_line(f"Thumbnails finished · {generated} written · {errors}")
|
||||
self.logger.info(payload["message"])
|
||||
job.publish(_frame("complete", payload))
|
||||
return "finished"
|
||||
|
||||
@staticmethod
|
||||
def _end_cancelled(job, generated, status_line):
|
||||
"""Close a cancelled job: its status line, log line and error frame."""
|
||||
job.update(status_line=status_line)
|
||||
job.log_line(f"Thumbnails cancelled · {_plural(generated, 'preview')} written")
|
||||
job.publish(
|
||||
_frame("error", {"error": CANCELLED_ERROR, "message": CANCELLED_ERROR})
|
||||
)
|
||||
return "cancelled"
|
||||
|
||||
def _record_fresh(self, job, plan, roots):
|
||||
"""List the already fresh previews under Skipped (no log line each)."""
|
||||
rows = []
|
||||
for src in plan["fresh_files"]:
|
||||
path, label, index = _scan_location(src, roots)
|
||||
rows.append(
|
||||
{
|
||||
"path": path,
|
||||
"root": label,
|
||||
"root_index": index,
|
||||
"detail": _DETAILS["skipped"],
|
||||
}
|
||||
)
|
||||
job.record_counted("skipped", plan["skipped"], rows)
|
||||
if plan["skipped"]:
|
||||
job.log_line(f"= {plan['skipped']:,} previews already up to date")
|
||||
|
||||
async def _generate_thumbnail_targets(self, job, options, plan, roots):
|
||||
"""Generate ``plan``'s targets in worker batches, publishing each file."""
|
||||
targets = plan["targets"]
|
||||
total = len(targets)
|
||||
video_count = sum(1 for _, _, is_video in targets if is_video)
|
||||
message = (
|
||||
f"Found {total - video_count} images and {video_count} videos to process"
|
||||
)
|
||||
if plan["skipped"]:
|
||||
message += f" ({plan['skipped']} already have thumbnails)"
|
||||
job.update(total=total)
|
||||
if total: # never "Processing file 0 of 0" (I3)
|
||||
job.update(status_line=_scan_status_line(0, total, 0, 0))
|
||||
job.publish(
|
||||
_frame(
|
||||
"start",
|
||||
{
|
||||
"total_images": total,
|
||||
"skipped": plan["skipped"],
|
||||
"remaining": plan["remaining"],
|
||||
"phase": "processing",
|
||||
"image_count": total - video_count,
|
||||
"video_count": video_count,
|
||||
"message": message,
|
||||
},
|
||||
)
|
||||
)
|
||||
stats = _new_thumbnail_stats()
|
||||
stats["skipped"] = plan["skipped"]
|
||||
start = time.monotonic()
|
||||
batch_size = _thumbnail_batch_size(_worker_threads())
|
||||
generate = {
|
||||
"encoding": options.encoding,
|
||||
"rebuild": options.rebuild,
|
||||
"stop": lambda: job.cancelled,
|
||||
}
|
||||
index = 0
|
||||
batches = 0
|
||||
for batch_start in range(0, total, batch_size):
|
||||
if job.cancelled:
|
||||
break
|
||||
batch = targets[batch_start : batch_start + batch_size]
|
||||
results = await self._run_in_executor(
|
||||
self._generate_batch, batch, options.size, **generate
|
||||
)
|
||||
for (src, _dst, _video), result in zip(batch, results):
|
||||
if result["action"] == "cancelled":
|
||||
continue
|
||||
index += 1
|
||||
_record_thumbnail_result(stats, result)
|
||||
self._record_thumbnail_outcome(job, src, result, roots)
|
||||
job.update(processed=index, progress=index * 100 // total)
|
||||
if result["action"] == "error":
|
||||
job.publish(
|
||||
_frame(
|
||||
"file_error",
|
||||
{"file": result["file"], "error": result["error"]},
|
||||
)
|
||||
)
|
||||
job.publish(
|
||||
_frame(
|
||||
"progress",
|
||||
_thumbnail_progress_payload(index, total, stats, result, start),
|
||||
)
|
||||
)
|
||||
batches += 1
|
||||
job.update(
|
||||
status_line=_scan_status_line(
|
||||
index, total, batches, time.monotonic() - start
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0) # let other requests run between batches
|
||||
return stats, start
|
||||
|
||||
@staticmethod
|
||||
def _record_thumbnail_outcome(job, src, result, roots):
|
||||
path, label, index = _scan_location(src, roots)
|
||||
where = f"{label}/{path}" if label else path
|
||||
if result["action"] == "error":
|
||||
group, detail = "errors", result.get("error") or "failed"
|
||||
elif result["action"] == "skipped":
|
||||
group, detail = "skipped", _DETAILS["skipped"]
|
||||
else:
|
||||
group, detail = "generated", _DETAILS["generated"]
|
||||
row = {"path": path, "root": label, "root_index": index, "detail": detail}
|
||||
if group == "skipped":
|
||||
job.record_counted(group, 1, [row])
|
||||
return
|
||||
job.record(group, row, f"{_LOG_PREFIXES[group]} {where} · {detail}")
|
||||
|
||||
# ── the legacy gallery stream ─────────────────────────────────────
|
||||
|
||||
async def generate_thumbnails_with_progress(self, request):
|
||||
"""The gallery's thumbnail stream (POST, server-sent events).
|
||||
|
||||
Starts the thumbnail job with the gallery's quality preset as the
|
||||
preview size (missing previews, medium encoding, as before), or
|
||||
attaches to the running thumbnail job, and streams its frames. A
|
||||
legacy hold (the blocking generate or clear) or a record-changing
|
||||
job refuses it with the stream's error frame. The job goes on when
|
||||
the client goes away.
|
||||
"""
|
||||
thumbnail_size = self._thumbnail_size(request.query.get("quality", "medium"))
|
||||
response = web.StreamResponse(
|
||||
status=200,
|
||||
headers={
|
||||
"Content-Type": "text/event-stream",
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
},
|
||||
)
|
||||
await response.prepare(request)
|
||||
# M3: read the registry after the last await, so a job that started
|
||||
# while the headers went out is attached to, not refused.
|
||||
job, started, busy = self._legacy_stream_job(thumbnail_size)
|
||||
if busy is not None:
|
||||
message = f"{busy.title} already running"
|
||||
await self._emit_sse(
|
||||
response, "error", {"error": message, "message": message}
|
||||
)
|
||||
return response
|
||||
# No await between the start and the subscription: the lead frame is
|
||||
# sent after it, so the stream misses none of the runner's frames.
|
||||
events = job.events(lead=_frame("job", {"id": job.id, "started": started}))
|
||||
try:
|
||||
async for payload in events:
|
||||
frame = (
|
||||
f"event: {payload['event']}\ndata: {json.dumps(payload['data'])}"
|
||||
)
|
||||
await response.write(f"{frame}\n\n".encode("utf-8"))
|
||||
except Exception as exc: # a write failed: the client went away
|
||||
self.logger.info(
|
||||
f"Thumbnail progress client disconnected ({exc}); "
|
||||
"the thumbnails continue in the background"
|
||||
)
|
||||
finally:
|
||||
# Also when aiohttp cancels the handler on disconnect: the job is
|
||||
# its own task and goes on.
|
||||
await events.aclose()
|
||||
return response
|
||||
|
||||
def _legacy_stream_job(self, thumbnail_size):
|
||||
"""``(job, started, busy)`` for the legacy stream, synchronously.
|
||||
|
||||
Attaches to the running cancellable thumbnail job, or starts one with
|
||||
the gallery's preset as the size (missing previews, medium encoding,
|
||||
as the gallery always wrote), or names the job in the way.
|
||||
"""
|
||||
running = self._jobs.active(THUMBNAILS_JOB_KIND)
|
||||
if running is not None and running.cancellable:
|
||||
return running, False, None
|
||||
busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded")
|
||||
if busy is not None:
|
||||
return None, False, busy
|
||||
# blocker() said free and nothing awaited since, so the start is admitted.
|
||||
options = ThumbnailOptions("missing", thumbnail_size[0], "medium")
|
||||
return self._start_thumbnail_job(options), True, None
|
||||
@@ -127,8 +127,6 @@ export async function openProduction(page,route,theme,motion='no-preference') {
|
||||
await expect(page.locator('#primary-nav .nav-count')).toHaveCount(3);
|
||||
// SAVED VIEWS' Favorites count (D51), the concept's starred images in the concept fixture.
|
||||
await expect(page.locator('#saved-nav a[data-route="favorites"] .nav-count')).toHaveCount(1);
|
||||
// SAVED VIEWS' Needs review count (ruling R5): the exact pending count.
|
||||
await expect(page.locator('#saved-nav a[data-route="review"] .nav-count')).toHaveCount(1);
|
||||
}
|
||||
|
||||
/** Computed outline of the focused element, after keyboard focus so `:focus-visible` applies. */
|
||||
|
||||
@@ -1,212 +0,0 @@
|
||||
// Needs review against the approved concept (#249 PR F, plan Task F3, Mock inventory M4: app.js renderReview
|
||||
// a749–760, style.css s407–415, s796–814 and their breakpoints, screenshot needs-review-tagging.png). Production runs
|
||||
// on the review fixture's concept-shaped queue (workspace_review_fixture.py `concept: true`: the concept's first three
|
||||
// image names and sizes, and the concept's first prompt on the open item), so every remaining difference is a recorded
|
||||
// deviation applied by mapConceptReview (D3, D29, D119) or a test failure. Paired captures of every state are attached
|
||||
// as artifacts; no screenshot baseline is added.
|
||||
import { test, expect } from './fixtures.mjs';
|
||||
import { openConcept, openProduction, inventoryOf, mutate, saveImageTags, diffInventory, SIZES, THEMES } from './concept-inventory.mjs';
|
||||
import { reviewInventory, mapConceptReview, taggingInventory, mapConceptTagging, diffTaggingInventory } from './tagging-canonical-comparison.mjs';
|
||||
|
||||
const CONCEPT_FILE = 'alpine_00001.png';
|
||||
const CONCEPT_TAGS = ['cinematic', 'blue_hour', 'architecture'];
|
||||
const SURFACE = ['h2', 'p.small', 'div.row-actions.spaced-15', 'div.field', 'section.positive.prompt-section',
|
||||
'section.negative.prompt-section', 'div.dialog-actions'];
|
||||
// The fake model's server default instructions (tests/browser/server.py TestModel.default_prompt), D84.
|
||||
const INSTRUCTIONS = 'Describe the image';
|
||||
|
||||
const panel = page => page.locator('#workspace-main section.review-panel');
|
||||
const tagDialog = page => page.getByRole('dialog', { name: 'Tag image', exact: true });
|
||||
|
||||
async function seed(request, data = {}) {
|
||||
await expect(await request.post('/__test__/review/reset', { data: { items: 3, concept: true, ...data } })).toBeOK();
|
||||
}
|
||||
/** Production's Needs review page with its first item's sampler paths loaded. */
|
||||
async function openReview(page, theme) {
|
||||
await openProduction(page, 'review', theme);
|
||||
await expect(panel(page).getByRole('combobox', { name: 'Sampler context', exact: true })).toBeEnabled();
|
||||
await settle(page);
|
||||
}
|
||||
async function settle(page) {
|
||||
await page.mouse.move(0, 0);
|
||||
await page.evaluate(async () => {
|
||||
await document.fonts.ready;
|
||||
await Promise.all([...document.images].map(image => image.decode().catch(() => {})));
|
||||
});
|
||||
}
|
||||
const conceptInventory = page => page.evaluate(reviewInventory, '#main');
|
||||
const productionInventory = page => page.evaluate(reviewInventory, '#workspace-main');
|
||||
const node = (tree, prefix) => tree.tag?.startsWith(prefix) ? tree : tree.children?.map(child => node(child, prefix)).find(Boolean) ?? null;
|
||||
/** Computed values of CSS tokens, read through a probe element in the open page. */
|
||||
const tokens = (page, declarations) => page.evaluate(text => {
|
||||
const probe = document.createElement('i');
|
||||
probe.style.cssText = text;
|
||||
document.body.append(probe);
|
||||
const style = getComputedStyle(probe), values = [style.borderTopColor, style.backgroundColor];
|
||||
probe.remove();
|
||||
return values;
|
||||
}, declarations);
|
||||
|
||||
async function compareAt(page, request, width, height, theme) {
|
||||
await seed(request);
|
||||
await page.setViewportSize({ width, height });
|
||||
await openConcept(page, 'review', theme);
|
||||
await settle(page);
|
||||
const concept = await conceptInventory(page);
|
||||
await openReview(page, theme);
|
||||
const actual = await productionInventory(page);
|
||||
return { concept, actual, reference: mapConceptReview(concept, actual) };
|
||||
}
|
||||
|
||||
test.beforeEach(async ({ page }) => {
|
||||
page.errors = [];
|
||||
page.on('pageerror', error => page.errors.push(error.message));
|
||||
});
|
||||
test.afterEach(async ({ page, request }) => {
|
||||
expect(page.errors).toEqual([]);
|
||||
await request.post('/__test__/jobs/restart');
|
||||
});
|
||||
|
||||
for (const [width, height] of SIZES) for (const theme of THEMES) {
|
||||
test(`concept and production Metadata review match at ${width}x${height} ${theme}`, async ({ page, request }) => {
|
||||
const { actual, reference } = await compareAt(page, request, width, height, theme);
|
||||
expect(diffInventory(reference, actual), `${width}x${height} ${theme}`).toEqual([]);
|
||||
// Pinned concept values (M4), so a mapping mistake on both sides cannot pass unseen.
|
||||
const main = actual.main;
|
||||
expect(main.children.map(child => child.tag)).toEqual(['div.heading', 'div.notice', 'div.gallery-layout']);
|
||||
const notice = node(main, 'div.notice');
|
||||
const [accent, soft] = await tokens(page, 'border-top:1px solid var(--ca-accent);background:var(--soft-blue)');
|
||||
expect([notice.borderTopColor, notice.borderLeftColor, notice.backgroundColor]).toEqual([accent, accent, soft]);
|
||||
expect([notice.borderTopWidth, notice.borderTopLeftRadius, notice.paddingTop, notice.paddingLeft, notice.marginBottom])
|
||||
.toEqual(['1px', '6px', '15px', '18px', '22px']);
|
||||
expect(notice.flexDirection).toBe(width <= 700 ? 'column' : 'row');
|
||||
expect(notice.children.map(child => child.children.map(item => [item.tag, item.text])))
|
||||
.toEqual([[['h2', '3 images need a decision'], ['p', 'Multiple sampler paths were found. The existing image links were preserved. Choose the correct context or leave them unresolved.']]]);
|
||||
const layout = node(main, 'div.gallery-layout');
|
||||
expect(layout.gridTemplateColumns).toBe(width <= 700 ? `${layout.width}px` : `${layout.width - 22 - 310}px 310px`);
|
||||
const surface = node(main, 'section.review-panel');
|
||||
expect(surface.children.map(child => child.tag)).toEqual(SURFACE);
|
||||
expect(surface.children[0].text).toBe('Resolve the association');
|
||||
expect(surface.children[1].text).toBe(CONCEPT_FILE);
|
||||
const actions = surface.children.at(-1);
|
||||
expect(actions.children.map(button => [button.text, button.primary])).toEqual([['Leave for later', false], ['Apply association', true]]);
|
||||
expect(node(surface, 'select').options.at(-1)).toEqual(['Leave unresolved', false]);
|
||||
expect(node(main, 'div.image-grid').children.map(card => node(card, 'h3').text))
|
||||
.toEqual(['alpine_00001.png', 'dunes_00001.png', 'forest_00001.png']);
|
||||
expect(actual.noPageScroll).toBe(true);
|
||||
});
|
||||
}
|
||||
|
||||
test('the gallery layout follows the concept at every breakpoint', async ({ page, request }) => {
|
||||
await seed(request);
|
||||
// 310px beside the queue, 275px and a 15px gap at ≤1250px, 280px at ≤950px, one column at ≤700px.
|
||||
for (const [width, columns, gap] of [[1600, '310px', '22px'], [1200, '275px', '15px'], [900, '280px', '15px'], [390, null, '15px']]) {
|
||||
await page.setViewportSize({ width, height: 1000 });
|
||||
await openConcept(page, 'review', 'dark');
|
||||
const concept = node((await conceptInventory(page)).main, 'div.gallery-layout');
|
||||
await openReview(page, 'dark');
|
||||
const actual = node((await productionInventory(page)).main, 'div.gallery-layout');
|
||||
expect([actual.gridTemplateColumns, actual.columnGap], `${width}px`).toEqual([concept.gridTemplateColumns, concept.columnGap]);
|
||||
expect([actual.gridTemplateColumns.split(' ').at(-1), actual.columnGap], `${width}px`).toEqual([columns ?? `${actual.width}px`, gap]);
|
||||
expect(actual.gridTemplateColumns.split(' ').length, `${width}px`).toBe(width <= 700 ? 1 : 2);
|
||||
}
|
||||
});
|
||||
|
||||
test('SAVED VIEWS rows match the concept with Needs review after Favorites', async ({ page, request }) => {
|
||||
await seed(request);
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
await openConcept(page, 'review', 'dark');
|
||||
const concept = (await inventoryOf(page, '#main')).chrome.navs.find(nav => nav.id === 'saved-nav');
|
||||
await openReview(page, 'dark');
|
||||
const actual = (await inventoryOf(page, '#workspace-main')).chrome.navs.find(nav => nav.id === 'saved-nav');
|
||||
// Sample data: the concept stars three images; this queue's fixture stars none. Needs review counts the queue (3).
|
||||
const reference = { ...concept, rows: concept.rows.map(row => row.text === 'Favorites' ? { ...row, count: { ...row.count, text: '0' } } : row) };
|
||||
expect(diffInventory(reference, actual)).toEqual([]);
|
||||
expect(actual.rows.map(row => [row.text, row.count?.text, row.current])).toEqual([['Favorites', '0', null], ['Needs review', '3', 'page']]);
|
||||
// The count's auto start margin follows the row's own text; every other count token is shared.
|
||||
const { text: favoriteCount, marginInlineStart: favoriteMargin, ...favoriteStyle } = actual.rows[0].count;
|
||||
const { text: reviewCount, marginInlineStart: reviewMargin, ...reviewStyle } = actual.rows[1].count;
|
||||
expect(reviewStyle).toEqual(favoriteStyle);
|
||||
expect(actual.rows[1].icon.d).toBe('M4 3h16v18H4zM8 7h8M8 12h8M8 17h5');
|
||||
});
|
||||
|
||||
test('review Tag dialog matches the needs-review-tagging reference', async ({ page, request }) => {
|
||||
await seed(request);
|
||||
await saveImageTags(request, `review/${CONCEPT_FILE}`, CONCEPT_TAGS);
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
await openConcept(page, 'review', 'dark');
|
||||
await page.locator('#main section.surface').getByRole('button', { name: 'Tag image', exact: true }).click();
|
||||
await expect(tagDialog(page)).toBeVisible();
|
||||
await settle(page);
|
||||
const opened = await page.locator('dialog#modal').evaluate(taggingInventory);
|
||||
await page.keyboard.press('Escape');
|
||||
await openReview(page, 'dark');
|
||||
await panel(page).getByRole('button', { name: 'Tag image', exact: true }).click();
|
||||
await expect(tagDialog(page).locator('img.tag-image-preview')).toHaveAttribute('src', /\/image-annotations\/capture\//);
|
||||
await expect(tagDialog(page).locator('#manual-image-tags')).toHaveValue(CONCEPT_TAGS.join(', '));
|
||||
await settle(page);
|
||||
const actual = await page.locator('dialog#modal').evaluate(taggingInventory);
|
||||
const numberField = opened.modelPanel.fields.find(field => field.control.id === 'image-general-threshold');
|
||||
const mapped = mapConceptTagging(opened, { instructions: INSTRUCTIONS, chip: opened.heading.size, numberField });
|
||||
// D119: an undecided queue item names no prompt, so the heading and the preview's alt are the file name.
|
||||
const reference = { ...mapped, heading: { ...mapped.heading, h3: { ...mapped.heading.h3, text: CONCEPT_FILE },
|
||||
preview: { ...mapped.heading.preview, alt: CONCEPT_FILE } } };
|
||||
expect(actual.scopeNote.text).toBe('Tags belong to this image. Resolve its prompt association separately in Needs review.');
|
||||
expect(diffTaggingInventory(reference, actual)).toEqual([]);
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(tagDialog(page)).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('the comparison notices a dropped notice border and reordered actions', async ({ page, request }) => {
|
||||
const { actual, reference } = await compareAt(page, request, 1600, 1100, 'dark');
|
||||
expect(diffInventory(reference, actual)).toEqual([]);
|
||||
const undoBorder = await mutate(page, { css: '#workspace-main .notice { border: 0 !important; }' });
|
||||
const borderless = await productionInventory(page);
|
||||
expect(diffInventory(reference, borderless).map(item => item.path)).toEqual(expect.arrayContaining([
|
||||
'main.children.1.borderTopWidth', 'main.children.1.borderTopStyle']));
|
||||
await undoBorder();
|
||||
await panel(page).locator('.dialog-actions').evaluate(row => row.prepend(row.lastElementChild));
|
||||
const reordered = await productionInventory(page);
|
||||
expect(diffInventory(reference, reordered).map(item => item.path)).toEqual(expect.arrayContaining([
|
||||
'main.children.2.children.1.children.6.children.0.text', 'main.children.2.children.1.children.6.children.0.primary']));
|
||||
await panel(page).locator('.dialog-actions').evaluate(row => row.prepend(row.lastElementChild));
|
||||
expect(diffInventory(reference, await productionInventory(page))).toEqual([]);
|
||||
});
|
||||
|
||||
// Paired captures for the PR (no baselines): the populated queue beside the concept in both themes and widths, then
|
||||
// the states the concept lacks (conflict, limited item, unavailable file, empty queue) as production captures.
|
||||
test('paired Needs review captures', async ({ page, request }, testInfo) => {
|
||||
test.setTimeout(120_000);
|
||||
const shot = async name => {
|
||||
await settle(page);
|
||||
const file = testInfo.outputPath(`${name}.png`);
|
||||
await page.screenshot({ path: file, fullPage: true });
|
||||
await testInfo.attach(name, { path: file, contentType: 'image/png' });
|
||||
};
|
||||
await seed(request);
|
||||
for (const [width, height] of SIZES) for (const theme of THEMES) {
|
||||
await page.setViewportSize({ width, height });
|
||||
await openConcept(page, 'review', theme);
|
||||
await shot(`concept-review-${width}-${theme}`);
|
||||
await openReview(page, theme);
|
||||
await shot(`production-review-${width}-${theme}`);
|
||||
}
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
await seed(request, { items: 0, conflict: true, limited: true, unavailable: true });
|
||||
await openReview(page, 'dark');
|
||||
await panel(page).getByRole('combobox', { name: 'Sampler context', exact: true }).selectOption({ index: 0 });
|
||||
const decided = page.waitForResponse(item => /\/decision$/.test(item.url()));
|
||||
await panel(page).getByRole('button', { name: 'Apply association', exact: true }).click();
|
||||
expect((await decided).status()).toBe(409);
|
||||
await expect(panel(page).locator('.warning-note')).toBeVisible();
|
||||
await shot('production-review-conflict');
|
||||
for (const [index, name] of [[1, 'limited'], [2, 'unavailable']]) {
|
||||
await page.locator('#workspace-main .review-queue article').nth(index).getByRole('button', { name: /^Inspect / }).click();
|
||||
await expect(panel(page).locator('.warning-note')).toBeVisible();
|
||||
await shot(`production-review-${name}`);
|
||||
}
|
||||
await seed(request, { items: 0 });
|
||||
await openProduction(page, 'review', 'dark');
|
||||
await expect(page.locator('#workspace-main .notice h2')).toHaveText('No images need a decision');
|
||||
await shot('production-review-empty');
|
||||
});
|
||||
@@ -40,7 +40,6 @@ from tests.browser.workspace_library_editor_fixture import (
|
||||
)
|
||||
from tests.browser.workspace_vocabulary_fixture import register_vocabulary_commands
|
||||
from tests.browser.workspace_tagging_fixture import register_tagging_commands
|
||||
from tests.browser.workspace_review_fixture import register_review_commands
|
||||
|
||||
CANONICAL_ROOT = Path(__file__).resolve().parents[2]
|
||||
CANONICAL_ASSETS = {
|
||||
@@ -198,7 +197,6 @@ def create_app(root: Path) -> web.Application:
|
||||
library_editor_commands = register_library_editor_commands(routes, api, root)
|
||||
register_vocabulary_commands(routes, api, root)
|
||||
register_tagging_commands(routes, api, root, model)
|
||||
register_review_commands(routes, api, root)
|
||||
reveal_stub = register_reveal_stub(routes, api, root)
|
||||
register_activity_commands(app, routes, api)
|
||||
register_system_commands(app, routes, api, root)
|
||||
@@ -228,8 +226,6 @@ def create_app(root: Path) -> web.Application:
|
||||
for table in (
|
||||
"saved_views",
|
||||
"image_favorites",
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_image_tags",
|
||||
"media_annotation_bytes",
|
||||
"media_annotation_files",
|
||||
@@ -595,13 +591,9 @@ def create_app(root: Path) -> web.Application:
|
||||
return web.json_response("dark")
|
||||
|
||||
async def cleanup(application: web.Application) -> None:
|
||||
# A scan of an ambiguous fixture file captures it for Needs review.
|
||||
api.close_annotation_captures()
|
||||
api.db.close_all()
|
||||
|
||||
seed()
|
||||
# Open Needs review change streams end first, as on the real host.
|
||||
app.on_shutdown.append(api.close_review_events)
|
||||
app.on_cleanup.append(cleanup)
|
||||
app.add_routes(routes)
|
||||
return app
|
||||
|
||||
@@ -68,12 +68,12 @@ function remainingFooter(footer,spans) {
|
||||
const OVERVIEW_GLYPH='M3 11 12 4l9 7M5 10v10h14V10M10 20v-6h4v6';
|
||||
|
||||
/**
|
||||
* The concept's SAVED VIEWS rows (app.js a331–335) as production lists them: Favorites with its count (D51), Needs
|
||||
* review with the exact pending count (ruling R5), then the saved views. The concept's "3" is sample data: the concept
|
||||
* fixture records no review items, so production's count reads "0".
|
||||
* The concept's SAVED VIEWS rows (app.js a331–335) as production lists them: Favorites with its count (D51), then the
|
||||
* saved views. D50: no "Needs review" row until the tag-review slice (maintainer ruling 2026-10-21).
|
||||
*/
|
||||
function savedNav(nav) {
|
||||
return {...nav,rows:nav.rows.map(row=>row.text==='Needs review'?{...row,count:{...row.count,text:'0'}}:row)};
|
||||
const rows=nav.rows.filter(row=>row.text!=='Needs review');
|
||||
return {...nav,rows:rows.map((row,index)=>index===rows.length-1?{...row,gapAfter:null}:row)};
|
||||
}
|
||||
|
||||
/** Recorded shell deviations only (spec §8). Everything else in the concept inventory must equal production. */
|
||||
@@ -243,9 +243,8 @@ for(const [width,height] of SIZES)test(`a saved view row matches the concept's $
|
||||
await openProduction(page,'images','dark');
|
||||
const actual=await save('Save','#workspace-main');
|
||||
expect(diffInventory(mapShellConcept(reference).chrome.navs,actual.chrome.navs)).toEqual([]);
|
||||
const referenceRows=reference.chrome.navs.find(nav=>nav.id==='saved-nav').rows;
|
||||
expect(actual.chrome.navs.find(nav=>nav.id==='saved-nav').rows.map(row=>[row.text,row.count]))
|
||||
.toEqual([['Favorites',referenceRows[0].count],['Needs review',{...referenceRows[1].count,text:'0'}],['Golden hour',null]]);
|
||||
.toEqual([['Favorites',reference.chrome.navs.find(nav=>nav.id==='saved-nav').rows[0].count],['Golden hour',null]]);
|
||||
});
|
||||
|
||||
test('the inventory reports a missing shell element by its path instead of failing',async({page})=>{
|
||||
|
||||
@@ -169,9 +169,9 @@ function mapOverviewConcept(reference, overview) {
|
||||
const relabel = { 'Database integrity': 'Not checked', 'Positive prompts': number(overview.library.positive_prompts),
|
||||
'Negative variants': number(overview.library.negative_variants), Images: number(overview.images.indexed) };
|
||||
const rows = {
|
||||
// D93: the real pending count; Review metadata opens Needs review (the concept's go-review).
|
||||
'3 ambiguous image links': row => ({ ...row,
|
||||
title: `${number(overview.needs_review.pending)} ambiguous image link${overview.needs_review.pending === 1 ? '' : 's'}` }),
|
||||
// D93 (Q6): no review queue yet, so no count and the action is unavailable (dimmed, as the Save row's D92).
|
||||
'3 ambiguous image links': row => ({ ...row, title: 'Ambiguous image links', description: 'Not available yet.',
|
||||
action: { ...row.action, opacity: '0.55', cursor: 'not-allowed' } }),
|
||||
// D93: no backup has been recorded (verified backups arrive with S8).
|
||||
'Backup status': row => ({ ...row, description: 'No retained backup record.' }),
|
||||
// D93: there is no paging, so the shared settings are grid and density.
|
||||
@@ -230,11 +230,12 @@ for (const theme of THEMES) for (const [width, height] of SIZES) {
|
||||
});
|
||||
}
|
||||
|
||||
// The recorded differences, by path: D93 the review row (its real count), the backup row, the
|
||||
// The recorded differences, by path: D93 the review row (title, Not available yet, dimmed action), the backup row, the
|
||||
// preferences row and the sources row, and metric 4; D96 the statistics row; D92 the chip, the status and the dimmed
|
||||
// Save; D122 the nav label. (The fixture's first three metrics are the concept's own counts.)
|
||||
const UNMAPPED_PATHS = [
|
||||
'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
|
||||
'columns.surfaces.0.rows.0.action.cursor', 'columns.surfaces.0.rows.0.action.opacity',
|
||||
'columns.surfaces.0.rows.0.description', 'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
|
||||
'columns.surfaces.1.rows.0.description', 'columns.surfaces.1.rows.1.title', 'columns.surfaces.1.rows.2.description',
|
||||
'heading.actions.0', 'heading.actions.length', 'metrics.items.3.value', 'nav.label',
|
||||
'save.button.cursor', 'save.button.opacity', 'save.status.text',
|
||||
@@ -775,131 +776,3 @@ test('add source comparison detects mutations', async ({ page, request }) => {
|
||||
await undoStyle();
|
||||
expect(diffInventory(expected, await page.evaluate(measureAddSource))).toEqual([]);
|
||||
});
|
||||
|
||||
/**
|
||||
* Runs in the page: the open "Thumbnail maintenance" dialog (the concept's `thumbnail-review`, app.js 1433–1437): its
|
||||
* title, the body's tree, each `.field` (label, control, options, selection, limits), the `p.small` line and the
|
||||
* actions, with the concept's geometry and tokens. The concept has no `p.small`, so a probe one stands in for its style.
|
||||
*/
|
||||
function measureThumbnailReview() {
|
||||
const dialog = document.querySelector('dialog#modal[open]');
|
||||
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
|
||||
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
|
||||
const text = node => node?.textContent.trim() ?? null;
|
||||
const body = dialog.querySelector('#modal-body');
|
||||
const fields = [...body.querySelectorAll(':scope > .field')].map(field => {
|
||||
const label = field.querySelector(':scope > label');
|
||||
const control = field.querySelector(':scope > select, :scope > input');
|
||||
return {
|
||||
tree: [...field.children].map(name),
|
||||
...css(field, 'display', 'flexDirection', 'rowGap', 'marginBottom'),
|
||||
label: { text: text(label), for: label.htmlFor, ...css(label, 'fontSize', 'fontWeight', 'color') },
|
||||
control: { id: control.id, tag: control.localName, type: control.type,
|
||||
options: control.options ? [...control.options].map(option => option.textContent) : null,
|
||||
selected: control.options ? control.selectedIndex : null,
|
||||
value: control.localName === 'input' ? control.value : null,
|
||||
min: control.getAttribute('min'), max: control.getAttribute('max'),
|
||||
...css(control, 'height', 'paddingLeft', 'fontSize', 'color', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
|
||||
};
|
||||
});
|
||||
let line = body.querySelector(':scope > p.small');
|
||||
const probe = !line;
|
||||
if (probe) { line = document.createElement('p'); line.className = 'small'; body.append(line); }
|
||||
const plan = { text: probe ? null : text(line), role: probe ? null : line.getAttribute('role'),
|
||||
...css(line, 'fontSize', 'color', 'marginTop', 'marginBottom') };
|
||||
if (probe) line.remove();
|
||||
const actions = body.querySelector(':scope > .dialog-actions');
|
||||
const rect = dialog.getBoundingClientRect();
|
||||
return {
|
||||
title: text(dialog.querySelector('#modal-title')),
|
||||
tree: [...body.children].map(name),
|
||||
fields,
|
||||
plan,
|
||||
actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
|
||||
buttons: [...actions.children].map(button => ({ text: text(button), className: button.className,
|
||||
...css(button, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) },
|
||||
width: Math.round(rect.width),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* D99: the planned counts line (`p.small[role=status]`) before the actions, from the server's plan for the chosen
|
||||
* options, and the primary "Start" (the concept's "Simulate thumbnail job").
|
||||
*/
|
||||
function mapThumbnailConcept(reference, plan) {
|
||||
const [cancel, start] = reference.actions.buttons;
|
||||
const tree = [...reference.tree];
|
||||
tree.splice(tree.indexOf('div.dialog-actions'), 0, 'p.small');
|
||||
return { ...reference, tree,
|
||||
plan: { ...reference.plan, role: 'status',
|
||||
text: `${number(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ${number(plan.fresh)} already up to date.` },
|
||||
actions: { ...reference.actions, buttons: [cancel, { ...start, text: 'Start' }] } };
|
||||
}
|
||||
|
||||
async function conceptThumbnailReview(page, theme) {
|
||||
await conceptSources(page, theme, { notice: false });
|
||||
const row = page.locator('#main .operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
|
||||
await row.getByRole('button', { name: 'Review options', exact: true }).click();
|
||||
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await page.mouse.move(0, 0);
|
||||
return page.evaluate(measureThumbnailReview);
|
||||
}
|
||||
|
||||
/** Production's dialog from Sources' Review options or Overview's Review thumbnails, once its plan line answered. */
|
||||
async function productionThumbnailReview(page, theme, entry) {
|
||||
if (entry === 'sources') {
|
||||
await productionSources(page, theme, { notice: false });
|
||||
const row = page.locator('#workspace-main .operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
|
||||
await row.getByRole('button', { name: 'Review options', exact: true }).click();
|
||||
} else {
|
||||
await openProduction(page, 'system/overview', theme);
|
||||
await page.locator('#workspace-main').getByRole('button', { name: 'Review thumbnails', exact: true }).click();
|
||||
}
|
||||
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await expect(page.locator('dialog#modal[open] #modal-body > p.small')).toHaveText(/ already up to date\.$/);
|
||||
await page.mouse.move(0, 0);
|
||||
return page.evaluate(measureThumbnailReview);
|
||||
}
|
||||
|
||||
const thumbnailPlan = request => serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium');
|
||||
|
||||
for (const entry of ['sources', 'overview']) for (const theme of THEMES) for (const [width, height] of SIZES) {
|
||||
test(`thumbnail dialog matches the concept from ${entry} ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
|
||||
await page.setViewportSize({ width, height });
|
||||
await watchedSources(request);
|
||||
const plan = await thumbnailPlan(request);
|
||||
const concept = await conceptThumbnailReview(page, theme);
|
||||
await page.screenshot({ path: testInfo.outputPath(`concept-thumbnail-review-${theme}-${width}.png`) });
|
||||
const production = await productionThumbnailReview(page, theme, entry);
|
||||
await page.screenshot({ path: testInfo.outputPath(`production-thumbnail-review-${entry}-${theme}-${width}.png`) });
|
||||
expect(diffInventory(mapThumbnailConcept(concept, plan), production)).toEqual([]);
|
||||
});
|
||||
}
|
||||
|
||||
test('thumbnail dialog comparison detects mutations', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
await watchedSources(request);
|
||||
const plan = await thumbnailPlan(request);
|
||||
const concept = await conceptThumbnailReview(page, 'dark');
|
||||
const expected = mapThumbnailConcept(concept, plan);
|
||||
const production = await productionThumbnailReview(page, 'dark', 'sources');
|
||||
expect(diffInventory(expected, production)).toEqual([]);
|
||||
// Unmapped, the concept differs only where D99 is recorded.
|
||||
expect(diffInventory(concept, production).map(item => item.path).sort())
|
||||
.toEqual(['actions.buttons.1.text', 'plan.role', 'plan.text', 'tree.3', 'tree.4', 'tree.length']);
|
||||
const paths = async () => diffInventory(expected, await page.evaluate(measureThumbnailReview)).map(item => item.path);
|
||||
// A changed option label, a swapped option order, a swapped action order and a changed style.
|
||||
await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium'; });
|
||||
expect(await paths()).toEqual(['fields.1.control.options.0']);
|
||||
await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium · optimized'; });
|
||||
await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['fields.0.control.options.0', 'fields.0.control.options.1']));
|
||||
await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); select.selectedIndex = 0; });
|
||||
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text']));
|
||||
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
|
||||
const undoStyle = await mutate(page, { css: 'dialog#modal #thumb-limit { border-color: rgb(255, 0, 0); } dialog#modal #modal-body > p.small { color: rgb(255, 0, 0); }' });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['fields.2.control.borderTopColor', 'plan.color']));
|
||||
await undoStyle();
|
||||
expect(diffInventory(expected, await page.evaluate(measureThumbnailReview))).toEqual([]);
|
||||
});
|
||||
|
||||
@@ -140,152 +140,3 @@ export function mapConceptTagging(reference,{instructions,limit=10,chip,numberFi
|
||||
export function withoutChipGeometry(inventory) {
|
||||
return {...inventory,suggestions:{...inventory.suggestions,rows:inventory.suggestions.rows.map(({paddingTop,paddingLeft,alignItems,...row})=>row)}};
|
||||
}
|
||||
|
||||
/**
|
||||
* Needs review (plan Task F3, Mock inventory M4: app.js renderReview a749–760, style.css s407–415, s796–814 and their
|
||||
* breakpoints): the page's main element as a tree, runs in the page. Every shown box below `main` to `depth` levels is
|
||||
* recorded with its tag and classes, its box relative to its parent, its fonts, edges and layout, a leaf's text, a
|
||||
* button's label and icon, a select's options; a `display: contents` node is replaced by its children (D20). A flex
|
||||
* row of actions also records the gaps between its children and its right inset, so a row whose button copy differs
|
||||
* (D29) still compares by order, spacing and alignment.
|
||||
*/
|
||||
export function reviewInventory(selector) {
|
||||
const main=document.querySelector(selector);
|
||||
// Half pixels: the concept's 13.5px line heights add up exactly, so a box that loses one line moves by 13.5px.
|
||||
const px=value=>Math.round(value*2)/2;
|
||||
const css=(node,...keys)=>{const style=getComputedStyle(node);return Object.fromEntries(keys.map(key=>[key,style[key]]));};
|
||||
const rect=node=>node.getBoundingClientRect();
|
||||
const shown=node=>node.getClientRects().length>0&&getComputedStyle(node).visibility!=='hidden';
|
||||
const relative=(node,origin)=>{const a=rect(node),b=rect(origin);return {left:px(a.left-b.left),top:px(a.top-b.top),width:px(a.width),height:px(a.height)};};
|
||||
const font=node=>css(node,'fontFamily','fontSize','fontWeight','lineHeight','letterSpacing','color','textTransform','textAlign');
|
||||
const edges=node=>css(node,'paddingTop','paddingRight','paddingBottom','paddingLeft','marginTop','marginRight','marginBottom','marginLeft',
|
||||
'borderTopWidth','borderTopStyle','borderTopColor','borderRightWidth','borderRightStyle','borderRightColor','borderBottomWidth',
|
||||
'borderBottomStyle','borderBottomColor','borderLeftWidth','borderLeftStyle','borderLeftColor','borderTopLeftRadius','backgroundColor',
|
||||
'boxShadow','outlineStyle');
|
||||
const layout=node=>css(node,'display','position','flexDirection','flexWrap','justifyContent','alignItems','columnGap','rowGap',
|
||||
'gridTemplateColumns','overflowX','overflowY','objectFit','minHeight','minWidth');
|
||||
const tagOf=node=>`${node.localName}${[...node.classList].sort().map(name=>`.${name}`).join('')}`;
|
||||
const textOf=node=>node.textContent.replace(/\s+/g,' ').trim();
|
||||
const boxed=node=>[...node.children].flatMap(child=>getComputedStyle(child).display==='contents'?boxed(child):[child]);
|
||||
const LEAVES=new Set(['button','select','input','img','svg','textarea','progress']);
|
||||
function nodeOf(node,origin,depth) {
|
||||
const base={tag:tagOf(node),...relative(node,origin),...font(node),...edges(node),...layout(node)};
|
||||
const name=node.localName;
|
||||
if(name==='svg')return {tag:base.tag,width:base.width,height:base.height,d:node.querySelector('path')?.getAttribute('d')??null,
|
||||
...css(node,'strokeWidth','color','verticalAlign')};
|
||||
if(name==='button')return {...base,text:textOf(node),label:node.getAttribute('aria-label'),pressed:node.getAttribute('aria-pressed'),
|
||||
disabled:node.getAttribute('aria-disabled')==='true'||node.disabled,primary:node.classList.contains('ca-primary'),
|
||||
children:boxed(node).filter(shown).map(child=>nodeOf(child,node,0))};
|
||||
if(name==='select')return {...base,id:node.id,options:[...node.options].map(option=>[option.textContent,option.disabled])};
|
||||
if(name==='input')return {...base,id:node.id,type:node.type,label:node.getAttribute('aria-label')};
|
||||
if(name==='img')return {...base,alt:node.getAttribute('alt')};
|
||||
const children=boxed(node).filter(shown);
|
||||
const item={...base,id:node.id||null,role:node.getAttribute('role'),label:node.getAttribute('aria-label'),
|
||||
text:children.length?null:textOf(node),htmlFor:node.htmlFor||null};
|
||||
if(/(^|\s)(dialog-actions|row-actions|image-card-actions)(\s|$)/.test(node.className)&&children.length) {
|
||||
const boxes=children.map(rect);
|
||||
item.gaps=boxes.slice(1).map((box,index)=>px(box.left-boxes[index].right));
|
||||
item.rightInset=px(rect(node).right-boxes.at(-1).right);
|
||||
}
|
||||
if(depth>0&&!LEAVES.has(name))item.children=children.map(child=>nodeOf(child,node,depth-1));
|
||||
return item;
|
||||
}
|
||||
return {main:nodeOf(main,main,8),noPageScroll:document.documentElement.scrollWidth<=innerWidth+1};
|
||||
}
|
||||
|
||||
/**
|
||||
* `tree` with `edit(node)` applied to every node `match` picks (a tag prefix or a predicate), depth first, new objects
|
||||
* only.
|
||||
*/
|
||||
function editNodes(tree,match,edit) {
|
||||
const children=tree.children?.map(child=>editNodes(child,match,edit));
|
||||
const node=children?{...tree,children}:tree;
|
||||
const picked=typeof match==='function'?match(node):node.tag?.startsWith(match);
|
||||
return picked?edit(node):node;
|
||||
}
|
||||
const findNode=(tree,prefix)=>tree.tag?.startsWith(prefix)?tree:tree.children?.map(child=>findNode(child,prefix)).find(Boolean)??null;
|
||||
|
||||
/**
|
||||
* The concept's Metadata review inventory (reviewInventory) with the recorded deviations applied, the remaining sample
|
||||
* data taken from production (`actual`) only where the concept's own copy cannot say it:
|
||||
* - D29: no "Sample review queue" chip in the notice; "Resolve the association" and "Apply association" without the
|
||||
* prototype's "sample";
|
||||
* - D119: queue cards carry no selection checkbox and no star (the overlay row stays, empty); a card's title is its
|
||||
* file name (an undecided item names no prompt) and its line is the extension only (D11 omits the unknown size and
|
||||
* seed, which a queue item does not carry); the image's alt is the file name; the prompt blocks' Copy and Send names
|
||||
* carry no prompt title;
|
||||
* - D3: the grid keeps the Images grid's `position: relative` (its cards stay in its flow);
|
||||
* - production's hooks and names: the `review-queue` / `review-panel` classes, the queue's list roles and name, the
|
||||
* panel heading's id (the region's name) and the prompt sections' region names;
|
||||
* - sample data: the Sampler context's candidate options (the file's own evidence; the last option, Leave unresolved,
|
||||
* stays compared) and the widths that follow from copy the deviations above change (a card's text column, the
|
||||
* panel's action buttons; their gaps and right inset stay compared).
|
||||
* Where that copy changes the height of a box, the boxes after it move by exactly that much: the stacked notice (390)
|
||||
* loses the chip and its row gap; a card whose concept line wraps (390: "1536 × 1152 · PNG · Seed 42001") takes
|
||||
* production's one-line height, its row of the grid shrinks by the most any of its cards lost, and in one column the
|
||||
* panel below the grid moves up by the grid's loss. The actions keep production's centred place in the card's info.
|
||||
*/
|
||||
export function mapConceptReview(reference,actual) {
|
||||
const named=(node,id)=>({...node,id});
|
||||
const actualCards=findNode(actual.main,'div.image-grid')?.children??[];
|
||||
const actualActions=findNode(actual.main,'div.dialog-actions');
|
||||
let cardIndex=0;
|
||||
let main={...reference.main,id:actual.main.id};
|
||||
let noticeLoss=0;
|
||||
main=editNodes(main,'div.notice',notice=>{
|
||||
const chip=notice.children.find(child=>child.tag==='span.chip');
|
||||
if(chip&¬ice.flexDirection==='column')noticeLoss=chip.height+parseFloat(notice.rowGap);
|
||||
return {...notice,height:notice.height-noticeLoss,children:notice.children.filter(child=>child!==chip)};
|
||||
});
|
||||
const noticeIndex=main.children.findIndex(child=>child.tag==='div.notice');
|
||||
main={...main,height:main.height-noticeLoss,children:main.children.map((child,index)=>index>noticeIndex?{...child,top:child.top-noticeLoss}:child)};
|
||||
const losses=[];
|
||||
main=editNodes(main,'div.image-grid',grid=>({...grid,tag:'div.image-grid.review-queue',position:'relative',role:'list',label:'Review queue'}));
|
||||
main=editNodes(main,'article.media-card',card=>{
|
||||
const twin=actualCards[cardIndex++];
|
||||
const filename=findNode(card,'button.image-open').label.replace(/^Open /,'');
|
||||
const art=editNodes(card.children[0],'div.image-overlays',overlays=>({...overlays,height:0,text:'',children:[]}));
|
||||
const withAlt=editNodes(art,'img',img=>({...img,alt:filename}));
|
||||
const info=card.children[1];
|
||||
const [copy,...rest]=info.children;
|
||||
const twinCopy=twin?.children[1]?.children[0];
|
||||
const [h3,p]=copy.children;
|
||||
const twinLine=twinCopy?.children[1];
|
||||
const loss=twinLine?p.height-twinLine.height:0;
|
||||
losses.push({top:card.top,loss});
|
||||
const mappedCopy={...copy,width:twinCopy?.width,height:copy.height-loss,children:[{...h3,text:filename,width:twinCopy?.children[0]?.width},
|
||||
{...p,text:'PNG',width:twinLine?.width,height:p.height-loss}]};
|
||||
const [actions,...after]=rest;
|
||||
const twinActions=twin?.children[1]?.children[1];
|
||||
const mappedActions=actions&&loss?{...actions,top:twinActions?.top}:actions;
|
||||
return {...card,role:'listitem',height:card.height-loss,
|
||||
children:[withAlt,{...info,height:info.height-loss,children:[mappedCopy,...(actions?[mappedActions]:[]),...after]},...card.children.slice(2)]};
|
||||
});
|
||||
// Each grid row loses what its most shortened card lost; later rows move up by the rows before them.
|
||||
const rowLoss=new Map();
|
||||
for(const {top,loss} of losses)rowLoss.set(top,Math.max(rowLoss.get(top)??0,loss));
|
||||
const lossBefore=top=>[...rowLoss].filter(([rowTop])=>rowTop<top).reduce((sum,[,loss])=>sum+loss,0);
|
||||
const gridLoss=[...rowLoss.values()].reduce((sum,loss)=>sum+loss,0);
|
||||
main=editNodes(main,'div.image-grid',grid=>({...grid,height:grid.height-gridLoss,
|
||||
children:grid.children.map(card=>({...card,top:card.top-lossBefore(card.top)}))}));
|
||||
main=editNodes(main,'div.gallery-layout',layout=>{
|
||||
if(!gridLoss||layout.gridTemplateColumns.trim().split(/\s+/).length>1)return layout;
|
||||
return {...layout,height:layout.height-gridLoss,children:layout.children.map((child,index)=>index?{...child,top:child.top-gridLoss}:child)};
|
||||
});
|
||||
if(gridLoss&&findNode(main,'div.gallery-layout').gridTemplateColumns.trim().split(/\s+/).length===1)main={...main,height:main.height-gridLoss};
|
||||
main=editNodes(main,'section.surface',surface=>{
|
||||
const [h2,...rest]=surface.children;
|
||||
return {...surface,tag:'section.review-panel.surface',children:[named({...h2,text:h2.text.replace('Resolve the sample association','Resolve the association')},'review-panel-title'),...rest]};
|
||||
});
|
||||
main=editNodes(main,'select',select=>{
|
||||
const twin=findNode(actual.main,'select');
|
||||
return select.id==='review-context'?{...select,options:[...twin.options.slice(0,-1),select.options.at(-1)]}:select;
|
||||
});
|
||||
main=editNodes(main,node=>/^section\..*\bprompt-section\b/.test(node.tag),section=>({...section,label:section.tag.includes('positive')?'Positive prompt':'Negative prompt'}));
|
||||
main=editNodes(main,node=>node.tag?.startsWith('button')&&/^(Copy|Send) (positive|negative) prompt/.test(node.label??''),
|
||||
button=>({...button,label:button.label.replace(/: .+$/,'')}));
|
||||
main=editNodes(main,'div.dialog-actions',actions=>({...actions,children:actions.children.map((button,index)=>({...button,
|
||||
text:button.text.replace('Apply sample association','Apply association'),
|
||||
left:actualActions?.children[index]?.left,width:actualActions?.children[index]?.width}))}));
|
||||
return {...reference,main};
|
||||
}
|
||||
|
||||
@@ -54,8 +54,8 @@ async function step(request, id, data) {
|
||||
await expect(await request.post(`/__test__/jobs/${id}/step`, { data })).toBeOK();
|
||||
}
|
||||
|
||||
async function finish(request, id, state = 'finished', extra = {}) {
|
||||
const response = await request.post(`/__test__/jobs/${id}/finish`, { data: { state, ...extra } });
|
||||
async function finish(request, id, state = 'finished') {
|
||||
const response = await request.post(`/__test__/jobs/${id}/finish`, { data: { state } });
|
||||
await expect(response).toBeOK();
|
||||
return (await response.json()).job;
|
||||
}
|
||||
@@ -1008,96 +1008,3 @@ test('the report download names the kind', async ({ page, request }) => {
|
||||
expect(report.outcomes.tagged.rows.map(row => [row.path, row.detail])).toEqual([['sub/a.png', 'Added 1 tag']]);
|
||||
await expect(toast(page)).toHaveText('Job report downloaded.');
|
||||
});
|
||||
|
||||
// D111 (System S5): each kind's own descriptor. A thumbnail job counts Generated (green), Skipped and Errors (red),
|
||||
// its pill says "Generating thumbnails" and its toasts "Thumbnails finished · N generated · M errors".
|
||||
const THUMB_JOB = { kind: 'thumbnails', total: 10, output_folders: 2 };
|
||||
|
||||
test('a thumbnail job shows Generated Skipped Errors, its pill verb and its toasts', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const job = await startJob(request, THUMB_JOB);
|
||||
await step(request, job.id, { processed: 4, records: [
|
||||
{ group: 'generated', path: 'sub/a.png', root: 'output', detail: 'Preview written' },
|
||||
{ group: 'skipped', path: 'b.png', root: 'input', detail: 'Preview already up to date' },
|
||||
{ group: 'errors', path: 'broken.png', root: 'output', detail: 'Not a valid image file' },
|
||||
] });
|
||||
await page.goto(`${workspace}#/images`);
|
||||
await expect(pill(page)).toHaveText('Generating thumbnails · 40% · 4 / 10');
|
||||
await page.goto(`${workspace}#/activity`);
|
||||
await expect(items(page).first().locator('p')).toHaveText(/^Started \d\d:\d\d · 2 output folders · closing this view does not cancel it$/);
|
||||
await expect(items(page).first().locator('progress')).toHaveAttribute('value', '40');
|
||||
await openJob(page, job.id);
|
||||
await expect(heading(page).locator('h1')).toHaveText('Thumbnail generation');
|
||||
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report', 'Cancel job']);
|
||||
expect(await tileValues(page)).toEqual([['stat added', 'Generated', '1'], ['stat', 'Skipped', '1'], ['stat bad', 'Errors', '1']]);
|
||||
await expect(outcomeButtons(page)).toHaveText(['Generated1', 'Errors1', 'Skipped1']);
|
||||
await expect(fileRows(page).first().locator('.path')).toHaveText('sub/a.png');
|
||||
expect(await fileRows(page).first().evaluate(node => node.localName)).toBe('button');
|
||||
await outcomeButtons(page).filter({ hasText: 'Errors' }).click();
|
||||
expect(await fileRows(page).first().evaluate(node => node.localName)).toBe('div');
|
||||
await expect(scanLog(page).locator('summary')).toHaveText('● Job log — raw detail, click to show');
|
||||
|
||||
await finish(request, job.id);
|
||||
await expect(toast(page)).toHaveText('Thumbnails finished · 1 generated · 1 error');
|
||||
await expect(heading(page).locator('p.muted')).toHaveText(/^Finished \d\d:\d\d · 1 generated · 1 error$/);
|
||||
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
|
||||
});
|
||||
|
||||
test('a held legacy job and the health check show no tiles, no Results and no made-up percentage', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const response = await request.post('/__test__/jobs/hold',
|
||||
{ data: { kind: 'thumbnails', title: 'Thumbnail clear (gallery page)', admission: 'guarded' } });
|
||||
await expect(response).toBeOK();
|
||||
const held = (await response.json()).job;
|
||||
await page.goto(`${workspace}#/images`);
|
||||
// D112: the pill names the held operation and shows no percentage; its bar is indeterminate, not an empty fill
|
||||
// (S5 review M5), and stays hidden from assistive tech, which reads the label.
|
||||
await expect(pill(page)).toHaveText('Thumbnail clear (gallery page)');
|
||||
await expect(pill(page)).toHaveAttribute('aria-label', 'Thumbnail clear (gallery page)');
|
||||
await expect(pill(page).locator('.activity-pill-bar')).toHaveClass('activity-pill-bar indeterminate');
|
||||
await expect(pill(page).locator('.activity-pill-bar')).toHaveAttribute('aria-hidden', 'true');
|
||||
expect(await pill(page).locator('.activity-pill-bar i').evaluate(node => node.style.width)).toBe('');
|
||||
await page.goto(`${workspace}#/activity`);
|
||||
const item = items(page).filter({ has: page.getByRole('link', { name: /Thumbnail clear \(gallery page\)$/ }) });
|
||||
await expect(item.locator('p')).toHaveText(/^Started \d\d:\d\d · closing this view does not cancel it$/);
|
||||
await expect(item.locator('progress')).toHaveCount(1);
|
||||
await expect(item.locator('progress')).not.toHaveAttribute('value');
|
||||
await expect(item.getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(0);
|
||||
await item.getByRole('link').click();
|
||||
await expect(heading(page).locator('h1')).toHaveText('Thumbnail clear (gallery page)');
|
||||
await expect(progressSection(page).locator('progress')).not.toHaveAttribute('value');
|
||||
await expect(progressSection(page).locator('dl.stat-row')).toHaveCount(0);
|
||||
await expect(resultsSection(page)).toHaveCount(0);
|
||||
await expect(scanLog(page)).toHaveCount(1);
|
||||
await finish(request, held.id);
|
||||
await expect(toast(page)).toHaveText('Thumbnail clear (gallery page) finished.');
|
||||
await expect(heading(page).locator('p.muted')).toHaveText(/^Finished \d\d:\d\d$/);
|
||||
await expect(progressSection(page).locator('progress')).toHaveAttribute('value', '100');
|
||||
|
||||
// The health check (a read job of its own kind): its status line, no tiles, no Cancel.
|
||||
const health = await startJob(request, { kind: 'health', total: 0 });
|
||||
await openJob(page, health.id);
|
||||
await expect(heading(page).locator('h1')).toHaveText('Health check');
|
||||
await expect(progressSection(page).locator('.panel-heading > span.small')).toHaveText('Checking the database and image links');
|
||||
await expect(progressSection(page).locator('progress')).not.toHaveAttribute('value');
|
||||
await expect(progressSection(page).locator('dl.stat-row')).toHaveCount(0);
|
||||
await expect(resultsSection(page)).toHaveCount(0);
|
||||
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
|
||||
await expect(pill(page)).toHaveText('Health check');
|
||||
await finish(request, health.id);
|
||||
await expect(toast(page)).toHaveText('Health check finished.');
|
||||
});
|
||||
|
||||
// S5 review I3: a thumbnail job with nothing to generate ends on the server's final line ("No previews to generate · N
|
||||
// already up to date"), never on a stale "Processing file 0 of 0" under the Completed chip.
|
||||
test('a finished thumbnail job with nothing to do says so', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const job = await startJob(request, { ...THUMB_JOB, total: 0 });
|
||||
await step(request, job.id, { total: 0, processed: 0 }); // the stale line an older server left
|
||||
await openJob(page, job.id);
|
||||
const line = progressSection(page).locator('.panel-heading > span.small');
|
||||
await finish(request, job.id, 'finished', { status_line: 'No previews to generate · 24 already up to date' });
|
||||
await expect(line).toHaveText('No previews to generate · 24 already up to date');
|
||||
await expect(page.getByText('Processing file 0 of 0')).toHaveCount(0);
|
||||
await expect(progressSection(page).locator('progress')).toHaveAttribute('value', '100');
|
||||
});
|
||||
|
||||
@@ -1,556 +0,0 @@
|
||||
// Needs review (#249 PR F, plan Tasks F1 and F2, Mock inventory M4): the SAVED VIEWS row and its count, the Metadata
|
||||
// review page over the real review routes, and Activity's Review metadata. Items are recorded by the real recorder
|
||||
// (tests/browser/workspace_review_fixture.py) through the monitor's entry point; the conflict item's file was linked
|
||||
// to another prompt after it was queued. An open page re-reads its queue from the review change stream (F2).
|
||||
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
|
||||
import { test, expect } from './fixtures.mjs';
|
||||
|
||||
const review = '/prompt_manager/workspace#/review';
|
||||
const main = page => page.locator('#workspace-main');
|
||||
const savedNav = page => page.getByRole('navigation', { name: 'Saved views', exact: true });
|
||||
const reviewRow = page => savedNav(page).locator('a[data-route="review"]');
|
||||
const panel = page => main(page).getByRole('region', { name: 'Resolve the association', exact: true });
|
||||
const context = page => panel(page).getByRole('combobox', { name: 'Sampler context', exact: true });
|
||||
const cards = page => main(page).locator('.review-queue article.media-card');
|
||||
const toast = page => page.locator('#workspace-toast');
|
||||
const NOTICE = 'Multiple sampler paths were found. The existing image links were preserved. Choose the correct context or leave them unresolved.';
|
||||
const EMPTY = 'Ambiguous images are added when a metadata scan or a new generation finds more than one sampler path.';
|
||||
const OPTIONS = ['Sampler 11 · Golden dunes at dusk / Soft negative, blur', 'Sampler 13 · Harbor lights at night / Soft negative, blur',
|
||||
'Leave unresolved'];
|
||||
|
||||
async function seed(request, items = 3, special = {}) {
|
||||
const response = await request.post('/__test__/review/reset', { data: { items, ...special } });
|
||||
await expect(response).toBeOK();
|
||||
return response.json();
|
||||
}
|
||||
async function pending(request) {
|
||||
return (await (await request.get('/prompt_manager/review/items?limit=1')).json()).total;
|
||||
}
|
||||
const links = async request => (await (await request.get('/__test__/review/links')).json()).links;
|
||||
const streams = async request => (await (await request.get('/__test__/review/streams')).json());
|
||||
const apply = page => panel(page).getByRole('button', { name: 'Apply association', exact: true });
|
||||
const warning = page => panel(page).locator('.warning-note');
|
||||
/** The page with its first item's sampler paths loaded. */
|
||||
async function open(page) {
|
||||
await page.goto(review);
|
||||
await expect(context(page)).toBeEnabled();
|
||||
}
|
||||
|
||||
test.beforeEach(async ({ page, browserName }) => {
|
||||
await startCoverage(page, browserName);
|
||||
page.reviewErrors = [];
|
||||
page.on('pageerror', error => page.reviewErrors.push(error.message));
|
||||
});
|
||||
test.afterEach(async ({ page, browserName, request }) => {
|
||||
await saveCoverage(page, browserName);
|
||||
expect(page.reviewErrors).toEqual([]);
|
||||
await request.post('/__test__/jobs/restart');
|
||||
});
|
||||
|
||||
test('queue shows persisted items and the nav count', async ({ page, request }) => {
|
||||
await seed(request, 3);
|
||||
await open(page);
|
||||
// SAVED VIEWS: Favorites, then Needs review with the exact pending count (ruling R5).
|
||||
await expect(savedNav(page).locator('a').nth(1)).toHaveAttribute('data-route', 'review');
|
||||
await expect(reviewRow(page)).toHaveText('Needs review3');
|
||||
await expect(reviewRow(page)).toHaveAttribute('aria-current', 'page');
|
||||
await expect(page).toHaveTitle('Review · Prompt Manager');
|
||||
await expect(page.locator('#workspace-breadcrumb')).toHaveText('Review');
|
||||
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Metadata review');
|
||||
await expect(main(page).locator('.heading p.muted')).toHaveText('Uncertain evidence stays visible until you decide.');
|
||||
await expect(main(page).locator('.notice h2')).toHaveText('3 images need a decision');
|
||||
await expect(main(page).locator('.notice p')).toHaveText(NOTICE);
|
||||
await expect(cards(page)).toHaveCount(3);
|
||||
await expect(cards(page).first()).toHaveClass(/selected/);
|
||||
await expect(panel(page).locator('p.small')).toHaveText('review_0000.png');
|
||||
await expect(context(page).locator('option')).toHaveText(OPTIONS);
|
||||
await expect(context(page)).toHaveValue(/^[0-9a-f]{64}$/);
|
||||
await expect(panel(page).locator('.prompt-section.positive .prompt-text')).toHaveText('Golden dunes at dusk');
|
||||
await expect(panel(page).locator('.prompt-section.negative .prompt-text')).toHaveText('Soft negative, blur');
|
||||
await context(page).selectOption({ label: OPTIONS[1] });
|
||||
await expect(panel(page).locator('.prompt-section.positive .prompt-text')).toHaveText('Harbor lights at night');
|
||||
await context(page).selectOption({ label: 'Leave unresolved' });
|
||||
await expect(panel(page).locator('.prompt-section.positive .prompt-text')).toHaveText('Multiple possible prompts. Source could not be determined.');
|
||||
// Info selects another item.
|
||||
await cards(page).nth(1).getByRole('button', { name: 'Inspect review_0001.png', exact: true }).click();
|
||||
await expect(panel(page).locator('p.small')).toHaveText('review_0001.png');
|
||||
await expect(cards(page).nth(1)).toHaveClass(/selected/);
|
||||
await expect(context(page)).toBeEnabled();
|
||||
// The card's image opens the shared viewer.
|
||||
await cards(page).nth(1).getByRole('button', { name: 'Open review_0001.png', exact: true }).click();
|
||||
await expect(page.locator('#viewer-dialog')).toBeVisible();
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(page.locator('#viewer-dialog')).toHaveCount(0);
|
||||
// Open activity.
|
||||
await main(page).getByRole('button', { name: 'Open activity', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity$/);
|
||||
});
|
||||
|
||||
test('apply candidate links the pair and the item leaves the queue and the count drops', async ({ page, request }) => {
|
||||
await seed(request, 2);
|
||||
await open(page);
|
||||
await expect(reviewRow(page)).toHaveText('Needs review2');
|
||||
const decided = page.waitForResponse(item => /\/review\/items\/[^/]+\/decision$/.test(item.url()));
|
||||
await panel(page).getByRole('button', { name: 'Apply association', exact: true }).click();
|
||||
expect((await decided).status()).toBe(200);
|
||||
await expect(toast(page)).toHaveText('Association accepted.');
|
||||
await expect(page).toHaveURL(/#\/images$/);
|
||||
await expect(reviewRow(page)).toHaveText('Needs review1');
|
||||
expect(await pending(request)).toBe(1);
|
||||
// The first sampler path's pair is linked to that file, and only to it.
|
||||
expect(await links(request)).toEqual([{ filename: 'review_0000.png', prompt: 'Golden dunes at dusk',
|
||||
pair_id: expect.any(Number), positive: 'Golden dunes at dusk', negative: 'Soft negative, blur' }]);
|
||||
});
|
||||
|
||||
test('conflict leaves links unchanged and explains', async ({ page, request }) => {
|
||||
// Queued first, then linked to another prompt (before a scan or monitor pass could settle it).
|
||||
await seed(request, 0, { conflict: true });
|
||||
const before = await links(request);
|
||||
expect(before).toEqual([{ filename: 'conflict.png', prompt: 'Misty forest path', pair_id: null, positive: null,
|
||||
negative: null }]);
|
||||
await open(page);
|
||||
await expect(panel(page).locator('p.small')).toHaveText('conflict.png');
|
||||
await expect(context(page).locator('option')).toHaveText(OPTIONS);
|
||||
const decided = page.waitForResponse(item => /\/decision$/.test(item.url()));
|
||||
await apply(page).click();
|
||||
const answer = await decided;
|
||||
expect(answer.status()).toBe(409);
|
||||
expect((await answer.json()).code).toBe('association-conflict');
|
||||
await expect(warning(page)).toHaveText('This image is already linked to another prompt or negative, so this sampler path cannot be attached safely. The existing links were preserved.');
|
||||
await expect(page).toHaveURL(/#\/review$/);
|
||||
await expect(context(page)).toBeEnabled();
|
||||
await expect(context(page)).toHaveValue(/^[0-9a-f]{64}$/);
|
||||
expect(await links(request)).toEqual(before);
|
||||
expect(await pending(request)).toBe(1);
|
||||
await expect(reviewRow(page)).toHaveText('Needs review1');
|
||||
// The file can still be left unresolved, which links nothing either.
|
||||
await context(page).selectOption({ label: 'Leave unresolved' });
|
||||
await apply(page).click();
|
||||
await expect(toast(page)).toHaveText('Left unresolved.');
|
||||
expect(await links(request)).toEqual(before);
|
||||
expect(await pending(request)).toBe(0);
|
||||
});
|
||||
|
||||
test('a limited item and an unavailable file can only be left unresolved', async ({ page, request }) => {
|
||||
await seed(request, 0, { limited: true, unavailable: true });
|
||||
await open(page);
|
||||
await expect(cards(page)).toHaveCount(2);
|
||||
await expect(panel(page).locator('p.small')).toHaveText('limited.png');
|
||||
await expect(context(page).locator('option')).toHaveText(['Leave unresolved']);
|
||||
await expect(warning(page)).toHaveText('This file has too many sampler paths to list. You can leave it unresolved.');
|
||||
// The unavailable file: no current root lists it, so its media and Tag are inert.
|
||||
const unavailable = cards(page).nth(1);
|
||||
await expect(unavailable.locator('.image-open')).toHaveAttribute('aria-disabled', 'true');
|
||||
await expect(unavailable.locator('.image-card-actions button').last()).toHaveAttribute('aria-disabled', 'true');
|
||||
// F3: its empty media button keeps the concept's 1.33 media area, so the card lines up with its neighbours.
|
||||
await expect(cards(page).first().locator('.image-open img')).toBeVisible();
|
||||
const sizes = await cards(page).evaluateAll(nodes => nodes.map(node => {
|
||||
const art = node.querySelector('.image-open').getBoundingClientRect();
|
||||
return [Math.round(art.width), Math.round(art.height), Math.round(node.getBoundingClientRect().height)];
|
||||
}));
|
||||
expect(sizes[1]).toEqual(sizes[0]);
|
||||
await unavailable.getByRole('button', { name: 'Inspect unavailable.png', exact: true }).click();
|
||||
await expect(panel(page).locator('p.small')).toHaveText('unavailable.png');
|
||||
await expect(warning(page)).toHaveText('This file is not under a current image source, so it cannot be opened. You can leave it unresolved.');
|
||||
await expect(panel(page).getByRole('button', { name: 'Tag image', exact: true })).toHaveAttribute('aria-disabled', 'true');
|
||||
const observes = [];
|
||||
page.on('request', item => { if (item.url().endsWith('/image-annotations/observe')) observes.push(item.url()); });
|
||||
await apply(page).click();
|
||||
await expect(toast(page)).toHaveText('Left unresolved.');
|
||||
expect(observes).toEqual([]);
|
||||
expect(await pending(request)).toBe(1);
|
||||
});
|
||||
|
||||
test('leave unresolved removes the item from pending', async ({ page, request }) => {
|
||||
await seed(request, 2);
|
||||
await open(page);
|
||||
await context(page).selectOption({ label: 'Leave unresolved' });
|
||||
await panel(page).getByRole('button', { name: 'Apply association', exact: true }).click();
|
||||
await expect(toast(page)).toHaveText('Left unresolved.');
|
||||
await expect(page).toHaveURL(/#\/images$/);
|
||||
expect(await pending(request)).toBe(1);
|
||||
});
|
||||
|
||||
test('leave for later writes nothing and goes to Images', async ({ page, request }) => {
|
||||
await seed(request, 2);
|
||||
await open(page);
|
||||
const writes = [];
|
||||
page.on('request', item => { if (item.method() !== 'GET') writes.push(item.url()); });
|
||||
await panel(page).getByRole('button', { name: 'Leave for later', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/images$/);
|
||||
expect(writes).toEqual([]);
|
||||
expect(await pending(request)).toBe(2);
|
||||
});
|
||||
|
||||
test('tag image from review saves tags and the item remains pending', async ({ page, request }) => {
|
||||
await seed(request, 1);
|
||||
await open(page);
|
||||
const before = await links(request);
|
||||
await panel(page).getByRole('button', { name: 'Tag image', exact: true }).click();
|
||||
const dialog = page.getByRole('dialog', { name: 'Tag image', exact: true });
|
||||
await expect(dialog).toBeVisible();
|
||||
await expect(dialog.locator('p.tag-scope-note')).toHaveText('Tags belong to this image. Resolve its prompt association separately in Needs review.');
|
||||
await dialog.getByRole('textbox', { name: 'Image tags · edit manually', exact: true }).fill('dunes');
|
||||
await dialog.getByRole('button', { name: 'Save manual tags', exact: true }).click();
|
||||
await expect(toast(page)).toHaveText('1 tags saved for review_0000.png. Prompt association still awaits review.');
|
||||
await expect(dialog).toHaveCount(0);
|
||||
expect(await pending(request)).toBe(1);
|
||||
await expect(page).toHaveURL(/#\/review$/);
|
||||
// The item is untouched: still listed, still open, nothing linked.
|
||||
await expect(cards(page)).toHaveCount(1);
|
||||
await expect(context(page)).toBeEnabled();
|
||||
expect(await links(request)).toEqual(before);
|
||||
const [item] = (await (await request.get('/prompt_manager/review/items')).json()).items;
|
||||
expect([item.state, item.revision]).toEqual(['pending', 0]);
|
||||
// The card's Tag opens the same dialog.
|
||||
await cards(page).first().getByRole('button', { name: 'Tag or auto-tag review_0000.png', exact: true }).click();
|
||||
await expect(dialog).toBeVisible();
|
||||
await expect(dialog.locator('p.tag-scope-note')).toBeVisible();
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(dialog).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('an empty queue shows the empty notice only', async ({ page, request }) => {
|
||||
await seed(request, 0);
|
||||
await page.goto(review);
|
||||
await expect(main(page).locator('.notice h2')).toHaveText('No images need a decision');
|
||||
await expect(main(page).locator('.notice p')).toHaveText(EMPTY);
|
||||
await expect(main(page).locator('.gallery-layout')).toHaveCount(0);
|
||||
await expect(reviewRow(page)).toHaveText('Needs review0');
|
||||
});
|
||||
|
||||
test('load more appends the next page', async ({ page, request }) => {
|
||||
await seed(request, 45);
|
||||
await open(page);
|
||||
await expect(cards(page)).toHaveCount(40);
|
||||
const continuation = main(page).locator('.infinite-continuation');
|
||||
await expect(continuation.locator('p')).toHaveText('40 of 45 images');
|
||||
await continuation.getByRole('button', { name: 'Load more images', exact: true }).click();
|
||||
await expect(cards(page)).toHaveCount(45);
|
||||
await expect(continuation).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('a scan job with review items offers Review metadata in Activity', async ({ page, request }) => {
|
||||
await seed(request, 1);
|
||||
const started = await (await request.post('/__test__/jobs/start', { data: { total: 10, review_items: 2 } })).json();
|
||||
await expect(await request.post(`/__test__/jobs/${started.job.id}/finish`, { data: {} })).toBeOK();
|
||||
const plain = await (await request.post('/__test__/jobs/start', { data: { total: 10 } })).json();
|
||||
await expect(await request.post(`/__test__/jobs/${plain.job.id}/finish`, { data: {} })).toBeOK();
|
||||
await page.goto('/prompt_manager/workspace#/activity');
|
||||
const item = main(page).locator(`.activity-item[data-job-id="${started.job.id}"]`);
|
||||
await expect(item.getByRole('button', { name: 'Review metadata', exact: true })).toBeVisible();
|
||||
await expect(main(page).locator(`.activity-item[data-job-id="${plain.job.id}"] button`)).toHaveCount(0);
|
||||
await page.goto(`/prompt_manager/workspace#/activity/${started.job.id}`);
|
||||
const actions = main(page).locator('.heading-actions button');
|
||||
await expect(actions).toHaveText(['Download report', 'Review metadata']);
|
||||
await actions.nth(1).click();
|
||||
await expect(page).toHaveURL(/#\/review$/);
|
||||
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Metadata review');
|
||||
});
|
||||
|
||||
test('a failed item read says so and Retry reads it again', async ({ page, request }) => {
|
||||
await seed(request, 1);
|
||||
await page.route(url => /\/prompt_manager\/review\/items\/[0-9a-f-]{36}$/.test(url.pathname),
|
||||
route => route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, code: 'review-failed', error: 'x' }) }),
|
||||
{ times: 1 });
|
||||
await page.goto(review);
|
||||
await expect(panel(page).locator('.warning-note')).toHaveText('This item could not be read. Select it again to retry.');
|
||||
await expect(panel(page).getByText('Could not read prompts from this file.', { exact: true })).toBeVisible();
|
||||
await expect(panel(page).getByRole('button', { name: 'Apply association', exact: true })).toHaveAttribute('aria-disabled', 'true');
|
||||
await panel(page).getByRole('button', { name: 'Retry prompt details', exact: true }).click();
|
||||
await expect(context(page)).toBeEnabled();
|
||||
await expect(context(page).locator('option')).toHaveText(OPTIONS);
|
||||
await expect(panel(page).locator('.warning-note')).toBeHidden();
|
||||
});
|
||||
|
||||
test('a failed queue read offers Retry', async ({ page, request }) => {
|
||||
await seed(request, 1);
|
||||
await page.route(url => url.pathname.endsWith('/prompt_manager/review/items') && url.searchParams.get('limit') === '40',
|
||||
route => route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, code: 'review-failed', error: 'x' }) }),
|
||||
{ times: 1 });
|
||||
await page.goto(review);
|
||||
await expect(main(page).getByRole('alert')).toHaveText('The review queue could not be read.');
|
||||
await main(page).getByRole('button', { name: 'Retry', exact: true }).click();
|
||||
await expect(context(page)).toBeEnabled();
|
||||
await expect(main(page).locator('.notice h2')).toHaveText('1 image needs a decision');
|
||||
});
|
||||
|
||||
test('keyboard focus labels and touch targets', async ({ page, request }) => {
|
||||
await seed(request, 2);
|
||||
await open(page);
|
||||
const first = cards(page).first();
|
||||
for (const [name, role] of [['Open review_0000.png', 'button'], ['Inspect review_0000.png', 'button'],
|
||||
['Tag or auto-tag review_0000.png', 'button']]) await expect(first.getByRole(role, { name, exact: true })).toBeVisible();
|
||||
await expect(main(page).getByRole('list', { name: 'Review queue', exact: true })).toBeVisible();
|
||||
await expect(panel(page)).toBeVisible();
|
||||
// Keyboard: Info on the second card opens it, and the panel's controls take focus in their order.
|
||||
const info = cards(page).nth(1).getByRole('button', { name: 'Inspect review_0001.png', exact: true });
|
||||
await info.focus();
|
||||
await page.keyboard.press('Enter');
|
||||
await expect(panel(page).locator('p.small')).toHaveText('review_0001.png');
|
||||
await expect(context(page)).toBeEnabled();
|
||||
await context(page).focus();
|
||||
await expect(context(page)).toBeFocused();
|
||||
const order = await panel(page).evaluate(node => [...node.querySelectorAll('button, select')]
|
||||
.filter(control => control.offsetParent !== null).map(control => control.getAttribute('aria-label') || control.textContent.trim()));
|
||||
expect(order[0]).toBe('Tag image');
|
||||
expect(order.slice(-2)).toEqual(['Leave for later', 'Apply association']);
|
||||
// Targets: the panel's controls are 40px on a coarse pointer (the concept's coarse buttons) and 24px otherwise (the
|
||||
// WCAG 2.2 minimum); the card's buttons keep the concept's compact size, never under 24px (its ≤700px rule).
|
||||
const coarse = await page.evaluate(() => matchMedia('(pointer: coarse)').matches);
|
||||
const sized = [
|
||||
[panel(page).getByRole('button', { name: 'Tag image', exact: true }), coarse ? 40 : 24],
|
||||
[context(page), coarse ? 40 : 24],
|
||||
[panel(page).getByRole('button', { name: 'Leave for later', exact: true }), coarse ? 40 : 24],
|
||||
[apply(page), coarse ? 40 : 24],
|
||||
[first.getByRole('button', { name: 'Inspect review_0000.png', exact: true }), 24],
|
||||
[first.getByRole('button', { name: 'Tag or auto-tag review_0000.png', exact: true }), 24],
|
||||
];
|
||||
for (const [control, minimum] of sized) {
|
||||
const box = await control.boundingBox();
|
||||
const label = await control.evaluate(node => node.getAttribute('aria-label') || node.textContent.trim().slice(0, 40));
|
||||
expect(box.height, label).toBeGreaterThanOrEqual(minimum);
|
||||
expect(box.width, label).toBeGreaterThanOrEqual(24);
|
||||
}
|
||||
// Enter on Leave for later returns to Images.
|
||||
await panel(page).getByRole('button', { name: 'Leave for later', exact: true }).focus();
|
||||
await page.keyboard.press('Enter');
|
||||
await expect(page).toHaveURL(/#\/images$/);
|
||||
});
|
||||
|
||||
test('an open page shows items the monitor adds and keeps the open choice', async ({ page, request }) => {
|
||||
await seed(request, 1);
|
||||
await open(page);
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(1);
|
||||
await context(page).selectOption({ label: OPTIONS[1] });
|
||||
await expect(reviewRow(page)).toHaveText('Needs review1');
|
||||
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
|
||||
await expect(cards(page)).toHaveCount(2);
|
||||
await expect(main(page).locator('.notice h2')).toHaveText('2 images need a decision');
|
||||
await expect(reviewRow(page)).toHaveText('Needs review2');
|
||||
await expect(cards(page).first()).toHaveClass(/selected/);
|
||||
await expect(panel(page).locator('p.small')).toHaveText('review_0000.png');
|
||||
await expect(context(page).locator('option:checked')).toHaveText(OPTIONS[1]);
|
||||
await expect(warning(page)).toBeHidden();
|
||||
});
|
||||
|
||||
test('an open page shows items a metadata scan adds', async ({ page, request }) => {
|
||||
await seed(request, 1);
|
||||
await expect(await request.post('/__test__/review/add', { data: { kind: 'file' } })).toBeOK();
|
||||
await open(page);
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(1);
|
||||
await expect(cards(page)).toHaveCount(1);
|
||||
const started = await request.post('/prompt_manager/jobs/scan', { data: { scope: 'new' } });
|
||||
expect(started.status()).toBe(202);
|
||||
await expect(cards(page)).toHaveCount(2);
|
||||
await expect(cards(page).nth(1).getByRole('button', { name: 'Inspect added_0000.png', exact: true })).toBeVisible();
|
||||
await expect(reviewRow(page)).toHaveText('Needs review2');
|
||||
});
|
||||
|
||||
test('a decision in another tab leaves the open page and opens the next image', async ({ page, request, context: browserContext }) => {
|
||||
await seed(request, 2);
|
||||
await open(page);
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(1);
|
||||
const other = await browserContext.newPage();
|
||||
await open(other);
|
||||
await context(other).selectOption({ label: 'Leave unresolved' });
|
||||
await apply(other).click();
|
||||
await expect(other).toHaveURL(/#\/images$/);
|
||||
await other.close();
|
||||
await expect(cards(page)).toHaveCount(1);
|
||||
await expect(panel(page).locator('p.small')).toHaveText('review_0001.png');
|
||||
await expect(warning(page)).toHaveText('review_0000.png was decided elsewhere, so the next image is open.');
|
||||
await expect(main(page).locator('.notice h2')).toHaveText('1 image needs a decision');
|
||||
await expect(reviewRow(page)).toHaveText('Needs review1');
|
||||
await expect(context(page)).toBeEnabled();
|
||||
});
|
||||
|
||||
/** The queue reads (`?limit=40`, cursors) and count reads (`?limit=1`) the page sends from now on. */
|
||||
function reviewReads(page) {
|
||||
const reads = [];
|
||||
page.on('request', item => {
|
||||
const url = new URL(item.url());
|
||||
if (url.pathname === '/prompt_manager/review/items') reads.push(url.searchParams.get('cursor') ? 'more' : url.search);
|
||||
});
|
||||
return reads;
|
||||
}
|
||||
|
||||
test('a visit reads its queue once, and a change reads it once more', async ({ page, request }) => {
|
||||
// F3 ruling: the stream's ready must not read a queue the first read already holds.
|
||||
await seed(request, 1);
|
||||
const reads = reviewReads(page);
|
||||
await open(page);
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(1);
|
||||
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
|
||||
await expect(cards(page)).toHaveCount(2);
|
||||
await expect(reviewRow(page)).toHaveText('Needs review2');
|
||||
// The page's first read and the change's re-read; the shell's count read at start, and the row follows the page.
|
||||
expect(reads.filter(search => search === '?limit=40')).toHaveLength(2);
|
||||
expect(reads.filter(search => search === '?limit=1')).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('a ready whose version cannot be read re-reads the queue once', async ({ page, request }) => {
|
||||
// The stream's ready names no readable version (an older server, a proxy that rewrote it): the page reads again.
|
||||
await seed(request, 1);
|
||||
const reads = reviewReads(page);
|
||||
// M1: `retry` keeps the browser from reconnecting the ended stream during the test, so exactly one re-read follows.
|
||||
await page.route(url => url.pathname === '/prompt_manager/review/events', route => route.fulfill({
|
||||
status: 200, headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-store' },
|
||||
body: 'retry: 3600000\nevent: ready\ndata: not json\n\n' }));
|
||||
let reread = 0;
|
||||
const second = page.waitForResponse(item => item.url().endsWith('/prompt_manager/review/items?limit=40') && ++reread === 2);
|
||||
await open(page);
|
||||
await second;
|
||||
await expect(cards(page)).toHaveCount(1);
|
||||
expect(reads.filter(search => search === '?limit=40')).toHaveLength(2);
|
||||
});
|
||||
|
||||
const live = page => main(page).locator('.heading .review-live');
|
||||
const NOT_LIVE = 'Live updates are paused: changes made elsewhere appear once the connection returns.';
|
||||
|
||||
test('a stream the browser gives up on is opened again, and the page says it is not live meanwhile', async ({ page, request }) => {
|
||||
// I3: a 503 (shutdown, or a proxy while the backend restarts) fails an EventSource for good.
|
||||
await seed(request, 1);
|
||||
let attempts = 0;
|
||||
await page.route(url => url.pathname === '/prompt_manager/review/events', route => {
|
||||
attempts += 1;
|
||||
if (attempts === 1) return route.fulfill({ status: 503, contentType: 'application/json', body: '{"success":false}' });
|
||||
return route.fallback();
|
||||
});
|
||||
await open(page);
|
||||
await expect(live(page)).toHaveText(NOT_LIVE);
|
||||
await expect(live(page)).toBeVisible();
|
||||
// The page opens the stream again (after 3 s) and is live once it sends ready.
|
||||
await expect.poll(async () => (await streams(request)).streams, { timeout: 15000 }).toBe(1);
|
||||
await expect(live(page)).toBeHidden();
|
||||
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
|
||||
await expect(cards(page)).toHaveCount(2);
|
||||
expect(attempts).toBe(2);
|
||||
});
|
||||
|
||||
/** Make the page hidden or visible as a tab switch does. */
|
||||
async function setVisible(page, visible) {
|
||||
await page.evaluate(shown => {
|
||||
Object.defineProperty(document, 'hidden', { configurable: true, get: () => !shown });
|
||||
Object.defineProperty(document, 'visibilityState', { configurable: true, get: () => (shown ? 'visible' : 'hidden') });
|
||||
document.dispatchEvent(new Event('visibilitychange'));
|
||||
}, visible);
|
||||
}
|
||||
|
||||
test('a hidden tab closes its stream and a visible one reads only what changed meanwhile', async ({ page, request }) => {
|
||||
// I4: background tabs must not hold one of the origin's six connections.
|
||||
await seed(request, 1);
|
||||
const reads = reviewReads(page);
|
||||
await open(page);
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(1);
|
||||
await setVisible(page, false);
|
||||
// The server drops a closed stream at its next write: this monitor addition is one.
|
||||
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(0);
|
||||
await expect(cards(page)).toHaveCount(1);
|
||||
expect(reads.filter(search => search === '?limit=40')).toHaveLength(1);
|
||||
// Visible again: the reopened stream's ready names a newer version, so the page reads once.
|
||||
const reread = page.waitForResponse(item => item.url().endsWith('/prompt_manager/review/items?limit=40'));
|
||||
await setVisible(page, true);
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(1);
|
||||
await reread;
|
||||
await expect(cards(page)).toHaveCount(2);
|
||||
await expect(live(page)).toBeHidden();
|
||||
expect(reads.filter(search => search === '?limit=40')).toHaveLength(2);
|
||||
});
|
||||
|
||||
test('a re-read refreshes a card whose media or target changed', async ({ page, request }) => {
|
||||
// M6 and M10: a root came or went between reads; the card follows, and a known thumbnail is its media.
|
||||
await seed(request, 2);
|
||||
let rewrite = false;
|
||||
await page.route(url => url.pathname === '/prompt_manager/review/items' && url.search === '?limit=40', async route => {
|
||||
if (!rewrite) return route.fallback();
|
||||
const response = await route.fetch();
|
||||
const body = await response.json();
|
||||
body.items[0] = { ...body.items[0], thumbnail_url: `${body.items[0].url}&thumb=1` };
|
||||
body.items[1] = { ...body.items[1], url: null, thumbnail_url: null, target: null, unavailable: true };
|
||||
return route.fulfill({ response, json: body });
|
||||
});
|
||||
await open(page);
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(1);
|
||||
const second = cards(page).nth(1);
|
||||
await expect(second.getByRole('button', { name: 'Open review_0001.png', exact: true })).not.toHaveAttribute('aria-disabled', 'true');
|
||||
rewrite = true;
|
||||
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
|
||||
await expect(cards(page)).toHaveCount(3);
|
||||
await expect(cards(page).first().locator('img')).toHaveAttribute('src', /&thumb=1$/);
|
||||
await expect(second.getByRole('button', { name: 'Open review_0001.png', exact: true })).toHaveAttribute('aria-disabled', 'true');
|
||||
await expect(second.locator('img')).toHaveCount(0);
|
||||
await expect(second.locator('.image-card-actions button').last()).toHaveAttribute('aria-disabled', 'true');
|
||||
await expect(cards(page).first()).toHaveClass(/selected/);
|
||||
});
|
||||
|
||||
test('leaving during the observe releases the capture it answers', async ({ page, request }) => {
|
||||
// M4: the server made the capture; the page left before the answer came.
|
||||
await seed(request, 1);
|
||||
await open(page);
|
||||
let held;
|
||||
const sent = new Promise(resolve => { held = resolve; });
|
||||
await page.route(url => url.pathname === '/prompt_manager/image-annotations/observe', async route => {
|
||||
const response = await route.fetch();
|
||||
held(() => route.fulfill({ response }));
|
||||
});
|
||||
await apply(page).click();
|
||||
const answer = await sent;
|
||||
expect((await streams(request)).captures).toBe(1);
|
||||
await page.evaluate(() => { window.location.hash = '#/tags'; });
|
||||
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Tags & categories');
|
||||
await answer();
|
||||
await expect.poll(async () => (await streams(request)).captures).toBe(0);
|
||||
});
|
||||
|
||||
test('a live re-read keeps the loaded depth and the scroll position', async ({ page, request }) => {
|
||||
// F3 ruling: a queue change re-reads as many items as Load more had shown, and the page does not jump.
|
||||
await seed(request, 45);
|
||||
await open(page);
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(1);
|
||||
await main(page).getByRole('button', { name: 'Load more images', exact: true }).click();
|
||||
await expect(cards(page)).toHaveCount(45);
|
||||
await cards(page).last().scrollIntoViewIfNeeded();
|
||||
const scrollTop = () => cards(page).first().evaluate(node => {
|
||||
for (let el = node.parentElement; el; el = el.parentElement) {
|
||||
const style = getComputedStyle(el);
|
||||
if (/(auto|scroll)/.test(style.overflowY) && el.scrollHeight > el.clientHeight) return el.scrollTop;
|
||||
}
|
||||
return document.scrollingElement.scrollTop;
|
||||
});
|
||||
const before = await scrollTop();
|
||||
expect(before).toBeGreaterThan(0);
|
||||
const reads = reviewReads(page);
|
||||
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
|
||||
await expect(main(page).locator('.notice h2')).toHaveText('46 images need a decision');
|
||||
await expect(cards(page)).toHaveCount(46);
|
||||
expect(await scrollTop()).toBe(before);
|
||||
expect(reads).toEqual(['?limit=40', 'more']);
|
||||
await expect(main(page).locator('.infinite-continuation')).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('teardown releases captures listeners and requests', async ({ page, request }) => {
|
||||
await seed(request, 1);
|
||||
await open(page);
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(1);
|
||||
const inFlight = new Set();
|
||||
page.on('request', item => { if (item.url().includes('/prompt_manager/review/')) inFlight.add(item); });
|
||||
page.on('requestfinished', item => inFlight.delete(item));
|
||||
page.on('requestfailed', item => inFlight.delete(item));
|
||||
// A decision whose answer never comes: leaving the page must still release its capture.
|
||||
let held;
|
||||
const sent = new Promise(resolve => { held = resolve; });
|
||||
await page.route(url => /\/decision$/.test(url.pathname), route => held(route));
|
||||
await apply(page).click();
|
||||
const route = await sent;
|
||||
expect((await streams(request)).captures).toBe(1);
|
||||
await page.evaluate(() => { window.location.hash = '#/tags'; });
|
||||
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Tags & categories');
|
||||
await expect.poll(async () => (await streams(request)).captures).toBe(0);
|
||||
await route.abort().catch(() => {});
|
||||
// The server drops a closed stream at its next write: a monitor addition is one.
|
||||
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
|
||||
await expect.poll(async () => (await streams(request)).streams).toBe(0);
|
||||
await expect.poll(() => inFlight.size).toBe(0);
|
||||
});
|
||||
@@ -372,7 +372,7 @@ test('Favorites is the first SAVED VIEWS row, with a real count that follows eve
|
||||
expect(await favoritesRow(page).evaluate(link => [...link.childNodes].map(node => node.nodeType === Node.TEXT_NODE
|
||||
? `text:${node.textContent}` : `${node.localName}.${node.getAttribute('class')}`)))
|
||||
.toEqual(['svg.icon', 'text:Favorites', 'span.nav-count']);
|
||||
expect(await savedNav(page).locator('a').evaluateAll(links => links.map(link => link.dataset.route))).toEqual(['favorites', 'review']);
|
||||
expect(await savedNav(page).locator('a').evaluateAll(links => links.map(link => link.dataset.route))).toEqual(['favorites']);
|
||||
|
||||
await star(page, '1:shared/same.png');
|
||||
await expect(toast(page)).toHaveText('Added to favorites.');
|
||||
|
||||
@@ -116,7 +116,7 @@ test('Scan changes opens it with that one source ("1 output folder") and starts
|
||||
const id = page.url().split('/').pop();
|
||||
// The job walks only that source (the count is recorded once the walk has listed it).
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.scope)
|
||||
.toMatchObject({ files: 'new', roots: [1], output_folders: 1 });
|
||||
.toEqual({ files: 'new', roots: [1], output_folders: 1 });
|
||||
await expect(page.locator('#workspace-main h1')).toHaveText('Scan for metadata');
|
||||
});
|
||||
|
||||
@@ -304,151 +304,16 @@ test('Refresh files calls /gallery/rescan and toasts its counts', async ({ page
|
||||
await expect(toast(page)).toHaveText('Files could not be refreshed. Try again.');
|
||||
});
|
||||
|
||||
const THUMBNAIL_PLAN_URL = '/prompt_manager/images/thumbnails/plan';
|
||||
const THUMBNAIL_JOB_URL = '/prompt_manager/jobs/thumbnails';
|
||||
const planCount = value => value.toLocaleString('en-US');
|
||||
|
||||
/** Open "Thumbnail maintenance" from Sources' Thumbnail generation → Review options. */
|
||||
async function openThumbnailReview(page) {
|
||||
test('Thumbnail generation hands off to the existing gallery until its review dialog', async ({ page }) => {
|
||||
const layout = await openSources(page);
|
||||
const thumbnails = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
|
||||
await expect(thumbnails.locator('p')).toHaveText('Generate missing previews; rebuilding all is a separate explicit option.');
|
||||
const popup = page.waitForEvent('popup');
|
||||
await thumbnails.getByRole('button', { name: 'Review options', exact: true }).click();
|
||||
await expect(dialog(page).locator('#modal-title')).toHaveText('Thumbnail maintenance');
|
||||
return dialog(page);
|
||||
}
|
||||
|
||||
/** The thumbnail requests this page sends: plan reads (their query) and job starts (their body). */
|
||||
function thumbnailRequests(page) {
|
||||
const seen = { plans: [], starts: [] };
|
||||
page.on('request', item => {
|
||||
const url = new URL(item.url());
|
||||
if (url.pathname === THUMBNAIL_PLAN_URL) seen.plans.push(Object.fromEntries(url.searchParams));
|
||||
if (item.method() === 'POST' && url.pathname === THUMBNAIL_JOB_URL) seen.starts.push(item.postDataJSON());
|
||||
});
|
||||
return seen;
|
||||
}
|
||||
|
||||
const missingLine = plan => `${planCount(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ` +
|
||||
`${planCount(plan.fresh)} already up to date.`;
|
||||
|
||||
test('Review options shows the plan counts for the chosen scope/quality/size before Start', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 3 } })).toBeOK();
|
||||
// Image loading quality (Appearance, Q3) preselects the dialog's quality.
|
||||
await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('gallerySettings', JSON.stringify({ imageQuality: 'high' })); });
|
||||
const seen = thumbnailRequests(page);
|
||||
const plan = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=missing&size=512&quality=high`);
|
||||
expect(plan.targets).toBe(3);
|
||||
const review = await openThumbnailReview(page);
|
||||
await expect(review.locator('#modal-body > *')).toHaveCount(5);
|
||||
await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']);
|
||||
await expect(review.locator('#thumb-mode option')).toHaveText(['Missing thumbnails only', 'Rebuild all (explicit opt-in)']);
|
||||
await expect(review.locator('#thumb-quality option')).toHaveText(['Medium · optimized', 'High · original', 'Low · fastest']);
|
||||
await expect(review.locator('#thumb-mode')).toHaveValue('missing');
|
||||
await expect(review.locator('#thumb-quality')).toHaveValue('high');
|
||||
await expect(review.locator('#thumb-limit')).toHaveValue('512');
|
||||
await expect(review.locator('#thumb-limit')).toHaveAttribute('min', '128');
|
||||
await expect(review.locator('#thumb-limit')).toHaveAttribute('max', '2048');
|
||||
const line = review.locator('#modal-body > p.small');
|
||||
await expect(line).toHaveAttribute('role', 'status');
|
||||
await expect(line).toHaveText(missingLine(plan));
|
||||
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'high' }]);
|
||||
await expect(review.locator('.dialog-actions > button')).toHaveText(['Cancel', 'Start']);
|
||||
// Reviewing starts nothing; Cancel closes.
|
||||
await review.getByRole('button', { name: 'Cancel', exact: true }).click();
|
||||
await expect(dialog(page)).toHaveCount(0);
|
||||
expect(seen.starts).toEqual([]);
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
|
||||
});
|
||||
|
||||
test('size outside 128–2048 reports validity and requests nothing', async ({ page, request }) => {
|
||||
const seen = thumbnailRequests(page);
|
||||
const review = await openThumbnailReview(page);
|
||||
const line = review.locator('#modal-body > p.small');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
const size = review.locator('#thumb-limit');
|
||||
for (const value of ['4096', '64', '']) {
|
||||
await size.fill(value);
|
||||
await expect(size).toHaveAttribute('aria-invalid', 'true');
|
||||
await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.');
|
||||
expect(await size.evaluate(input => input.validity.valid)).toBe(false);
|
||||
// Start keeps the dialog open and sends nothing; neither does a scope change.
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(dialog(page).locator('#modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await review.locator('#thumb-mode').selectOption('rebuild');
|
||||
await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.');
|
||||
await review.locator('#thumb-mode').selectOption('missing');
|
||||
}
|
||||
expect(seen.starts).toEqual([]);
|
||||
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }]);
|
||||
await size.fill('1024');
|
||||
await expect(size).not.toHaveAttribute('aria-invalid');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
|
||||
});
|
||||
|
||||
test('Start closes the dialog, toasts, and opens the job in Activity', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK();
|
||||
const seen = thumbnailRequests(page);
|
||||
const review = await openThumbnailReview(page);
|
||||
await review.locator('#thumb-quality').selectOption('low');
|
||||
await review.locator('#thumb-limit').fill('256');
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(dialog(page)).toHaveCount(0);
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
// The toast says it started (or, for a run this quick, that it already finished); Activity shows its progress.
|
||||
await expect(toast(page)).toHaveText(/^(Thumbnail generation started\.|Thumbnails finished · 2 generated · 0 errors)$/);
|
||||
expect(seen.starts).toEqual([{ mode: 'missing', size: 256, quality: 'low' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
const { job } = await serverJson(request, `/prompt_manager/jobs/${id}`);
|
||||
expect(job).toMatchObject({ kind: 'thumbnails', title: 'Thumbnail generation', scope: { mode: 'missing', size: 256, quality: 'low' } });
|
||||
await expect(page.locator('#workspace-main h1')).toHaveText('Thumbnail generation');
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
});
|
||||
|
||||
test('Rebuild all is explicit and the plan line says every preview is regenerated', async ({ page, request }) => {
|
||||
const seen = thumbnailRequests(page);
|
||||
const rebuild = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=rebuild&size=512&quality=medium`);
|
||||
expect(rebuild.targets).toBeGreaterThan(0);
|
||||
const review = await openThumbnailReview(page);
|
||||
// Missing thumbnails only is the default; rebuilding is the explicit second option.
|
||||
await expect(review.locator('#thumb-mode')).toHaveValue('missing');
|
||||
const line = review.locator('#modal-body > p.small');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
await review.locator('#thumb-mode').selectOption('rebuild');
|
||||
await expect(line).toHaveText(`Every preview is regenerated: ${planCount(rebuild.targets)} files.`);
|
||||
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }, { mode: 'rebuild', size: '512', quality: 'medium' }]);
|
||||
// Back and forth reads each scope once.
|
||||
await review.locator('#thumb-mode').selectOption('missing');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
await review.locator('#thumb-mode').selectOption('rebuild');
|
||||
await expect(line).toHaveText(`Every preview is regenerated: ${planCount(rebuild.targets)} files.`);
|
||||
expect(seen.plans).toHaveLength(2);
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
expect(seen.starts).toEqual([{ mode: 'rebuild', size: 512, quality: 'medium' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
});
|
||||
|
||||
test('a plan that cannot be read says so; Start still starts', async ({ page }) => {
|
||||
await page.route(`**${THUMBNAIL_PLAN_URL}?*`, route => route.fulfill({ status: 500, json: { success: false, error: 'disk' } }));
|
||||
const review = await openThumbnailReview(page);
|
||||
await expect(review.locator('#modal-body > p.small')).toHaveText('The previews could not be counted.');
|
||||
await expect(review.getByRole('button', { name: 'Start', exact: true })).not.toHaveAttribute('aria-disabled');
|
||||
await review.getByRole('button', { name: 'Cancel', exact: true }).click();
|
||||
});
|
||||
|
||||
test('a refused start shows the busy link toast', async ({ page, request }) => {
|
||||
const running = await (await request.post('/__test__/jobs/start', { data: { title: 'Scan for metadata' } })).json();
|
||||
const review = await openThumbnailReview(page);
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(dialog(page)).toHaveCount(0);
|
||||
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
|
||||
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
|
||||
await expect(page).toHaveURL(`${workspace}#/system/sources`);
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs.map(job => job.kind)).toEqual(['scan']);
|
||||
await request.post(`/__test__/jobs/${running.job.id}/finish`, { data: { state: 'finished' } });
|
||||
const gallery = await popup;
|
||||
expect(new URL(gallery.url()).pathname).toBe('/prompt_manager/gallery');
|
||||
await gallery.close();
|
||||
await expect(statusOf(page)).toHaveText('No unsaved changes.');
|
||||
});
|
||||
|
||||
test('a save refused by a running scan shows the busy link toast', async ({ page, request }) => {
|
||||
|
||||
@@ -55,8 +55,6 @@ test('overview shows real counts and Not checked', async ({ page, request }) =>
|
||||
expect([overview.library.positive_prompts, overview.images.indexed, thumbnails.pending]).toEqual([6, 24, 0]);
|
||||
// I2: the Overview never walks the sources for thumbnails; the row reads the status route itself.
|
||||
expect(overview).not.toHaveProperty('thumbnails');
|
||||
// The fixture records no ambiguous file: the Needs review queue is empty.
|
||||
expect(overview.needs_review).toEqual({ pending: 0 });
|
||||
const layout = await openOverview(page);
|
||||
expect(await layout.locator('.summary-metrics > .metric').evaluateAll(items => items.map(item =>
|
||||
[item.querySelector('strong').textContent, item.querySelector('span').textContent]))).toEqual([
|
||||
@@ -66,7 +64,7 @@ test('overview shows real counts and Not checked', async ({ page, request }) =>
|
||||
['Not checked', 'Database integrity'],
|
||||
]);
|
||||
expect(await rowsOf(layout, 'Library health')).toEqual([
|
||||
['0 ambiguous image links', 'Review metadata before assigning a positive / negative pair.', 'Review metadata'],
|
||||
['Ambiguous image links', 'Not available yet.', 'Review metadata'],
|
||||
['Backup status', 'No retained backup record.', 'View backups'],
|
||||
['No missing thumbnails', 'Generate only missing previews when necessary.', 'Review thumbnails'],
|
||||
]);
|
||||
@@ -140,32 +138,9 @@ test('thumbnail row reflects the status endpoint', async ({ page, request }) =>
|
||||
const layout = await openOverview(page);
|
||||
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
|
||||
await expect(row.locator('h3')).toHaveText('3 missing thumbnails');
|
||||
});
|
||||
|
||||
test('Overview Review thumbnails opens the same dialog', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK();
|
||||
const plan = await serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium');
|
||||
const starts = [];
|
||||
page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/thumbnails')) starts.push(item.postDataJSON()); });
|
||||
const layout = await openOverview(page);
|
||||
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
|
||||
await expect(row.locator('h3')).toHaveText('2 missing thumbnails');
|
||||
// Until the thumbnail review dialog (S5), Review thumbnails opens Sources.
|
||||
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
|
||||
const review = page.locator('dialog#modal[open]');
|
||||
await expect(review.locator('#modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await expect(page).toHaveURL(`${workspace}#/system/overview`);
|
||||
await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']);
|
||||
await expect(review.locator('#modal-body > p.small')).toHaveText(`2 previews to generate · ${number(plan.fresh)} already up to date.`);
|
||||
// Escape closes it and focus returns to the row's button.
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(review).toHaveCount(0);
|
||||
await expect(row.getByRole('button', { name: 'Review thumbnails', exact: true })).toBeFocused();
|
||||
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
expect(starts).toEqual([{ mode: 'missing', size: 512, quality: 'medium' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
await expect(page).toHaveURL(`${workspace}#/system/sources`);
|
||||
});
|
||||
|
||||
test('the thumbnail row paints when its own status read answers', async ({ page, request }) => {
|
||||
@@ -191,17 +166,14 @@ test('a failed thumbnail read is unavailable, never zero', async ({ page }) => {
|
||||
await expect(layout.locator('.metric').nth(2).locator('strong')).toHaveText('24');
|
||||
});
|
||||
|
||||
test('ambiguous row reads the pending count and Review metadata opens Needs review', async ({ page }) => {
|
||||
test('ambiguous row reads Not available yet (Q6)', async ({ page }) => {
|
||||
const layout = await openOverview(page);
|
||||
const row = layout.locator('.operation').filter({ hasText: 'ambiguous image links' });
|
||||
await expect(row.locator('h3')).toHaveText('0 ambiguous image links');
|
||||
await expect(row.locator('p')).toHaveText('Review metadata before assigning a positive / negative pair.');
|
||||
const row = layout.locator('.operation').filter({ hasText: 'Ambiguous image links' });
|
||||
await expect(row.locator('p')).toHaveText('Not available yet.');
|
||||
const review = row.getByRole('button', { name: 'Review metadata', exact: true });
|
||||
await expect(review).not.toHaveAttribute('aria-disabled');
|
||||
await review.click();
|
||||
// The concept's go-review: the Needs review page.
|
||||
await expect(page).toHaveURL(`${workspace}#/review`);
|
||||
await expect(page.getByRole('heading', { name: 'Metadata review', exact: true })).toBeVisible();
|
||||
await expect(review).toHaveAttribute('aria-disabled', 'true');
|
||||
await review.click({ force: true });
|
||||
await expect(page).toHaveURL(`${workspace}#/system/overview`);
|
||||
});
|
||||
|
||||
test('View backups / Appearance / Manage sources switch sections', async ({ page }) => {
|
||||
|
||||
@@ -22,8 +22,6 @@ const routes = [
|
||||
['system', 'System', null],
|
||||
// #249 D51: SAVED VIEWS' Favorites, Images filtered to favorites.
|
||||
['favorites', 'Favorites', null],
|
||||
// #249 PR F (ruling R5): SAVED VIEWS' Needs review, the concept's Metadata review page (its own heading).
|
||||
['review', 'Needs review', null, 'Metadata review'],
|
||||
// #249 Activity-1: the concept's Activity page in the tools-and-system group.
|
||||
['activity', 'Activity', null],
|
||||
];
|
||||
@@ -79,10 +77,10 @@ test('workspace starts with real persisted statistics without mutating data', as
|
||||
});
|
||||
|
||||
test('all shell destinations deep-link, reload and expose working existing applications', async ({ page, request }) => {
|
||||
for (const [route, label, legacy, title = label] of routes) {
|
||||
for (const [route, label, legacy] of routes) {
|
||||
await page.goto(`${workspace}#/${route}`);
|
||||
const main = page.locator('#workspace-main');
|
||||
await expect(main.getByRole('heading', { level: 1, name: title, exact: true })).toBeVisible();
|
||||
await expect(main.getByRole('heading', { level: 1, name: label, exact: true })).toBeVisible();
|
||||
await expect(sidebarLink(page, label)).toHaveAttribute('aria-current', 'page');
|
||||
if (route === 'tags' || route === 'inspect') await expect(main.locator('.handoff')).toHaveCount(0);
|
||||
if (route === 'inspect') await expect(main.locator('#local-file')).toBeAttached();
|
||||
@@ -91,7 +89,7 @@ test('all shell destinations deep-link, reload and expose working existing appli
|
||||
await expect(await request.get(legacy.split('#')[0])).toBeOK();
|
||||
}
|
||||
await page.reload();
|
||||
await expect(main.getByRole('heading', { level: 1, name: title, exact: true })).toBeVisible();
|
||||
await expect(main.getByRole('heading', { level: 1, name: label, exact: true })).toBeVisible();
|
||||
}
|
||||
// D29 and #249 E3: the legacy gallery and Library are reached from Overview's tool cards, not from their pages.
|
||||
await page.goto(`${workspace}#/overview`);
|
||||
@@ -311,9 +309,9 @@ test('overview reports loading, offers retry after failure and renders real empt
|
||||
test('small screens keep navigation and the theme accessible without horizontal page overflow', async ({ page }) => {
|
||||
await page.setViewportSize({ width: 375, height: 812 });
|
||||
await page.goto(`${workspace}#/overview`);
|
||||
for (const [, label, , title = label] of routes) {
|
||||
for (const [, label] of routes) {
|
||||
await sidebarLink(page, label).click();
|
||||
await expect(page.locator('#workspace-main h1')).toHaveText(title);
|
||||
await expect(page.locator('#workspace-main h1')).toHaveText(label);
|
||||
}
|
||||
await expect(page.getByRole('combobox', { name: 'Theme', exact: true })).toBeVisible();
|
||||
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true);
|
||||
|
||||
@@ -5,23 +5,18 @@ reading files, so a test sets its progress and counters exactly:
|
||||
|
||||
- ``POST /__test__/jobs/start`` starts one (``kind``, ``title``, ``total``,
|
||||
``output_folders``; for ``kind: "image-tags"`` the image tagging job's
|
||||
groups and its ``{mode, images}`` scope instead; for ``kind:
|
||||
"thumbnails"`` the thumbnail job's groups, ``guarded`` admission and its
|
||||
``{mode, size, quality, output_folders}`` scope; for ``kind: "health"`` the
|
||||
health check's: no groups, ``read``, not cancellable, with its status
|
||||
line) and returns its summary, or 409 while one runs;
|
||||
groups and its ``{mode, images}`` scope instead) and returns its summary,
|
||||
or 409 while one runs;
|
||||
- ``POST /__test__/jobs/{id}/step`` records ``records`` (``{group, path,
|
||||
root, root_index, detail}`` file outcomes, counted, listed and logged as the scan
|
||||
records them) and ``bulk`` (``{group, count, root}``: ``count`` numbered
|
||||
rows), sets ``processed`` (and ``total``) and ``counters``, then publishes
|
||||
the progress to every subscriber;
|
||||
- ``POST /__test__/jobs/{id}/finish`` ends it as ``finished``, ``cancelled``
|
||||
or ``failed``, with the runner's final ``status_line`` when given; a cancel
|
||||
through the real route ends it ``cancelled``;
|
||||
- ``POST /__test__/jobs/hold`` opens a legacy-page hold (``kind``, ``title``,
|
||||
``admission``, default ``records``): a running, non-cancellable job with no
|
||||
outcome groups, as a legacy restore, maintenance or thumbnail clear request
|
||||
takes, until ``finish`` ends it;
|
||||
or ``failed``; a cancel through the real route ends it ``cancelled``;
|
||||
- ``POST /__test__/jobs/hold`` opens a legacy-page hold (``kind``, ``title``):
|
||||
a running, non-cancellable ``records`` job, as a legacy restore or
|
||||
maintenance request takes, until ``finish`` ends it;
|
||||
- ``POST /__test__/jobs/restart`` acts out a ComfyUI restart: the open job
|
||||
streams drop without a final event and the registry starts empty.
|
||||
"""
|
||||
@@ -40,9 +35,6 @@ PREFIXES = {"added": "+", "known": "=", "review": "?", "errors": "!"}
|
||||
PREFIXES["no_metadata"] = "-"
|
||||
# The image tagging job's groups (#249 PR D), logged with the runner's prefixes.
|
||||
PREFIXES.update({"tagged": "+", "unchanged": "=", "skipped": "-"})
|
||||
# The thumbnail job's (#249 System S5).
|
||||
PREFIXES["generated"] = "+"
|
||||
HEALTH_STATUS_LINE = "Checking the database and image links"
|
||||
ROOT_INDEXES = {"output": 0, "input": 1}
|
||||
|
||||
|
||||
@@ -88,12 +80,9 @@ def outcome_records(data):
|
||||
|
||||
def register_activity_commands(app, routes, api):
|
||||
from py.api.jobs import (
|
||||
HEALTH_JOB_KIND,
|
||||
IMAGE_TAG_GROUP_NAMES,
|
||||
IMAGE_TAG_JOB_KIND,
|
||||
SCAN_GROUP_NAMES,
|
||||
THUMBNAIL_GROUP_NAMES,
|
||||
THUMBNAILS_JOB_KIND,
|
||||
JobBusy,
|
||||
JobRegistry,
|
||||
)
|
||||
@@ -149,21 +138,7 @@ def register_activity_commands(app, routes, api):
|
||||
data = await body(request)
|
||||
ending = asyncio.get_running_loop().create_future()
|
||||
kind = str(data.get("kind", "scan"))
|
||||
admission, cancellable = "records", True
|
||||
if kind == THUMBNAILS_JOB_KIND:
|
||||
title = "Thumbnail generation"
|
||||
scope = {
|
||||
"mode": str(data.get("mode", "missing")),
|
||||
"size": count(data.get("size"), 512),
|
||||
"quality": str(data.get("quality", "medium")),
|
||||
"output_folders": count(data.get("output_folders"), 2),
|
||||
}
|
||||
groups = THUMBNAIL_GROUP_NAMES
|
||||
admission = "guarded"
|
||||
elif kind == HEALTH_JOB_KIND:
|
||||
title, scope, groups = "Health check", {}, ()
|
||||
admission, cancellable = "read", False
|
||||
elif kind == IMAGE_TAG_JOB_KIND:
|
||||
if kind == IMAGE_TAG_JOB_KIND:
|
||||
title = "Add tags to images"
|
||||
scope = {
|
||||
"mode": str(data.get("mode", "add")),
|
||||
@@ -175,20 +150,13 @@ def register_activity_commands(app, routes, api):
|
||||
scope = {
|
||||
"files": str(data.get("files", "new")),
|
||||
"output_folders": count(data.get("output_folders"), 2),
|
||||
# A real scan's count of its files in the Needs review queue.
|
||||
**(
|
||||
{"review_items": count(data["review_items"])}
|
||||
if "review_items" in data
|
||||
else {}
|
||||
),
|
||||
}
|
||||
groups = SCAN_GROUP_NAMES
|
||||
try:
|
||||
job = api._jobs.start(
|
||||
kind,
|
||||
str(data.get("title", title)),
|
||||
admission=admission,
|
||||
cancellable=cancellable,
|
||||
record_changing=True,
|
||||
scope=scope,
|
||||
runner=fake_runner(ending),
|
||||
groups=groups,
|
||||
@@ -196,8 +164,6 @@ def register_activity_commands(app, routes, api):
|
||||
except JobBusy:
|
||||
raise web.HTTPConflict()
|
||||
job.update(total=count(data.get("total")))
|
||||
if kind == HEALTH_JOB_KIND:
|
||||
job.update(status_line=HEALTH_STATUS_LINE)
|
||||
endings[job.id] = ending
|
||||
return web.json_response({"success": True, "job": job.summary()})
|
||||
|
||||
@@ -212,7 +178,7 @@ def register_activity_commands(app, routes, api):
|
||||
async with api._jobs.hold(
|
||||
str(data.get("kind", "restore")),
|
||||
str(data.get("title", "Database restore (admin page)")),
|
||||
admission=str(data.get("admission", "records")),
|
||||
admission="records",
|
||||
) as job:
|
||||
entered.set_result(job)
|
||||
state = await ending
|
||||
@@ -260,12 +226,9 @@ def register_activity_commands(app, routes, api):
|
||||
@routes.post("/__test__/jobs/{job_id}/finish")
|
||||
async def finish_job(request):
|
||||
job = job_or_404(request)
|
||||
data = await body(request)
|
||||
state = data.get("state", "finished")
|
||||
state = (await body(request)).get("state", "finished")
|
||||
if state not in FINAL_STATES:
|
||||
raise web.HTTPBadRequest()
|
||||
if isinstance(data.get("status_line"), str): # the runner's final line
|
||||
job.update(status_line=data["status_line"])
|
||||
ending = endings[job.id]
|
||||
if not ending.done():
|
||||
ending.set_result(state)
|
||||
|
||||
@@ -176,8 +176,6 @@ _TABLES = (
|
||||
"execution_context_texts",
|
||||
"saved_views",
|
||||
"image_favorites",
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_image_tags",
|
||||
"media_annotation_bytes",
|
||||
"media_annotation_files",
|
||||
|
||||
@@ -28,8 +28,6 @@ EDITOR_METADATA = {
|
||||
_TABLES = (
|
||||
"saved_views",
|
||||
"image_favorites",
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_image_tags",
|
||||
"media_annotation_bytes",
|
||||
"media_annotation_files",
|
||||
|
||||
@@ -1,368 +0,0 @@
|
||||
"""Owned, deterministic Needs review items for the workspace page (#249 PR F).
|
||||
|
||||
Each file is a PNG whose workflow has two samplers with their own positive
|
||||
encoders and one shared negative, so the resolver finds it ``ambiguous``. Its
|
||||
loader node carries ComfyUI's ``Infinity`` stamp, which the legacy pages'
|
||||
parser (the scan's fallback) cannot read, so nothing links the file and it is
|
||||
a Needs review case (#284: only ambiguous, unlinked files are queued). It is
|
||||
recorded through the producers' entry point (``record_review_file``). No path
|
||||
or SQL comes from a request.
|
||||
|
||||
``POST /__test__/review/reset`` also takes the special items F2 needs, queued
|
||||
after the plain ones in this order:
|
||||
|
||||
- ``conflict``: an item whose file was linked to another prompt ("Misty
|
||||
forest path") after it was queued (the next scan or monitor pass would
|
||||
remove the item); applying a sampler path then conflicts with that link.
|
||||
- ``limited``: a graph of 17 samplers, each with its own prompt, which the
|
||||
recorder stores ``limited`` (more than 16 candidates, so none are stored).
|
||||
An A1111 block with two negative lines is not queued any more: it offers
|
||||
no candidate to choose (#284).
|
||||
- ``unavailable``: a file recorded under the ``temp`` directory while it was a
|
||||
gallery root; the root is then removed again, so no current root lists it.
|
||||
|
||||
``concept: true`` (F3, the concept comparison) names and sizes the plain
|
||||
files as the concept's first three images (app.js ``images``) and gives
|
||||
their first sampler the concept's first prompt (``alpine``: its positive and
|
||||
its first negative, the one the concept's panel shows), so the panel's text
|
||||
wraps as the concept's does.
|
||||
|
||||
``POST /__test__/review/add {kind}`` adds one file while a page is open:
|
||||
``monitor`` records a new ambiguous, unlinked output from a worker thread
|
||||
through ``record_review_file`` (the monitor's own entry point; the monitor
|
||||
never queues an output it linked itself), ``file`` only writes one (a
|
||||
metadata scan records it). ``GET
|
||||
/__test__/review/links`` answers each review file's prompt and pair links, and
|
||||
``GET /__test__/review/streams`` the open change streams and captures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from aiohttp import web
|
||||
from PIL import Image
|
||||
from PIL.PngImagePlugin import PngInfo
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from py.api import PromptManagerAPI
|
||||
|
||||
POSITIVES = ("Golden dunes at dusk", "Harbor lights at night")
|
||||
NEGATIVE = "Soft negative, blur"
|
||||
# The prompt the conflict item's file is linked to after it was queued.
|
||||
FALLBACK = "Misty forest path"
|
||||
SPECIAL = ("conflict", "limited", "unavailable")
|
||||
# The concept's first three images (concept app.js `images`) and first prompt.
|
||||
CONCEPT_IMAGES = (
|
||||
("alpine_00001.png", (1536, 1152)),
|
||||
("dunes_00001.png", (1024, 1536)),
|
||||
("forest_00001.png", (1280, 1280)),
|
||||
)
|
||||
CONCEPT_POSITIVE = (
|
||||
"A remote alpine observatory at blue hour, warm light through glass, dramatic "
|
||||
"mountain peaks, cinematic composition, atmospheric mist, fine detail"
|
||||
)
|
||||
CONCEPT_NEGATIVE = "blurry, low quality, low resolution"
|
||||
# More candidates than an item stores (16): the ``limited`` item.
|
||||
LIMITED_SAMPLERS = 17
|
||||
# More than one queue page (40), so Load more has a second page.
|
||||
MAX_ITEMS = 45
|
||||
_TABLES = (
|
||||
"saved_views",
|
||||
"image_favorites",
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_image_tags",
|
||||
"media_annotation_bytes",
|
||||
"media_annotation_files",
|
||||
"image_prompt_pairs",
|
||||
"prompt_pair_tags",
|
||||
"prompt_pairs",
|
||||
"generated_images",
|
||||
"prompt_tags",
|
||||
"tags",
|
||||
"prompt_library_state",
|
||||
"prompts",
|
||||
)
|
||||
|
||||
|
||||
def _graph(
|
||||
encoder_class: str = "CLIPTextEncode",
|
||||
positives: tuple = POSITIVES,
|
||||
negative: str = NEGATIVE,
|
||||
) -> dict:
|
||||
graph = {
|
||||
# ``Infinity`` (as ComfyUI writes it): the legacy parser cannot read the
|
||||
# graph, so the scan's fallback links nothing and the file is queued.
|
||||
"1": {
|
||||
"class_type": "CheckpointLoaderSimple",
|
||||
"inputs": {"ckpt_name": "m"},
|
||||
"is_changed": [float("inf")],
|
||||
},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": negative}},
|
||||
}
|
||||
for index, text in enumerate(positives):
|
||||
encoder, sampler = str(10 + 2 * index), str(11 + 2 * index)
|
||||
graph[encoder] = {"class_type": encoder_class, "inputs": {"text": text}}
|
||||
graph[sampler] = {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {
|
||||
"model": ["1", 0],
|
||||
"positive": [encoder, 0],
|
||||
"negative": ["2", 0],
|
||||
},
|
||||
}
|
||||
return graph
|
||||
|
||||
|
||||
def _png(
|
||||
path: Path,
|
||||
colour: tuple[int, int, int],
|
||||
*,
|
||||
graph: dict | None = None,
|
||||
parameters: str | None = None,
|
||||
size: tuple[int, int] = (96, 64),
|
||||
) -> None:
|
||||
info = PngInfo()
|
||||
if parameters is not None:
|
||||
info.add_text("parameters", parameters)
|
||||
else:
|
||||
info.add_text("prompt", json.dumps(graph or _graph()))
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with Image.new("RGB", size, colour) as image:
|
||||
image.save(path, pnginfo=info)
|
||||
|
||||
|
||||
def _record(api: PromptManagerAPI, path: Path, expected: str) -> None:
|
||||
outcome = api.record_review_file(str(path))
|
||||
if outcome != expected:
|
||||
raise RuntimeError(f"{path.name} was {outcome}, not {expected}")
|
||||
|
||||
|
||||
def _stamp(path: Path, age: float) -> None:
|
||||
stamp = time.time() - age
|
||||
os.utime(path, (stamp, stamp))
|
||||
|
||||
|
||||
class _Monitor:
|
||||
"""The monitor's recorder over this server's database (one per server)."""
|
||||
|
||||
def __init__(self, api: PromptManagerAPI) -> None:
|
||||
self.api = api
|
||||
self.lock = threading.Lock()
|
||||
|
||||
def new_output(self, path: Path) -> None:
|
||||
"""The monitor's Needs review pass for a new output, from a worker thread."""
|
||||
outcome = []
|
||||
with self.lock:
|
||||
worker = threading.Thread(
|
||||
target=lambda: outcome.append(self.api.record_review_file(str(path)))
|
||||
)
|
||||
worker.start()
|
||||
worker.join()
|
||||
if outcome != ["recorded"]:
|
||||
raise RuntimeError(f"{path.name} was {outcome}, not recorded")
|
||||
|
||||
|
||||
def _monitored_output(api: PromptManagerAPI, monitor: _Monitor, path: Path) -> None:
|
||||
_png(path, (150, 60, 40))
|
||||
monitor.new_output(path)
|
||||
|
||||
|
||||
def _linked_after_queueing(api: PromptManagerAPI, path: Path) -> None:
|
||||
from utils.hashing import generate_prompt_hash
|
||||
|
||||
_png(path, (150, 60, 40))
|
||||
_record(api, path, "recorded")
|
||||
prompt = api.db.get_prompt_by_hash(generate_prompt_hash(FALLBACK))
|
||||
prompt_id = (
|
||||
prompt["id"]
|
||||
if prompt is not None
|
||||
else api.db.save_prompt(FALLBACK, prompt_hash=generate_prompt_hash(FALLBACK))
|
||||
)
|
||||
api.db.link_image_to_prompt(prompt_id, str(path))
|
||||
|
||||
|
||||
def _unavailable(api: PromptManagerAPI, root: Path, path: Path) -> None:
|
||||
from py.config import GalleryConfig
|
||||
|
||||
roots = list(GalleryConfig.MONITORING_DIRECTORIES)
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [*roots, str(root / "temp")]
|
||||
try:
|
||||
_record(api, path, "recorded")
|
||||
finally:
|
||||
GalleryConfig.MONITORING_DIRECTORIES = roots
|
||||
|
||||
|
||||
def seed_review(
|
||||
api: PromptManagerAPI,
|
||||
root: Path,
|
||||
items: int,
|
||||
special: tuple = (),
|
||||
monitor: _Monitor | None = None,
|
||||
concept: bool = False,
|
||||
) -> dict:
|
||||
"""Replace this server's disposable rows and files with ``items`` queued files."""
|
||||
with api.db.model.get_connection() as conn:
|
||||
for table in _TABLES:
|
||||
conn.execute(f"DELETE FROM {table}")
|
||||
api.close_annotation_captures()
|
||||
for name in ("output", "input"):
|
||||
shutil.rmtree(root / name)
|
||||
(root / name).mkdir()
|
||||
shutil.rmtree(root / "temp" / "review", ignore_errors=True)
|
||||
output = root / "output"
|
||||
files = []
|
||||
for number in range(items):
|
||||
relative = f"review/review_{number:04}.png"
|
||||
options = {}
|
||||
if concept:
|
||||
name, size = CONCEPT_IMAGES[number % len(CONCEPT_IMAGES)]
|
||||
relative = f"review/{name}" if number < len(CONCEPT_IMAGES) else relative
|
||||
graph = _graph(
|
||||
positives=(CONCEPT_POSITIVE, POSITIVES[1]), negative=CONCEPT_NEGATIVE
|
||||
)
|
||||
options = {"graph": graph, "size": size}
|
||||
_png(output / relative, (40 + number * 4, 90, 140), **options)
|
||||
_stamp(output / relative, 60 + number)
|
||||
_record(api, output / relative, "recorded")
|
||||
files.append({"path": relative, "filename": Path(relative).name})
|
||||
for kind in SPECIAL:
|
||||
if kind not in special:
|
||||
continue
|
||||
relative = f"review/{kind}.png"
|
||||
path = (root / "temp" if kind == "unavailable" else output) / relative
|
||||
if kind == "conflict":
|
||||
_linked_after_queueing(api, path)
|
||||
elif kind == "limited":
|
||||
many = tuple(f"{POSITIVES[0]} {n}" for n in range(LIMITED_SAMPLERS))
|
||||
_png(path, (90, 90, 90), graph=_graph(positives=many))
|
||||
_record(api, path, "limited")
|
||||
else:
|
||||
_png(path, (20, 140, 90))
|
||||
_unavailable(api, root, path)
|
||||
files.append({"path": relative, "filename": path.name, "kind": kind})
|
||||
api.invalidate_gallery_cache()
|
||||
return {
|
||||
"files": files,
|
||||
"positives": list(POSITIVES),
|
||||
"negative": NEGATIVE,
|
||||
"fallback": FALLBACK,
|
||||
}
|
||||
|
||||
|
||||
def add_review_file(
|
||||
api: PromptManagerAPI, root: Path, kind: str, monitor: _Monitor
|
||||
) -> dict:
|
||||
"""One more ambiguous output: through the monitor, or only written."""
|
||||
output = root / "output"
|
||||
number = (
|
||||
len(list((output / "added").glob("*.png")))
|
||||
if (output / "added").exists()
|
||||
else 0
|
||||
)
|
||||
relative = f"added/added_{number:04}.png"
|
||||
if kind == "monitor":
|
||||
_monitored_output(api, monitor, output / relative)
|
||||
else:
|
||||
_png(output / relative, (200, 200, 40))
|
||||
api.invalidate_gallery_cache()
|
||||
return {"path": relative, "filename": Path(relative).name}
|
||||
|
||||
|
||||
_LINKS = (
|
||||
"SELECT gi.filename, p.text AS prompt, pp.id AS pair_id, pos.text AS positive, "
|
||||
"neg.text AS negative FROM generated_images gi "
|
||||
"JOIN prompts p ON p.id = gi.prompt_id "
|
||||
"LEFT JOIN image_prompt_pairs ipp ON ipp.image_id = gi.id "
|
||||
"LEFT JOIN prompt_pairs pp ON pp.id = ipp.pair_id "
|
||||
"LEFT JOIN prompts pos ON pos.id = pp.positive_prompt_id "
|
||||
"LEFT JOIN prompts neg ON neg.id = pp.negative_prompt_id "
|
||||
"ORDER BY gi.filename, gi.id, pp.id"
|
||||
)
|
||||
|
||||
|
||||
def review_links(api: PromptManagerAPI) -> list:
|
||||
"""Every file's prompt link and pair links, as plain rows."""
|
||||
conn = api.db.model.get_connection()
|
||||
return [dict(row) for row in conn.execute(_LINKS)]
|
||||
|
||||
|
||||
def register_review_commands(
|
||||
routes: web.RouteTableDef, api: PromptManagerAPI, root: Path
|
||||
) -> None:
|
||||
"""Register the review fixture's commands; no path or SQL comes from a request."""
|
||||
monitors = []
|
||||
|
||||
def monitor() -> _Monitor:
|
||||
if not monitors:
|
||||
monitors.append(_Monitor(api))
|
||||
return monitors[0]
|
||||
|
||||
@routes.post("/__test__/review/reset")
|
||||
async def reset(request: web.Request) -> web.Response:
|
||||
data = await request.json() if request.can_read_body else {}
|
||||
if (
|
||||
request.query
|
||||
or not isinstance(data, dict)
|
||||
or set(data) - {"items", "concept", *SPECIAL}
|
||||
):
|
||||
raise web.HTTPBadRequest()
|
||||
items = data.get("items", 3)
|
||||
if isinstance(items, bool) or not isinstance(items, int):
|
||||
raise web.HTTPBadRequest()
|
||||
if not 0 <= items <= MAX_ITEMS:
|
||||
raise web.HTTPBadRequest()
|
||||
if any(
|
||||
not isinstance(data.get(kind, False), bool)
|
||||
for kind in (*SPECIAL, "concept")
|
||||
):
|
||||
raise web.HTTPBadRequest()
|
||||
special = tuple(kind for kind in SPECIAL if data.get(kind))
|
||||
manifest = await api._run_in_executor(
|
||||
seed_review,
|
||||
api,
|
||||
root,
|
||||
items,
|
||||
special,
|
||||
monitor(),
|
||||
data.get("concept", False),
|
||||
)
|
||||
return web.json_response({"success": True, **manifest})
|
||||
|
||||
@routes.post("/__test__/review/add")
|
||||
async def add(request: web.Request) -> web.Response:
|
||||
data = await request.json() if request.can_read_body else None
|
||||
if request.query or not isinstance(data, dict) or set(data) != {"kind"}:
|
||||
raise web.HTTPBadRequest()
|
||||
if data["kind"] not in ("monitor", "file"):
|
||||
raise web.HTTPBadRequest()
|
||||
added = await api._run_in_executor(
|
||||
add_review_file, api, root, data["kind"], monitor()
|
||||
)
|
||||
return web.json_response({"success": True, **added})
|
||||
|
||||
@routes.get("/__test__/review/links")
|
||||
async def links(request: web.Request) -> web.Response:
|
||||
if request.query:
|
||||
raise web.HTTPBadRequest()
|
||||
return web.json_response(
|
||||
{"links": await api._run_in_executor(review_links, api)}
|
||||
)
|
||||
|
||||
@routes.get("/__test__/review/streams")
|
||||
async def streams(request: web.Request) -> web.Response:
|
||||
"""Open review change streams and held captures (teardown checks)."""
|
||||
if request.query:
|
||||
raise web.HTTPBadRequest()
|
||||
store = api._captures
|
||||
captures = store.stats()["captures"] if store is not None else 0
|
||||
return web.json_response(
|
||||
{"streams": api.review_stream_count(), "captures": captures}
|
||||
)
|
||||
@@ -44,8 +44,6 @@ MAX_EXTRA_IMAGES = 160
|
||||
_TABLES = (
|
||||
"saved_views",
|
||||
"image_favorites",
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_image_tags",
|
||||
"media_annotation_bytes",
|
||||
"media_annotation_files",
|
||||
|
||||
@@ -114,8 +114,6 @@ def seed_workspace_visual(api: PromptManagerAPI, root: Path) -> None:
|
||||
for table in (
|
||||
"saved_views",
|
||||
"image_favorites",
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_image_tags",
|
||||
"media_annotation_bytes",
|
||||
"media_annotation_files",
|
||||
|
||||
@@ -81,8 +81,6 @@ MAX_EXTRA_CATEGORIES = 200
|
||||
_TABLES = (
|
||||
"saved_views",
|
||||
"image_favorites",
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_image_tags",
|
||||
"media_annotation_bytes",
|
||||
"media_annotation_files",
|
||||
|
||||
@@ -110,123 +110,3 @@ test('saving gallery settings retains the remembered thumbnail dismissal', () =>
|
||||
f.manager.saveSettings();
|
||||
assert.equal(JSON.parse(f.storage.get('gallerySettings')).thumbnailPromptDismissed, 'remembered-state');
|
||||
});
|
||||
|
||||
// #249 System S5: the stream's work is now an Activity job that outlives the page's connection, so the gallery's
|
||||
// Cancel asks the server to stop the job. The stream's first frame, `job {id, started}`, names the job; the gallery
|
||||
// cancels by that id, only a job this stream started, and a Cancel before the id waits for it (review I1).
|
||||
const JOB = 'b'.repeat(32);
|
||||
|
||||
function streamFixture({ modal = false, postFails = false } = {}) {
|
||||
const posts = [];
|
||||
const warnings = [];
|
||||
const gets = [];
|
||||
const listeners = {};
|
||||
const source = { closed: 0, readyState: 1, CLOSED: 2, url: null,
|
||||
addEventListener(type, handler) { (listeners[type] ||= []).push(handler); },
|
||||
close() { this.closed += 1; } };
|
||||
const element = () => ({ classList: { add() {}, remove() {} }, style: {}, textContent: '', handlers: {},
|
||||
addEventListener(type, handler) { this.handlers[type] = handler; },
|
||||
removeEventListener(type, handler) { if (this.handlers[type] === handler) delete this.handlers[type]; },
|
||||
click() { this.handlers.click?.(); } });
|
||||
const elements = {};
|
||||
const document = { addEventListener() {}, getElementById: id => elements[id] ||= element(),
|
||||
createElement: () => element(), body: { appendChild() {}, removeChild() {} } };
|
||||
const context = vm.createContext({ document, window: {},
|
||||
console: { debug() {}, warn: (...args) => warnings.push(args), error() {} },
|
||||
setTimeout() {}, localStorage: { setItem() {} },
|
||||
SseStream: { connect: (url, options) => { source.url = url; source.options = options; return source; } } });
|
||||
const file = path.join(__dirname, '../../web/js/gallery.js');
|
||||
const Manager = vm.runInContext(fs.readFileSync(file, 'utf8') + '\nGalleryManager;', context, { filename: file });
|
||||
const manager = Object.create(Manager.prototype);
|
||||
Object.assign(manager, { getSettings: () => ({}),
|
||||
api: { get: async url => { gets.push(url); return { success: true, jobs: [] }; },
|
||||
post: async (url, body) => {
|
||||
posts.push([url, body]);
|
||||
if (postFails) throw new Error('offline');
|
||||
return { success: true };
|
||||
} } });
|
||||
const emit = (type, data) => (listeners[type] || []).forEach(handler => handler({ data: JSON.stringify(data) }));
|
||||
const ui = { addStatusMessage() {}, progressText: element(), progressPercent: element(), progressBar: element(),
|
||||
progressDetails: element(), progressETA: element(), isCancelled: () => false,
|
||||
onCancel: handler => { ui.cancel = handler; } };
|
||||
const result = modal ? manager.generateThumbnailsWithProgressInModal(ui)
|
||||
: manager.generateThumbnailsWithProgress({ quality: 'medium' });
|
||||
const outcome = result.then(value => ({ value }), error => ({ error: error.message }));
|
||||
const cancel = () => (modal ? ui.cancel() : elements.cancelThumbnailsBtn.click());
|
||||
const settle = () => new Promise(resolve => setImmediate(resolve));
|
||||
return { emit, source, posts, gets, warnings, outcome, cancel, settle };
|
||||
}
|
||||
|
||||
const cancelUrl = id => `/prompt_manager/jobs/${id}/cancel`;
|
||||
|
||||
test('Cancel before the job id arrives cancels that job once it does', async () => {
|
||||
const f = streamFixture();
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.deepEqual(f.posts, []);
|
||||
assert.equal(f.source.closed, 0); // kept open until the stream names its job
|
||||
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
|
||||
f.emit('status', { phase: 'scanning', message: 'Scanning…' });
|
||||
f.emit('job', { id: JOB, started: true });
|
||||
await f.settle();
|
||||
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
|
||||
assert.equal(f.source.closed, 1);
|
||||
f.emit('job', { id: 'c'.repeat(32), started: true }); // a stray second frame changes nothing
|
||||
await f.settle();
|
||||
assert.equal(f.posts.length, 1);
|
||||
assert.deepEqual(f.gets, []); // never looks a job up by kind
|
||||
});
|
||||
|
||||
test('Cancel after the id cancels it at once, and a double Cancel posts once', async () => {
|
||||
const f = streamFixture();
|
||||
f.emit('job', { id: JOB, started: true });
|
||||
f.cancel();
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
|
||||
assert.equal(f.source.closed, 1);
|
||||
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
|
||||
assert.deepEqual(f.gets, []);
|
||||
});
|
||||
|
||||
test("Cancel leaves another tab's or System's job running and only stops following it", async () => {
|
||||
const f = streamFixture();
|
||||
f.emit('job', { id: JOB, started: false });
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.deepEqual(f.posts, []);
|
||||
assert.deepEqual(f.gets, []);
|
||||
assert.equal(f.source.closed, 1);
|
||||
assert.match((await f.outcome).error, /keeps running/);
|
||||
});
|
||||
|
||||
test('a Cancel the stream never names a job for posts nothing and closes when it ends', async () => {
|
||||
const f = streamFixture();
|
||||
f.cancel();
|
||||
f.emit('error', { error: 'Scan for metadata already running' });
|
||||
await f.settle();
|
||||
assert.deepEqual(f.posts, []);
|
||||
assert.equal(f.source.closed, 1);
|
||||
});
|
||||
|
||||
test('a cancel post that fails is reported, not thrown', async () => {
|
||||
const f = streamFixture({ postFails: true });
|
||||
f.emit('job', { id: JOB, started: true });
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.equal(f.posts.length, 1);
|
||||
assert.equal(f.warnings.length, 1);
|
||||
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
|
||||
});
|
||||
|
||||
test('the startup dialog cancels its own job by id too', async () => {
|
||||
const f = streamFixture({ modal: true });
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.deepEqual(f.posts, []);
|
||||
f.emit('job', { id: JOB, started: true });
|
||||
await f.settle();
|
||||
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
|
||||
assert.equal(f.source.closed, 1);
|
||||
assert.match((await f.outcome).error, /cancelled/);
|
||||
});
|
||||
|
||||
@@ -1,148 +0,0 @@
|
||||
// Per-kind Activity descriptors (#249 System S5 Task 13, D111): each job kind's pill verb, toasts, line, tiles,
|
||||
// outcomes, openable rows and labels, from one table (web/js/workspace/activity-kinds.mjs).
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
|
||||
const source = import('../../web/js/workspace/activity-kinds.mjs');
|
||||
const store = import('../../web/js/workspace/activity-store.mjs');
|
||||
|
||||
const ID = 'a'.repeat(32);
|
||||
|
||||
function job(overrides = {}) {
|
||||
return { id: ID, kind: 'scan', title: 'Scan for metadata', state: 'running', started_at: 1, finished_at: null,
|
||||
progress: 42, status_line: '', processed: 8240, total: 19622,
|
||||
counters: { added: 1, known: 0, review: 0, no_metadata: 0, errors: 1 }, scope: {}, listed: {}, log_length: 0,
|
||||
cancellable: true, ...overrides };
|
||||
}
|
||||
|
||||
test('scan descriptors are unchanged', async () => {
|
||||
const { kindOf } = await source;
|
||||
const scan = kindOf('scan');
|
||||
assert.equal(scan.pillVerb, 'Scanning');
|
||||
assert.equal(scan.startToast(job()), 'Scan started.');
|
||||
assert.deepEqual(scan.tiles, [['added', 'Added to Library', 'added'], ['known', 'Already known', ''],
|
||||
['review', 'Needs review', 'warn'], ['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad']]);
|
||||
assert.deepEqual(scan.outcomes, [['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
|
||||
['no_metadata', 'No usable metadata'], ['known', 'Already known']]);
|
||||
assert.deepEqual([...scan.openable].sort(), ['added', 'known']);
|
||||
assert.deepEqual([scan.log, scan.cancelLabel], ['Scan log', 'Cancel scan']);
|
||||
assert.equal(scan.finishToast(job({ state: 'finished' })), 'Scan finished · 1 added · 1 error');
|
||||
assert.equal(scan.finishToast(job({ state: 'cancelled' })), 'Scan cancelled · 1 added');
|
||||
assert.equal(scan.finishToast(job({ state: 'failed' })), 'Scan failed · 1 added · 1 error');
|
||||
assert.equal(scan.line(job()), '1 added · 1 error');
|
||||
assert.equal(scan.kept(job()), '1 added');
|
||||
assert.equal(scan.scope(job({ scope: { output_folders: 2 } })), '2 output folders');
|
||||
// The store keeps answering with the same table.
|
||||
const { jobView, finishToast, startToast, pillParts, outcomeLine } = await store;
|
||||
assert.equal(jobView('scan'), scan);
|
||||
assert.equal(finishToast(job({ state: 'finished' })), 'Scan finished · 1 added · 1 error');
|
||||
assert.equal(startToast(job()), 'Scan started.');
|
||||
assert.equal(pillParts(job()).text, 'Scanning · 42% · 8,240 / 19,622');
|
||||
assert.equal(outcomeLine(job()), '1 added · 1 error');
|
||||
});
|
||||
|
||||
test('thumbnail job shows its own tiles and toasts', async () => {
|
||||
const { kindOf, viewOf } = await source;
|
||||
const { finishToast, startToast, pillParts, outcomeLine } = await store;
|
||||
const thumbs = job({ kind: 'thumbnails', title: 'Thumbnail generation', processed: 4, total: 10, progress: 40,
|
||||
counters: { generated: 1420, skipped: 3, errors: 1 }, scope: { mode: 'missing', size: 512, quality: 'medium',
|
||||
output_folders: 2 } });
|
||||
const view = kindOf('thumbnails');
|
||||
assert.equal(viewOf(thumbs), view);
|
||||
assert.deepEqual(view.tiles, [['generated', 'Generated', 'added'], ['skipped', 'Skipped', ''], ['errors', 'Errors', 'bad']]);
|
||||
assert.deepEqual(view.outcomes, [['generated', 'Generated'], ['errors', 'Errors'], ['skipped', 'Skipped']]);
|
||||
assert.deepEqual([...view.openable].sort(), ['generated', 'skipped']);
|
||||
assert.deepEqual([view.log, view.cancelLabel], ['Job log', 'Cancel job']);
|
||||
assert.equal(pillParts(thumbs).text, 'Generating thumbnails · 40% · 4 / 10');
|
||||
assert.equal(startToast(thumbs), 'Thumbnail generation started.');
|
||||
assert.equal(finishToast({ ...thumbs, state: 'finished' }), 'Thumbnails finished · 1,420 generated · 1 error');
|
||||
assert.equal(finishToast({ ...thumbs, state: 'failed' }), 'Thumbnails failed · 1,420 generated · 1 error');
|
||||
assert.equal(finishToast({ ...thumbs, state: 'cancelled' }), 'Thumbnails cancelled · 1,420 generated');
|
||||
assert.equal(finishToast({ ...thumbs, counters: { generated: 0, skipped: 0, errors: 2 }, state: 'finished' }),
|
||||
'Thumbnails finished · 0 generated · 2 errors');
|
||||
assert.equal(outcomeLine(thumbs), '1,420 generated · 1 error');
|
||||
assert.equal(view.kept(thumbs), '1,420 generated');
|
||||
assert.equal(view.scope(thumbs), '2 output folders');
|
||||
});
|
||||
|
||||
test('non-cancellable jobs show no Cancel job', async () => {
|
||||
const { canCancel } = await source;
|
||||
assert.equal(canCancel(job()), true);
|
||||
assert.equal(canCancel(job({ cancellable: false })), false);
|
||||
for (const state of ['finished', 'cancelled', 'failed', 'lost']) assert.equal(canCancel(job({ state })), false);
|
||||
// The health check is a read job the server cannot stop.
|
||||
assert.equal(canCancel(job({ kind: 'health', title: 'Health check', cancellable: false, counters: {} })), false);
|
||||
});
|
||||
|
||||
test('indeterminate progress has no value attribute', async () => {
|
||||
const { indeterminate, showProgress } = await source;
|
||||
const { pillParts } = await store;
|
||||
function node() {
|
||||
const attributes = new Map();
|
||||
return { attributes, textContent: '', set value(v) { attributes.set('value', String(v)); },
|
||||
get value() { return Number(attributes.get('value') ?? 0); },
|
||||
setAttribute(key, v) { attributes.set(key, String(v)); }, removeAttribute(key) { attributes.delete(key); } };
|
||||
}
|
||||
// Running with no total yet: the server has not said how much there is, so there is no percentage.
|
||||
const health = job({ kind: 'health', title: 'Health check', progress: 0, processed: 0, total: 0, counters: {},
|
||||
cancellable: false, status_line: 'Checking the database and image links' });
|
||||
assert.equal(indeterminate(health), true);
|
||||
const bar = node();
|
||||
bar.value = 37;
|
||||
showProgress(bar, health);
|
||||
assert.equal(bar.attributes.has('value'), false);
|
||||
assert.equal(bar.textContent, '');
|
||||
assert.equal(pillParts(health).text, 'Health check');
|
||||
assert.deepEqual(pillParts(health), { label: 'Health check', files: '', text: 'Health check' });
|
||||
// A known total is determinate, and so is any ended job.
|
||||
const scan = job();
|
||||
assert.equal(indeterminate(scan), false);
|
||||
showProgress(bar, scan);
|
||||
assert.equal(bar.attributes.get('value'), '42');
|
||||
assert.equal(bar.textContent, '42%');
|
||||
assert.equal(indeterminate({ ...health, state: 'finished', progress: 100 }), false);
|
||||
// A scan before its file count is known is indeterminate too, never "0%".
|
||||
assert.equal(pillParts(job({ total: 0, processed: 0, progress: 0 })).text, 'Scanning');
|
||||
});
|
||||
|
||||
test('held legacy jobs read "(admin page)" with no outcomes', async () => {
|
||||
const { viewOf } = await source;
|
||||
const { jobDetail } = await import('../../web/js/workspace/activity.mjs');
|
||||
const { finishToast, outcomeLine } = await store;
|
||||
// A hold has no outcome groups (counters {}), whatever its kind: thumbnails held by the gallery's clear included.
|
||||
for (const [kind, title] of [['restore', 'Database restore (admin page)'],
|
||||
['thumbnails', 'Thumbnail clear (gallery page)'], ['maintenance', 'Database maintenance (admin page)']]) {
|
||||
const held = job({ kind, title, counters: {}, cancellable: false, progress: 0, processed: 0, total: 0 });
|
||||
const view = viewOf(held);
|
||||
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []], kind);
|
||||
assert.equal(outcomeLine(held), '');
|
||||
assert.match(jobDetail(held), /^Started \d\d:\d\d · closing this view does not cancel it$/);
|
||||
const ended = { ...held, state: 'finished', finished_at: 2 };
|
||||
assert.match(jobDetail(ended), /^Finished \d\d:\d\d$/);
|
||||
assert.equal(finishToast(ended), `${title} finished.`);
|
||||
assert.ok(title.endsWith('page)'));
|
||||
}
|
||||
});
|
||||
|
||||
test('unknown and generic kinds show no tiles of another kind', async () => {
|
||||
const { kindOf, viewOf } = await source;
|
||||
for (const kind of ['maintenance', '', 'constructor', '__proto__', 'health']) {
|
||||
const view = kindOf(kind);
|
||||
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []], kind);
|
||||
assert.deepEqual([view.log, view.cancelLabel], ['Job log', 'Cancel job']);
|
||||
assert.equal(view.pillVerb, null);
|
||||
assert.equal(view.line(job({ kind })), '');
|
||||
assert.equal(view.kept(job({ kind })), '');
|
||||
}
|
||||
const backup = job({ kind: 'backup', title: 'Backup' });
|
||||
assert.equal(kindOf('backup').startToast(backup), 'Backup started.');
|
||||
assert.equal(kindOf('backup').finishToast({ ...backup, state: 'finished' }), 'Backup finished.');
|
||||
assert.equal(kindOf('backup').finishToast({ ...backup, state: 'cancelled' }), 'Backup was cancelled.');
|
||||
assert.equal(kindOf('backup').finishToast({ ...backup, state: 'failed' }), 'Backup failed.');
|
||||
// A kind's tiles are only the groups its job counts: no zero tile for a group the server never reported.
|
||||
const partial = viewOf(job({ counters: { added: 3, errors: 0 } }));
|
||||
assert.deepEqual(partial.tiles, [['added', 'Added to Library', 'added'], ['errors', 'Errors', 'bad']]);
|
||||
assert.deepEqual(partial.outcomes, [['added', 'Added to Library'], ['errors', 'Errors']]);
|
||||
assert.deepEqual([...partial.openable], ['added']);
|
||||
assert.equal(partial.log, 'Scan log');
|
||||
});
|
||||
@@ -289,10 +289,8 @@ test('summaries are validated and the copy follows the spec', async () => {
|
||||
|
||||
const scan = parseJob(summary({ progress: 42, processed: 8240, total: 19622, counters: { added: 1, errors: 1 } }));
|
||||
assert.deepEqual(pillParts(scan), { label: 'Scanning · 42%', files: ' · 8,240 / 19,622', text: 'Scanning · 42% · 8,240 / 19,622' });
|
||||
// D111: without a total the progress is indeterminate, so the pill shows no percentage.
|
||||
assert.equal(pillParts({ ...scan, total: 0 }).text, 'Scanning');
|
||||
// A kind without a pill verb shows its title (D111: thumbnails have their own, see workspace-activity-kinds.test.js).
|
||||
assert.equal(pillParts({ ...scan, kind: 'backup', title: 'Backup' }).text, 'Backup · 42% · 8,240 / 19,622');
|
||||
assert.equal(pillParts({ ...scan, total: 0 }).text, 'Scanning · 42%');
|
||||
assert.equal(pillParts({ ...scan, kind: 'thumbnails', title: 'Thumbnails' }).text, 'Thumbnails · 42% · 8,240 / 19,622');
|
||||
assert.equal(finishToast({ ...scan, state: 'finished' }), 'Scan finished · 1 added · 1 error');
|
||||
assert.equal(finishToast({ ...scan, state: 'cancelled' }), 'Scan cancelled · 1 added');
|
||||
assert.equal(finishToast({ ...scan, state: 'failed' }), 'Scan failed · 1 added · 1 error');
|
||||
@@ -334,8 +332,7 @@ test('image-tags jobs get their own tiles outcomes and pill verb', async () => {
|
||||
'Add tags to images finished · 0 tagged · 0 errors');
|
||||
});
|
||||
|
||||
// D111 (S5): an unknown kind no longer borrows the scan's tiles (S1 review M6, S2 review M3); it has none.
|
||||
test('unknown kinds get the generic view without the scan tiles', async () => {
|
||||
test('unknown kinds keep the generic view', async () => {
|
||||
const { jobView } = await source;
|
||||
const scan = jobView('scan');
|
||||
assert.deepEqual(scan.tiles, [['added', 'Added to Library', 'added'], ['known', 'Already known', ''],
|
||||
@@ -346,7 +343,7 @@ test('unknown kinds get the generic view without the scan tiles', async () => {
|
||||
assert.deepEqual([scan.log, scan.cancel], ['Scan log', 'Cancel scan']);
|
||||
for (const kind of ['maintenance', '', 'constructor', '__proto__']) {
|
||||
const view = jobView(kind);
|
||||
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []]);
|
||||
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [scan.tiles, scan.outcomes, [...scan.openable]]);
|
||||
assert.deepEqual([view.log, view.cancel], ['Job log', 'Cancel job']);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -1,788 +0,0 @@
|
||||
// Run with: node --test tests/js/workspace-review-state.test.js
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { createApiClient } = require('../../web/js/api-client.js');
|
||||
|
||||
// Only an absent module is tolerated, so RED is an assertion failure, not a crash.
|
||||
const source = import('../../web/js/workspace/review-state.mjs').catch(error => {
|
||||
if (error.code !== 'ERR_MODULE_NOT_FOUND') throw error;
|
||||
return {};
|
||||
});
|
||||
async function load() {
|
||||
const module = await source;
|
||||
assert.equal(typeof module.createReviewController, 'function', 'review-state.mjs is not implemented');
|
||||
return module;
|
||||
}
|
||||
|
||||
const EPOCH = '11111111-1111-4111-8111-111111111111';
|
||||
const SCOPE = 'c'.repeat(64);
|
||||
const SHA = 'd'.repeat(64);
|
||||
const PRINT = 'e'.repeat(64);
|
||||
const HASH_A = 'a'.repeat(64);
|
||||
const HASH_B = 'b'.repeat(64);
|
||||
const HASH_C = 'f'.repeat(64);
|
||||
const ITEMS = '/prompt_manager/review/items';
|
||||
const OBSERVE = '/prompt_manager/image-annotations/observe';
|
||||
const CAPTURE = '/prompt_manager/image-annotations/capture/';
|
||||
|
||||
const flush = () => new Promise(resolve => setImmediate(resolve));
|
||||
async function settle(times = 8) { for (let i = 0; i < times; i++) await flush(); }
|
||||
function response(status, body) {
|
||||
return { ok: status >= 200 && status < 300, status, statusText: `Status ${status}`, json: async () => body };
|
||||
}
|
||||
const fail = (status, code, error = `failure ${code}`, extra = {}) => response(status, { success: false, code, error, ...extra });
|
||||
|
||||
function item(id, overrides = {}) {
|
||||
const path = `dunes/${id}.png`;
|
||||
return {
|
||||
id, state: 'pending', revision: 0, created_at: `2026-10-22T10:00:0${id.length}+00:00`, filename: `${id}.png`,
|
||||
target: { root_index: 0, root_scope: SCOPE, path }, unavailable: false, candidate_count: 3, limited: false,
|
||||
url: `/prompt_manager/images/serve/${path}?root=0&root_scope=${SCOPE}`, is_video: false, ...overrides,
|
||||
};
|
||||
}
|
||||
const candidate = (hash, sampler, label, positive, negative, state = 'resolved') => ({
|
||||
hash, sampler_id: sampler, state, reason: '', label,
|
||||
positive: positive === null ? null : { text: positive }, negative: negative === null ? null : { text: negative },
|
||||
});
|
||||
function detailOf(entry, overrides = {}) {
|
||||
return {
|
||||
success: true, library_epoch: EPOCH, ...entry, total: 2, sha256: SHA, dependency_fingerprint: PRINT,
|
||||
candidates: [
|
||||
candidate(HASH_C, '9', 'No positive / Soft negative', null, 'blur', 'unresolved'),
|
||||
candidate(HASH_A, '12', 'Dunes / Soft negative', 'golden dunes', 'blur'),
|
||||
candidate(HASH_B, '28', 'Dunes / Harsh negative', 'golden dunes', 'noise'),
|
||||
],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
const page = (items, overrides = {}) => ({ success: true, library_epoch: EPOCH, total: items.length, items, next_cursor: null, ...overrides });
|
||||
const observed = (token = 'capture-1') => ({
|
||||
success: true, library_epoch: EPOCH, target: item('one').target, filename: 'one.png', title: 'one.png', tags: [], tag_revision: 0,
|
||||
capture: { token, sha256: SHA, preview_url: CAPTURE + token, size: 10, width: 8, height: 8, media_type: 'image',
|
||||
expires_at: '2026-10-22T12:00:00+00:00', generate: { supported: true, reason: null } },
|
||||
});
|
||||
|
||||
/** Real createApiClient over a fetch whose answers come from routes['METHOD path'](call). */
|
||||
function harness(routes = {}) {
|
||||
const calls = [];
|
||||
const fetch = (url, init = {}) => {
|
||||
const call = { url, method: init.method, body: init.body === undefined ? undefined : JSON.parse(init.body) };
|
||||
calls.push(call);
|
||||
const path = url.startsWith(CAPTURE) ? CAPTURE : url.split('?')[0];
|
||||
const route = routes[`${call.method} ${path}`];
|
||||
return new Promise((resolve, reject) => {
|
||||
init.signal?.addEventListener('abort', () => reject(new DOMException('Aborted', 'AbortError')));
|
||||
Promise.resolve().then(() => (route ? route(call) : response(200, { success: true }))).then(resolve, reject);
|
||||
});
|
||||
};
|
||||
return {
|
||||
api: createApiClient(fetch), calls,
|
||||
of: (method, prefix) => calls.filter(call => call.method === method && call.url.startsWith(prefix)),
|
||||
};
|
||||
}
|
||||
const QUEUE = [item('one'), item('two')];
|
||||
function routes(extra = {}) {
|
||||
return {
|
||||
[`GET ${ITEMS}`]: () => response(200, page(QUEUE)),
|
||||
[`GET ${ITEMS}/one`]: () => response(200, detailOf(QUEUE[0])),
|
||||
[`GET ${ITEMS}/two`]: () => response(200, detailOf(QUEUE[1], { candidates: [], limited: true, candidate_count: 0 })),
|
||||
[`POST ${OBSERVE}`]: () => response(200, observed()),
|
||||
[`POST ${ITEMS}/one/decision`]: call => response(200, { success: true, library_epoch: EPOCH, total: 1,
|
||||
item: { ...QUEUE[0], state: call.body.choice === 'unresolved' ? 'unresolved' : 'selected', revision: 1 } }),
|
||||
...extra,
|
||||
};
|
||||
}
|
||||
async function loaded(extra) {
|
||||
const { createReviewController } = await load();
|
||||
const h = harness(routes(extra));
|
||||
const c = createReviewController({ api: h.api });
|
||||
await c.load();
|
||||
await settle();
|
||||
return { c, h };
|
||||
}
|
||||
|
||||
test('notice titles for 0, 1 and 3', async () => {
|
||||
const { noticeTitle } = await load();
|
||||
assert.equal(noticeTitle(0), 'No images need a decision');
|
||||
assert.equal(noticeTitle(1), '1 image needs a decision');
|
||||
assert.equal(noticeTitle(3), '3 images need a decision');
|
||||
assert.equal(noticeTitle(1234), '1,234 images need a decision');
|
||||
});
|
||||
|
||||
test('loads pending items and the exact count', async () => {
|
||||
const { c, h } = await loaded();
|
||||
assert.equal(c.state.status, 'ready');
|
||||
assert.equal(c.state.total, 2);
|
||||
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
|
||||
const [list] = h.of('GET', ITEMS);
|
||||
assert.equal(list.url, `${ITEMS}?limit=40`);
|
||||
// The first item is selected, as the concept selects its first queue image.
|
||||
assert.equal(c.state.selectedId, 'one');
|
||||
assert.ok(Object.isFrozen(c.state));
|
||||
});
|
||||
|
||||
test('selecting a card loads detail and option labels, first workable preselected, unusable disabled', async () => {
|
||||
const { c } = await loaded();
|
||||
assert.equal(c.state.detailStatus, 'ready');
|
||||
assert.deepEqual(c.state.options.map(option => [option.label, option.disabled]), [
|
||||
['Sampler 9 · No positive / Soft negative', true],
|
||||
['Sampler 12 · Dunes / Soft negative', false],
|
||||
['Sampler 28 · Dunes / Harsh negative', false],
|
||||
['Leave unresolved', false],
|
||||
]);
|
||||
assert.equal(c.state.options.at(-1).value, 'unresolved');
|
||||
assert.equal(c.state.choice, HASH_A);
|
||||
assert.deepEqual(c.state.prompts, { positive: { status: 'resolved', text: 'golden dunes' }, negative: { status: 'resolved', text: 'blur' } });
|
||||
c.choose(HASH_C);
|
||||
assert.equal(c.state.choice, HASH_A, 'a disabled option cannot be chosen');
|
||||
c.choose(HASH_B);
|
||||
assert.equal(c.state.prompts.negative.text, 'noise');
|
||||
c.choose('unresolved');
|
||||
assert.equal(c.state.prompts.positive.status, 'ambiguous');
|
||||
await c.select('two');
|
||||
await settle();
|
||||
assert.equal(c.state.selectedId, 'two');
|
||||
assert.deepEqual(c.state.options.map(option => option.label), ['Leave unresolved']);
|
||||
assert.equal(c.state.choice, 'unresolved');
|
||||
assert.match(c.state.warning, /too many sampler paths/i);
|
||||
});
|
||||
|
||||
test('apply sends the selected candidate hash or unresolved with a fresh capture', async () => {
|
||||
const { c, h } = await loaded();
|
||||
c.choose(HASH_B);
|
||||
const done = await c.apply();
|
||||
assert.equal(done, true);
|
||||
const [observe] = h.of('POST', OBSERVE);
|
||||
assert.deepEqual(observe.body, { target: QUEUE[0].target });
|
||||
const [decision] = h.of('POST', `${ITEMS}/one/decision`);
|
||||
assert.deepEqual(decision.body, { expected_epoch: EPOCH, expected_revision: 0, capture_token: 'capture-1', sha256: SHA,
|
||||
dependency_fingerprint: PRINT, choice: HASH_B });
|
||||
assert.equal(c.state.phase, 'done');
|
||||
assert.equal(c.state.total, 1);
|
||||
assert.equal(c.state.toast, 'Association accepted.');
|
||||
// The server releases a used capture itself.
|
||||
assert.equal(h.of('DELETE', CAPTURE).length, 0);
|
||||
|
||||
const again = await loaded();
|
||||
again.c.choose('unresolved');
|
||||
assert.equal(await again.c.apply(), true);
|
||||
const [unresolved] = again.h.of('POST', `${ITEMS}/one/decision`);
|
||||
assert.equal(unresolved.body.choice, 'unresolved');
|
||||
assert.equal(unresolved.body.capture_token, 'capture-1');
|
||||
assert.equal(again.c.state.toast, 'Left unresolved.');
|
||||
});
|
||||
|
||||
test('an unavailable file can only be left unresolved, without a capture', async () => {
|
||||
const gone = item('one', { target: null, unavailable: true, url: null });
|
||||
const { c, h } = await loaded({
|
||||
[`GET ${ITEMS}`]: () => response(200, page([gone])),
|
||||
[`GET ${ITEMS}/one`]: () => response(200, detailOf(gone)),
|
||||
});
|
||||
assert.ok(c.state.options.slice(0, -1).every(option => option.disabled));
|
||||
assert.equal(c.state.choice, 'unresolved');
|
||||
assert.match(c.state.warning, /not under a current image source/i);
|
||||
assert.equal(await c.apply(), true);
|
||||
assert.equal(h.of('POST', OBSERVE).length, 0);
|
||||
const [decision] = h.of('POST', `${ITEMS}/one/decision`);
|
||||
assert.equal(decision.body.capture_token, null);
|
||||
assert.equal(decision.body.sha256, SHA);
|
||||
});
|
||||
|
||||
test('association conflict keeps the selection and shows the inline warning', async () => {
|
||||
const message = 'This image is already linked to another prompt or negative, so this sampler path cannot be attached safely.';
|
||||
const { c, h } = await loaded({ [`POST ${ITEMS}/one/decision`]: () => fail(409, 'association-conflict', message) });
|
||||
assert.equal(await c.apply(), false);
|
||||
assert.equal(c.state.selectedId, 'one');
|
||||
assert.equal(c.state.choice, HASH_A);
|
||||
assert.equal(c.state.phase, 'idle');
|
||||
assert.equal(c.state.warning, message);
|
||||
// The refused decision's capture is released.
|
||||
assert.equal(h.of('DELETE', CAPTURE).length, 1);
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'no reload');
|
||||
});
|
||||
|
||||
test('storage-changed requires reload', async () => {
|
||||
let lists = 0;
|
||||
const { c, h } = await loaded({
|
||||
[`GET ${ITEMS}`]: () => { lists += 1; return response(200, page(QUEUE)); },
|
||||
[`POST ${ITEMS}/one/decision`]: () => fail(409, 'storage-changed', 'Prompt storage was restored; reload the review queue.'),
|
||||
});
|
||||
assert.equal(await c.apply(), false);
|
||||
await settle();
|
||||
assert.equal(lists, 2, 'the queue is read again under the new epoch');
|
||||
assert.equal(c.state.warning, 'Prompt storage was restored; reload the review queue.');
|
||||
assert.equal(c.state.canRetry, false);
|
||||
assert.equal(h.of('DELETE', CAPTURE).length, 1);
|
||||
});
|
||||
|
||||
test('lost response keeps the selection and retry resends once', async () => {
|
||||
let attempts = 0;
|
||||
const { c, h } = await loaded({
|
||||
[`POST ${ITEMS}/one/decision`]: call => {
|
||||
attempts += 1;
|
||||
if (attempts === 1) return Promise.reject(new TypeError('Failed to fetch'));
|
||||
return response(200, { success: true, library_epoch: EPOCH, total: 1, item: { ...QUEUE[0], state: 'selected', revision: 1 } });
|
||||
},
|
||||
});
|
||||
assert.equal(await c.apply(), false);
|
||||
assert.equal(c.state.selectedId, 'one');
|
||||
assert.equal(c.state.canRetry, true);
|
||||
assert.match(c.state.warning, /may not have been saved/i);
|
||||
assert.equal(h.of('DELETE', CAPTURE).length, 0, 'the capture is kept for the retry');
|
||||
const [first] = h.of('POST', `${ITEMS}/one/decision`);
|
||||
assert.equal(await c.retry(), true);
|
||||
const sent = h.of('POST', `${ITEMS}/one/decision`);
|
||||
assert.equal(sent.length, 2);
|
||||
assert.deepEqual(sent[1].body, first.body, 'the identical decision is resent');
|
||||
assert.equal(h.of('POST', OBSERVE).length, 1, 'no second capture');
|
||||
assert.equal(await c.retry(), false, 'nothing left to resend');
|
||||
assert.equal(h.of('POST', `${ITEMS}/one/decision`).length, 2);
|
||||
});
|
||||
|
||||
test('a retry answered by review-conflict reloads the queue', async () => {
|
||||
let attempts = 0, lists = 0;
|
||||
const { c } = await loaded({
|
||||
[`GET ${ITEMS}`]: () => { lists += 1; return response(200, page(lists === 1 ? QUEUE : [QUEUE[1]])); },
|
||||
[`POST ${ITEMS}/one/decision`]: () => {
|
||||
attempts += 1;
|
||||
if (attempts === 1) return Promise.reject(new TypeError('Failed to fetch'));
|
||||
return fail(409, 'review-conflict', 'This item was decided or changed since it was opened. Reload the queue.');
|
||||
},
|
||||
});
|
||||
await c.apply();
|
||||
assert.equal(await c.retry(), false);
|
||||
await settle();
|
||||
assert.equal(lists, 2);
|
||||
assert.equal(c.state.selectedId, 'two');
|
||||
});
|
||||
|
||||
test('a review-conflict on a still pending item reads its detail again', async () => {
|
||||
let details = 0, decisions = 0;
|
||||
const { c, h } = await loaded({
|
||||
[`GET ${ITEMS}/one`]: () => { details += 1; return response(200, detailOf({ ...QUEUE[0], revision: details - 1 })); },
|
||||
[`POST ${ITEMS}/one/decision`]: () => {
|
||||
decisions += 1;
|
||||
return decisions === 1 ? fail(409, 'review-conflict', 'This item was decided or changed since it was opened. Reload the queue.')
|
||||
: response(200, { success: true, library_epoch: EPOCH, total: 1, item: { ...QUEUE[0], state: 'selected', revision: 2 } });
|
||||
},
|
||||
});
|
||||
assert.equal(await c.apply(), false);
|
||||
await settle();
|
||||
assert.equal(details, 2, 'the item is read again');
|
||||
assert.equal(c.state.detail.revision, 1);
|
||||
assert.equal(c.state.warning, 'This item was decided or changed since it was opened. Reload the queue.');
|
||||
assert.equal(await c.apply(), true);
|
||||
assert.equal(h.of('POST', `${ITEMS}/one/decision`).at(-1).body.expected_revision, 1);
|
||||
});
|
||||
|
||||
test('leave for later writes nothing', async () => {
|
||||
const { c, h } = await loaded();
|
||||
c.leave();
|
||||
assert.equal(c.state.phase, 'done');
|
||||
assert.equal(c.state.toast, null);
|
||||
assert.equal(h.of('POST', '/').length, 0);
|
||||
assert.equal(h.of('DELETE', '/').length, 0);
|
||||
});
|
||||
|
||||
test('load more pages with the cursor and keeps the selection', async () => {
|
||||
const more = [item('three')];
|
||||
const { c, h } = await loaded({
|
||||
[`GET ${ITEMS}`]: call => response(200, call.url.includes('cursor=')
|
||||
? page(more, { total: 3 }) : page(QUEUE, { total: 3, next_cursor: 'abc_-1' })),
|
||||
});
|
||||
assert.equal(c.state.nextCursor, 'abc_-1');
|
||||
await c.loadMore();
|
||||
assert.equal(h.of('GET', ITEMS).at(-1).url, `${ITEMS}?limit=40&cursor=abc_-1`);
|
||||
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two', 'three']);
|
||||
assert.equal(c.state.nextCursor, null);
|
||||
assert.equal(c.state.selectedId, 'one');
|
||||
});
|
||||
|
||||
test('a failed queue read is an error state and load retries', async () => {
|
||||
let lists = 0;
|
||||
const { c } = await loaded({ [`GET ${ITEMS}`]: () => { lists += 1; return lists === 1 ? fail(500, 'review-failed') : response(200, page(QUEUE)); } });
|
||||
assert.equal(c.state.status, 'error');
|
||||
await c.load();
|
||||
await settle();
|
||||
assert.equal(c.state.status, 'ready');
|
||||
});
|
||||
|
||||
test('tag save toast carries the review suffix and the item stays pending', async () => {
|
||||
const { c, h } = await loaded({
|
||||
['GET /prompt_manager/autotag/models']: () => response(200, { success: true, models: {}, default_prompt: 'x',
|
||||
wd14_general_threshold: 0.35, wd14_character_threshold: 0.85, suggestion_limit: 10 }),
|
||||
['PUT /prompt_manager/image-annotations/tags']: call => response(200, { success: true, library_epoch: EPOCH, sha256: SHA,
|
||||
tags: call.body.tags, tag_revision: 1, replayed: false }),
|
||||
});
|
||||
const { createImageTaggingController } = await import('../../web/js/workspace/image-tagging-state.mjs');
|
||||
const before = h.calls.filter(call => call.url.startsWith(ITEMS)).length;
|
||||
// The Tag dialog opened from Needs review is the shared controller with the review source.
|
||||
const tagging = createImageTaggingController({ api: h.api, target: QUEUE[0].target, source: 'review' });
|
||||
assert.equal(tagging.state.showScopeNote, true);
|
||||
await tagging.open();
|
||||
await settle();
|
||||
tagging.setManual('dunes');
|
||||
await tagging.save();
|
||||
await settle();
|
||||
assert.equal(tagging.state.toast, '1 tags saved for one.png. Prompt association still awaits review.');
|
||||
tagging.dispose();
|
||||
assert.equal(h.calls.filter(call => call.url.startsWith(ITEMS)).length, before, 'a tag save neither decides nor re-reads the queue');
|
||||
assert.deepEqual(c.state.items.map(entry => [entry.id, entry.state]), [['one', 'pending'], ['two', 'pending']]);
|
||||
assert.equal(c.state.phase, 'idle');
|
||||
});
|
||||
|
||||
test('dispose releases a kept capture and stops reads', async () => {
|
||||
const { c, h } = await loaded({ [`POST ${ITEMS}/one/decision`]: () => Promise.reject(new TypeError('Failed to fetch')) });
|
||||
await c.apply();
|
||||
c.dispose();
|
||||
await settle();
|
||||
const released = h.of('DELETE', CAPTURE);
|
||||
assert.equal(released.length, 1);
|
||||
assert.equal(released[0].url, CAPTURE + 'capture-1');
|
||||
assert.equal(await c.retry(), false);
|
||||
});
|
||||
|
||||
// F2: an open page re-reads its queue when the server says it changed (the review change stream).
|
||||
function refreshOf(c) {
|
||||
assert.equal(typeof c.refresh, 'function', 'refresh is not implemented');
|
||||
return c.refresh();
|
||||
}
|
||||
test('refresh re-reads the queue quietly and keeps the selection and choice', async () => {
|
||||
let queue = QUEUE;
|
||||
const { c, h } = await loaded({ [`GET ${ITEMS}`]: () => response(200, page(queue)) });
|
||||
c.choose(HASH_B);
|
||||
const statuses = [];
|
||||
c.subscribe(state => statuses.push(state.status));
|
||||
queue = [...QUEUE, item('three')];
|
||||
await refreshOf(c);
|
||||
await settle();
|
||||
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two', 'three']);
|
||||
assert.equal(c.state.total, 3);
|
||||
assert.equal(c.state.selectedId, 'one');
|
||||
assert.equal(c.state.choice, HASH_B);
|
||||
assert.equal(c.state.warning, '');
|
||||
assert.ok(statuses.every(status => status === 'ready'), `no loading state: ${statuses}`);
|
||||
assert.equal(h.of('GET', `${ITEMS}/one`).length, 1, 'the open item is not read again');
|
||||
});
|
||||
|
||||
test('refreshes asked for during a refresh collapse into one more read', async () => {
|
||||
let release;
|
||||
const held = new Promise(resolve => { release = resolve; });
|
||||
let lists = 0;
|
||||
const { c, h } = await loaded({ [`GET ${ITEMS}`]: async () => {
|
||||
lists += 1;
|
||||
if (lists === 2) await held;
|
||||
return response(200, page(QUEUE));
|
||||
} });
|
||||
const first = refreshOf(c);
|
||||
await settle();
|
||||
refreshOf(c);
|
||||
refreshOf(c);
|
||||
release();
|
||||
await first;
|
||||
await settle();
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 3, 'the load, the refresh and one collapsed re-read');
|
||||
assert.equal(c.state.status, 'ready');
|
||||
});
|
||||
|
||||
test('a refresh after the open item was decided elsewhere opens the next one and says so', async () => {
|
||||
let queue = QUEUE;
|
||||
const { c, h } = await loaded({
|
||||
[`GET ${ITEMS}`]: () => response(200, page(queue)),
|
||||
[`GET ${ITEMS}/three`]: () => response(200, detailOf(item('three'))),
|
||||
});
|
||||
queue = [item('three')];
|
||||
await refreshOf(c);
|
||||
await settle();
|
||||
assert.equal(c.state.selectedId, 'three');
|
||||
assert.equal(c.state.detailStatus, 'ready');
|
||||
assert.equal(c.state.warning, 'one.png was decided elsewhere, so the next image is open.');
|
||||
assert.equal(h.of('GET', `${ITEMS}/three`).length, 1);
|
||||
});
|
||||
|
||||
test('a refresh to an empty queue clears the selection', async () => {
|
||||
let queue = QUEUE;
|
||||
const { c } = await loaded({ [`GET ${ITEMS}`]: () => response(200, page(queue)) });
|
||||
queue = [];
|
||||
await refreshOf(c);
|
||||
await settle();
|
||||
assert.equal(c.state.total, 0);
|
||||
assert.deepEqual(c.state.items, []);
|
||||
assert.equal(c.state.selectedId, null);
|
||||
assert.equal(c.state.status, 'ready');
|
||||
});
|
||||
|
||||
test('a failed refresh keeps the queue and says so', async () => {
|
||||
let lists = 0;
|
||||
const { c } = await loaded({ [`GET ${ITEMS}`]: () => { lists += 1; return lists === 1 ? response(200, page(QUEUE)) : fail(500, 'review-failed'); } });
|
||||
await refreshOf(c);
|
||||
await settle();
|
||||
assert.equal(c.state.status, 'ready');
|
||||
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
|
||||
assert.equal(c.state.warning, 'The review queue could not be read again. Reload the page to see the latest images.');
|
||||
});
|
||||
|
||||
test('dispose during a decision releases its capture', async () => {
|
||||
const { c, h } = await loaded({ [`POST ${ITEMS}/one/decision`]: () => new Promise(() => {}) });
|
||||
const applying = c.apply();
|
||||
await settle();
|
||||
assert.equal(c.state.phase, 'applying');
|
||||
c.dispose();
|
||||
assert.equal(await applying, false);
|
||||
await settle();
|
||||
assert.deepEqual(h.of('DELETE', CAPTURE).map(call => call.url), [CAPTURE + 'capture-1']);
|
||||
await refreshOf(c);
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'a disposed controller reads nothing');
|
||||
});
|
||||
|
||||
test('a change heard during the first read re-reads once after it, and a failed first read stays an error', async () => {
|
||||
const { createReviewController } = await load();
|
||||
let release;
|
||||
let lists = 0;
|
||||
const h = harness(routes({ [`GET ${ITEMS}`]: async () => {
|
||||
lists += 1;
|
||||
if (lists === 1) await new Promise(resolve => { release = resolve; });
|
||||
return response(200, page(QUEUE));
|
||||
} }));
|
||||
const c = createReviewController({ api: h.api });
|
||||
const first = c.load();
|
||||
await settle();
|
||||
refreshOf(c);
|
||||
assert.equal(lists, 1, 'the first read is not interrupted');
|
||||
release();
|
||||
await first;
|
||||
await settle();
|
||||
assert.equal(lists, 2, 'one re-read after the first read');
|
||||
assert.equal(c.state.status, 'ready');
|
||||
|
||||
const failing = harness(routes({ [`GET ${ITEMS}`]: () => fail(500, 'review-failed') }));
|
||||
const broken = createReviewController({ api: failing.api });
|
||||
const reading = broken.load();
|
||||
refreshOf(broken);
|
||||
await reading;
|
||||
await settle();
|
||||
assert.equal(broken.state.status, 'error');
|
||||
assert.equal(failing.of('GET', `${ITEMS}?`).length, 1, 'Retry is the way back, not a re-read');
|
||||
});
|
||||
|
||||
// F3 rulings: a live re-read keeps the loaded depth, and the stream's `ready` reads nothing when the first read is current.
|
||||
|
||||
/** A keyset pager over `source()`: pages of 40 with an index cursor, each answering `version()`. */
|
||||
function pager(source, version = () => 0, { onPage } = {}) {
|
||||
return call => {
|
||||
const query = new URL(call.url, 'http://x').searchParams;
|
||||
const cursor = query.get('cursor');
|
||||
const all = source();
|
||||
const from = cursor ? Number(cursor.slice(1)) : 0;
|
||||
const answer = onPage?.(cursor);
|
||||
if (answer) return answer;
|
||||
const items = all.slice(from, from + 40);
|
||||
const next = from + 40 < all.length ? `c${from + 40}` : null;
|
||||
return response(200, page(items, { total: all.length, next_cursor: next, review_revision: version() }));
|
||||
};
|
||||
}
|
||||
const many = (count, prefix = 'i') => Array.from({ length: count }, (_, index) => item(`${prefix}${String(index).padStart(3, '0')}`));
|
||||
function syncOf(c, version) {
|
||||
assert.equal(typeof c.sync, 'function', 'sync is not implemented');
|
||||
return c.sync(version);
|
||||
}
|
||||
const version = revision => ({ library_epoch: EPOCH, review_revision: revision });
|
||||
|
||||
test('a refresh keeps the loaded depth: it re-reads as many items as were loaded', async () => {
|
||||
let queue = many(100);
|
||||
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => queue) });
|
||||
await c.loadMore();
|
||||
assert.equal(c.state.items.length, 80);
|
||||
const before = h.of('GET', `${ITEMS}?`).length;
|
||||
queue = [...queue, item('late')];
|
||||
await refreshOf(c);
|
||||
await settle();
|
||||
assert.equal(c.state.items.length, 80, 'the Load more page is kept');
|
||||
assert.deepEqual(c.state.items.map(entry => entry.id), queue.slice(0, 80).map(entry => entry.id));
|
||||
assert.equal(c.state.total, 101);
|
||||
assert.equal(c.state.nextCursor, 'c80', 'Load more continues after the kept depth');
|
||||
assert.deepEqual(h.of('GET', `${ITEMS}?`).slice(before).map(call => call.url),
|
||||
[`${ITEMS}?limit=40`, `${ITEMS}?limit=40&cursor=c40`]);
|
||||
assert.equal(c.state.selectedId, 'i000');
|
||||
});
|
||||
|
||||
test('a refresh after decisions elsewhere fills the loaded depth from what is left', async () => {
|
||||
let queue = many(85);
|
||||
const { c } = await loaded({ [`GET ${ITEMS}`]: pager(() => queue) });
|
||||
await c.loadMore();
|
||||
await c.loadMore();
|
||||
assert.equal(c.state.items.length, 85);
|
||||
queue = queue.filter((_, index) => index % 10 !== 5);
|
||||
await refreshOf(c);
|
||||
await settle();
|
||||
assert.equal(c.state.items.length, queue.length);
|
||||
assert.equal(c.state.nextCursor, null);
|
||||
});
|
||||
|
||||
test('a deep refresh whose next page meets a restore reads again from the first page', async () => {
|
||||
let restored = false;
|
||||
let queue = many(60);
|
||||
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => queue, () => 0, {
|
||||
onPage: cursor => {
|
||||
if (cursor && restored) { restored = false; return fail(409, 'storage-changed', 'The library was restored.'); }
|
||||
return null;
|
||||
},
|
||||
}) });
|
||||
await c.loadMore();
|
||||
const before = h.of('GET', `${ITEMS}?`).length;
|
||||
restored = true;
|
||||
queue = many(60, 'r');
|
||||
await refreshOf(c);
|
||||
await settle();
|
||||
assert.deepEqual(h.of('GET', `${ITEMS}?`).slice(before).map(call => call.url),
|
||||
[`${ITEMS}?limit=40`, `${ITEMS}?limit=40&cursor=c40`, `${ITEMS}?limit=40`, `${ITEMS}?limit=40&cursor=c40`]);
|
||||
assert.equal(c.state.items.length, 60);
|
||||
assert.equal(c.state.items[0].id, 'r000');
|
||||
assert.notEqual(c.state.warning, 'The review queue could not be read again. Reload the page to see the latest images.',
|
||||
'a restore met mid-read is not a failed re-read');
|
||||
assert.equal(c.state.status, 'ready');
|
||||
});
|
||||
|
||||
test('ready naming the version the first read answered reads nothing more', async () => {
|
||||
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => QUEUE, () => 4) });
|
||||
await syncOf(c, version(4));
|
||||
await settle();
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'one list read per visit');
|
||||
assert.equal(h.of('GET', `${ITEMS}/one`).length, 1);
|
||||
});
|
||||
|
||||
test('ready naming a newer version, another epoch or no version re-reads once', async () => {
|
||||
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => QUEUE, () => 4) });
|
||||
await syncOf(c, version(5));
|
||||
await settle();
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 2);
|
||||
await syncOf(c, { library_epoch: '22222222-2222-4222-8222-222222222222', review_revision: 4 });
|
||||
await settle();
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 3);
|
||||
await syncOf(c, {});
|
||||
await settle();
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 4, 'an unknown version is read, as before versions existed');
|
||||
});
|
||||
|
||||
test('ready heard during the first read re-reads only when that read is older', async () => {
|
||||
const { createReviewController } = await load();
|
||||
for (const [readRevision, readyRevision, reads] of [[4, 4, 1], [5, 4, 1], [4, 5, 2]]) {
|
||||
let release;
|
||||
let lists = 0;
|
||||
const answer = pager(() => QUEUE, () => (lists === 1 ? readRevision : readyRevision));
|
||||
const h = harness(routes({ [`GET ${ITEMS}`]: async call => {
|
||||
lists += 1;
|
||||
if (lists === 1) await new Promise(resolve => { release = resolve; });
|
||||
return answer(call);
|
||||
} }));
|
||||
const c = createReviewController({ api: h.api });
|
||||
const first = c.load();
|
||||
await settle();
|
||||
syncOf(c, version(readyRevision));
|
||||
release();
|
||||
await first;
|
||||
await settle();
|
||||
assert.equal(lists, reads, `first read r${readRevision}, ready r${readyRevision}`);
|
||||
c.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('a change heard during the first read still re-reads after a current ready', async () => {
|
||||
const { createReviewController } = await load();
|
||||
let release;
|
||||
let lists = 0;
|
||||
const h = harness(routes({ [`GET ${ITEMS}`]: async call => {
|
||||
lists += 1;
|
||||
if (lists === 1) await new Promise(resolve => { release = resolve; });
|
||||
return pager(() => QUEUE, () => 4)(call);
|
||||
} }));
|
||||
const c = createReviewController({ api: h.api });
|
||||
const first = c.load();
|
||||
await settle();
|
||||
syncOf(c, version(4));
|
||||
refreshOf(c);
|
||||
release();
|
||||
await first;
|
||||
await settle();
|
||||
assert.equal(lists, 2);
|
||||
});
|
||||
|
||||
// Final review round (I1, I2, M3–M5, M7).
|
||||
test('I1: a change heard during a refused Apply is re-read once the phase is idle again', async () => {
|
||||
let queue = [item('one')], releaseDecision;
|
||||
const { c, h } = await loaded({
|
||||
[`GET ${ITEMS}`]: () => response(200, page(queue, { review_revision: queue.length })),
|
||||
[`POST ${ITEMS}/one/decision`]: () => new Promise(resolve => {
|
||||
releaseDecision = () => resolve(fail(409, 'association-conflict', 'conflict'));
|
||||
}),
|
||||
});
|
||||
const applying = c.apply();
|
||||
await settle();
|
||||
assert.equal(c.state.phase, 'applying');
|
||||
queue = [item('one'), item('two')]; // the monitor records an item while the decision is in flight
|
||||
c.refresh();
|
||||
await settle();
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'no read while the decision is in flight');
|
||||
releaseDecision();
|
||||
await applying;
|
||||
await settle();
|
||||
assert.equal(c.state.phase, 'idle');
|
||||
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
|
||||
assert.equal(c.state.total, 2);
|
||||
assert.equal(c.state.warning, 'conflict', 'the refusal stays the warning');
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 2, 'exactly one owed re-read');
|
||||
});
|
||||
|
||||
test('I1: a change heard during an Apply whose observe is refused is re-read too', async () => {
|
||||
let queue = [item('one')], releaseObserve;
|
||||
const { c } = await loaded({
|
||||
[`GET ${ITEMS}`]: () => response(200, page(queue)),
|
||||
[`POST ${OBSERVE}`]: () => new Promise(resolve => {
|
||||
releaseObserve = () => resolve(fail(409, 'capture-capacity', 'busy'));
|
||||
}),
|
||||
});
|
||||
const applying = c.apply();
|
||||
await settle();
|
||||
queue = [item('one'), item('two')];
|
||||
c.refresh();
|
||||
releaseObserve();
|
||||
await applying;
|
||||
await settle();
|
||||
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
|
||||
});
|
||||
|
||||
const forty = Array.from({ length: 41 }, (_, i) => item(`i${String(i).padStart(2, '0')}`));
|
||||
function deepRoutes(state) {
|
||||
return {
|
||||
[`GET ${ITEMS}`]: call => {
|
||||
const cursor = new URL(call.url, 'http://x').searchParams.get('cursor');
|
||||
if (cursor) {
|
||||
if (state.moreGate) return state.moreGate.then(() => state.more ?? response(200, page([forty[40]], { total: 41 })));
|
||||
return state.more ?? response(200, page([forty[40]], { total: 41 }));
|
||||
}
|
||||
const body = response(200, page(state.first, { total: state.first.length + 1, review_revision: state.revision ?? 1, next_cursor: 'cur' }));
|
||||
return state.gate ? state.gate.then(() => state.fail ?? body) : body;
|
||||
},
|
||||
[`GET ${ITEMS}/i00`]: () => response(200, detailOf(forty[0])),
|
||||
};
|
||||
}
|
||||
|
||||
test('I2: Load more during a re-read waits for it and continues from the re-read queue', async () => {
|
||||
const state = { first: forty.slice(0, 40) };
|
||||
const { c } = await loaded(deepRoutes(state));
|
||||
// Another tab decides i05; the re-read is slow and the user clicks Load more meanwhile.
|
||||
let open;
|
||||
state.first = forty.slice(0, 40).filter(entry => entry.id !== 'i05');
|
||||
state.gate = new Promise(resolve => { open = resolve; });
|
||||
const refreshing = c.refresh();
|
||||
await settle();
|
||||
const more = c.loadMore();
|
||||
await settle();
|
||||
assert.equal(c.state.more, 'loading', 'the click shows as loading while the re-read runs');
|
||||
open();
|
||||
await Promise.all([refreshing, more]);
|
||||
await settle();
|
||||
const ids = c.state.items.map(entry => entry.id);
|
||||
assert.ok(!ids.includes('i05'), 'the re-read was applied');
|
||||
assert.equal(ids.at(-1), 'i40', 'Load more appended the next page after the re-read');
|
||||
assert.equal(c.state.more, 'idle');
|
||||
});
|
||||
|
||||
test('I2: a re-read started during Load more takes over the click and reads one page deeper', async () => {
|
||||
const state = { first: forty.slice(0, 40) };
|
||||
const { c, h } = await loaded(deepRoutes(state));
|
||||
let openMore;
|
||||
state.moreGate = new Promise(resolve => { openMore = resolve; });
|
||||
const more = c.loadMore();
|
||||
await settle();
|
||||
state.moreGate = null;
|
||||
state.first = forty.slice(0, 40).filter(entry => entry.id !== 'i05');
|
||||
await c.refresh();
|
||||
openMore();
|
||||
await more;
|
||||
await settle();
|
||||
const ids = c.state.items.map(entry => entry.id);
|
||||
assert.ok(!ids.includes('i05'));
|
||||
assert.equal(ids.at(-1), 'i40', 'the aborted Load more page is part of the re-read');
|
||||
assert.equal(c.state.more, 'idle');
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 4, 'load, the aborted more, and the re-read of two pages');
|
||||
});
|
||||
|
||||
test('M7: a failed re-read that took over Load more clears its loading state', async () => {
|
||||
const state = { first: forty.slice(0, 40) };
|
||||
const { c } = await loaded(deepRoutes(state));
|
||||
state.moreGate = new Promise(() => {});
|
||||
c.loadMore();
|
||||
await settle();
|
||||
assert.equal(c.state.more, 'loading');
|
||||
state.moreGate = null;
|
||||
state.gate = Promise.resolve();
|
||||
state.fail = fail(500, 'review-failed');
|
||||
await c.refresh();
|
||||
await settle();
|
||||
assert.equal(c.state.more, 'idle', 'Load more can be clicked again');
|
||||
assert.equal(c.state.warning, COPY_REFRESH_FAILED);
|
||||
});
|
||||
const COPY_REFRESH_FAILED = 'The review queue could not be read again. Reload the page to see the latest images.';
|
||||
|
||||
test('M3: Apply fences on the epoch of the detail it decides', async () => {
|
||||
const OTHER = '22222222-2222-4222-8222-222222222222';
|
||||
const { c, h } = await loaded({ [`POST ${OBSERVE}`]: () => response(200, { ...observed(), library_epoch: OTHER }) });
|
||||
assert.equal(await c.apply(), false);
|
||||
await settle();
|
||||
assert.equal(h.of('POST', `${ITEMS}/one/decision`).length, 0, 'a restored storage sends no decision');
|
||||
assert.deepEqual(h.of('DELETE', CAPTURE).map(call => call.url), [CAPTURE + 'capture-1']);
|
||||
assert.equal(h.of('GET', `${ITEMS}?`).length, 2, 'the queue is read again');
|
||||
assert.equal(h.of('GET', `${ITEMS}/one`).length, 2, 'the open item is read again under the new storage');
|
||||
assert.equal(c.state.phase, 'idle');
|
||||
});
|
||||
|
||||
test('M3: the decision carries the detail epoch', async () => {
|
||||
const { c, h } = await loaded();
|
||||
assert.equal(await c.apply(), true);
|
||||
assert.equal(h.of('POST', `${ITEMS}/one/decision`)[0].body.expected_epoch, EPOCH);
|
||||
});
|
||||
|
||||
test('M3: a quiet re-read under another epoch reads the open item again', async () => {
|
||||
const OTHER = '22222222-2222-4222-8222-222222222222';
|
||||
let epoch = EPOCH;
|
||||
const { c, h } = await loaded({
|
||||
[`GET ${ITEMS}`]: () => response(200, page(QUEUE, { library_epoch: epoch })),
|
||||
[`GET ${ITEMS}/one`]: () => response(200, detailOf(QUEUE[0], { library_epoch: epoch })),
|
||||
});
|
||||
epoch = OTHER;
|
||||
await c.refresh();
|
||||
await settle();
|
||||
assert.equal(h.of('GET', `${ITEMS}/one`).length, 2);
|
||||
assert.equal(c.state.detail.library_epoch, OTHER);
|
||||
});
|
||||
|
||||
test('M4: leaving during the observe releases the capture it then answers', async () => {
|
||||
let answer;
|
||||
const { c, h } = await loaded({ [`POST ${OBSERVE}`]: () => new Promise(resolve => { answer = resolve; }) });
|
||||
const applying = c.apply();
|
||||
await settle();
|
||||
c.dispose();
|
||||
answer(response(200, observed('late')));
|
||||
assert.equal(await applying, false);
|
||||
await settle();
|
||||
assert.deepEqual(h.of('DELETE', CAPTURE).map(call => call.url), [CAPTURE + 'late']);
|
||||
assert.equal(h.of('POST', `${ITEMS}/one/decision`).length, 0);
|
||||
});
|
||||
|
||||
test('M5: an unanswered decision that landed finishes as recorded, not as decided elsewhere', async () => {
|
||||
let queue = QUEUE;
|
||||
const { c, h } = await loaded({
|
||||
[`GET ${ITEMS}`]: () => response(200, page(queue)),
|
||||
[`POST ${ITEMS}/one/decision`]: () => Promise.reject(new TypeError('Failed to fetch')),
|
||||
});
|
||||
await c.apply();
|
||||
assert.equal(c.state.canRetry, true);
|
||||
queue = [QUEUE[1]]; // the server did record it: its change arrives
|
||||
await c.refresh();
|
||||
await settle();
|
||||
assert.equal(c.state.phase, 'done');
|
||||
assert.equal(c.state.toast, 'Association accepted.');
|
||||
assert.doesNotMatch(c.state.warning, /decided elsewhere/);
|
||||
assert.equal(h.of('DELETE', CAPTURE).length, 0, 'the server used the capture');
|
||||
});
|
||||
@@ -4,7 +4,7 @@ const source = import('../../web/js/workspace/state.mjs');
|
||||
|
||||
test('workspace routes resolve only supported deep links', async () => {
|
||||
const { resolveRoute } = await source;
|
||||
for (const route of ['overview', 'images', 'library', 'tags', 'inspect', 'system', 'activity', 'review']) {
|
||||
for (const route of ['overview', 'images', 'library', 'tags', 'inspect', 'system', 'activity']) {
|
||||
assert.equal(resolveRoute(`#/${route}`).id, route);
|
||||
}
|
||||
for (const hash of ['', '#', '#/unknown', '#/images/other', '#/<script>', '#/activity/abc', `#/activity/${'A'.repeat(32)}`]) {
|
||||
@@ -24,7 +24,7 @@ test('tags is a native route titled Tags; only unbuilt routes hand off to legacy
|
||||
}
|
||||
assert.equal(routeTitle(resolveRoute('#/library/7')), 'Prompt library');
|
||||
// System is native from S2 (#249 System): its undelivered sections hand off per section (D94), not the route.
|
||||
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity', 'review', 'system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
|
||||
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity', 'system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
|
||||
});
|
||||
|
||||
test('system resolves its sections; #/overview stays the production hub', async () => {
|
||||
@@ -157,10 +157,9 @@ test('positive library deep links preserve the selected identity without duplica
|
||||
|
||||
test('nav counts reject negative, non-integer and missing totals', async () => {
|
||||
const { summarizeNavCounts } = await source;
|
||||
const counts = summarizeNavCounts({images: {total: 24}, favorites: {total: 3}, library: {total: 0}, tags: {total: 7},
|
||||
review: {total: 3}});
|
||||
const counts = summarizeNavCounts({images: {total: 24}, favorites: {total: 3}, library: {total: 0}, tags: {total: 7}});
|
||||
assert.ok(counts instanceof Map);
|
||||
assert.deepEqual([...counts], [['images', 24], ['favorites', 3], ['library', 0], ['tags', 7], ['review', 3]]);
|
||||
assert.deepEqual([...counts], [['images', 24], ['favorites', 3], ['library', 0], ['tags', 7]]);
|
||||
assert.equal(counts.has('inspect'), false, 'Inspect a file has no count');
|
||||
for (const invalid of [undefined, null, {}, {total: -1}, {total: 1.5}, {total: '3'}, {total: NaN},
|
||||
{total: Infinity}, {total: Number.MAX_SAFE_INTEGER + 1}, {total: null}, [], 'x']) {
|
||||
@@ -170,8 +169,7 @@ test('nav counts reject negative, non-integer and missing totals', async () => {
|
||||
assert.equal(result.get('library'), 2);
|
||||
assert.equal(result.get('tags'), null);
|
||||
}
|
||||
assert.deepEqual([...summarizeNavCounts()], [['images', null], ['favorites', null], ['library', null], ['tags', null],
|
||||
['review', null]]);
|
||||
assert.deepEqual([...summarizeNavCounts()], [['images', null], ['favorites', null], ['library', null], ['tags', null]]);
|
||||
});
|
||||
|
||||
test('nav counts settle independently so a slow images read never delays the others', async () => {
|
||||
@@ -187,14 +185,14 @@ test('nav counts settle independently so a slow images read never delays the oth
|
||||
const settled = [];
|
||||
const done = loadNavCounts(api, new AbortController().signal, (id, count) => settled.push([id, count]));
|
||||
await new Promise(resolve => setTimeout(resolve, 0));
|
||||
assert.deepEqual(settled.sort(), [['favorites', 2], ['library', 3], ['review', null], ['tags', null]]);
|
||||
assert.deepEqual(settled.sort(), [['favorites', 2], ['library', 3], ['tags', null]]);
|
||||
releaseImages({success: true, total: 40});
|
||||
const counts = await done;
|
||||
assert.deepEqual(settled.at(-1), ['images', 40]);
|
||||
assert.deepEqual([...counts], [['images', 40], ['favorites', 2], ['library', 3], ['tags', null], ['review', null]]);
|
||||
assert.deepEqual([...counts], [['images', 40], ['favorites', 2], ['library', 3], ['tags', null]]);
|
||||
});
|
||||
|
||||
test('nav counts issue five bounded requests and drop only the failed count', async () => {
|
||||
test('nav counts issue four bounded requests and drop only the failed count', async () => {
|
||||
const { loadNavCounts } = await source;
|
||||
const calls = [];
|
||||
const signal = new AbortController().signal;
|
||||
@@ -202,15 +200,13 @@ test('nav counts issue five bounded requests and drop only the failed count', as
|
||||
calls.push([url, options.signal]);
|
||||
if (url.startsWith('/prompt_manager/library')) throw new Error('Unavailable');
|
||||
if (url.startsWith('/prompt_manager/vocabulary/tags')) return {success: true, total: 5};
|
||||
if (url.startsWith('/prompt_manager/review/items')) return {success: true, total: 2};
|
||||
return {success: true, total: 12};
|
||||
}};
|
||||
const counts = await loadNavCounts(api, signal);
|
||||
assert.deepEqual([...counts], [['images', 12], ['favorites', 12], ['library', null], ['tags', 5], ['review', 2]]);
|
||||
assert.deepEqual([...counts], [['images', 12], ['favorites', 12], ['library', null], ['tags', 5]]);
|
||||
const urls = calls.map(([url]) => new URL(url, 'http://host'));
|
||||
assert.deepEqual(urls.map(url => url.pathname),
|
||||
['/prompt_manager/images/output', '/prompt_manager/images/output', '/prompt_manager/library', '/prompt_manager/vocabulary/tags',
|
||||
'/prompt_manager/review/items']);
|
||||
['/prompt_manager/images/output', '/prompt_manager/images/output', '/prompt_manager/library', '/prompt_manager/vocabulary/tags']);
|
||||
for (const url of urls) assert.equal(url.searchParams.get('limit'), '1');
|
||||
assert.equal(urls[0].searchParams.get('mode'), 'workspace');
|
||||
assert.equal(urls[0].searchParams.has('favorites'), false);
|
||||
@@ -292,22 +288,3 @@ test('saved-view links resolve to their id and re-render only when the id change
|
||||
assert.deepEqual(seen,[['saved',3],['saved',4],['images',undefined]]);
|
||||
dispose();
|
||||
});
|
||||
|
||||
test('Needs review is a native SAVED VIEWS route titled Review, with its own count', async () => {
|
||||
const { resolveRoute, isHandoff, routeTitle, routes, loadNavCounts, REVIEW_WRITE_COUNTS, NAV_COUNT_ROUTES } = await source;
|
||||
const route = resolveRoute('#/review');
|
||||
assert.equal(route.id, 'review');
|
||||
assert.equal(route.label, 'Needs review');
|
||||
assert.equal(routeTitle(route), 'Review');
|
||||
assert.equal(isHandoff(route), false);
|
||||
// No href: Overview lists no legacy card for it.
|
||||
assert.equal(routes.find(item => item.id === 'review').href, undefined);
|
||||
assert.ok(NAV_COUNT_ROUTES.includes('review'));
|
||||
assert.deepEqual([...REVIEW_WRITE_COUNTS], ['review']);
|
||||
assert.ok(Object.isFrozen(REVIEW_WRITE_COUNTS));
|
||||
const calls = [];
|
||||
const api = {get: async url => { calls.push(url); return {success: true, total: 4}; }};
|
||||
const counts = await loadNavCounts(api, new AbortController().signal, () => {}, REVIEW_WRITE_COUNTS);
|
||||
assert.deepEqual(calls, ['/prompt_manager/review/items?limit=1']);
|
||||
assert.deepEqual([...counts], [['review', 4]]);
|
||||
});
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
// System › thumbnail review (#249 System S5 Task 14): the concept's "Thumbnail maintenance" dialog (app.js 1433–1437,
|
||||
// `thumbnail-review`) on the thumbnail job (S5 Task 12). Pure parts: the option values the job takes, the preview size
|
||||
// check, the Image loading quality preselection (Q3), the D99 plan line and the start outcome.
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
|
||||
const reviewModule = import('../../web/js/workspace/thumbnail-review.mjs');
|
||||
|
||||
test('the concept options map to the job values in the concept order', async () => {
|
||||
const { SCOPE_OPTIONS, QUALITY_OPTIONS, SIZE_LIMIT } = await reviewModule;
|
||||
assert.deepEqual(SCOPE_OPTIONS, [['missing', 'Missing thumbnails only'], ['rebuild', 'Rebuild all (explicit opt-in)']]);
|
||||
assert.deepEqual(QUALITY_OPTIONS, [['medium', 'Medium · optimized'], ['high', 'High · original'], ['low', 'Low · fastest']]);
|
||||
assert.deepEqual(SIZE_LIMIT, { min: 128, max: 2048, value: 512 });
|
||||
});
|
||||
|
||||
test('size outside 128–2048 reports validity: only whole pixels in range are a size', async () => {
|
||||
const { previewSize } = await reviewModule;
|
||||
assert.equal(previewSize('512'), 512);
|
||||
assert.equal(previewSize('128'), 128);
|
||||
assert.equal(previewSize('2048'), 2048);
|
||||
for (const text of ['', ' ', '127', '2049', '4096', '512.5', '-512', 'abc', '1e3', '0x200']) {
|
||||
assert.equal(previewSize(text), null, text);
|
||||
}
|
||||
});
|
||||
|
||||
test('Image loading quality preselects the dialog quality (Q3); anything else is Medium', async () => {
|
||||
const { preselectedQuality } = await reviewModule;
|
||||
assert.equal(preselectedQuality('low'), 'low');
|
||||
assert.equal(preselectedQuality('high'), 'high');
|
||||
assert.equal(preselectedQuality('medium'), 'medium');
|
||||
assert.equal(preselectedQuality(undefined), 'medium');
|
||||
assert.equal(preselectedQuality('original'), 'medium');
|
||||
});
|
||||
|
||||
test('the plan line counts what a missing-only run generates and what is already up to date (D99)', async () => {
|
||||
const { planText } = await reviewModule;
|
||||
assert.equal(planText({ mode: 'missing', targets: 12, fresh: 30, remaining: 0 }), '12 previews to generate · 30 already up to date.');
|
||||
assert.equal(planText({ mode: 'missing', targets: 1, fresh: 1, remaining: 0 }), '1 preview to generate · 1 already up to date.');
|
||||
assert.equal(planText({ mode: 'missing', targets: 0, fresh: 0, remaining: 0 }), '0 previews to generate · 0 already up to date.');
|
||||
assert.equal(planText({ mode: 'missing', targets: 10000, fresh: 5, remaining: 2500 }),
|
||||
'10,000 previews to generate · 5 already up to date · 2,500 more on the next run.');
|
||||
});
|
||||
|
||||
test('Rebuild all says every preview is regenerated (D99)', async () => {
|
||||
const { planText } = await reviewModule;
|
||||
assert.equal(planText({ mode: 'rebuild', targets: 42, fresh: 30, remaining: 0 }), 'Every preview is regenerated: 42 files.');
|
||||
assert.equal(planText({ mode: 'rebuild', targets: 1, fresh: 1, remaining: 0 }), 'Every preview is regenerated: 1 file.');
|
||||
assert.equal(planText({ mode: 'rebuild', targets: 10000, fresh: 9000, remaining: 3 }),
|
||||
'Every preview is regenerated: 10,000 files · 3 more on the next run.');
|
||||
});
|
||||
|
||||
test('an answer without counts is a failed plan, never zeros', async () => {
|
||||
const { planText, PLAN_FAILED } = await reviewModule;
|
||||
assert.equal(planText(null), PLAN_FAILED);
|
||||
assert.equal(planText({ mode: 'missing', targets: 2 }), PLAN_FAILED);
|
||||
assert.equal(planText({ mode: 'missing', targets: -1, fresh: 0, remaining: 0 }), PLAN_FAILED);
|
||||
assert.equal(planText({ mode: 'other', targets: 1, fresh: 0, remaining: 0 }), PLAN_FAILED);
|
||||
});
|
||||
|
||||
const scopeOf = () => ({ signal: new AbortController().signal });
|
||||
|
||||
test('Start starts the thumbnail job with the chosen options and opens it in Activity', async () => {
|
||||
const { startThumbnails } = await reviewModule;
|
||||
const calls = [];
|
||||
const navigated = [];
|
||||
const store = { start: async (kind, body) => { calls.push([kind, body]); return { job: { id: 'a'.repeat(32) } }; } };
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'rebuild', size: 1024, quality: 'high' },
|
||||
navigate: hash => navigated.push(hash), notify: () => assert.fail('no toast of its own') });
|
||||
assert.deepEqual(calls, [['thumbnails', { mode: 'rebuild', size: 1024, quality: 'high' }]]);
|
||||
assert.deepEqual(navigated, [`#/activity/${'a'.repeat(32)}`]);
|
||||
});
|
||||
|
||||
test('a refused start shows the busy link toast and stays', async () => {
|
||||
const { startThumbnails } = await reviewModule;
|
||||
const toasts = [];
|
||||
const store = { start: async () => ({ busy: { id: 'b'.repeat(32), title: 'Scan for metadata' } }) };
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' },
|
||||
navigate: () => assert.fail('stays'), notify: message => toasts.push(message) });
|
||||
assert.deepEqual(toasts, [['Another job is running: ', { text: 'Scan for metadata', href: `#/activity/${'b'.repeat(32)}` }, '.']]);
|
||||
});
|
||||
|
||||
test('a failed start says so; a start after the section left says nothing', async () => {
|
||||
const { startThumbnails, START_FAILED } = await reviewModule;
|
||||
const toasts = [];
|
||||
const store = { start: async () => { throw new Error('offline'); } };
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' },
|
||||
navigate: () => assert.fail('stays'), notify: message => toasts.push(message) });
|
||||
assert.deepEqual(toasts, [START_FAILED]);
|
||||
const gone = new AbortController();
|
||||
gone.abort();
|
||||
await startThumbnails({ store, scope: { signal: gone.signal }, options: { mode: 'missing', size: 512, quality: 'medium' },
|
||||
navigate: () => assert.fail('stays'), notify: () => assert.fail('silent') });
|
||||
});
|
||||
@@ -30,11 +30,6 @@ FILES = "media_annotation_files"
|
||||
BYTES = "media_annotation_bytes"
|
||||
IMAGE_TAGS = "media_image_tags"
|
||||
TABLES = (METADATA, FILES, BYTES, IMAGE_TAGS)
|
||||
REVIEW_TABLES = (
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_review_metadata",
|
||||
)
|
||||
# The exact DDL the plan pins; the module must produce precisely these objects.
|
||||
DDL = {
|
||||
METADATA: (
|
||||
@@ -207,14 +202,10 @@ class AnnotationSchemaTests(unittest.TestCase):
|
||||
)
|
||||
|
||||
def drop_group(self, path):
|
||||
"""Remove the group from a closed database file (an old backup).
|
||||
|
||||
The review queue group cites annotation bytes and predates nothing, so
|
||||
an old backup lacks it as well.
|
||||
"""
|
||||
"""Remove the group from a closed database file (an old backup)."""
|
||||
with closing(sqlite3.connect(path)) as raw, raw:
|
||||
raw.execute("PRAGMA foreign_keys=OFF")
|
||||
for table in REVIEW_TABLES + tuple(reversed(TABLES)):
|
||||
for table in reversed(TABLES):
|
||||
raw.execute(f"DROP TABLE {table}")
|
||||
|
||||
def test_fresh_install_and_reopen_is_idempotent(self):
|
||||
|
||||
+3
-10
@@ -107,9 +107,6 @@ class AdminAPITestCase(AioHTTPTestCase):
|
||||
for task in tasks:
|
||||
task.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
if api is not None:
|
||||
# A scan of an ambiguous file captures it for Needs review.
|
||||
api.close_annotation_captures()
|
||||
PromptModel.close_all_instances() # restore may have swapped api.db
|
||||
assert_closed(self, self.tmpdir)
|
||||
shutil.rmtree(self.tmpdir, ignore_errors=True)
|
||||
@@ -2242,9 +2239,7 @@ class TestActivityJobs(AdminAPITestCase):
|
||||
for name in names:
|
||||
(self.output_dir / name).write_bytes(b"not an image")
|
||||
|
||||
def fake_ingest(
|
||||
media_file, metadata, detailed=False, ambiguous=None, untraced=None
|
||||
):
|
||||
def fake_ingest(media_file, metadata, detailed=False):
|
||||
key = Path(media_file).relative_to(self.output_dir).as_posix()
|
||||
outcome = names[key]
|
||||
if isinstance(outcome, Exception):
|
||||
@@ -2373,10 +2368,8 @@ class TestActivityJobs(AdminAPITestCase):
|
||||
first_done, release = threading.Event(), threading.Event()
|
||||
original = self.api._ingest_scan_batch_sync
|
||||
|
||||
def gated(batch_results, roots=(), outcomes=None, review=None):
|
||||
counts = original(
|
||||
batch_results, roots=roots, outcomes=outcomes, review=review
|
||||
)
|
||||
def gated(batch_results, roots=(), outcomes=None):
|
||||
counts = original(batch_results, roots=roots, outcomes=outcomes)
|
||||
first_done.set()
|
||||
release.wait(5)
|
||||
return counts
|
||||
|
||||
@@ -1045,15 +1045,7 @@ class TestFailures(AnnotationAPITestCase):
|
||||
await self.api.cleanup_annotation_captures(None)
|
||||
self.assertFalse(os.path.exists(directory))
|
||||
self.assertNotIn(identity, observers._restore_observers)
|
||||
# The shutdown cleanup is final: a later observation (or a monitor
|
||||
# recording still in flight) creates no new store or directory.
|
||||
status, body = await self.observe({"target": self.target("sub/a.png")})
|
||||
self.assertEqual((status, body["code"]), (500, "capture-error"), body)
|
||||
self.assertIsNone(self.api._captures)
|
||||
self.assertNotIn(identity, observers._restore_observers)
|
||||
# An explicit, non-final close still lets a later observation start
|
||||
# a fresh store (the tests' own store swaps rely on it).
|
||||
self.api._captures_closed = False
|
||||
# A later observation starts a fresh store.
|
||||
await self.observe_ok()
|
||||
self.assertNotEqual(self.api._captures.directory, directory)
|
||||
|
||||
|
||||
@@ -471,19 +471,6 @@ class TestUiRoutes(CoreRoutesTestCase):
|
||||
self.assertIn(("POST", "/prompt_manager/jobs/scan"), table)
|
||||
self.assertIn(("POST", "/prompt_manager/jobs/image-tags"), table)
|
||||
|
||||
async def test_review_routes_are_registered(self):
|
||||
table = {
|
||||
(route.method, route.resource.canonical)
|
||||
for route in self.app.router.routes()
|
||||
if route.resource is not None
|
||||
}
|
||||
for entry in (
|
||||
("GET", "/prompt_manager/review/items"),
|
||||
("GET", "/prompt_manager/review/items/{item_id}"),
|
||||
("POST", "/prompt_manager/review/items/{item_id}/decision"),
|
||||
):
|
||||
self.assertIn(entry, table)
|
||||
|
||||
|
||||
class TestStaticRoutes(CoreRoutesTestCase):
|
||||
|
||||
|
||||
@@ -413,11 +413,11 @@ class TestThumbnailProgressStream(ThumbnailTestCase):
|
||||
lock = threading.Lock()
|
||||
original = images_module._write_image_thumbnail
|
||||
|
||||
def recording(src, dst, size, *args):
|
||||
def recording(src, dst, size):
|
||||
with lock:
|
||||
seen.add(threading.current_thread().name)
|
||||
time.sleep(0.05)
|
||||
return original(src, dst, size, *args)
|
||||
return original(src, dst, size)
|
||||
|
||||
images_module._write_image_thumbnail = recording
|
||||
self.addCleanup(setattr, images_module, "_write_image_thumbnail", original)
|
||||
@@ -1939,11 +1939,7 @@ class TestGalleryRescan(ImageAPITestCase):
|
||||
|
||||
|
||||
class TestOutputScanHelpers(ImageAPITestCase):
|
||||
"""Stable ids for filesystem entries and no symlink reads.
|
||||
|
||||
The thumbnail stream no longer stops when its client goes away: see
|
||||
``tests/test_thumbnail_jobs.py`` (the work is a job that outlives it).
|
||||
"""
|
||||
"""Stable ids for filesystem entries, no symlink reads, stop on disconnect."""
|
||||
|
||||
def test_output_entry_id_is_a_stable_digest_of_the_path(self):
|
||||
import hashlib
|
||||
@@ -1968,6 +1964,28 @@ class TestOutputScanHelpers(ImageAPITestCase):
|
||||
names = sorted(p.name for p in images_module._iter_media_files(output, 100))
|
||||
self.assertEqual(names, ["real.png"])
|
||||
|
||||
async def test_thumbnail_stream_stops_when_the_client_disconnects(self):
|
||||
generated = []
|
||||
|
||||
def fake_generate(src, dst, size, is_video):
|
||||
generated.append(src)
|
||||
return {
|
||||
"action": "generated",
|
||||
"file": src.name,
|
||||
"dir": src.parent.name,
|
||||
"type": "image",
|
||||
}
|
||||
|
||||
self.api._generate_one = fake_generate
|
||||
|
||||
class GoneResponse:
|
||||
async def write(self, data):
|
||||
raise ConnectionResetError("client went away")
|
||||
|
||||
targets = [(Path(f"/x/{i}.png"), Path(f"/x/t{i}.jpg"), False) for i in range(3)]
|
||||
await self.api._stream_thumbnails(GoneResponse(), targets, (256, 256))
|
||||
self.assertEqual(generated, [])
|
||||
|
||||
|
||||
class TestThumbnailStartupStatus(ThumbnailTestCase):
|
||||
"""Startup counts and generation cover the same configured collection."""
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,364 +0,0 @@
|
||||
"""Needs review change notices (#249 PR F2).
|
||||
|
||||
``GET /prompt_manager/review/events`` is a server-sent event stream: one
|
||||
``ready`` event when it opens, then one ``change`` event after every
|
||||
committed queue change (an item recorded by the scan or the monitor, a
|
||||
spelling refresh, a decision from any tab) and after a restore. An open
|
||||
Needs review page re-reads its queue on ``change``; nothing polls. A refused
|
||||
decision and a duplicate recording change nothing and send nothing.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sqlite3
|
||||
import threading
|
||||
|
||||
try:
|
||||
from tests.test_api_review import ReviewRoutesTestCase
|
||||
from tests.test_review import unlinkable_graph, write_png
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_api_review import ReviewRoutesTestCase
|
||||
from test_review import unlinkable_graph, write_png
|
||||
|
||||
try: # guarded so a missing module fails assertions, not test loading
|
||||
from database import review_changes
|
||||
except ImportError:
|
||||
review_changes = None
|
||||
try:
|
||||
from py.api import review_events
|
||||
except ImportError:
|
||||
review_events = None
|
||||
|
||||
EVENTS_URL = "/prompt_manager/review/events"
|
||||
# A safety bound for one read; every wait below ends on an event first.
|
||||
READ_SECONDS = 10
|
||||
|
||||
|
||||
class ReviewEventsTestCase(ReviewRoutesTestCase):
|
||||
def _require_events(self):
|
||||
self.assertIsNotNone(review_changes, "database.review_changes is missing")
|
||||
self.assertIsNotNone(review_events, "py.api.review_events is missing")
|
||||
|
||||
async def _stream(self):
|
||||
resp = await self.client.get(EVENTS_URL)
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(resp.headers["Content-Type"], "text/event-stream")
|
||||
self.assertEqual(resp.headers["Cache-Control"], "no-store")
|
||||
self.addAsyncCleanup(self._close, resp)
|
||||
self.assertEqual(await self._event(resp), "ready")
|
||||
return resp
|
||||
|
||||
@staticmethod
|
||||
async def _close(resp):
|
||||
resp.close()
|
||||
|
||||
async def _event(self, resp):
|
||||
"""The next event's name (keepalive comments are skipped)."""
|
||||
return (await self._event_with_data(resp))[0]
|
||||
|
||||
async def _event_with_data(self, resp):
|
||||
"""The next event's name and its parsed ``data`` line."""
|
||||
name, data = None, None
|
||||
while True:
|
||||
line = await asyncio.wait_for(resp.content.readline(), READ_SECONDS)
|
||||
self.assertTrue(line, "the stream ended")
|
||||
text = line.decode("utf-8").rstrip("\n")
|
||||
if text.startswith("event: "):
|
||||
name = text[len("event: ") :]
|
||||
elif text.startswith("data: "):
|
||||
data = json.loads(text[len("data: ") :])
|
||||
elif text == "" and name is not None:
|
||||
return name, data
|
||||
|
||||
async def _ready_version(self):
|
||||
resp = await self.client.get(EVENTS_URL)
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.addAsyncCleanup(self._close, resp)
|
||||
name, data = await self._event_with_data(resp)
|
||||
self.assertEqual(name, "ready")
|
||||
return data
|
||||
|
||||
def _queueable(self, name="two.png", color=(255, 0, 0)):
|
||||
"""A file the producers queue: ambiguous, and nothing links it."""
|
||||
path = self.output_dir / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
write_png(path, prompt=unlinkable_graph("cat", "dog"), color=color)
|
||||
return path
|
||||
|
||||
async def _record_in_thread(self, path):
|
||||
"""Record as the monitor does: from a worker thread, not the loop."""
|
||||
self._roots()
|
||||
outcomes = []
|
||||
thread = threading.Thread(
|
||||
target=lambda: outcomes.append(self.api.record_review_file(str(path)))
|
||||
)
|
||||
thread.start()
|
||||
await asyncio.get_running_loop().run_in_executor(None, thread.join)
|
||||
return outcomes[0]
|
||||
|
||||
|
||||
class TestReviewEvents(ReviewEventsTestCase):
|
||||
async def test_monitor_recording_sends_one_change(self):
|
||||
self._require_events()
|
||||
stream = await self._stream()
|
||||
self.assertEqual(await self._record_in_thread(self._queueable()), "recorded")
|
||||
self.assertEqual(await self._event(stream), "change")
|
||||
|
||||
async def test_duplicate_recording_and_refused_decision_send_nothing(self):
|
||||
# A spy on the notice itself: a wrong notice would otherwise pass as
|
||||
# the next real change's event.
|
||||
self._require_events()
|
||||
from unittest.mock import patch
|
||||
|
||||
from database import review as review_db
|
||||
from database import review_decisions
|
||||
|
||||
calls = []
|
||||
original = review_changes.notify_review_change
|
||||
|
||||
def spy(db_path):
|
||||
calls.append(db_path)
|
||||
original(db_path)
|
||||
|
||||
path = self._queueable("two.png")
|
||||
self._record(path)
|
||||
with (
|
||||
patch.object(review_db, "notify_review_change", spy),
|
||||
patch.object(review_decisions, "notify_review_change", spy),
|
||||
):
|
||||
# The same bytes again: nothing changes.
|
||||
self.assertEqual(await self._record_in_thread(path), "duplicate")
|
||||
self.assertEqual(calls, [], "a duplicate recording sent a notice")
|
||||
item, detail, observed = await self._open()
|
||||
# A stale revision: refused with zero writes.
|
||||
body = {
|
||||
**self._body(detail, observed, "unresolved"),
|
||||
"expected_revision": 9,
|
||||
}
|
||||
status, data = await self._decide(item["id"], body)
|
||||
self.assertEqual(status, 409, data)
|
||||
self.assertEqual(calls, [], "a refused decision sent a notice")
|
||||
self._record(self._ambiguous("other.png", color=(0, 0, 9)))
|
||||
self.assertEqual(len(calls), 1, "a real recording sends one notice")
|
||||
self.assertEqual(len(self._items()), 2)
|
||||
|
||||
async def test_a_burst_of_changes_coalesces_into_one_more_event(self):
|
||||
# M9: a busy scan records file after file; an open page hears the
|
||||
# first at once and the rest as one trailing ``change``.
|
||||
self._require_events()
|
||||
from unittest.mock import patch
|
||||
|
||||
path = self.api.db.model.db_path
|
||||
with (
|
||||
patch.object(review_events, "COALESCE_SECONDS", 0.3),
|
||||
patch.object(review_events, "HEARTBEAT_SECONDS", 0.6),
|
||||
):
|
||||
stream = await self._stream()
|
||||
review_changes.notify_review_change(path)
|
||||
self.assertEqual(await self._event(stream), "change")
|
||||
for _ in range(5):
|
||||
review_changes.notify_review_change(path)
|
||||
await asyncio.sleep(0.02)
|
||||
# Every change event up to the next keepalive: exactly one.
|
||||
changes = 0
|
||||
while True:
|
||||
line = await asyncio.wait_for(stream.content.readline(), READ_SECONDS)
|
||||
self.assertTrue(line, "the stream ended")
|
||||
if line == b": keepalive\n":
|
||||
break
|
||||
changes += line == b"event: change\n"
|
||||
self.assertEqual(changes, 1)
|
||||
|
||||
async def test_decision_from_any_tab_sends_a_change(self):
|
||||
self._require_events()
|
||||
self._record(self._ambiguous())
|
||||
stream = await self._stream()
|
||||
item, detail, observed = await self._open()
|
||||
status, data = await self._decide(
|
||||
item["id"], self._body(detail, observed, "unresolved")
|
||||
)
|
||||
self.assertEqual(status, 200, data)
|
||||
self.assertEqual(await self._event(stream), "change")
|
||||
|
||||
async def test_restore_sends_a_change(self):
|
||||
self._require_events()
|
||||
stream = await self._stream()
|
||||
from database.execution_history import notify_history_restore
|
||||
|
||||
notify_history_restore(self.api.db.model.db_path)
|
||||
self.assertEqual(await self._event(stream), "change")
|
||||
|
||||
async def test_scan_recording_sends_a_change(self):
|
||||
self._require_events()
|
||||
self._roots()
|
||||
self._queueable("sub/two.png")
|
||||
stream = await self._stream()
|
||||
job = await self._scan("all")
|
||||
self.assertEqual(job["scope"]["review_items"], 1)
|
||||
self.assertEqual(await self._event(stream), "change")
|
||||
|
||||
async def test_a_linked_file_leaving_the_queue_sends_a_change(self):
|
||||
"""The queue heals (#284 maintainer test): a pending item whose file is
|
||||
linked now leaves the queue, and the open page hears about it."""
|
||||
self._require_events()
|
||||
self._roots()
|
||||
path = self._ambiguous("two.png")
|
||||
self._record(path) # as the pre-fix scan queued fallback-linked files
|
||||
stream = await self._stream()
|
||||
job = await self._scan("new")
|
||||
self.assertEqual(job["scope"]["review_settled"], 1)
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertEqual(await self._event(stream), "change")
|
||||
|
||||
async def test_ready_carries_the_version_a_list_read_answers(self):
|
||||
# F3 (no double read per visit): the page skips the re-read on
|
||||
# ``ready`` when its first read already answered this version.
|
||||
self._require_events()
|
||||
self._record(self._ambiguous())
|
||||
page = await self._list()
|
||||
self.assertIsInstance(page.get("review_revision"), int, page)
|
||||
version = await self._ready_version()
|
||||
self.assertEqual(
|
||||
version,
|
||||
{
|
||||
"library_epoch": page["library_epoch"],
|
||||
"review_revision": page["review_revision"],
|
||||
},
|
||||
)
|
||||
|
||||
async def test_a_queue_change_moves_the_version_ready_carries(self):
|
||||
self._require_events()
|
||||
before = await self._list()
|
||||
self._record(self._ambiguous())
|
||||
after = await self._list()
|
||||
self.assertIn("review_revision", after)
|
||||
self.assertGreater(after["review_revision"], before["review_revision"])
|
||||
version = await self._ready_version()
|
||||
self.assertEqual(version["review_revision"], after["review_revision"])
|
||||
self.assertNotEqual(version["review_revision"], before["review_revision"])
|
||||
|
||||
async def test_ready_names_no_version_when_it_cannot_be_read(self):
|
||||
# The page then re-reads, as it did before versions existed.
|
||||
self._require_events()
|
||||
|
||||
def broken():
|
||||
raise sqlite3.DatabaseError("locked")
|
||||
|
||||
self.api.db.review_queue_version = broken
|
||||
self.assertEqual(await self._ready_version(), {})
|
||||
|
||||
async def test_a_stream_counts_as_open_once_it_has_sent_ready(self):
|
||||
# The browser fixture waits on this count before it changes the
|
||||
# queue, so a change after it is a ``change`` event and never part of
|
||||
# the version ``ready`` named.
|
||||
self._require_events()
|
||||
reading, release = asyncio.Event(), asyncio.Event()
|
||||
original = self.api._queue_version
|
||||
|
||||
async def held():
|
||||
reading.set()
|
||||
await release.wait()
|
||||
return await original()
|
||||
|
||||
self.api._queue_version = held
|
||||
opening = asyncio.ensure_future(self._ready_version())
|
||||
await asyncio.wait_for(reading.wait(), READ_SECONDS)
|
||||
self.assertEqual(self.api.review_stream_count(), 0)
|
||||
release.set()
|
||||
await asyncio.wait_for(opening, READ_SECONDS)
|
||||
self.assertEqual(self.api.review_stream_count(), 1)
|
||||
|
||||
async def test_change_events_carry_no_data(self):
|
||||
self._require_events()
|
||||
stream = await self._stream()
|
||||
self._record(self._ambiguous())
|
||||
self.assertEqual(await self._event_with_data(stream), ("change", {}))
|
||||
|
||||
async def test_closing_the_stream_unsubscribes(self):
|
||||
self._require_events()
|
||||
path = self.api.db.model.db_path
|
||||
stream = await self._stream()
|
||||
self.assertEqual(review_changes.observer_count(path), 1)
|
||||
self.assertEqual(self.api.review_stream_count(), 1)
|
||||
stream.close()
|
||||
# The server notices the closed socket at its next write: a change.
|
||||
self._record(self._ambiguous())
|
||||
for _ in range(100):
|
||||
if not self.api.review_stream_count():
|
||||
break
|
||||
await asyncio.sleep(0.02)
|
||||
self.assertEqual(self.api.review_stream_count(), 0)
|
||||
self.assertEqual(review_changes.observer_count(path), 0)
|
||||
|
||||
async def test_shutdown_ends_every_stream(self):
|
||||
self._require_events()
|
||||
stream = await self._stream()
|
||||
await self.api.close_review_events(None)
|
||||
line = await asyncio.wait_for(stream.content.read(), READ_SECONDS)
|
||||
self.assertEqual(line, b"")
|
||||
self.assertEqual(self.api.review_stream_count(), 0)
|
||||
# A stream opened after shutdown ends at once.
|
||||
resp = await self.client.get(EVENTS_URL)
|
||||
self.assertEqual(resp.status, 503)
|
||||
resp.close()
|
||||
|
||||
async def test_keepalive_comment_is_skipped_by_clients(self):
|
||||
self._require_events()
|
||||
with self.patch_heartbeat(0.05):
|
||||
stream = await self._stream()
|
||||
line = await asyncio.wait_for(stream.content.readline(), READ_SECONDS)
|
||||
self.assertEqual(line, b": keepalive\n")
|
||||
|
||||
def patch_heartbeat(self, seconds):
|
||||
from unittest.mock import patch
|
||||
|
||||
return patch.object(review_events, "HEARTBEAT_SECONDS", seconds)
|
||||
|
||||
|
||||
class TestReviewChangeRegistry(ReviewEventsTestCase):
|
||||
def test_a_failing_observer_never_fails_a_write(self):
|
||||
self._require_events()
|
||||
path = self.api.db.model.db_path
|
||||
seen = []
|
||||
|
||||
def broken():
|
||||
raise RuntimeError("observer failed")
|
||||
|
||||
unsubscribe_broken = review_changes.subscribe_review_changes(path, broken)
|
||||
unsubscribe = review_changes.subscribe_review_changes(
|
||||
path, lambda: seen.append(1)
|
||||
)
|
||||
try:
|
||||
with self.assertLogs(
|
||||
"prompt_manager.database.review_changes", level="WARNING"
|
||||
):
|
||||
self._record(self._ambiguous())
|
||||
self.assertEqual(seen, [1])
|
||||
finally:
|
||||
unsubscribe_broken()
|
||||
unsubscribe()
|
||||
self.assertEqual(review_changes.observer_count(path), 0)
|
||||
|
||||
def test_notice_comes_after_the_commit(self):
|
||||
self._require_events()
|
||||
path = self.api.db.model.db_path
|
||||
seen = []
|
||||
|
||||
def committed():
|
||||
# A separate connection sees only committed rows.
|
||||
other = sqlite3.connect(path)
|
||||
try:
|
||||
seen.append(
|
||||
other.execute("SELECT COUNT(*) FROM media_review_items").fetchone()[
|
||||
0
|
||||
]
|
||||
)
|
||||
finally:
|
||||
other.close()
|
||||
|
||||
unsubscribe = review_changes.subscribe_review_changes(path, committed)
|
||||
try:
|
||||
self._record(self._ambiguous())
|
||||
finally:
|
||||
unsubscribe()
|
||||
self.assertEqual(seen, [1])
|
||||
@@ -466,9 +466,6 @@ class TestRestoreFromFile(BackupTestCase):
|
||||
|
||||
def test_restore_old_backup_without_annotations(self):
|
||||
tables = (
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_review_metadata",
|
||||
"media_image_tags",
|
||||
"media_annotation_bytes",
|
||||
"media_annotation_files",
|
||||
@@ -505,70 +502,6 @@ class TestRestoreFromFile(BackupTestCase):
|
||||
0,
|
||||
)
|
||||
|
||||
def test_restore_annotations_era_backup_without_review_queue(self):
|
||||
tables = (
|
||||
"media_review_decisions",
|
||||
"media_review_items",
|
||||
"media_review_metadata",
|
||||
)
|
||||
src = self._make_source_db(["predates the review queue"])
|
||||
with closing(sqlite3.connect(src)) as conn:
|
||||
for table in tables:
|
||||
conn.execute(f"DROP TABLE {table}")
|
||||
conn.commit()
|
||||
self.assertTrue(PromptModel.verify_database_file(src)[0])
|
||||
ok, reason = PromptModel.verify_database_file(src, require_library_state=True)
|
||||
self.assertFalse(ok)
|
||||
self.assertEqual(reason, "Missing review queue schema")
|
||||
|
||||
self.db.model.restore_from_file(src)
|
||||
|
||||
self.assertEqual(_texts(self.path), ["predates the review queue"])
|
||||
with closing(sqlite3.connect(self.path)) as conn:
|
||||
self.assertEqual(
|
||||
conn.execute("SELECT * FROM media_review_metadata").fetchall(),
|
||||
[(1, 1, 0)],
|
||||
)
|
||||
self.assertEqual(
|
||||
conn.execute("SELECT COUNT(*) FROM media_review_items").fetchone()[0],
|
||||
0,
|
||||
)
|
||||
ok, reason = PromptModel.verify_database_file(
|
||||
self.path, require_library_state=True
|
||||
)
|
||||
self.assertTrue(ok, reason)
|
||||
with closing(sqlite3.connect(src)) as conn: # the source is untouched
|
||||
self.assertEqual(
|
||||
conn.execute(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE name LIKE 'media_review%'"
|
||||
).fetchone()[0],
|
||||
0,
|
||||
)
|
||||
|
||||
def test_restore_rejects_review_queue_without_annotations(self):
|
||||
src = self._make_source_db(["review without annotations"])
|
||||
with closing(sqlite3.connect(src)) as conn:
|
||||
conn.execute("PRAGMA foreign_keys=OFF")
|
||||
for table in (
|
||||
"media_image_tags",
|
||||
"media_annotation_bytes",
|
||||
"media_annotation_files",
|
||||
"media_annotation_metadata",
|
||||
):
|
||||
conn.execute(f"DROP TABLE {table}")
|
||||
conn.commit()
|
||||
before = Path(src).read_bytes()
|
||||
live_before = _texts(self.path)
|
||||
ok, reason = PromptModel.verify_database_file(src)
|
||||
self.assertFalse(ok)
|
||||
self.assertIn("image annotation", reason)
|
||||
|
||||
with self.assertRaises(ValueError):
|
||||
self.db.model.restore_from_file(src)
|
||||
|
||||
self.assertEqual(Path(src).read_bytes(), before)
|
||||
self.assertEqual(_texts(self.path), live_before)
|
||||
|
||||
def test_restore_preserves_favorites_and_closes_handles(self):
|
||||
self.db.set_image_favorite("output", "live.png")
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
|
||||
@@ -101,35 +101,6 @@ class TestJobFanOut(unittest.IsolatedAsyncioTestCase):
|
||||
|
||||
self.assertEqual(await _collect(job), [{"type": "complete"}])
|
||||
|
||||
async def test_lead_comes_first_and_nothing_published_meanwhile_is_lost(self):
|
||||
"""S5 I1: the ``job`` frame leads, with the subscription already in place."""
|
||||
job = Job("thumbnails", "Thumbnail generation")
|
||||
job.publish({"type": "status"})
|
||||
lead = {"type": "job"}
|
||||
seen = []
|
||||
async with contextlib.aclosing(job.events(lead=lead)) as events:
|
||||
async for payload in events:
|
||||
seen.append(payload)
|
||||
if payload is lead:
|
||||
# Published while the consumer is still sending the lead.
|
||||
job.publish({"type": "progress"})
|
||||
job.publish({"type": "complete"})
|
||||
job.finish()
|
||||
self.assertEqual(
|
||||
seen,
|
||||
[lead, {"type": "status"}, {"type": "progress"}, {"type": "complete"}],
|
||||
)
|
||||
|
||||
async def test_lead_on_a_finished_job_then_its_last_event(self):
|
||||
job = Job("thumbnails", "Thumbnail generation")
|
||||
job.publish({"type": "complete"})
|
||||
job.finish()
|
||||
seen = []
|
||||
async with contextlib.aclosing(job.events(lead={"type": "job"})) as events:
|
||||
async for payload in events:
|
||||
seen.append(payload)
|
||||
self.assertEqual(seen, [{"type": "job"}, {"type": "complete"}])
|
||||
|
||||
async def test_departed_subscriber_is_forgotten(self):
|
||||
job = Job("scan", "Scan")
|
||||
job.publish({"type": "progress", "progress": 1})
|
||||
@@ -180,30 +151,6 @@ class TestJobFanOut(unittest.IsolatedAsyncioTestCase):
|
||||
|
||||
class TestJobRegistry(unittest.IsolatedAsyncioTestCase):
|
||||
|
||||
async def test_a_compatible_kind_never_blocks(self):
|
||||
"""S5 M7: a caller may name running kinds that cannot conflict with it."""
|
||||
registry = JobRegistry()
|
||||
gate = asyncio.Event()
|
||||
thumbs = registry.start(
|
||||
"thumbnails",
|
||||
"Thumbnail generation",
|
||||
admission="guarded",
|
||||
runner=_gated_runner(gate),
|
||||
)
|
||||
self.assertIs(registry.blocker("probe", admission="records"), thumbs)
|
||||
self.assertIsNone(
|
||||
registry.blocker("probe", admission="records", compatible=("thumbnails",))
|
||||
)
|
||||
backup = registry.start(
|
||||
"backup", "Backup", admission="guarded", runner=_gated_runner(gate)
|
||||
)
|
||||
self.assertIs(
|
||||
registry.blocker("probe", admission="records", compatible=("thumbnails",)),
|
||||
backup,
|
||||
)
|
||||
gate.set()
|
||||
await asyncio.wait_for(asyncio.gather(thumbs.task, backup.task), 2)
|
||||
|
||||
async def test_start_lists_and_finishes_a_job(self):
|
||||
registry = JobRegistry()
|
||||
gate = asyncio.Event()
|
||||
@@ -474,28 +421,6 @@ class TestJobOutcomesAndLog(unittest.TestCase):
|
||||
SCAN_GROUP_NAMES, ("added", "known", "review", "no_metadata", "errors")
|
||||
)
|
||||
|
||||
def test_record_counted_counts_exactly_lists_capped_and_logs_nothing(self):
|
||||
job = Job("thumbnails", "Thumbnail generation", groups=("skipped",))
|
||||
rows = [
|
||||
{"path": os.path.abspath(f"out{os.sep}f{i}.png"), "root": "output"}
|
||||
for i in range(MAX_OUTCOME_ROWS + 5)
|
||||
]
|
||||
job.record_counted("skipped", 10_000, rows)
|
||||
job.record_counted("skipped", 1, [{"path": "late.png", "root_index": True}])
|
||||
self.assertEqual(job.counters["skipped"], 10_001)
|
||||
self.assertEqual(len(job.outcomes["skipped"]), MAX_OUTCOME_ROWS)
|
||||
self.assertEqual(job.outcomes["skipped"][0]["path"], "f0.png")
|
||||
self.assertIsNone(job.outcomes["skipped"][0]["root_index"])
|
||||
self.assertEqual(job.log, [])
|
||||
with self.assertRaises(ValueError):
|
||||
job.record_counted("bogus", 1)
|
||||
|
||||
def test_thumbnail_job_groups(self):
|
||||
self.assertEqual(
|
||||
jobs_module.THUMBNAIL_GROUP_NAMES, ("generated", "skipped", "errors")
|
||||
)
|
||||
self.assertEqual(jobs_module.THUMBNAILS_JOB_KIND, "thumbnails")
|
||||
|
||||
def test_image_tag_job_kind_and_groups(self):
|
||||
# #249 D1: bulk Add tags / Auto-tag, counter-tile order.
|
||||
self.assertEqual(getattr(jobs_module, "IMAGE_TAG_JOB_KIND", None), "image-tags")
|
||||
|
||||
@@ -1744,7 +1744,6 @@ class LibraryCounts(unittest.TestCase):
|
||||
"negative_variants": 0,
|
||||
"images": 0,
|
||||
"rated_images": 0,
|
||||
"review_pending": 0,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -471,94 +471,6 @@ class CaptureStoreTestCase(unittest.TestCase):
|
||||
self.assertEqual(results["capture"].size, 60)
|
||||
self.capture(workers, "b.png")
|
||||
|
||||
def _hold_worker(self, store, relative, **options):
|
||||
"""Start a capture that holds a worker until ``proceed`` is set."""
|
||||
entered, proceed = threading.Event(), threading.Event()
|
||||
original = CaptureStore._read
|
||||
results = {}
|
||||
|
||||
def blocking(store_, stream, size):
|
||||
if threading.current_thread() is thread and not entered.is_set():
|
||||
entered.set()
|
||||
self.assertTrue(proceed.wait(WAIT))
|
||||
return original(store_, stream, size)
|
||||
|
||||
def worker():
|
||||
try:
|
||||
results["capture"] = self.capture(store, relative, **options)
|
||||
except Exception as error: # pragma: no cover - reported below
|
||||
results["error"] = error
|
||||
|
||||
patcher = patch.object(CaptureStore, "_read", blocking)
|
||||
patcher.start()
|
||||
self.addCleanup(patcher.stop)
|
||||
thread = threading.Thread(target=worker)
|
||||
thread.start()
|
||||
self.assertTrue(entered.wait(WAIT))
|
||||
|
||||
def finish():
|
||||
proceed.set()
|
||||
thread.join(WAIT)
|
||||
self.assertFalse(thread.is_alive())
|
||||
self.assertNotIn("error", results)
|
||||
return results["capture"]
|
||||
|
||||
self.addCleanup(proceed.set)
|
||||
return finish
|
||||
|
||||
def test_background_capture_never_takes_the_last_worker(self):
|
||||
for name in ("a", "b", "c"):
|
||||
self.write(f"{name}.png", bytes(60))
|
||||
store = self.make_store(max_workers=2)
|
||||
finish = self._hold_worker(store, "a.png", background=True)
|
||||
# The second background capture waits, then fails: one worker is free
|
||||
# but it is the interactive request's.
|
||||
started = time.monotonic()
|
||||
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.05):
|
||||
with self.assertRaises(CaptureCapacity):
|
||||
self.capture(store, "b.png", background=True)
|
||||
self.assertGreaterEqual(time.monotonic() - started, 0.04)
|
||||
# An interactive capture (Tag dialog, Apply) still gets the last one.
|
||||
interactive = self.capture(store, "c.png")
|
||||
self.assertEqual(interactive.size, 60)
|
||||
finish()
|
||||
|
||||
def test_background_capture_waits_for_a_worker_to_free(self):
|
||||
for name in ("a", "b"):
|
||||
self.write(f"{name}.png", bytes(60))
|
||||
store = self.make_store(max_workers=2)
|
||||
finish = self._hold_worker(store, "a.png", background=True)
|
||||
waited = {}
|
||||
|
||||
def background():
|
||||
waited["capture"] = self.capture(store, "b.png", background=True)
|
||||
|
||||
thread = threading.Thread(target=background)
|
||||
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", WAIT):
|
||||
thread.start()
|
||||
finish()
|
||||
thread.join(WAIT)
|
||||
self.assertFalse(thread.is_alive())
|
||||
self.assertEqual(waited["capture"].size, 60)
|
||||
|
||||
def test_single_worker_store_admits_no_background_capture(self):
|
||||
self.write("a.png", bytes(60))
|
||||
store = self.make_store(max_workers=1)
|
||||
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
|
||||
with self.assertRaises(CaptureCapacity):
|
||||
self.capture(store, "a.png", background=True)
|
||||
self.assertEqual(self.capture(store, "a.png").size, 60)
|
||||
|
||||
def test_background_capture_leaves_the_last_capture_slot(self):
|
||||
for name in ("a", "b", "c"):
|
||||
self.write(f"{name}.png", bytes(60))
|
||||
store = self.make_store(max_captures=2)
|
||||
self.capture(store, "a.png")
|
||||
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
|
||||
with self.assertRaises(CaptureCapacity):
|
||||
self.capture(store, "b.png", background=True)
|
||||
self.assertEqual(self.capture(store, "c.png").size, 60)
|
||||
|
||||
# -- lifetime ------------------------------------------------------------
|
||||
|
||||
def test_idle_and_hard_expiry_and_in_use_retention(self):
|
||||
|
||||
@@ -160,69 +160,3 @@ class TestPairIngestion(unittest.TestCase):
|
||||
info = self.db.get_image_prompt_info(self.path)
|
||||
self.assertEqual(info["pair_context"]["positive_text"], "cat, lora_trigger")
|
||||
self.assertEqual(info["pair_context"]["text_source"], "execution")
|
||||
|
||||
def test_public_ingest_resolved_pair_contract_unchanged(self):
|
||||
"""The public ingester keeps its result shape and its own transaction;
|
||||
the extracted ``_ingest_resolved_pair_in`` runs in the caller's."""
|
||||
from database import pair_ingestion
|
||||
|
||||
self.assertTrue(
|
||||
hasattr(pair_ingestion, "_ingest_resolved_pair_in"),
|
||||
"_ingest_resolved_pair_in is not extracted",
|
||||
)
|
||||
for state in ("ambiguous", "unresolved", "conflict"):
|
||||
self.assertEqual(
|
||||
ingest_resolved_pair(self.db, self.path, {}, {"state": state}),
|
||||
{"status": state, "pairs_added": 0, "links_added": 0},
|
||||
)
|
||||
first = ingest_resolved_pair(self.db, self.path, {}, resolved())
|
||||
self.assertEqual(
|
||||
set(first),
|
||||
{"status", "pairs_added", "links_added", "pair_id", "image_id"},
|
||||
)
|
||||
self.assertEqual(first["status"], "added")
|
||||
(link,) = self.db.get_image_prompt_pairs(first["image_id"])
|
||||
self.assertEqual(
|
||||
link["provenance"],
|
||||
{
|
||||
"resolver_version": 1,
|
||||
"positive": resolved()["positive"],
|
||||
"negative": resolved()["negative"],
|
||||
},
|
||||
)
|
||||
conn = self.db.model.get_connection()
|
||||
self.assertFalse(conn.in_transaction)
|
||||
# Inside the caller's transaction, with the caller's provenance; an
|
||||
# exception afterwards rolls the link back with the caller's writes.
|
||||
provenance = {"kind": "user-selection", "resolver_version": 1}
|
||||
with self.assertRaises(RuntimeError):
|
||||
with self.db._pair_transaction() as held:
|
||||
inner = pair_ingestion._ingest_resolved_pair_in(
|
||||
held,
|
||||
self.db,
|
||||
self.path + "2",
|
||||
{},
|
||||
resolved("dog"),
|
||||
provenance=provenance,
|
||||
)
|
||||
self.assertEqual(inner["status"], "added")
|
||||
raise RuntimeError("caller failed")
|
||||
self.assertIsNone(self.db.get_image_prompt_info(self.path + "2"))
|
||||
with self.db._pair_transaction() as held:
|
||||
inner = pair_ingestion._ingest_resolved_pair_in(
|
||||
held,
|
||||
self.db,
|
||||
self.path + "2",
|
||||
{},
|
||||
resolved("dog"),
|
||||
provenance=provenance,
|
||||
)
|
||||
(link,) = self.db.get_image_prompt_pairs(inner["image_id"])
|
||||
self.assertEqual(link["provenance"], provenance)
|
||||
with self.db._pair_transaction() as held:
|
||||
conflict = pair_ingestion._ingest_resolved_pair_in(
|
||||
held, self.db, self.path, {}, resolved("bird")
|
||||
)
|
||||
self.assertEqual(
|
||||
conflict, {"status": "conflict", "pairs_added": 0, "links_added": 0}
|
||||
)
|
||||
|
||||
@@ -82,8 +82,7 @@ class TestPairScan(AdminAPITestCase):
|
||||
"class_type": "BasicGuider",
|
||||
"inputs": {"width": 2048, "height": 2048},
|
||||
}
|
||||
# Ambiguous with nothing to choose: unresolved, never a review item.
|
||||
self.assertEqual(self._scanned("untraceable.png", g), "unresolved")
|
||||
self.assertEqual(self._scanned("untraceable.png", g), "ambiguous")
|
||||
self.assertEqual(self.api.db.list_prompt_pairs()["total"], 0)
|
||||
self.assertEqual(self._texts(), [])
|
||||
|
||||
@@ -160,10 +159,9 @@ class TestPairScan(AdminAPITestCase):
|
||||
"\nSteps: 20, Sampler: Euler"
|
||||
)
|
||||
}
|
||||
# No candidate to choose: reported unresolved, still not guessed.
|
||||
self.assertEqual(
|
||||
self.api._ingest_scanned_file(self.output_dir / "ambiguous.png", meta),
|
||||
"unresolved",
|
||||
"ambiguous",
|
||||
)
|
||||
self.assertEqual(self.api.db.get_recent_prompts()["prompts"], [])
|
||||
|
||||
|
||||
@@ -258,10 +258,6 @@ class PairSchemaTests(unittest.TestCase):
|
||||
],
|
||||
[(1, 1, 0)],
|
||||
)
|
||||
self.assertEqual(
|
||||
[tuple(r) for r in conn.execute("SELECT * FROM media_review_metadata")],
|
||||
[(1, 1, 0)],
|
||||
)
|
||||
self.assertTrue(PromptModel.verify_database_file(live.db_path)[0])
|
||||
self.assertTrue(
|
||||
PromptModel.verify_database_file(live.db_path, require_library_state=True)[
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,335 +0,0 @@
|
||||
"""Needs review queues a file only when there is something to choose.
|
||||
|
||||
The maintainer's live queue (#284 / #288) held 39 items whose 11 candidates
|
||||
all read "unresolved", "No positive / No negative": the ingestion resolver
|
||||
found several sampler branches but could trace no prompt on any of them.
|
||||
An ambiguous result is reviewable only when at least two candidates carry a
|
||||
resolved prompt text. With none, the file could not be traced; with exactly
|
||||
one, the other branches may still have made the file, so the one text is
|
||||
not certain either (the resolver never flattens unknown semantics). Both
|
||||
stay unresolved and are never queued, and a rescan removes the pending,
|
||||
undecided items such files already have.
|
||||
"""
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
try:
|
||||
from tests.test_review import DELIMITERS, write_png
|
||||
from tests.test_review_outcome import OutcomeTestCase
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_review import DELIMITERS, write_png
|
||||
from test_review_outcome import OutcomeTestCase
|
||||
|
||||
from database.review_schema import evidence_offers_choice
|
||||
from utils import review_capture
|
||||
|
||||
|
||||
def opaque_graph(*opaque, traced=()):
|
||||
"""Several KSamplers: ``opaque`` positives come from an encoder the
|
||||
ingestion resolver does not know, ``traced`` ones from CLIPTextEncode.
|
||||
|
||||
The first sampler is opaque, so the scan's browser-parser fallback links
|
||||
nothing either.
|
||||
"""
|
||||
graph = {
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
|
||||
}
|
||||
sides = [("OpaqueTextEncode", "prompt", t) for t in opaque]
|
||||
sides += [("CLIPTextEncode", "text", t) for t in traced]
|
||||
for index, (encoder_class, key, text) in enumerate(sides):
|
||||
encoder, sampler = str(10 + 2 * index), str(11 + 2 * index)
|
||||
graph[encoder] = {"class_type": encoder_class, "inputs": {key: text}}
|
||||
graph[sampler] = {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {
|
||||
"model": ["1", 0],
|
||||
"positive": [encoder, 0],
|
||||
"negative": ["2", 0],
|
||||
},
|
||||
}
|
||||
return graph
|
||||
|
||||
|
||||
def _always_reviewable(_resolution):
|
||||
return True
|
||||
|
||||
|
||||
class ChoosableTestCase(OutcomeTestCase):
|
||||
def _record_pre_fix(self, name):
|
||||
"""An item as the pre-fix recorder stored it (no choosable test)."""
|
||||
with patch.object(review_capture, "reviewable", _always_reviewable):
|
||||
self._record(name)
|
||||
|
||||
def _write(self, name, graph):
|
||||
write_png(self.output_dir / name, prompt=graph)
|
||||
return self.output_dir / name
|
||||
|
||||
|
||||
class TestChoosableCandidates(unittest.TestCase):
|
||||
def _resolution(self, *states):
|
||||
candidates = []
|
||||
for index, state in enumerate(states):
|
||||
side = {"text": f"t{index}"} if state == "resolved" else None
|
||||
candidates.append(
|
||||
{"sampler_id": str(index), "state": state, "positive": side}
|
||||
)
|
||||
return {"state": "ambiguous", "candidates": candidates}
|
||||
|
||||
def test_two_resolved_candidates_are_reviewable(self):
|
||||
resolution = self._resolution("resolved", "resolved", "unresolved")
|
||||
self.assertTrue(review_capture.reviewable(resolution))
|
||||
|
||||
def test_none_or_one_resolved_candidate_is_not_reviewable(self):
|
||||
for states in ((), ("unresolved",) * 11, ("resolved", "unresolved")):
|
||||
with self.subTest(states=states):
|
||||
resolution = self._resolution(*states)
|
||||
self.assertFalse(review_capture.reviewable(resolution))
|
||||
|
||||
def test_blank_or_missing_text_does_not_count(self):
|
||||
resolution = self._resolution("resolved", "resolved")
|
||||
resolution["candidates"][0]["positive"] = {"text": " "}
|
||||
self.assertFalse(review_capture.reviewable(resolution))
|
||||
resolution["candidates"][0]["positive"] = None
|
||||
self.assertFalse(review_capture.reviewable(resolution))
|
||||
|
||||
def test_nested_ambiguous_stage_counts_its_own_candidates(self):
|
||||
inner = self._resolution("resolved", "resolved")["candidates"]
|
||||
resolution = {
|
||||
"state": "ambiguous",
|
||||
"candidates": [{"state": "ambiguous", "candidates": inner}],
|
||||
}
|
||||
self.assertTrue(review_capture.reviewable(resolution))
|
||||
|
||||
def test_a_result_that_is_not_ambiguous_is_not_reviewable(self):
|
||||
resolution = self._resolution("resolved", "resolved")
|
||||
resolution["state"] = "resolved"
|
||||
self.assertFalse(review_capture.reviewable(resolution))
|
||||
|
||||
|
||||
class TestStoredEvidenceOffersChoice(unittest.TestCase):
|
||||
def test_unlimited_evidence_needs_two_choosable_candidates(self):
|
||||
side = {"text": "cat"}
|
||||
two = [{"state": "resolved", "positive": side}] * 2
|
||||
self.assertTrue(evidence_offers_choice({"candidates": two}))
|
||||
one = two[:1] + [{"state": "unresolved", "positive": None}]
|
||||
self.assertFalse(evidence_offers_choice({"candidates": one}))
|
||||
|
||||
def test_limited_evidence_offers_a_choice_only_with_two_candidates(self):
|
||||
self.assertTrue(
|
||||
evidence_offers_choice({"limited": True, "candidate_total": 18})
|
||||
)
|
||||
for total in (0, 1, None, "18", True):
|
||||
with self.subTest(total=total):
|
||||
evidence = {"limited": True, "candidate_total": total}
|
||||
self.assertFalse(evidence_offers_choice(evidence))
|
||||
|
||||
def test_malformed_evidence_offers_nothing(self):
|
||||
for evidence in (None, [], {"candidates": "x"}, {}):
|
||||
with self.subTest(evidence=evidence):
|
||||
self.assertFalse(evidence_offers_choice(evidence))
|
||||
|
||||
|
||||
class TestNothingToChooseIsNotQueued(ChoosableTestCase):
|
||||
async def test_no_candidate_with_text_could_not_be_traced(self):
|
||||
self._roots()
|
||||
self._write("none.png", opaque_graph("cat", "dog"))
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertEqual(self._pending(), 0)
|
||||
self.assertEqual(job["scope"]["review_items"], 0)
|
||||
(row,) = job["outcomes"]["review"]
|
||||
self.assertEqual(row["path"], "none.png")
|
||||
self.assertIn("could not be traced", row["detail"])
|
||||
self.assertEqual(job["counters"]["review"], 1, job["counters"])
|
||||
self.assertEqual(self._capture_files(), [])
|
||||
|
||||
async def test_exactly_one_candidate_with_text_stays_unresolved(self):
|
||||
"""One traced branch beside untraced ones is not certain: no link."""
|
||||
self._roots()
|
||||
self._write("one.png", opaque_graph("cat", traced=("dog",)))
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
(row,) = job["outcomes"]["review"]
|
||||
self.assertIn("could not be traced", row["detail"])
|
||||
self.assertEqual(job["counters"].get("added", 0), 0, job["counters"])
|
||||
self.assertEqual(job["counters"].get("known", 0), 0, job["counters"])
|
||||
|
||||
async def test_two_candidates_with_text_are_queued(self):
|
||||
self._roots()
|
||||
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
(item,) = self._items()
|
||||
self.assertEqual(os.path.basename(item["locator"]), "two.png")
|
||||
self.assertEqual(item["candidate_count"], 3)
|
||||
self.assertEqual(job["scope"]["review_items"], 1)
|
||||
(row,) = job["outcomes"]["review"]
|
||||
self.assertIn("more than one prompt could match", row["detail"])
|
||||
|
||||
async def test_parameters_delimiter_ambiguity_is_not_queued(self):
|
||||
"""Two A1111 negative delimiters offer no candidate to choose."""
|
||||
self._roots()
|
||||
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
(row,) = job["outcomes"]["review"]
|
||||
self.assertIn("could not be traced", row["detail"])
|
||||
|
||||
def test_recorder_reports_untraced_and_stores_nothing(self):
|
||||
self._roots()
|
||||
self._write("none.png", opaque_graph("cat", "dog"))
|
||||
self._write("one.png", opaque_graph("cat", traced=("dog",)))
|
||||
for name in ("none.png", "one.png"):
|
||||
with self.subTest(name=name):
|
||||
outcome = review_capture.record_ambiguous_file(
|
||||
self.api.db, self.api._captures, self.output_dir, 0, name
|
||||
)
|
||||
self.assertEqual(outcome, "untraced")
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertEqual(self._capture_files(), [])
|
||||
|
||||
def test_monitor_does_not_queue_an_untraced_output(self):
|
||||
self._roots()
|
||||
path = self._write("none.png", opaque_graph("cat", "dog"))
|
||||
|
||||
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
|
||||
self.assertEqual(self._items(), [])
|
||||
|
||||
|
||||
class TestUntracedItemsLeaveTheQueue(ChoosableTestCase):
|
||||
async def test_rescan_removes_pending_items_without_a_choice(self):
|
||||
"""The maintainer's 39 items: queued before the rule, gone on rescan."""
|
||||
self._roots()
|
||||
self._write("none.png", opaque_graph("cat", "dog"))
|
||||
self._write("one.png", opaque_graph("cat", traced=("dog",)))
|
||||
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
|
||||
for name in ("none.png", "one.png", "two.png"):
|
||||
self._record_pre_fix(name)
|
||||
revision = self._review_revision()
|
||||
self.assertEqual(self._pending(), 3)
|
||||
|
||||
job = await self._scan("all")
|
||||
|
||||
(item,) = self._items()
|
||||
self.assertEqual(os.path.basename(item["locator"]), "two.png")
|
||||
self.assertEqual(self._pending(), 1)
|
||||
self.assertGreater(self._review_revision(), revision)
|
||||
self.assertEqual(job["scope"]["review_untraced"], 2)
|
||||
self.assertEqual(job["scope"]["review_items"], 1)
|
||||
self.assertIn(
|
||||
"= 2 images left Needs review · nothing to choose", "\n".join(job["log"])
|
||||
)
|
||||
|
||||
async def test_decided_items_of_untraced_files_are_never_touched(self):
|
||||
self._roots()
|
||||
for name in ("sel.png", "unres.png"):
|
||||
self._write(name, opaque_graph("cat", traced=(name, name + "2")))
|
||||
self._record_pre_fix(name)
|
||||
items = {os.path.basename(i["locator"]): i for i in self._items()}
|
||||
self._decide(items["sel.png"], "selected")
|
||||
self._decide(items["unres.png"], "unresolved")
|
||||
before = (self._items(), self._decisions())
|
||||
# The files now carry graphs without a choice, queued pre-fix.
|
||||
for name in ("sel.png", "unres.png", "pending.png"):
|
||||
self._write(name, opaque_graph("a cow", "dog"))
|
||||
self._record_pre_fix(name)
|
||||
self.assertEqual(self._pending(), 3)
|
||||
|
||||
job = await self._scan("all")
|
||||
|
||||
self.assertEqual((self._items(), self._decisions()), before)
|
||||
self.assertEqual(self._pending(), 0)
|
||||
self.assertEqual(job["scope"]["review_untraced"], 3)
|
||||
|
||||
async def test_an_item_with_a_choice_survives_beside_an_untraced_one(self):
|
||||
"""Removal judges each item by its own evidence: an item that offers
|
||||
a choice is not removed although the file's new bytes offer none."""
|
||||
self._roots()
|
||||
self._write("f.png", opaque_graph("x", traced=("cat", "dog")))
|
||||
self._record_pre_fix("f.png")
|
||||
(choosable,) = self._items()
|
||||
self._write("f.png", opaque_graph("a cow", "dog"))
|
||||
self._record_pre_fix("f.png")
|
||||
self.assertEqual(self._pending(), 2)
|
||||
|
||||
job = await self._scan("all")
|
||||
|
||||
self.assertEqual([i["id"] for i in self._items()], [choosable["id"]])
|
||||
self.assertEqual(job["scope"]["review_untraced"], 1)
|
||||
|
||||
async def test_limited_items_go_only_without_enough_candidates(self):
|
||||
"""A pre-fix delimiter item (no candidate) goes; a limited item of 18
|
||||
candidates may hold a choice and stays, though the bytes now offer
|
||||
none."""
|
||||
self._roots()
|
||||
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
|
||||
self._record_pre_fix("delims.png")
|
||||
many = [f"prompt {n}" for n in range(17)]
|
||||
self._write("many.png", opaque_graph("x", traced=many))
|
||||
self._record_pre_fix("many.png")
|
||||
kept = [i["id"] for i in self._items() if "many" in i["locator"]]
|
||||
self._write("many.png", opaque_graph("a cow", "dog"))
|
||||
|
||||
job = await self._scan("all")
|
||||
|
||||
self.assertEqual([i["id"] for i in self._items()], kept)
|
||||
self.assertEqual(job["scope"]["review_untraced"], 1)
|
||||
|
||||
async def test_choosable_pending_item_stays(self):
|
||||
self._roots()
|
||||
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
|
||||
await self._scan()
|
||||
(item,) = self._items()
|
||||
|
||||
job = await self._scan("all")
|
||||
|
||||
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
|
||||
self.assertEqual(job["scope"]["review_untraced"], 0)
|
||||
|
||||
async def test_a_failed_removal_is_logged_and_counted_not_fatal(self):
|
||||
import logging
|
||||
import sqlite3
|
||||
|
||||
self._roots()
|
||||
self._write("none.png", opaque_graph("cat", "dog"))
|
||||
self._record_pre_fix("none.png")
|
||||
with (
|
||||
patch.object(
|
||||
self.api.db,
|
||||
"settle_untraced_review_items",
|
||||
side_effect=sqlite3.OperationalError("database is locked"),
|
||||
),
|
||||
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
|
||||
):
|
||||
job = await self._scan("all")
|
||||
|
||||
self.assertEqual(job["state"], "finished")
|
||||
self.assertEqual(job["scope"]["review_errors"], 1)
|
||||
self.assertTrue(any("could not be removed" in line for line in logs.output))
|
||||
self.assertEqual(self._pending(), 1)
|
||||
await self._scan("all")
|
||||
self.assertEqual(self._pending(), 0)
|
||||
|
||||
def test_monitor_removes_the_pending_item_of_an_untraced_output(self):
|
||||
self._roots()
|
||||
path = self._write("none.png", opaque_graph("cat", "dog"))
|
||||
self._record_pre_fix("none.png")
|
||||
# A pre-fix item of other bytes keeps the stat shortcut out of the way.
|
||||
self._write("none.png", opaque_graph("a cow", "dog"))
|
||||
|
||||
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
|
||||
self.assertEqual(self._items(), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,374 +0,0 @@
|
||||
"""Needs review takes only files whose final outcome is ambiguous and unlinked.
|
||||
|
||||
The maintainer's live scan (#284 / #288) queued every file the strict
|
||||
resolver found ambiguous, also those the legacy browser-parser fallback then
|
||||
linked and those already linked, so the queue (10,232) was far larger than
|
||||
the scan's Needs review group. The rule now: a file is queued only when the
|
||||
resolver finds it ambiguous, the fallback does not link it and no saved row
|
||||
links it. A pending, undecided item whose file is linked (by the fallback, a
|
||||
later resolution or the monitor) is removed on the next scan or monitor
|
||||
pass; an item that carries a decision is never touched.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sqlite3
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
try:
|
||||
from tests.test_review import (
|
||||
DELIMITERS,
|
||||
NOW,
|
||||
ReviewTestCase,
|
||||
ambiguous_graph,
|
||||
resolved_graph,
|
||||
unlinkable_graph,
|
||||
write_png,
|
||||
)
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_review import (
|
||||
DELIMITERS,
|
||||
NOW,
|
||||
ReviewTestCase,
|
||||
ambiguous_graph,
|
||||
resolved_graph,
|
||||
unlinkable_graph,
|
||||
write_png,
|
||||
)
|
||||
|
||||
from utils.hashing import generate_prompt_hash
|
||||
from utils.image_monitor import ImageGenerationHandler
|
||||
from utils.prompt_tracker import PromptTracker
|
||||
from utils import review_capture
|
||||
|
||||
|
||||
def nan_stamped(graph):
|
||||
"""ComfyUI's ``is_changed: [NaN]`` stamp (written as a bare NaN literal)."""
|
||||
for node in graph.values():
|
||||
node["is_changed"] = [float("nan")]
|
||||
return graph
|
||||
|
||||
|
||||
class OutcomeTestCase(ReviewTestCase):
|
||||
def _pending(self):
|
||||
return self.api.db.library_counts()["review_pending"]
|
||||
|
||||
def _record(self, name):
|
||||
"""An item as the pre-fix producers recorded it (any ambiguous file)."""
|
||||
outcome = review_capture.record_ambiguous_file(
|
||||
self.api.db, self.api._captures, self.output_dir, 0, name
|
||||
)
|
||||
self.assertIn(outcome, ("recorded", "limited"))
|
||||
|
||||
def _decide(self, item, outcome):
|
||||
conn = self.api.db.model.get_connection()
|
||||
candidate = None
|
||||
if outcome == "selected":
|
||||
candidate = json.loads(item["evidence"])["candidates"][0]["hash"]
|
||||
with conn:
|
||||
conn.execute(
|
||||
"UPDATE media_review_items SET state=?, revision=1 WHERE id=?",
|
||||
(outcome, item["id"]),
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO media_review_decisions(id,item_id,decision_revision,"
|
||||
"created_at,outcome,candidate_hash,evidence) VALUES(?,?,?,?,?,?,?)",
|
||||
(
|
||||
f"00000000-0000-4000-8000-{len(self._decisions()) + 1:012d}",
|
||||
item["id"],
|
||||
1,
|
||||
NOW,
|
||||
outcome,
|
||||
candidate,
|
||||
'{"kind":"user-selection"}',
|
||||
),
|
||||
)
|
||||
|
||||
def _decisions(self):
|
||||
conn = self.api.db.model.get_connection()
|
||||
return [
|
||||
tuple(row)
|
||||
for row in conn.execute(
|
||||
"SELECT id,item_id,outcome FROM media_review_decisions ORDER BY id"
|
||||
)
|
||||
]
|
||||
|
||||
def _link(self, path, text="already saved"):
|
||||
prompt_id = self.api.db.save_prompt(
|
||||
text=text, prompt_hash=generate_prompt_hash(text)
|
||||
)
|
||||
self.api.db.link_image_to_prompt(prompt_id, str(path))
|
||||
return prompt_id
|
||||
|
||||
@staticmethod
|
||||
def _rows(job, group):
|
||||
return sorted(row["path"] for row in job["outcomes"].get(group, []))
|
||||
|
||||
|
||||
class TestOnlyUnlinkedAmbiguousFilesAreQueued(OutcomeTestCase):
|
||||
async def test_nan_stamped_graph_the_fallback_links_is_not_queued(self):
|
||||
self._roots()
|
||||
path = self.output_dir / "nan.png"
|
||||
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
|
||||
self.assertIn(b"NaN", path.read_bytes())
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(job["counters"]["added"], 1, job["counters"])
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertEqual(job["scope"]["review_items"], 0)
|
||||
self.assertEqual(self._pending(), 0)
|
||||
|
||||
async def test_fallback_link_to_a_known_prompt_is_not_queued(self):
|
||||
self._roots()
|
||||
write_png(self.output_dir / "a.png", prompt=ambiguous_graph("cat", "dog"))
|
||||
await self._scan()
|
||||
write_png(self.output_dir / "b.png", prompt=ambiguous_graph("cat", "dog"))
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertEqual(job["scope"]["review_items"], 0)
|
||||
|
||||
async def test_already_linked_file_is_not_queued(self):
|
||||
self._roots()
|
||||
path = self.output_dir / "linked.png"
|
||||
write_png(path, prompt=unlinkable_graph("cat", "dog"))
|
||||
self._link(path)
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertEqual(job["scope"]["review_items"], 0)
|
||||
self.assertEqual(self._rows(job, "review"), [])
|
||||
self.assertEqual(self._rows(job, "known"), ["linked.png"])
|
||||
|
||||
async def test_genuinely_ambiguous_file_is_queued(self):
|
||||
self._roots()
|
||||
path = self.output_dir / "two.png"
|
||||
write_png(path, prompt=nan_stamped(unlinkable_graph("cat", "dog")))
|
||||
# Beside it, a file the fallback links: only the first is queued.
|
||||
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
(item,) = self._items()
|
||||
self.assertEqual(os.path.basename(item["locator"]), "two.png")
|
||||
self.assertEqual(item["state"], "pending")
|
||||
self.assertEqual(item["candidate_count"], 3)
|
||||
self.assertEqual(job["scope"]["review_items"], 1)
|
||||
self.assertEqual(self._rows(job, "review"), ["two.png"])
|
||||
self.assertEqual(self._pending(), 1)
|
||||
self.assertIn("? 1 image added to Needs review", "\n".join(job["log"]))
|
||||
|
||||
async def test_counter_and_queue_agree_with_the_review_group(self):
|
||||
"""``review_items`` and the queue total equal the review group less
|
||||
its unresolved files, on the first scan and on a rescan."""
|
||||
self._roots()
|
||||
for name in ("u1.png", "u2.png"):
|
||||
write_png(self.output_dir / name, prompt=unlinkable_graph("cat", name))
|
||||
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
|
||||
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
|
||||
unresolved = {"1": {"class_type": "KSampler", "inputs": {}}}
|
||||
write_png(self.output_dir / "none.png", prompt=unresolved)
|
||||
write_png(self.output_dir / "one.png", prompt=resolved_graph())
|
||||
|
||||
for _ in range(2):
|
||||
job = await self._scan()
|
||||
review = self._rows(job, "review")
|
||||
unresolved_rows = sorted(
|
||||
row["path"]
|
||||
for row in job["outcomes"]["review"]
|
||||
if "could not be traced" in row["detail"]
|
||||
)
|
||||
queued = [name for name in review if name not in unresolved_rows]
|
||||
# delims.png has nothing to choose: "could not be traced".
|
||||
self.assertEqual(queued, ["u1.png", "u2.png"], job)
|
||||
self.assertIn("delims.png", unresolved_rows)
|
||||
self.assertEqual(job["scope"]["review_items"], 2)
|
||||
self.assertEqual(self._pending(), 2)
|
||||
self.assertIn("? 2 images already in Needs review", "\n".join(job["log"]))
|
||||
|
||||
|
||||
class TestQueueSelfHeals(OutcomeTestCase):
|
||||
async def test_pending_item_of_a_fallback_linked_file_is_removed(self):
|
||||
"""The maintainer's case: queued by the old rule, linked on rescan."""
|
||||
self._roots()
|
||||
path = self.output_dir / "nan.png"
|
||||
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
|
||||
self._record("nan.png")
|
||||
revision = self._review_revision()
|
||||
self.assertEqual(self._pending(), 1)
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertEqual(self._pending(), 0)
|
||||
self.assertGreater(self._review_revision(), revision)
|
||||
self.assertEqual(job["scope"]["review_settled"], 1)
|
||||
self.assertIn("= 1 image left Needs review · now linked", "\n".join(job["log"]))
|
||||
|
||||
async def test_pending_item_of_an_already_linked_file_is_removed(self):
|
||||
self._roots()
|
||||
path = self.output_dir / "linked.png"
|
||||
write_png(path, prompt=unlinkable_graph("cat", "dog"))
|
||||
self._record("linked.png")
|
||||
self._link(path)
|
||||
|
||||
await self._scan()
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertEqual(self._pending(), 0)
|
||||
|
||||
async def test_pending_item_of_a_skipped_known_file_is_removed(self):
|
||||
"""A file the "new" scope skips as known still settles its old item."""
|
||||
self._roots()
|
||||
path = self.output_dir / "one.png"
|
||||
write_png(path, prompt=unlinkable_graph("cat", "dog"))
|
||||
self._record("one.png")
|
||||
write_png(path, prompt=resolved_graph())
|
||||
first = await self._scan()
|
||||
self.assertEqual(self._rows(first, "added"), ["one.png"])
|
||||
# An item a pre-fix scan left for the file's old bytes.
|
||||
self.assertEqual(self._items(), [])
|
||||
write_png(path, prompt=unlinkable_graph("cat", "dog"), color=(0, 9, 0))
|
||||
self._record("one.png")
|
||||
write_png(path, prompt=resolved_graph())
|
||||
second = await self._scan()
|
||||
self.assertEqual(self._rows(second, "known"), ["one.png"])
|
||||
self.assertEqual(self._items(), [])
|
||||
|
||||
async def test_a_failed_settle_is_logged_and_counted_not_fatal(self):
|
||||
self._roots()
|
||||
write_png(self.output_dir / "nan.png", prompt=ambiguous_graph("cat", "dog"))
|
||||
self._record("nan.png")
|
||||
|
||||
with (
|
||||
patch.object(
|
||||
self.api.db,
|
||||
"settle_linked_review_items",
|
||||
side_effect=sqlite3.OperationalError("database is locked"),
|
||||
),
|
||||
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
|
||||
):
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(job["state"], "finished")
|
||||
self.assertEqual(job["scope"]["review_errors"], 1)
|
||||
self.assertEqual(job["scope"]["review_settled"], 0)
|
||||
self.assertTrue(any("could not be removed" in line for line in logs.output))
|
||||
self.assertEqual(self._pending(), 1)
|
||||
# The next scan settles it.
|
||||
await self._scan()
|
||||
self.assertEqual(self._pending(), 0)
|
||||
|
||||
async def test_still_unlinked_pending_item_stays(self):
|
||||
self._roots()
|
||||
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
|
||||
await self._scan()
|
||||
(item,) = self._items()
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
|
||||
self.assertEqual(job["scope"].get("review_settled", 0), 0)
|
||||
|
||||
async def test_decided_items_are_never_touched(self):
|
||||
self._roots()
|
||||
for name in ("sel.png", "unres.png"):
|
||||
write_png(self.output_dir / name, prompt=ambiguous_graph("cat", name))
|
||||
self._record(name)
|
||||
items = {os.path.basename(i["locator"]): i for i in self._items()}
|
||||
self._decide(items["sel.png"], "selected")
|
||||
self._decide(items["unres.png"], "unresolved")
|
||||
before = (self._items(), self._decisions())
|
||||
# A pending neighbour the fallback links: it alone leaves the queue.
|
||||
write_png(self.output_dir / "pending.png", prompt=ambiguous_graph("x", "y"))
|
||||
self._record("pending.png")
|
||||
self.assertEqual(self._pending(), 1)
|
||||
|
||||
job = await self._scan("all")
|
||||
|
||||
self.assertEqual((self._items(), self._decisions()), before)
|
||||
self.assertEqual(self._pending(), 0)
|
||||
self.assertEqual(job["scope"]["review_settled"], 1)
|
||||
|
||||
|
||||
class TestMonitorRule(OutcomeTestCase):
|
||||
def _handler(self):
|
||||
handler = ImageGenerationHandler(self.api.db, PromptTracker(self.api.db))
|
||||
handler.review_recorder = self.api.record_review_file
|
||||
return handler
|
||||
|
||||
def _context(self):
|
||||
prompt_id = self.api.db.save_prompt(
|
||||
text="latest", prompt_hash=generate_prompt_hash("latest")
|
||||
)
|
||||
return {"id": prompt_id, "text": "latest", "fallback": True}
|
||||
|
||||
async def test_monitor_linked_ambiguous_output_is_not_queued(self):
|
||||
self._roots()
|
||||
path = self.output_dir / "two.png"
|
||||
graph = unlinkable_graph("cat", "dog")
|
||||
write_png(path, prompt=graph)
|
||||
|
||||
self._handler().link_image_to_prompt(
|
||||
str(path), self._context(), {"prompt": graph}
|
||||
)
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
|
||||
async def test_monitor_link_removes_the_files_pending_item(self):
|
||||
self._roots()
|
||||
path = self.output_dir / "two.png"
|
||||
graph = unlinkable_graph("cat", "dog")
|
||||
write_png(path, prompt=graph)
|
||||
self._record("two.png")
|
||||
self.assertEqual(self._pending(), 1)
|
||||
|
||||
self._handler().link_image_to_prompt(
|
||||
str(path), self._context(), {"prompt": graph}
|
||||
)
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertEqual(self._pending(), 0)
|
||||
|
||||
async def test_monitor_never_touches_a_decided_item(self):
|
||||
self._roots()
|
||||
path = self.output_dir / "two.png"
|
||||
graph = unlinkable_graph("cat", "dog")
|
||||
write_png(path, prompt=graph)
|
||||
self._record("two.png")
|
||||
(item,) = self._items()
|
||||
self._decide(item, "unresolved")
|
||||
before = (self._items(), self._decisions())
|
||||
other = self.output_dir / "other.png"
|
||||
write_png(other, prompt=graph, color=(0, 0, 9))
|
||||
self._record("other.png")
|
||||
handler, context = self._handler(), self._context()
|
||||
|
||||
handler.link_image_to_prompt(str(path), context, {"prompt": graph})
|
||||
handler.link_image_to_prompt(str(other), context, {"prompt": graph})
|
||||
|
||||
self.assertEqual((self._items(), self._decisions()), before)
|
||||
self.assertEqual(self._pending(), 0)
|
||||
|
||||
async def test_record_review_file_applies_the_scan_rule(self):
|
||||
self._roots()
|
||||
write_png(self.output_dir / "fallback.png", prompt=ambiguous_graph("a", "b"))
|
||||
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
|
||||
linked = self.output_dir / "linked.png"
|
||||
write_png(linked, prompt=unlinkable_graph("cat", "dog"))
|
||||
self._link(linked)
|
||||
|
||||
record = self.api.record_review_file
|
||||
self.assertEqual(record(str(self.output_dir / "fallback.png")), "skipped")
|
||||
self.assertEqual(record(str(linked)), "linked")
|
||||
self.assertEqual(record(str(self.output_dir / "two.png")), "recorded")
|
||||
self.assertEqual(len(self._items()), 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,918 +0,0 @@
|
||||
"""Needs review schema group (#249): install, validation and restore safety."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from contextlib import closing
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
try:
|
||||
from tests.open_handles import assert_closed
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from open_handles import assert_closed
|
||||
|
||||
from database.models import PromptModel
|
||||
from database.operations import PromptDatabase
|
||||
|
||||
try: # guarded so a missing module fails assertions, not test loading
|
||||
from database.review_schema import (
|
||||
canonical_evidence,
|
||||
evidence_digest,
|
||||
migrate_review_state,
|
||||
review_schema_error,
|
||||
)
|
||||
except ImportError:
|
||||
canonical_evidence = None
|
||||
evidence_digest = None
|
||||
migrate_review_state = None
|
||||
review_schema_error = None
|
||||
|
||||
METADATA = "media_review_metadata"
|
||||
ITEMS = "media_review_items"
|
||||
DECISIONS = "media_review_decisions"
|
||||
TABLES = (METADATA, ITEMS, DECISIONS)
|
||||
ANNOTATION_TABLES = (
|
||||
"media_annotation_metadata",
|
||||
"media_annotation_files",
|
||||
"media_annotation_bytes",
|
||||
"media_image_tags",
|
||||
)
|
||||
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
|
||||
# The exact DDL the plan pins; the module must produce precisely these objects.
|
||||
DDL = {
|
||||
METADATA: (
|
||||
f"CREATE TABLE {METADATA}(singleton_id INTEGER PRIMARY KEY "
|
||||
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
|
||||
"CHECK(schema_version=1), review_index_revision INTEGER NOT NULL "
|
||||
"DEFAULT 0 CHECK(review_index_revision>=0))"
|
||||
),
|
||||
ITEMS: (
|
||||
f"CREATE TABLE {ITEMS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
|
||||
"byte_id TEXT NOT NULL REFERENCES media_annotation_bytes(id) "
|
||||
"ON DELETE RESTRICT, spelled_path TEXT NOT NULL CHECK(length(CAST("
|
||||
"spelled_path AS BLOB)) BETWEEN 1 AND 16384), "
|
||||
"resolver_version INTEGER NOT NULL, "
|
||||
"evidence_hash TEXT NOT NULL CHECK(length(evidence_hash)=64 AND "
|
||||
"evidence_hash NOT GLOB '*[^0-9a-f]*'), evidence TEXT NOT NULL "
|
||||
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
|
||||
"candidate_count INTEGER NOT NULL CHECK(candidate_count BETWEEN 0 AND 16), "
|
||||
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), state TEXT NOT NULL "
|
||||
"CHECK(state IN ('pending','selected','unresolved')), revision INTEGER "
|
||||
"NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, updated_at TEXT "
|
||||
"NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
|
||||
),
|
||||
DECISIONS: (
|
||||
f"CREATE TABLE {DECISIONS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
|
||||
f"item_id TEXT NOT NULL REFERENCES {ITEMS}(id) ON DELETE RESTRICT, "
|
||||
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
|
||||
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
|
||||
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
|
||||
"(length(candidate_hash)=64 AND candidate_hash NOT GLOB '*[^0-9a-f]*')), "
|
||||
"evidence TEXT NOT NULL "
|
||||
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
|
||||
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
|
||||
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
|
||||
"UNIQUE(item_id, decision_revision), "
|
||||
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
|
||||
),
|
||||
}
|
||||
INDEXES = {
|
||||
"idx_media_review_items_queue": (
|
||||
f"CREATE INDEX idx_media_review_items_queue ON {ITEMS}"
|
||||
"(state, created_at, id)"
|
||||
),
|
||||
"idx_media_review_decisions_pair": (
|
||||
f"CREATE INDEX idx_media_review_decisions_pair ON {DECISIONS}(pair_id)"
|
||||
),
|
||||
"idx_media_review_decisions_image": (
|
||||
f"CREATE INDEX idx_media_review_decisions_image ON {DECISIONS}(image_id)"
|
||||
),
|
||||
}
|
||||
AUTOINDEXES = {ITEMS: 2, DECISIONS: 2}
|
||||
LEGACY_TABLES = (
|
||||
"prompts",
|
||||
"generated_images",
|
||||
"tags",
|
||||
"prompt_tags",
|
||||
"prompt_pairs",
|
||||
"prompt_pair_tags",
|
||||
"image_favorites",
|
||||
) + ANNOTATION_TABLES
|
||||
FILE_ID = "0f8e4b2a-1c3d-4e5f-8a9b-0c1d2e3f4a5b"
|
||||
BYTE_ID = "6a1b2c3d-4e5f-4a6b-8c9d-0e1f2a3b4c5d"
|
||||
ITEM_ID = "1d2e3f4a-5b6c-4d7e-8f9a-0b1c2d3e4f5a"
|
||||
DECISION_ID = "2e3f4a5b-6c7d-4e8f-9a0b-1c2d3e4f5a6b"
|
||||
OTHER_ID = "7b2c3d4e-5f6a-4b7c-9d0e-1f2a3b4c5d6e"
|
||||
DIGEST = hashlib.sha256(b"img").hexdigest()
|
||||
CANDIDATE = hashlib.sha256(b"candidate").hexdigest()
|
||||
NOW = "2026-10-22T00:00:00.000000+00:00"
|
||||
EVIDENCE = '{"candidates":[],"sha256":"' + DIGEST + '"}'
|
||||
EVIDENCE_HASH = hashlib.sha256(EVIDENCE.encode("ascii")).hexdigest()
|
||||
|
||||
|
||||
def _compact(sql):
|
||||
return re.sub(r"\s+", "", sql or "").lower()
|
||||
|
||||
|
||||
def expected_catalog():
|
||||
objects = {("table", name, _compact(sql)) for name, sql in DDL.items()}
|
||||
objects |= {("index", name, _compact(sql)) for name, sql in INDEXES.items()}
|
||||
objects |= {
|
||||
("index", f"sqlite_autoindex_{table}_{n}", "")
|
||||
for table, count in AUTOINDEXES.items()
|
||||
for n in range(1, count + 1)
|
||||
}
|
||||
return objects
|
||||
|
||||
|
||||
def catalog(conn):
|
||||
names = tuple(TABLES) + tuple(INDEXES)
|
||||
marks = ",".join("?" * len(names))
|
||||
table_marks = ",".join("?" * len(TABLES))
|
||||
return {
|
||||
(kind, name, _compact(sql))
|
||||
for kind, name, sql in conn.execute(
|
||||
"SELECT type,name,sql FROM sqlite_master "
|
||||
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
|
||||
names + tuple(TABLES),
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
def rows(conn, sql, *params):
|
||||
"""Plain tuples whatever row factory *conn* uses."""
|
||||
return [tuple(row) for row in conn.execute(sql, params)]
|
||||
|
||||
|
||||
def metadata(conn):
|
||||
return rows(conn, f"SELECT schema_version,review_index_revision FROM {METADATA}")
|
||||
|
||||
|
||||
def table_rows(conn, tables):
|
||||
found = {}
|
||||
for table in tables:
|
||||
present = conn.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,)
|
||||
).fetchone()
|
||||
if present:
|
||||
found[table] = rows(conn, f"SELECT * FROM {table}") # nosec B608
|
||||
return found
|
||||
|
||||
|
||||
def insert_byte(conn, file_id=FILE_ID, byte_id=BYTE_ID, locator="/out/img.png"):
|
||||
conn.execute(
|
||||
"INSERT INTO media_annotation_files VALUES(?,?,?)", (file_id, locator, NOW)
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO media_annotation_bytes VALUES(?,?,?,?,?,?,?,0)",
|
||||
(byte_id, file_id, DIGEST, 3, 1_700_000_000_000_000_000, NOW, NOW),
|
||||
)
|
||||
|
||||
|
||||
def insert_item(conn, *, item_id=ITEM_ID, evidence=EVIDENCE, candidates=2, **extra):
|
||||
values = {
|
||||
"id": item_id,
|
||||
"byte_id": BYTE_ID,
|
||||
"spelled_path": "/out/img.png",
|
||||
"resolver_version": 1,
|
||||
"evidence_hash": hashlib.sha256(evidence.encode("utf-8")).hexdigest(),
|
||||
"evidence": evidence,
|
||||
"candidate_count": candidates,
|
||||
"limited": 0,
|
||||
"state": "pending",
|
||||
"revision": 0,
|
||||
"created_at": NOW,
|
||||
"updated_at": NOW,
|
||||
}
|
||||
values.update(extra)
|
||||
columns = ",".join(values)
|
||||
marks = ",".join("?" * len(values))
|
||||
conn.execute(
|
||||
f"INSERT INTO {ITEMS}({columns}) VALUES({marks})", # nosec B608
|
||||
tuple(values.values()),
|
||||
)
|
||||
|
||||
|
||||
def insert_decision(conn, *, decision_id=DECISION_ID, **extra):
|
||||
values = {
|
||||
"id": decision_id,
|
||||
"item_id": ITEM_ID,
|
||||
"decision_revision": 1,
|
||||
"created_at": NOW,
|
||||
"outcome": "selected",
|
||||
"candidate_hash": CANDIDATE,
|
||||
"evidence": EVIDENCE,
|
||||
"pair_id": None,
|
||||
"image_id": None,
|
||||
}
|
||||
values.update(extra)
|
||||
columns = ",".join(values)
|
||||
marks = ",".join("?" * len(values))
|
||||
conn.execute(
|
||||
f"INSERT INTO {DECISIONS}({columns}) VALUES({marks})", # nosec B608
|
||||
tuple(values.values()),
|
||||
)
|
||||
|
||||
|
||||
def insert_group_rows(conn, **item):
|
||||
insert_byte(conn)
|
||||
insert_item(conn, **item)
|
||||
insert_decision(conn)
|
||||
|
||||
|
||||
def populate_legacy(conn):
|
||||
conn.execute(
|
||||
"INSERT INTO prompts(id,text,hash,category,notes,run_count) VALUES"
|
||||
"(7,'positive','p','keep','notes',12)"
|
||||
)
|
||||
conn.execute("INSERT INTO prompts(id,text,hash) VALUES(9,'negative','n')")
|
||||
conn.execute(
|
||||
"INSERT INTO generated_images(id,prompt_id,image_path,filename,file_path) "
|
||||
"VALUES(4,7,'/tmp/image.png','image.png','/tmp/image.png')"
|
||||
)
|
||||
conn.execute("INSERT INTO tags(id,name) VALUES(3,'keep-tag')")
|
||||
conn.execute("INSERT INTO prompt_tags VALUES(7,3)")
|
||||
conn.execute(
|
||||
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
|
||||
"VALUES(6,7,9)"
|
||||
)
|
||||
conn.execute("INSERT INTO prompt_pair_tags VALUES(6,3)")
|
||||
conn.execute("INSERT INTO image_favorites VALUES('root','a.png',?)", (NOW,))
|
||||
|
||||
|
||||
def full_capacity_evidence():
|
||||
"""Sixteen candidates of control, lone-surrogate, astral and combining
|
||||
text, exactly 8 MiB."""
|
||||
text = '\x00\x07\x1f
\ud800 é é "q" \\ \U0001f9ea'
|
||||
candidates = [
|
||||
{
|
||||
"hash": hashlib.sha256(str(n).encode()).hexdigest(),
|
||||
"negative": {"text": text * 3},
|
||||
"positive": {"text": text * 5},
|
||||
"sampler_id": str(n),
|
||||
"state": "resolved",
|
||||
}
|
||||
for n in range(16)
|
||||
]
|
||||
value = {"candidates": candidates, "pad": ""}
|
||||
base = len(canonical_evidence(value).encode("utf-8"))
|
||||
value["pad"] = "x" * (MAX_EVIDENCE_BYTES - base)
|
||||
return value
|
||||
|
||||
|
||||
class ReviewSchemaTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.addCleanup(assert_closed, self, self.temp.name)
|
||||
self.addCleanup(PromptModel.reset_schema_cache)
|
||||
self.models = []
|
||||
self.addCleanup(self.close_models)
|
||||
self.path = str(Path(self.temp.name) / "live.db")
|
||||
|
||||
def close_models(self):
|
||||
for model in self.models:
|
||||
model.close_all()
|
||||
|
||||
def model(self, path=None):
|
||||
model = PromptModel(path or self.path)
|
||||
self.models.append(model)
|
||||
return model
|
||||
|
||||
def require_module(self):
|
||||
self.assertIsNotNone(
|
||||
review_schema_error, "database.review_schema is not implemented"
|
||||
)
|
||||
|
||||
def drop_tables(self, path, tables=TABLES):
|
||||
"""Remove tables from a closed database file (an old backup)."""
|
||||
with closing(sqlite3.connect(path)) as raw, raw:
|
||||
raw.execute("PRAGMA foreign_keys=OFF")
|
||||
for table in reversed(tables):
|
||||
raw.execute(f"DROP TABLE {table}")
|
||||
|
||||
def test_fresh_install_and_reopen_is_idempotent(self):
|
||||
self.require_module()
|
||||
model = self.model()
|
||||
with model.get_connection() as conn:
|
||||
self.assertEqual(catalog(conn), expected_catalog())
|
||||
self.assertIsNone(review_schema_error(conn, allow_missing=False))
|
||||
self.assertEqual(metadata(conn), [(1, 0)])
|
||||
insert_group_rows(conn)
|
||||
snapshot = table_rows(conn, TABLES)
|
||||
self.assertEqual(
|
||||
PromptModel.verify_database_file(self.path, require_library_state=True),
|
||||
(True, "ok"),
|
||||
)
|
||||
model.close_all()
|
||||
for _ in range(2):
|
||||
PromptModel.reset_schema_cache()
|
||||
model = self.model()
|
||||
with model.get_connection() as conn:
|
||||
self.assertEqual(catalog(conn), expected_catalog())
|
||||
self.assertIsNone(review_schema_error(conn, allow_missing=False))
|
||||
self.assertEqual(table_rows(conn, TABLES), snapshot)
|
||||
# Running the migration again on a current database changes nothing.
|
||||
migrate_review_state(conn)
|
||||
self.assertEqual(table_rows(conn, TABLES), snapshot)
|
||||
model.close_all()
|
||||
# Items and decisions keep the byte record and the item they cite.
|
||||
PromptModel.reset_schema_cache()
|
||||
model = self.model()
|
||||
with model.get_connection() as conn:
|
||||
with self.assertRaises(sqlite3.IntegrityError):
|
||||
conn.execute("DELETE FROM media_annotation_files")
|
||||
with self.assertRaises(sqlite3.IntegrityError):
|
||||
conn.execute(f"DELETE FROM {ITEMS}")
|
||||
self.assertEqual(table_rows(conn, TABLES), snapshot)
|
||||
|
||||
def test_startup_validates_stored_evidence_once(self):
|
||||
"""A reopen streams the evidence once and the annotation group twice
|
||||
(its own install), never again for the review group (I3)."""
|
||||
self.require_module()
|
||||
import database.annotation_schema as annotation_module
|
||||
import database.review_schema as review_module
|
||||
|
||||
model = self.model()
|
||||
with model.get_connection() as conn:
|
||||
insert_group_rows(conn)
|
||||
model.close_all()
|
||||
for _ in range(2):
|
||||
PromptModel.reset_schema_cache()
|
||||
with (
|
||||
patch.object(
|
||||
review_module,
|
||||
"_evidence_error",
|
||||
wraps=review_module._evidence_error,
|
||||
) as evidence,
|
||||
patch.object(
|
||||
review_module,
|
||||
"annotation_schema_error",
|
||||
wraps=review_module.annotation_schema_error,
|
||||
) as nested,
|
||||
patch.object(
|
||||
annotation_module,
|
||||
"_value_error",
|
||||
wraps=annotation_module._value_error,
|
||||
) as annotation_rows,
|
||||
):
|
||||
model = self.model()
|
||||
model.get_connection()
|
||||
self.assertEqual(evidence.call_count, 1)
|
||||
self.assertEqual(nested.call_count, 0)
|
||||
self.assertEqual(annotation_rows.call_count, 2)
|
||||
model.close_all()
|
||||
|
||||
def test_old_database_without_group_migrates_without_touching_legacy_rows(self):
|
||||
self.require_module()
|
||||
for label, dropped in (
|
||||
("annotations only", TABLES),
|
||||
("predates annotations", ANNOTATION_TABLES + TABLES),
|
||||
):
|
||||
with self.subTest(label):
|
||||
path = str(Path(self.temp.name) / f"{label.replace(' ', '-')}.db")
|
||||
model = self.model(path)
|
||||
with model.get_connection() as conn:
|
||||
populate_legacy(conn)
|
||||
if dropped == TABLES:
|
||||
insert_byte(conn)
|
||||
model.close_all()
|
||||
self.drop_tables(path, dropped)
|
||||
with closing(sqlite3.connect(path)) as raw:
|
||||
self.assertIsNone(review_schema_error(raw))
|
||||
self.assertEqual(
|
||||
review_schema_error(raw, allow_missing=False),
|
||||
"Missing review queue schema",
|
||||
)
|
||||
legacy_rows_before = table_rows(raw, LEGACY_TABLES)
|
||||
self.assertTrue(PromptModel.verify_database_file(path)[0])
|
||||
self.assertFalse(
|
||||
PromptModel.verify_database_file(path, require_library_state=True)[
|
||||
0
|
||||
]
|
||||
)
|
||||
PromptModel.reset_schema_cache()
|
||||
|
||||
model = self.model(path)
|
||||
|
||||
with model.get_connection() as conn:
|
||||
self.assertEqual(catalog(conn), expected_catalog())
|
||||
self.assertIsNone(review_schema_error(conn, allow_missing=False))
|
||||
self.assertEqual(metadata(conn), [(1, 0)])
|
||||
counts = [
|
||||
conn.execute(f"SELECT COUNT(*) FROM {t}").fetchone()[0]
|
||||
for t in (ITEMS, DECISIONS)
|
||||
]
|
||||
legacy_rows_after = {
|
||||
table: value
|
||||
for table, value in table_rows(conn, LEGACY_TABLES).items()
|
||||
if table in legacy_rows_before
|
||||
}
|
||||
self.assertEqual(counts, [0, 0])
|
||||
self.assertEqual(legacy_rows_after, legacy_rows_before)
|
||||
self.assertEqual(
|
||||
PromptModel.verify_database_file(path, require_library_state=True),
|
||||
(True, "ok"),
|
||||
)
|
||||
model.close_all()
|
||||
|
||||
def test_annotations_only_installation_is_valid(self):
|
||||
self.require_module()
|
||||
model = self.model()
|
||||
with model.get_connection() as conn:
|
||||
insert_byte(conn)
|
||||
model.close_all()
|
||||
self.drop_tables(self.path)
|
||||
with closing(sqlite3.connect(self.path)) as raw:
|
||||
self.assertIsNone(review_schema_error(raw))
|
||||
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
|
||||
self.assertEqual(
|
||||
PromptModel.verify_database_file(self.path, require_library_state=True),
|
||||
(False, "Missing review queue schema"),
|
||||
)
|
||||
|
||||
def test_review_group_without_annotation_group_is_invalid(self):
|
||||
self.require_module()
|
||||
model = self.model()
|
||||
model.close_all()
|
||||
self.drop_tables(self.path, ANNOTATION_TABLES)
|
||||
with closing(sqlite3.connect(self.path)) as raw:
|
||||
for allow_missing in (True, False):
|
||||
error = review_schema_error(raw, allow_missing=allow_missing)
|
||||
self.assertIsNotNone(error)
|
||||
self.assertIn("image annotation", error.lower())
|
||||
self.assertFalse(PromptModel.verify_database_file(self.path)[0])
|
||||
PromptModel.reset_schema_cache()
|
||||
with self.assertRaises(sqlite3.DatabaseError):
|
||||
self.model()
|
||||
# An invalid annotation group invalidates the review group beside it.
|
||||
path = self.tampered(
|
||||
"bad annotations",
|
||||
("UPDATE media_annotation_bytes SET sha256=upper(sha256)",),
|
||||
checks_off=True,
|
||||
)
|
||||
with closing(sqlite3.connect(path)) as raw:
|
||||
self.assertIn("image annotation", review_schema_error(raw).lower())
|
||||
|
||||
def tampered(self, name, statements, *, checks_off=False, rows_in=True):
|
||||
"""A closed current database whose group was altered after the fact."""
|
||||
path = str(Path(self.temp.name) / f"{name.replace(' ', '-')}.db")
|
||||
model = self.model(path)
|
||||
with model.get_connection() as conn:
|
||||
populate_legacy(conn)
|
||||
if rows_in:
|
||||
insert_group_rows(conn)
|
||||
model.close_all()
|
||||
with closing(sqlite3.connect(path)) as raw, raw:
|
||||
raw.execute("PRAGMA foreign_keys=OFF")
|
||||
if checks_off:
|
||||
raw.execute("PRAGMA ignore_check_constraints=ON")
|
||||
for statement in statements:
|
||||
if isinstance(statement, tuple):
|
||||
raw.execute(*statement)
|
||||
else:
|
||||
raw.execute(statement)
|
||||
return path
|
||||
|
||||
def assert_rejected_unrepaired(self, path):
|
||||
with closing(sqlite3.connect(path)) as raw:
|
||||
schema_before = raw.execute(
|
||||
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
|
||||
).fetchall()
|
||||
rows_before = table_rows(raw, TABLES)
|
||||
for allow_missing in (False, True):
|
||||
error = review_schema_error(raw, allow_missing=allow_missing)
|
||||
self.assertIsNotNone(error)
|
||||
self.assertIn("review", error.lower())
|
||||
self.assertFalse(PromptModel.verify_database_file(path)[0])
|
||||
self.assertFalse(
|
||||
PromptModel.verify_database_file(path, require_library_state=True)[0]
|
||||
)
|
||||
PromptModel.reset_schema_cache()
|
||||
with self.assertRaises(sqlite3.DatabaseError):
|
||||
self.model(path)
|
||||
PromptModel.reset_schema_cache()
|
||||
with closing(sqlite3.connect(path)) as raw:
|
||||
self.assertEqual(
|
||||
raw.execute(
|
||||
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
|
||||
).fetchall(),
|
||||
schema_before,
|
||||
)
|
||||
self.assertEqual(table_rows(raw, TABLES), rows_before)
|
||||
|
||||
def test_partial_or_malformed_group_is_rejected_without_repair(self):
|
||||
self.require_module()
|
||||
cases = {
|
||||
"missing metadata": (f"DROP TABLE {METADATA}",),
|
||||
"missing items": (f"DROP TABLE {DECISIONS}", f"DROP TABLE {ITEMS}"),
|
||||
"missing decisions": (f"DROP TABLE {DECISIONS}",),
|
||||
"reserved view": (
|
||||
f"DROP TABLE {DECISIONS}",
|
||||
f"CREATE VIEW {DECISIONS} AS SELECT 'x' AS id",
|
||||
),
|
||||
"extra trigger": (
|
||||
f"CREATE TRIGGER media_review_trigger AFTER INSERT ON {ITEMS} "
|
||||
"BEGIN SELECT 1; END",
|
||||
),
|
||||
"extra index": (
|
||||
f"CREATE INDEX idx_media_review_extra ON {ITEMS}(updated_at)",
|
||||
),
|
||||
"extra column": (
|
||||
f"DROP TABLE {DECISIONS}",
|
||||
DDL[DECISIONS].replace("evidence TEXT", "note TEXT, evidence TEXT"),
|
||||
),
|
||||
"wrong primary key": (
|
||||
f"DROP TABLE {DECISIONS}",
|
||||
DDL[DECISIONS].replace("id TEXT PRIMARY KEY", "id TEXT", 1),
|
||||
),
|
||||
"wrong foreign key": (
|
||||
f"DROP TABLE {DECISIONS}",
|
||||
DDL[DECISIONS].replace(
|
||||
"generated_images(id) ON DELETE SET NULL",
|
||||
"generated_images(id) ON DELETE CASCADE",
|
||||
),
|
||||
INDEXES["idx_media_review_decisions_pair"],
|
||||
INDEXES["idx_media_review_decisions_image"],
|
||||
),
|
||||
"wrong check": (
|
||||
f"DROP TABLE {METADATA}",
|
||||
DDL[METADATA].replace("schema_version=1", "schema_version=2"),
|
||||
f"INSERT INTO {METADATA} VALUES(1,2,0)",
|
||||
),
|
||||
"missing unique": (
|
||||
"DROP INDEX idx_media_review_decisions_pair",
|
||||
"DROP INDEX idx_media_review_decisions_image",
|
||||
f"DROP TABLE {DECISIONS}",
|
||||
DDL[DECISIONS].replace(" UNIQUE(item_id, decision_revision),", ""),
|
||||
INDEXES["idx_media_review_decisions_pair"],
|
||||
INDEXES["idx_media_review_decisions_image"],
|
||||
),
|
||||
"missing queue index": ("DROP INDEX idx_media_review_items_queue",),
|
||||
"wrong index columns": (
|
||||
"DROP INDEX idx_media_review_items_queue",
|
||||
INDEXES["idx_media_review_items_queue"].replace(", id", ""),
|
||||
),
|
||||
"orphan item": ((f"UPDATE {ITEMS} SET byte_id=?", (OTHER_ID,)),),
|
||||
"orphan decision": ((f"UPDATE {DECISIONS} SET item_id=?", (OTHER_ID,)),),
|
||||
"dangling pair": (f"UPDATE {DECISIONS} SET pair_id=404",),
|
||||
"bad item uuid": (
|
||||
(f"UPDATE {ITEMS} SET id=?", ("x" * 36,)),
|
||||
(f"UPDATE {DECISIONS} SET item_id=?", ("x" * 36,)),
|
||||
),
|
||||
"uppercase decision uuid": (f"UPDATE {DECISIONS} SET id=upper(id)",),
|
||||
"null decision id": (f"UPDATE {DECISIONS} SET id=NULL",),
|
||||
"blob ids": (f"UPDATE {DECISIONS} SET id=CAST(id AS BLOB)",),
|
||||
"no metadata row": (f"DELETE FROM {METADATA}",),
|
||||
"long created_at": ((f"UPDATE {ITEMS} SET created_at=?", ("x" * 65,)),),
|
||||
"blob updated_at": (f"UPDATE {ITEMS} SET updated_at=X'00'",),
|
||||
"text resolver version": (f"UPDATE {ITEMS} SET resolver_version='one'",),
|
||||
"blob spelled path": (
|
||||
f"UPDATE {ITEMS} SET spelled_path=CAST(spelled_path AS BLOB)",
|
||||
),
|
||||
"evidence hash mismatch": (
|
||||
(f"UPDATE {ITEMS} SET evidence_hash=?", (CANDIDATE,)),
|
||||
),
|
||||
}
|
||||
for label, statements in cases.items():
|
||||
with self.subTest(label):
|
||||
self.assert_rejected_unrepaired(self.tampered(label, statements))
|
||||
unchecked = {
|
||||
"two metadata rows": (f"INSERT INTO {METADATA} VALUES(2,1,0)",),
|
||||
"wrong schema version": (f"UPDATE {METADATA} SET schema_version=2",),
|
||||
"negative index revision": (
|
||||
f"UPDATE {METADATA} SET review_index_revision=-1",
|
||||
),
|
||||
"uppercase evidence hash": (
|
||||
f"UPDATE {ITEMS} SET evidence_hash=upper(evidence_hash)",
|
||||
),
|
||||
"bad state": (f"UPDATE {ITEMS} SET state='done'",),
|
||||
"limited two": (f"UPDATE {ITEMS} SET limited=2",),
|
||||
"empty spelled path": (f"UPDATE {ITEMS} SET spelled_path=''",),
|
||||
"negative revision": (f"UPDATE {ITEMS} SET revision=-1",),
|
||||
"negative candidates": (f"UPDATE {ITEMS} SET candidate_count=-1",),
|
||||
"bad outcome": (f"UPDATE {DECISIONS} SET outcome='maybe'",),
|
||||
"short candidate hash": (
|
||||
f"UPDATE {DECISIONS} SET candidate_hash=substr(candidate_hash,1,63)",
|
||||
),
|
||||
"negative decision revision": (
|
||||
f"UPDATE {DECISIONS} SET decision_revision=-1",
|
||||
),
|
||||
}
|
||||
for label, statements in unchecked.items():
|
||||
with self.subTest(label):
|
||||
path = self.tampered(label, statements, checks_off=True)
|
||||
self.assert_rejected_unrepaired(path)
|
||||
|
||||
def test_selected_decision_requires_candidate_hash(self):
|
||||
self.require_module()
|
||||
model = self.model()
|
||||
with model.get_connection() as conn:
|
||||
insert_byte(conn)
|
||||
insert_item(conn)
|
||||
for outcome, candidate in (("selected", None), ("unresolved", CANDIDATE)):
|
||||
with self.subTest(outcome):
|
||||
with self.assertRaises(sqlite3.IntegrityError):
|
||||
insert_decision(conn, outcome=outcome, candidate_hash=candidate)
|
||||
insert_decision(conn, outcome="unresolved", candidate_hash=None)
|
||||
insert_decision(conn, decision_id=OTHER_ID, decision_revision=2)
|
||||
self.assertIsNone(review_schema_error(conn, allow_missing=False))
|
||||
self.assertEqual(
|
||||
rows(conn, f"SELECT outcome FROM {DECISIONS} ORDER BY 1"),
|
||||
[("selected",), ("unresolved",)],
|
||||
)
|
||||
with self.assertRaises(sqlite3.IntegrityError): # one per revision
|
||||
insert_decision(conn, decision_id=ITEM_ID, decision_revision=2)
|
||||
model.close_all()
|
||||
path = self.tampered(
|
||||
"selected without hash",
|
||||
(f"UPDATE {DECISIONS} SET candidate_hash=NULL",),
|
||||
checks_off=True,
|
||||
)
|
||||
self.assert_rejected_unrepaired(path)
|
||||
|
||||
def test_oversize_or_noncanonical_evidence_rejected(self):
|
||||
self.require_module()
|
||||
self.assertEqual(
|
||||
canonical_evidence({"b": [1, "é"], "a": None}),
|
||||
'{"a":null,"b":[1,"\\u00e9"]}',
|
||||
)
|
||||
self.assertEqual(
|
||||
evidence_digest(EVIDENCE), hashlib.sha256(EVIDENCE.encode()).hexdigest()
|
||||
)
|
||||
# The store's non-finite policy (utils.json_strict): written as null.
|
||||
self.assertEqual(
|
||||
canonical_evidence({"x": float("nan"), "y": [float("-inf")]}),
|
||||
'{"x":null,"y":[null]}',
|
||||
)
|
||||
for bad in (
|
||||
[],
|
||||
"text",
|
||||
{"x": object()},
|
||||
{"x": object(), "y": float("nan")},
|
||||
{1: 1, "a": 2},
|
||||
):
|
||||
with self.subTest(value=repr(bad)), self.assertRaises(ValueError):
|
||||
canonical_evidence(bad)
|
||||
with self.assertRaises(ValueError):
|
||||
canonical_evidence({"pad": "x" * MAX_EVIDENCE_BYTES})
|
||||
oversize = '{"pad":"' + "x" * (MAX_EVIDENCE_BYTES - 9) + '"}'
|
||||
self.assertEqual(len(oversize), MAX_EVIDENCE_BYTES + 1)
|
||||
model = self.model()
|
||||
with model.get_connection() as conn:
|
||||
insert_byte(conn)
|
||||
with self.assertRaises(sqlite3.IntegrityError):
|
||||
insert_item(conn, evidence=oversize)
|
||||
model.close_all()
|
||||
noncanonical = {
|
||||
"spaces": '{"candidates": [], "sha256": "x"}',
|
||||
"unsorted": '{"sha256":"x","candidates":[]}',
|
||||
"raw unicode": '{"text":"é"}',
|
||||
"duplicate key": '{"a":1,"a":2}',
|
||||
"not an object": "[]",
|
||||
"not json": "{",
|
||||
"nan": '{"a":NaN}',
|
||||
"empty": "",
|
||||
}
|
||||
for label, evidence in noncanonical.items():
|
||||
for table in (ITEMS, DECISIONS):
|
||||
with self.subTest(label, table=table):
|
||||
digest = hashlib.sha256(evidence.encode("utf-8")).hexdigest()
|
||||
statements = [
|
||||
(f"UPDATE {table} SET evidence=?", (evidence,)),
|
||||
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
|
||||
]
|
||||
if table == DECISIONS:
|
||||
statements.pop()
|
||||
self.assert_rejected_unrepaired(
|
||||
self.tampered(f"{label} {table}", statements)
|
||||
)
|
||||
for table in (ITEMS, DECISIONS):
|
||||
with self.subTest("oversize", table=table):
|
||||
digest = hashlib.sha256(oversize.encode("ascii")).hexdigest()
|
||||
statements = [
|
||||
(f"UPDATE {table} SET evidence=?", (oversize,)),
|
||||
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
|
||||
]
|
||||
path = self.tampered(f"oversize {table}", statements, checks_off=True)
|
||||
self.assert_rejected_unrepaired(path)
|
||||
|
||||
def test_seventeen_candidates_rejected(self):
|
||||
self.require_module()
|
||||
model = self.model()
|
||||
with model.get_connection() as conn:
|
||||
insert_byte(conn)
|
||||
with self.assertRaises(sqlite3.IntegrityError):
|
||||
insert_item(conn, candidates=17)
|
||||
insert_item(conn, candidates=16)
|
||||
self.assertIsNone(review_schema_error(conn, allow_missing=False))
|
||||
model.close_all()
|
||||
path = self.tampered(
|
||||
"seventeen", (f"UPDATE {ITEMS} SET candidate_count=17",), checks_off=True
|
||||
)
|
||||
self.assert_rejected_unrepaired(path)
|
||||
|
||||
def test_prompt_or_image_deletion_nulls_navigation_and_keeps_decision(self):
|
||||
self.require_module()
|
||||
db = PromptDatabase(self.path)
|
||||
self.models.append(db.model)
|
||||
with db.model.get_connection() as conn:
|
||||
populate_legacy(conn)
|
||||
conn.execute(
|
||||
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
|
||||
"VALUES(8,9,NULL)"
|
||||
)
|
||||
insert_byte(conn)
|
||||
insert_item(conn)
|
||||
insert_decision(conn, pair_id=8, image_id=4)
|
||||
conn.commit()
|
||||
self.assertTrue(db.delete_prompt_pair(8))
|
||||
with db.model.get_connection() as conn:
|
||||
self.assertEqual(
|
||||
rows(conn, f"SELECT pair_id, image_id FROM {DECISIONS}"), [(None, 4)]
|
||||
)
|
||||
with db.model.get_connection() as conn:
|
||||
conn.execute("DELETE FROM prompt_pairs WHERE id=6")
|
||||
conn.commit()
|
||||
self.assertTrue(db.delete_prompt(7))
|
||||
with db.model.get_connection() as conn:
|
||||
self.assertEqual(
|
||||
rows(
|
||||
conn,
|
||||
f"SELECT id, item_id, outcome, candidate_hash, pair_id, image_id "
|
||||
f"FROM {DECISIONS}",
|
||||
),
|
||||
[(DECISION_ID, ITEM_ID, "selected", CANDIDATE, None, None)],
|
||||
)
|
||||
self.assertEqual(rows(conn, f"SELECT id FROM {ITEMS}"), [(ITEM_ID,)])
|
||||
self.assertIsNone(review_schema_error(conn, allow_missing=False))
|
||||
# The SET NULL actions look decisions up by index, not by a scan.
|
||||
for column in ("pair_id", "image_id"):
|
||||
plan = " ".join(
|
||||
str(row[3])
|
||||
for row in conn.execute(
|
||||
f"EXPLAIN QUERY PLAN UPDATE {DECISIONS} " # nosec B608
|
||||
f"SET {column}=NULL WHERE {column}=?",
|
||||
(1,),
|
||||
)
|
||||
)
|
||||
self.assertIn(f"idx_media_review_decisions_{column[:-3]}", plan)
|
||||
|
||||
def test_full_capacity_unicode_control_pair_round_trips_within_8_mib(self):
|
||||
self.require_module()
|
||||
value = full_capacity_evidence()
|
||||
evidence = canonical_evidence(value)
|
||||
self.assertEqual(len(evidence.encode("utf-8")), MAX_EVIDENCE_BYTES)
|
||||
self.assertTrue(evidence.isascii())
|
||||
source_path = str(Path(self.temp.name) / "source.db")
|
||||
source = self.model(source_path)
|
||||
with source.get_connection() as conn:
|
||||
conn.execute("PRAGMA wal_autocheckpoint=0")
|
||||
insert_byte(conn)
|
||||
insert_item(conn, evidence=evidence, candidates=16)
|
||||
insert_decision(conn, evidence=evidence)
|
||||
conn.commit()
|
||||
source_rows = table_rows(conn, TABLES)
|
||||
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
|
||||
live = self.model()
|
||||
backup_file = str(Path(self.temp.name) / "backup.db")
|
||||
self.assertTrue(source.backup_database(backup_file))
|
||||
for restore_from in (source_path, backup_file):
|
||||
with self.subTest(Path(restore_from).name):
|
||||
live.restore_from_file(restore_from)
|
||||
with live.get_connection() as conn:
|
||||
self.assertEqual(table_rows(conn, TABLES), source_rows)
|
||||
self.assertIsNone(review_schema_error(conn, allow_missing=False))
|
||||
stored = conn.execute(
|
||||
f"SELECT i.evidence, d.evidence FROM {ITEMS} i "
|
||||
f"JOIN {DECISIONS} d ON d.item_id = i.id"
|
||||
).fetchone()
|
||||
self.assertEqual(json.loads(stored[0]), value)
|
||||
self.assertEqual(json.loads(stored[1]), value)
|
||||
PromptModel.close_all_instances()
|
||||
assert_closed(self, self.temp.name)
|
||||
|
||||
def test_install_failure_rolls_back_and_retries(self):
|
||||
self.require_module()
|
||||
with patch(
|
||||
"database.review_schema.review_schema_error",
|
||||
side_effect=[None, "injected final validation failure"],
|
||||
):
|
||||
with self.assertRaisesRegex(sqlite3.DatabaseError, "injected"):
|
||||
self.model()
|
||||
with closing(sqlite3.connect(self.path)) as raw:
|
||||
self.assertEqual(catalog(raw), set())
|
||||
self.assertIsNotNone(
|
||||
raw.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE name='media_annotation_bytes'"
|
||||
).fetchone()
|
||||
)
|
||||
PromptModel.reset_schema_cache()
|
||||
model = self.model()
|
||||
self.assertEqual(
|
||||
PromptModel.verify_database_file(model.db_path, require_library_state=True),
|
||||
(True, "ok"),
|
||||
)
|
||||
with model.get_connection() as conn:
|
||||
self.assertEqual(metadata(conn), [(1, 0)])
|
||||
|
||||
def test_wal_backup_restore_round_trips_group(self):
|
||||
self.require_module()
|
||||
source_path = str(Path(self.temp.name) / "source.db")
|
||||
source = self.model(source_path)
|
||||
with source.get_connection() as conn:
|
||||
conn.execute("PRAGMA wal_autocheckpoint=0")
|
||||
insert_group_rows(conn)
|
||||
conn.execute(f"UPDATE {METADATA} SET review_index_revision=3")
|
||||
conn.commit()
|
||||
source_rows = table_rows(conn, TABLES)
|
||||
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
|
||||
live = self.model()
|
||||
with live.get_connection() as conn:
|
||||
insert_byte(conn)
|
||||
insert_item(conn, item_id=OTHER_ID)
|
||||
source_before = Path(source_path).read_bytes()
|
||||
backup_file = str(Path(self.temp.name) / "backup.db")
|
||||
self.assertTrue(source.backup_database(backup_file))
|
||||
with closing(sqlite3.connect(backup_file)) as raw:
|
||||
self.assertEqual(table_rows(raw, TABLES), source_rows)
|
||||
self.assertIsNone(review_schema_error(raw, allow_missing=False))
|
||||
|
||||
safety = live.restore_from_file(source_path)
|
||||
|
||||
self.assertEqual(Path(source_path).read_bytes(), source_before)
|
||||
with live.get_connection() as conn:
|
||||
self.assertEqual(table_rows(conn, TABLES), source_rows)
|
||||
self.assertEqual(metadata(conn), [(1, 3)])
|
||||
with closing(sqlite3.connect(safety)) as raw:
|
||||
self.assertEqual(rows(raw, f"SELECT id FROM {ITEMS}"), [(OTHER_ID,)])
|
||||
PromptModel.reset_schema_cache()
|
||||
live.restore_from_file(backup_file)
|
||||
with live.get_connection() as conn:
|
||||
self.assertEqual(table_rows(conn, TABLES), source_rows)
|
||||
PromptModel.close_all_instances()
|
||||
assert_closed(self, self.temp.name)
|
||||
|
||||
def test_staged_failure_preserves_source_live_safety_and_handles(self):
|
||||
self.require_module()
|
||||
live = self.model()
|
||||
with live.get_connection() as conn:
|
||||
insert_byte(conn)
|
||||
insert_item(conn, item_id=OTHER_ID)
|
||||
live_rows = table_rows(conn, TABLES)
|
||||
bad = self.tampered(
|
||||
"restore bad", (f"UPDATE {ITEMS} SET state='done'",), checks_off=True
|
||||
)
|
||||
old = str(Path(self.temp.name) / "old.db")
|
||||
self.model(old).close_all()
|
||||
self.drop_tables(old)
|
||||
for source, patched in ((bad, False), (old, True)):
|
||||
with self.subTest(source=Path(source).name):
|
||||
before = Path(source).read_bytes()
|
||||
safety = sorted(Path(self.temp.name).glob("live.db.backup_*"))
|
||||
if patched:
|
||||
context = patch(
|
||||
"database.models.migrate_review_state",
|
||||
side_effect=sqlite3.OperationalError("staging failed"),
|
||||
)
|
||||
expected = sqlite3.OperationalError
|
||||
else:
|
||||
context = patch.object(PromptModel, "backup_database")
|
||||
expected = ValueError
|
||||
with context as hook, self.assertRaises(expected):
|
||||
live.restore_from_file(source)
|
||||
if not patched:
|
||||
hook.assert_not_called()
|
||||
self.assertEqual(Path(source).read_bytes(), before)
|
||||
self.assertEqual(
|
||||
sorted(Path(self.temp.name).glob("live.db.backup_*")), safety
|
||||
)
|
||||
assert_closed(self, source)
|
||||
with live.get_connection() as conn:
|
||||
self.assertEqual(table_rows(conn, TABLES), live_rows)
|
||||
self.assertFalse(
|
||||
PromptModel.verify_database_file(old, require_library_state=True)[0]
|
||||
)
|
||||
self.assertTrue(PromptModel.verify_database_file(old)[0])
|
||||
|
||||
def test_verify_requires_group_only_with_require_library_state(self):
|
||||
self.require_module()
|
||||
self.model().close_all()
|
||||
self.assertEqual(
|
||||
PromptModel.verify_database_file(self.path, require_library_state=True),
|
||||
(True, "ok"),
|
||||
)
|
||||
self.drop_tables(self.path)
|
||||
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
|
||||
self.assertEqual(
|
||||
PromptModel.verify_database_file(self.path, require_library_state=True),
|
||||
(False, "Missing review queue schema"),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -38,11 +38,6 @@ def _graph(text):
|
||||
}
|
||||
|
||||
|
||||
def _scope_part(scope):
|
||||
"""The per-source scope keys this module pins (other features add counters)."""
|
||||
return {key: scope[key] for key in ("files", "roots", "output_folders")}
|
||||
|
||||
|
||||
class ScanScopeTestCase(AdminAPITestCase):
|
||||
"""Helpers: link files the way the scan and the monitor do, spy on reads."""
|
||||
|
||||
@@ -347,8 +342,7 @@ class TestPerSourceScope(ScanScopeTestCase):
|
||||
self.assertEqual(walked, [["b"]])
|
||||
self.assertEqual(reads, ["in_b.png"])
|
||||
self.assertEqual(
|
||||
_scope_part(job["scope"]),
|
||||
{"files": "all", "roots": [1], "output_folders": 1},
|
||||
job["scope"], {"files": "all", "roots": [1], "output_folders": 1}
|
||||
)
|
||||
self.assertEqual(
|
||||
job["outcomes"]["added"],
|
||||
@@ -384,8 +378,7 @@ class TestPerSourceScope(ScanScopeTestCase):
|
||||
|
||||
unscoped = await self._scan({"scope": "all"})
|
||||
self.assertEqual(
|
||||
_scope_part(unscoped["scope"]),
|
||||
{"files": "all", "roots": None, "output_folders": 2},
|
||||
unscoped["scope"], {"files": "all", "roots": None, "output_folders": 2}
|
||||
)
|
||||
self.assertIn("Scan started · 2 output folders · all files", unscoped["log"][0])
|
||||
|
||||
@@ -459,7 +452,7 @@ class TestPerSourceScope(ScanScopeTestCase):
|
||||
self.assertEqual(final["directories"], ["output/a", "output/b"])
|
||||
self.assertEqual(final["added"], 2)
|
||||
self.assertEqual(
|
||||
_scope_part(self.api._scan_job.scope),
|
||||
self.api._scan_job.scope,
|
||||
{"files": "all", "roots": None, "output_folders": 2},
|
||||
)
|
||||
|
||||
|
||||
@@ -292,7 +292,7 @@ class TestOverviewRoute(SystemAPITestCase):
|
||||
body = await self.json("GET", "/prompt_manager/system/overview")
|
||||
self.assertIsNone(body["health"])
|
||||
self.assertIsNone(body["backup"])
|
||||
self.assertEqual(body["needs_review"], {"pending": 0})
|
||||
self.assertEqual(body["needs_review"], {"pending": None})
|
||||
self.save("Any")
|
||||
await self.run_health()
|
||||
report = (await self.json("GET", "/prompt_manager/system/health"))["report"]
|
||||
|
||||
@@ -1,846 +0,0 @@
|
||||
"""Thumbnail generation as an Activity job (#249 System S5, Task 12).
|
||||
|
||||
``GET /prompt_manager/images/thumbnails/plan`` counts what a run would do and
|
||||
writes nothing. ``POST /prompt_manager/jobs/thumbnails`` starts the
|
||||
``thumbnails`` job (admission ``guarded``, cancellable at a file boundary).
|
||||
The legacy gallery stream ``POST /images/generate-thumbnails/progress`` starts
|
||||
or attaches to that job and streams its unchanged events from it, so a client
|
||||
that goes away no longer stops the work.
|
||||
|
||||
Ordering is driven by threading/asyncio events, never by timers.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
try:
|
||||
from tests.test_api_images import ThumbnailTestCase, make_png, parse_sse
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_api_images import ThumbnailTestCase, make_png, parse_sse
|
||||
|
||||
from PIL import Image # noqa: E402
|
||||
|
||||
import py.api.images as images_module # noqa: E402
|
||||
from py.api.thumbnail_jobs import thumbnail_options # noqa: E402
|
||||
from py.config import PromptManagerConfig # noqa: E402
|
||||
|
||||
PLAN_URL = "/prompt_manager/images/thumbnails/plan"
|
||||
JOB_URL = "/prompt_manager/jobs/thumbnails"
|
||||
|
||||
|
||||
def _tree_state(root):
|
||||
"""{relative path: (size, mtime_ns, sha256)} of every file under ``root``."""
|
||||
state = {}
|
||||
for folder, _dirs, files in os.walk(root):
|
||||
for name in files:
|
||||
path = Path(folder) / name
|
||||
data = path.read_bytes()
|
||||
stat = path.stat()
|
||||
state[path.relative_to(root).as_posix()] = (
|
||||
stat.st_size,
|
||||
stat.st_mtime_ns,
|
||||
hashlib.sha256(data).hexdigest(),
|
||||
)
|
||||
return state
|
||||
|
||||
|
||||
class ThumbnailJobCase(ThumbnailTestCase):
|
||||
"""Helpers: start, gate and wait for thumbnail jobs."""
|
||||
|
||||
async def _start(self, **body):
|
||||
resp = await self.client.post(JOB_URL, json=body)
|
||||
return resp.status, await resp.json()
|
||||
|
||||
async def _finished(self, job_id):
|
||||
job = self.api._jobs.get(job_id)
|
||||
await asyncio.wait_for(job.task, 10)
|
||||
return job
|
||||
|
||||
def _gate_generation(self):
|
||||
"""Block every file in the worker until released; returns (entered, release)."""
|
||||
entered = threading.Event()
|
||||
release = threading.Event()
|
||||
original = self.api._generate_one
|
||||
|
||||
def gated(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(10)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
self.api._generate_one = gated
|
||||
self.addCleanup(release.set)
|
||||
return entered, release
|
||||
|
||||
async def _wait(self, event):
|
||||
loop = asyncio.get_running_loop()
|
||||
self.assertTrue(await loop.run_in_executor(None, event.wait, 10))
|
||||
|
||||
@staticmethod
|
||||
def _watch_subscribers(job):
|
||||
"""(subscribed, left): threading events for a stream joining or leaving."""
|
||||
subscribed, left = threading.Event(), threading.Event()
|
||||
|
||||
class Watched(set):
|
||||
def add(self, item):
|
||||
super().add(item)
|
||||
subscribed.set()
|
||||
|
||||
def discard(self, item):
|
||||
super().discard(item)
|
||||
left.set()
|
||||
|
||||
job._subscribers = Watched(job._subscribers)
|
||||
return subscribed, left
|
||||
|
||||
def _status_lines(self):
|
||||
"""Every status line any job is given, in order."""
|
||||
from py.api.jobs import Job
|
||||
|
||||
lines = []
|
||||
original = Job.update
|
||||
|
||||
def update(job, **kwargs):
|
||||
if kwargs.get("status_line") is not None:
|
||||
lines.append(kwargs["status_line"])
|
||||
return original(job, **kwargs)
|
||||
|
||||
patcher = patch.object(Job, "update", update)
|
||||
patcher.start()
|
||||
self.addCleanup(patcher.stop)
|
||||
return lines
|
||||
|
||||
def _one_worker(self):
|
||||
original = PromptManagerConfig.WORKER_THREADS
|
||||
PromptManagerConfig.WORKER_THREADS = 1 # batches of 4
|
||||
self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original)
|
||||
|
||||
def _run_scan(self):
|
||||
gate = asyncio.Event()
|
||||
|
||||
async def runner(job):
|
||||
await gate.wait()
|
||||
|
||||
job = self.api._jobs.start(
|
||||
"scan", "Scan for metadata", admission="records", scope={}, runner=runner
|
||||
)
|
||||
|
||||
async def release():
|
||||
gate.set()
|
||||
await asyncio.wait_for(job.task, 5)
|
||||
|
||||
self.addAsyncCleanup(release)
|
||||
return job
|
||||
|
||||
|
||||
class TestThumbnailPlan(ThumbnailJobCase):
|
||||
async def test_plan_is_read_only_and_counts_cap(self):
|
||||
for name in ("a", "b", "c"):
|
||||
make_png(self.output_dir / f"{name}.png", size=(64, 64))
|
||||
fresh = make_png(self._thumb_path("a"))
|
||||
source_time = (self.output_dir / "a.png").stat().st_mtime
|
||||
os.utime(fresh, (source_time + 10, source_time + 10))
|
||||
before = _tree_state(self.comfy_dir)
|
||||
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 1):
|
||||
missing = await self.client.get(
|
||||
PLAN_URL, params={"mode": "missing", "size": "512", "quality": "high"}
|
||||
)
|
||||
rebuild = await self.client.get(
|
||||
PLAN_URL, params={"mode": "rebuild", "size": "256", "quality": "low"}
|
||||
)
|
||||
|
||||
self.assertEqual(missing.status, 200)
|
||||
self.assertEqual(
|
||||
await missing.json(),
|
||||
{
|
||||
"success": True,
|
||||
"targets": 1,
|
||||
"fresh": 1,
|
||||
"remaining": 1,
|
||||
"roots": ["output"],
|
||||
"mode": "missing",
|
||||
"size": 512,
|
||||
"quality": "high",
|
||||
},
|
||||
)
|
||||
data = await rebuild.json()
|
||||
# Rebuild regenerates fresh previews too: all three are work, one fits the cap.
|
||||
self.assertEqual((data["targets"], data["fresh"], data["remaining"]), (1, 1, 2))
|
||||
self.assertEqual(
|
||||
(data["mode"], data["size"], data["quality"]), ("rebuild", 256, "low")
|
||||
)
|
||||
self.assertEqual(_tree_state(self.comfy_dir), before)
|
||||
self.assertEqual(self.api._jobs.list(), [])
|
||||
|
||||
async def test_plan_defaults_to_missing_previews_at_the_concept_size(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
data = await (await self.client.get(PLAN_URL)).json()
|
||||
self.assertEqual(
|
||||
(data["mode"], data["size"], data["quality"], data["targets"]),
|
||||
("missing", 512, "medium", 1),
|
||||
)
|
||||
|
||||
async def test_size_bounds_validated(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
for params in (
|
||||
{"size": "127"},
|
||||
{"size": "2049"},
|
||||
{"size": "512.5"},
|
||||
{"size": "big"},
|
||||
{"mode": "all"},
|
||||
{"quality": "ultra"},
|
||||
):
|
||||
with self.subTest(params=params):
|
||||
resp = await self.client.get(PLAN_URL, params=params)
|
||||
body = await resp.json()
|
||||
self.assertEqual(resp.status, 400)
|
||||
self.assertEqual(
|
||||
(body["success"], body["code"]), (False, "invalid-request")
|
||||
)
|
||||
for body in (
|
||||
{"size": 127},
|
||||
{"size": 2049},
|
||||
{"size": "512"},
|
||||
{"size": 512.0},
|
||||
{"size": True},
|
||||
{"mode": "rebuild-all"},
|
||||
{"quality": None},
|
||||
[],
|
||||
):
|
||||
with self.subTest(body=body):
|
||||
resp = await self.client.post(JOB_URL, json=body)
|
||||
self.assertEqual(resp.status, 400)
|
||||
self.assertEqual((await resp.json())["code"], "invalid-request")
|
||||
for edge in (128, 2048):
|
||||
with self.subTest(edge=edge):
|
||||
resp = await self.client.get(PLAN_URL, params={"size": str(edge)})
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(self.api._jobs.list(), [])
|
||||
self.assertFalse(self._thumb_path("a").exists())
|
||||
|
||||
|
||||
class TestThumbnailJob(ThumbnailJobCase):
|
||||
async def test_job_generates_missing_previews_at_the_chosen_edge(self):
|
||||
make_png(self.output_dir / "a.png", size=(800, 400))
|
||||
make_png(self.output_dir / "sub" / "b.png", size=(64, 64))
|
||||
(self.output_dir / "bad.png").write_bytes(b"\x89PNG not really")
|
||||
|
||||
status, body = await self._start(mode="missing", size=200, quality="medium")
|
||||
|
||||
self.assertEqual(status, 202)
|
||||
summary = body["job"]
|
||||
self.assertEqual(
|
||||
(summary["kind"], summary["title"], summary["admission"]),
|
||||
("thumbnails", "Thumbnail generation", "guarded"),
|
||||
)
|
||||
self.assertTrue(summary["cancellable"])
|
||||
self.assertEqual(set(summary["counters"]), {"generated", "skipped", "errors"})
|
||||
self.assertEqual(
|
||||
summary["scope"], {"mode": "missing", "size": 200, "quality": "medium"}
|
||||
)
|
||||
job = await self._finished(summary["id"])
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertEqual(job.counters, {"generated": 2, "skipped": 0, "errors": 1})
|
||||
self.assertEqual((job.processed, job.total, job.progress), (3, 3, 100))
|
||||
self.assertEqual(job.scope["output_folders"], 1)
|
||||
with Image.open(self._thumb_path("a")) as thumb:
|
||||
self.assertEqual(thumb.size, (200, 100))
|
||||
rows = {row["path"]: row for row in job.outcomes["generated"]}
|
||||
self.assertEqual(set(rows), {"a.png", "sub/b.png"})
|
||||
self.assertEqual(
|
||||
(rows["a.png"]["root"], rows["a.png"]["root_index"]), ("output", 0)
|
||||
)
|
||||
self.assertEqual(job.outcomes["errors"][0]["path"], "bad.png")
|
||||
self.assertTrue(any("Thumbnails finished" in line for line in job.log))
|
||||
# Rows and log lines never carry the server's absolute layout.
|
||||
self.assertNotIn(str(self.comfy_dir), repr(job.detail()))
|
||||
|
||||
async def test_fresh_previews_are_counted_as_skipped(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
make_png(self.output_dir / "b.png")
|
||||
fresh = make_png(self._thumb_path("a"))
|
||||
source_time = (self.output_dir / "a.png").stat().st_mtime
|
||||
os.utime(fresh, (source_time + 10, source_time + 10))
|
||||
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.counters, {"generated": 1, "skipped": 1, "errors": 0})
|
||||
self.assertEqual([r["path"] for r in job.outcomes["skipped"]], ["a.png"])
|
||||
|
||||
async def test_rebuild_never_touches_originals(self):
|
||||
for name in ("a", "b"):
|
||||
make_png(self.output_dir / f"{name}.png", size=(400, 400))
|
||||
_, body = await self._start(size=128)
|
||||
await self._finished(body["job"]["id"])
|
||||
originals = {
|
||||
name: hashlib.sha256(
|
||||
(self.output_dir / f"{name}.png").read_bytes()
|
||||
).hexdigest()
|
||||
for name in ("a", "b")
|
||||
}
|
||||
|
||||
_, body = await self._start(mode="rebuild", size=256, quality="high")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.counters, {"generated": 2, "skipped": 0, "errors": 0})
|
||||
for name in ("a", "b"):
|
||||
digest = hashlib.sha256(
|
||||
(self.output_dir / f"{name}.png").read_bytes()
|
||||
).hexdigest()
|
||||
self.assertEqual(digest, originals[name], name)
|
||||
with Image.open(self._thumb_path(name)) as thumb:
|
||||
self.assertEqual(thumb.size, (256, 256))
|
||||
|
||||
async def test_rebuild_keeps_old_preview_when_generation_fails(self):
|
||||
source = make_png(self.output_dir / "a.png", size=(64, 64))
|
||||
_, body = await self._start()
|
||||
await self._finished(body["job"]["id"])
|
||||
preview = self._thumb_path("a").read_bytes()
|
||||
source.write_bytes(b"\x89PNG broken now")
|
||||
later = self._thumb_path("a").stat().st_mtime + 10
|
||||
os.utime(self._thumb_path("a"), (later, later)) # still fresh
|
||||
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
self.assertEqual(self._thumb_path("a").read_bytes(), preview)
|
||||
leftovers = [p.name for p in self._thumb_path("a").parent.iterdir()]
|
||||
self.assertEqual(leftovers, ["a_thumb.png"]) # no temp file left
|
||||
|
||||
async def test_rebuild_keeps_an_old_video_preview_instead_of_a_placeholder(self):
|
||||
clip = self.output_dir / "clip.mp4"
|
||||
clip.write_bytes(b"\x00")
|
||||
preview = make_png(self.output_dir / "thumbnails" / "clip_thumb.jpg")
|
||||
old = preview.read_bytes()
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
):
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
self.assertEqual(preview.read_bytes(), old)
|
||||
|
||||
async def test_quality_sets_the_encoder_not_the_edge(self):
|
||||
noise = bytes(
|
||||
(i * 7919 + (i >> 3) * 104729) % 251 for i in range(600 * 600 * 3)
|
||||
)
|
||||
with Image.frombytes("RGB", (600, 600), noise) as img:
|
||||
img.save(self.output_dir / "photo.jpg", "JPEG", quality=95)
|
||||
sizes = {}
|
||||
for quality in ("low", "high"):
|
||||
_, body = await self._start(mode="rebuild", size=300, quality=quality)
|
||||
await self._finished(body["job"]["id"])
|
||||
thumb = self._thumb_path("photo", ".jpg")
|
||||
with Image.open(thumb) as image:
|
||||
self.assertEqual(image.size, (300, 300))
|
||||
sizes[quality] = thumb.stat().st_size
|
||||
self.assertLess(sizes["low"], sizes["high"])
|
||||
|
||||
async def test_no_output_folder_fails_the_job(self):
|
||||
self.api._get_all_output_dirs = lambda: []
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.state, "failed")
|
||||
self.assertEqual(job.status_line, "ComfyUI output directory not found")
|
||||
|
||||
async def test_cancel_stops_at_file_boundary_keeping_written_previews(self):
|
||||
for i in range(6):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
original = PromptManagerConfig.WORKER_THREADS
|
||||
PromptManagerConfig.WORKER_THREADS = 1
|
||||
self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original)
|
||||
written = []
|
||||
real_write = images_module._write_image_thumbnail
|
||||
job_ids = []
|
||||
|
||||
def counting(src, dst, *args, **kwargs):
|
||||
real_write(src, dst, *args, **kwargs)
|
||||
written.append(dst)
|
||||
if len(written) == 2:
|
||||
self.assertTrue(self.api._jobs.cancel(job_ids[0]))
|
||||
|
||||
with patch.object(images_module, "_write_image_thumbnail", counting):
|
||||
_, body = await self._start()
|
||||
job_ids.append(body["job"]["id"])
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertEqual(job.counters["generated"], 2)
|
||||
self.assertEqual(len(written), 2)
|
||||
for dst in written:
|
||||
self.assertTrue(dst.is_file())
|
||||
made = sorted((self.output_dir / "thumbnails").glob("*_thumb.png"))
|
||||
self.assertEqual(made, sorted(written))
|
||||
self.assertTrue(any("Thumbnails cancelled" in line for line in job.log))
|
||||
|
||||
async def test_a_second_thumbnail_job_is_refused_naming_the_first(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
entered, release = self._gate_generation()
|
||||
_, first = await self._start()
|
||||
await self._wait(entered)
|
||||
|
||||
status, body = await self._start(mode="rebuild")
|
||||
|
||||
self.assertEqual((status, body["code"]), (409, "busy"))
|
||||
self.assertEqual(body["running"]["id"], first["job"]["id"])
|
||||
release.set()
|
||||
await self._finished(first["job"]["id"])
|
||||
|
||||
async def test_thumbnail_job_refused_while_scan_runs(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
self._run_scan()
|
||||
|
||||
status, body = await self._start()
|
||||
|
||||
self.assertEqual((status, body["code"]), (409, "busy"))
|
||||
self.assertEqual(body["running"]["title"], "Scan for metadata")
|
||||
self.assertFalse(self._thumb_path("a").exists())
|
||||
|
||||
async def test_scan_refused_while_thumbnail_job_runs(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
entered, release = self._gate_generation()
|
||||
_, started = await self._start()
|
||||
await self._wait(entered)
|
||||
|
||||
resp = await self.client.post("/prompt_manager/jobs/scan")
|
||||
body = await resp.json()
|
||||
|
||||
self.assertEqual((resp.status, body["code"]), (409, "busy"))
|
||||
self.assertEqual(body["running"]["title"], "Thumbnail generation")
|
||||
release.set()
|
||||
await self._finished(started["job"]["id"])
|
||||
|
||||
async def test_clear_refused_while_thumbnail_job_runs(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
kept = make_png(self.output_dir / "thumbnails" / "old_thumb.png")
|
||||
entered, release = self._gate_generation()
|
||||
_, started = await self._start()
|
||||
await self._wait(entered)
|
||||
|
||||
resp = await self.client.post("/prompt_manager/images/clear-thumbnails")
|
||||
|
||||
self.assertEqual(resp.status, 409)
|
||||
self.assertEqual(
|
||||
await resp.json(),
|
||||
{"success": False, "error": "Thumbnail generation already running"},
|
||||
)
|
||||
self.assertTrue(kept.exists())
|
||||
release.set()
|
||||
await self._finished(started["job"]["id"])
|
||||
|
||||
async def test_every_exit_sets_a_truthful_final_status_line(self):
|
||||
"""I3: finished, nothing to do, cancelled and failed each say what happened."""
|
||||
lines = self._status_lines()
|
||||
make_png(self.output_dir / "a.png")
|
||||
(self.output_dir / "bad.png").write_bytes(b"\x89PNG nope")
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.status_line, "1 preview written · 1 error")
|
||||
|
||||
# Every preview fresh: the job finishes with nothing to do.
|
||||
source_time = (self.output_dir / "a.png").stat().st_mtime
|
||||
bad_thumb = make_png(self._thumb_path("bad"))
|
||||
for thumb in (self._thumb_path("a"), bad_thumb):
|
||||
os.utime(thumb, (source_time + 10, source_time + 10))
|
||||
os.utime(self.output_dir / "bad.png", (source_time, source_time))
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual((job.state, job.total), ("finished", 0))
|
||||
self.assertEqual(
|
||||
job.status_line, "No previews to generate · 2 already up to date"
|
||||
)
|
||||
self.assertNotIn("Processing file 0 of 0", lines)
|
||||
|
||||
async def test_an_empty_library_says_there_is_nothing_to_preview(self):
|
||||
lines = self._status_lines()
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(
|
||||
(job.state, job.status_line), ("finished", "No images or videos to preview")
|
||||
)
|
||||
self.assertNotIn("Processing file 0 of 0", lines)
|
||||
|
||||
async def test_a_cancelled_job_says_what_it_kept(self):
|
||||
for i in range(6):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
self._one_worker()
|
||||
real_write = images_module._write_image_thumbnail
|
||||
job_ids = []
|
||||
written = []
|
||||
|
||||
def counting(src, dst, *args, **kwargs):
|
||||
real_write(src, dst, *args, **kwargs)
|
||||
written.append(dst)
|
||||
if len(written) == 1:
|
||||
self.api._jobs.cancel(job_ids[0])
|
||||
|
||||
with patch.object(images_module, "_write_image_thumbnail", counting):
|
||||
_, body = await self._start()
|
||||
job_ids.append(body["job"]["id"])
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertEqual(job.status_line, "Cancelled · 1 preview written")
|
||||
|
||||
async def test_an_internal_error_says_the_job_failed(self):
|
||||
def broken(*_args, **_kwargs):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
self.api._plan_gallery_thumbnails = broken
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(
|
||||
(job.state, job.status_line),
|
||||
("failed", "Thumbnail generation stopped on an internal error."),
|
||||
)
|
||||
|
||||
async def test_cancel_during_planning_stops_the_walk_and_starts_nothing(self):
|
||||
"""M2: the walk checks Cancel per file; no ``start`` frame is published."""
|
||||
for i in range(10):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
self.addCleanup(release.set)
|
||||
checked = []
|
||||
real_fresh = images_module._thumbnail_is_fresh
|
||||
|
||||
def gated_fresh(src, dst):
|
||||
checked.append(src.name)
|
||||
if len(checked) == 1:
|
||||
entered.set()
|
||||
release.wait(10)
|
||||
return real_fresh(src, dst)
|
||||
|
||||
with patch.object(images_module, "_thumbnail_is_fresh", gated_fresh):
|
||||
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
|
||||
await self._wait(entered)
|
||||
job = self.api._jobs.active("thumbnails")
|
||||
self.assertTrue(self.api._jobs.cancel(job.id))
|
||||
release.set()
|
||||
events = parse_sse(await (await stream).text())
|
||||
await asyncio.wait_for(job.task, 10)
|
||||
|
||||
self.assertEqual(checked, ["p0.png"]) # the walk stopped at the next file
|
||||
self.assertEqual([name for name, _ in events], ["job", "status", "error"])
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertEqual(job.status_line, "Cancelled before any preview was written")
|
||||
self.assertEqual(job.total, 0)
|
||||
self.assertFalse((self.output_dir / "thumbnails").exists())
|
||||
|
||||
async def test_a_locked_preview_is_a_readable_coded_error(self):
|
||||
"""M6: Windows refuses to replace a preview another process has open."""
|
||||
make_png(self.output_dir / "a.png")
|
||||
locked = PermissionError(13, "Access is denied")
|
||||
locked.winerror = 5
|
||||
|
||||
def refuse(*_args, **_kwargs):
|
||||
raise locked
|
||||
|
||||
with patch.object(images_module, "_write_image_thumbnail", refuse):
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
[row] = job.outcomes["errors"]
|
||||
self.assertEqual(row["detail"], images_module.PREVIEW_IN_USE)
|
||||
self.assertNotIn("WinError", repr(job.detail()))
|
||||
self.assertTrue(any(images_module.PREVIEW_IN_USE in line for line in job.log))
|
||||
|
||||
async def test_a_locked_video_preview_is_the_same_coded_error(self):
|
||||
clip = self.output_dir / "clip.mp4"
|
||||
clip.write_bytes(b"\x00")
|
||||
locked = PermissionError(13, "Access is denied")
|
||||
locked.winerror = 32
|
||||
|
||||
def refuse(*_args, **_kwargs):
|
||||
raise locked
|
||||
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
patch.object(images_module, "_atomic_save", refuse),
|
||||
):
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
[row] = job.outcomes["errors"]
|
||||
self.assertEqual(row["detail"], images_module.PREVIEW_IN_USE)
|
||||
|
||||
async def test_a_permission_error_off_windows_names_the_permission(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
|
||||
def refuse(*_args, **_kwargs):
|
||||
raise PermissionError(13, "Permission denied")
|
||||
|
||||
with patch.object(images_module, "_write_image_thumbnail", refuse):
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
[row] = job.outcomes["errors"]
|
||||
self.assertEqual(row["detail"], images_module.PREVIEW_NOT_WRITABLE)
|
||||
|
||||
async def test_ffmpeg_fallback_keeps_quality_and_aspect_ratio(self):
|
||||
"""M1: without OpenCV, ffmpeg keeps the chosen quality and the aspect ratio."""
|
||||
(self.output_dir / "clip.mp4").write_bytes(b"\x00")
|
||||
commands = []
|
||||
|
||||
def fake_ffmpeg(cmd, **_kwargs):
|
||||
commands.append(cmd)
|
||||
Path(cmd[-1]).write_bytes(b"jpeg")
|
||||
return type("Done", (), {"returncode": 0, "stderr": ""})()
|
||||
|
||||
for quality, qscale in (("high", "2"), ("medium", "4"), ("low", "8")):
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=fake_ffmpeg),
|
||||
):
|
||||
_, body = await self._start(mode="rebuild", size=640, quality=quality)
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.counters["generated"], 1, quality)
|
||||
cmd = commands[-1]
|
||||
self.assertNotIn("-s", cmd)
|
||||
self.assertEqual(
|
||||
cmd[cmd.index("-vf") + 1],
|
||||
"scale=640:640:force_original_aspect_ratio=decrease",
|
||||
)
|
||||
self.assertEqual(cmd[cmd.index("-q:v") + 1], qscale, quality)
|
||||
|
||||
async def test_the_placeholder_uses_the_chosen_quality(self):
|
||||
(self.output_dir / "clip.mp4").write_bytes(b"\x00")
|
||||
saved = []
|
||||
real_save = images_module._atomic_save
|
||||
|
||||
def recording(img, dst, fmt=None, **kwargs):
|
||||
saved.append(kwargs)
|
||||
return real_save(img, dst, fmt, **kwargs)
|
||||
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
patch.object(images_module, "_atomic_save", recording),
|
||||
):
|
||||
_, body = await self._start(quality="low")
|
||||
await self._finished(body["job"]["id"])
|
||||
self.assertEqual(saved, [images_module.THUMBNAIL_ENCODINGS["low"]])
|
||||
|
||||
|
||||
# The legacy stream's frames and their keys, as the gallery page reads them.
|
||||
LEGACY_KEYS = {
|
||||
"job": {"id", "started"},
|
||||
"status": {"phase", "message"},
|
||||
"start": {
|
||||
"total_images",
|
||||
"skipped",
|
||||
"remaining",
|
||||
"phase",
|
||||
"image_count",
|
||||
"video_count",
|
||||
"message",
|
||||
},
|
||||
"progress": {
|
||||
"processed",
|
||||
"total_images",
|
||||
"generated",
|
||||
"skipped",
|
||||
"error_count",
|
||||
"percentage",
|
||||
"rate",
|
||||
"eta",
|
||||
"elapsed",
|
||||
"current_file",
|
||||
"file_type",
|
||||
"action",
|
||||
},
|
||||
"complete": {
|
||||
"count",
|
||||
"skipped",
|
||||
"total_images",
|
||||
"remaining",
|
||||
"errors",
|
||||
"error_count",
|
||||
"elapsed_time",
|
||||
"processing_rate",
|
||||
"message",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class TestLegacyStreamOnTheJob(ThumbnailJobCase):
|
||||
async def test_legacy_progress_events_unchanged_on_success(self):
|
||||
make_png(self.output_dir / "a.png", size=(400, 400))
|
||||
make_png(self.output_dir / "b.png")
|
||||
|
||||
resp = await self.client.post(f"{self.PROGRESS_URL}?quality=low")
|
||||
events = parse_sse(await resp.text())
|
||||
|
||||
self.assertEqual(
|
||||
[name for name, _ in events],
|
||||
["job", "status", "start", "progress", "progress", "complete"],
|
||||
)
|
||||
for name, data in events:
|
||||
self.assertEqual(set(data), LEGACY_KEYS[name], name)
|
||||
complete = events[-1][1]
|
||||
self.assertEqual((complete["count"], complete["total_images"]), (2, 2))
|
||||
self.assertEqual(
|
||||
events[1][1],
|
||||
{"phase": "scanning", "message": "Scanning for images and videos..."},
|
||||
)
|
||||
with Image.open(self._thumb_path("a")) as thumb:
|
||||
self.assertEqual(max(thumb.size), 150) # the legacy preset still sizes
|
||||
[job] = self.api._jobs.list()
|
||||
self.assertEqual(
|
||||
(job.kind, job.title, job.state, job.cancellable),
|
||||
("thumbnails", "Thumbnail generation", "finished", True),
|
||||
)
|
||||
self.assertEqual(job.counters["generated"], 2)
|
||||
# I1: the first frame names the job this stream started, for its Cancel.
|
||||
self.assertEqual(events[0][1], {"id": job.id, "started": True})
|
||||
|
||||
async def test_legacy_file_errors_still_stream(self):
|
||||
make_png(self.output_dir / "good.png")
|
||||
(self.output_dir / "bad.png").write_bytes(b"\x89PNG nope")
|
||||
|
||||
resp = await self.client.post(self.PROGRESS_URL)
|
||||
events = parse_sse(await resp.text())
|
||||
|
||||
self.assertEqual(
|
||||
[d for e, d in events if e == "file_error"],
|
||||
[{"file": "bad.png", "error": "Not a valid image file"}],
|
||||
)
|
||||
|
||||
async def test_thumbnail_job_survives_client_disconnect(self):
|
||||
"""I2: the stream sees its client go after batch 1; every batch still runs.
|
||||
|
||||
One worker thread makes batches of 4, so 9 files take three batches. The
|
||||
first file waits until the client is gone; the second batch waits until
|
||||
the stream has seen it go (its subscription dropped) and records whether
|
||||
the job was still running then. A stream that stopped the work on
|
||||
disconnect would leave 4 previews; the job writes all 9.
|
||||
"""
|
||||
for i in range(9):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
self._one_worker()
|
||||
first, closed = threading.Event(), threading.Event()
|
||||
state = {"calls": 0, "left": None, "running_when_left": None}
|
||||
original = self.api._generate_one
|
||||
|
||||
def gated(*args, **kwargs):
|
||||
state["calls"] += 1
|
||||
if state["calls"] == 1:
|
||||
first.set()
|
||||
closed.wait(10)
|
||||
elif state["calls"] == 5: # the second batch
|
||||
state["running_when_left"] = state["left"].wait(10)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
self.api._generate_one = gated
|
||||
self.addCleanup(closed.set)
|
||||
|
||||
resp = await self.client.post(self.PROGRESS_URL)
|
||||
self.assertEqual(resp.status, 200)
|
||||
await self._wait(first)
|
||||
job = self.api._jobs.active("thumbnails")
|
||||
_subscribed, state["left"] = self._watch_subscribers(job)
|
||||
resp.close() # the gallery tab went away
|
||||
closed.set()
|
||||
await self._wait(state["left"]) # the server saw the client go
|
||||
self.assertEqual(job.subscriber_count, 0)
|
||||
await asyncio.wait_for(job.task, 10)
|
||||
|
||||
self.assertEqual(job.counters["generated"], 9)
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertTrue(state["running_when_left"])
|
||||
for i in range(9):
|
||||
self.assertTrue(self._thumb_path(f"p{i}").is_file())
|
||||
|
||||
async def test_legacy_stream_attaches_to_a_running_job(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
entered, release = self._gate_generation()
|
||||
_, started = await self._start(mode="missing", size=256)
|
||||
await self._wait(entered)
|
||||
|
||||
job = self.api._jobs.get(started["job"]["id"])
|
||||
subscribed, _left = self._watch_subscribers(job)
|
||||
attached = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
|
||||
await self._wait(subscribed) # M4: an event with a ceiling, never a poll
|
||||
release.set()
|
||||
events = parse_sse(await (await attached).text())
|
||||
|
||||
self.assertEqual(events[-1][0], "complete")
|
||||
# I1: the stream names the job it did not start: the gallery never cancels it.
|
||||
self.assertEqual(events[0], ("job", {"id": job.id, "started": False}))
|
||||
# One job: the stream followed the running one with its own options.
|
||||
[listed] = self.api._jobs.list()
|
||||
self.assertIs(listed, job)
|
||||
self.assertEqual(job.scope["size"], 256)
|
||||
|
||||
async def test_cancel_from_activity_ends_the_legacy_stream_with_an_error(self):
|
||||
for i in range(2):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
entered, release = self._gate_generation()
|
||||
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
|
||||
await self._wait(entered)
|
||||
job = self.api._jobs.active("thumbnails")
|
||||
self.assertTrue(self.api._jobs.cancel(job.id))
|
||||
release.set()
|
||||
|
||||
events = parse_sse(await (await stream).text())
|
||||
|
||||
self.assertEqual(events[-1][0], "error")
|
||||
self.assertEqual(events[-1][1]["error"], "Thumbnail generation was cancelled")
|
||||
await asyncio.wait_for(job.task, 10)
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
|
||||
async def test_a_job_started_while_the_stream_prepares_is_attached(self):
|
||||
"""M3: a job that starts during ``prepare`` is followed, not refused."""
|
||||
from aiohttp import web
|
||||
|
||||
make_png(self.output_dir / "a.png")
|
||||
entered, release = self._gate_generation()
|
||||
real_prepare = web.StreamResponse.prepare
|
||||
raced = []
|
||||
|
||||
async def prepare(response, request):
|
||||
if not raced and type(response) is web.StreamResponse:
|
||||
options, _ = thumbnail_options({"size": 256})
|
||||
raced.append(self.api._start_thumbnail_job(options))
|
||||
return await real_prepare(response, request)
|
||||
|
||||
with patch.object(web.StreamResponse, "prepare", prepare):
|
||||
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
|
||||
await self._wait(entered)
|
||||
release.set()
|
||||
events = parse_sse(await (await stream).text())
|
||||
|
||||
[job] = raced
|
||||
self.assertEqual(events[0], ("job", {"id": job.id, "started": False}))
|
||||
self.assertEqual(events[-1][0], "complete")
|
||||
self.assertEqual(self.api._jobs.list(), [job])
|
||||
|
||||
async def test_a_refused_stream_names_no_job(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
self._run_scan()
|
||||
events = parse_sse(await (await self.client.post(self.PROGRESS_URL)).text())
|
||||
self.assertEqual(
|
||||
events,
|
||||
[
|
||||
(
|
||||
"error",
|
||||
{
|
||||
"error": "Scan for metadata already running",
|
||||
"message": "Scan for metadata already running",
|
||||
},
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+5
-35
@@ -42,10 +42,10 @@
|
||||
"statements": 98
|
||||
},
|
||||
"gallery.js": {
|
||||
"lines": 45,
|
||||
"branches": 64,
|
||||
"functions": 48,
|
||||
"statements": 45
|
||||
"lines": 41,
|
||||
"branches": 63,
|
||||
"functions": 44,
|
||||
"statements": 41
|
||||
},
|
||||
"html-escape.js": {
|
||||
"lines": 98,
|
||||
@@ -161,12 +161,6 @@
|
||||
"functions": 98,
|
||||
"statements": 97
|
||||
},
|
||||
"workspace/activity-kinds.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 95,
|
||||
"functions": 98,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/activity-scan.mjs": {
|
||||
"lines": 96,
|
||||
"branches": 78,
|
||||
@@ -181,7 +175,7 @@
|
||||
},
|
||||
"workspace/activity.mjs": {
|
||||
"lines": 96,
|
||||
"branches": 91,
|
||||
"branches": 87,
|
||||
"functions": 98,
|
||||
"statements": 96
|
||||
},
|
||||
@@ -221,12 +215,6 @@
|
||||
"functions": 98,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/image-card.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 98,
|
||||
"functions": 98,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/image-layout.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 98,
|
||||
@@ -449,18 +437,6 @@
|
||||
"functions": 80,
|
||||
"statements": 95
|
||||
},
|
||||
"workspace/review-state.mjs": {
|
||||
"lines": 96,
|
||||
"branches": 84,
|
||||
"functions": 98,
|
||||
"statements": 96
|
||||
},
|
||||
"workspace/review.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 89,
|
||||
"functions": 93,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/saved-views.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 95,
|
||||
@@ -515,12 +491,6 @@
|
||||
"functions": 98,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/thumbnail-review.mjs": {
|
||||
"lines": 96,
|
||||
"branches": 93,
|
||||
"functions": 98,
|
||||
"statements": 96
|
||||
},
|
||||
"workspace/toast.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 98,
|
||||
|
||||
+1
-39
@@ -27,7 +27,7 @@ import os
|
||||
import queue
|
||||
import time
|
||||
import threading
|
||||
from typing import Callable, Optional, Dict, Any, Tuple
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
from watchdog.observers import Observer
|
||||
from watchdog.events import FileSystemEventHandler
|
||||
|
||||
@@ -75,10 +75,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
|
||||
self.prompt_tracker = prompt_tracker
|
||||
self.metadata_extractor = ComfyUIMetadataExtractor()
|
||||
self.logger = get_logger("prompt_manager.image_monitor")
|
||||
# The Needs review pass for ambiguous outputs (the API's
|
||||
# ``record_review_file``, wired at start-up): a linked output is
|
||||
# never queued and its pending items leave the queue. None: no pass.
|
||||
self.review_recorder: Optional[Callable[[str], str]] = None
|
||||
|
||||
# Read from GalleryConfig if available, otherwise use defaults
|
||||
try:
|
||||
@@ -583,29 +579,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to link image to prompt: {e}")
|
||||
return
|
||||
if resolution["state"] == "ambiguous":
|
||||
self._record_review(image_path)
|
||||
|
||||
def _record_review(self, image_path: str) -> None:
|
||||
"""The Needs review pass for an ambiguous output, best effort.
|
||||
|
||||
The output is linked by now, so it is never queued; a pending item
|
||||
an earlier scan left for its path leaves the queue. The link above
|
||||
stays whatever happens here: a failure is logged, never raised.
|
||||
"""
|
||||
recorder = getattr(self, "review_recorder", None)
|
||||
if recorder is None:
|
||||
return
|
||||
try:
|
||||
outcome = recorder(image_path)
|
||||
except Exception:
|
||||
self.logger.warning(
|
||||
f"{os.path.basename(image_path)} could not be added to Needs review",
|
||||
exc_info=True,
|
||||
)
|
||||
return
|
||||
self.logger.debug(f"Needs review {outcome}: {os.path.basename(image_path)}")
|
||||
|
||||
|
||||
class ImageMonitor:
|
||||
@@ -635,16 +608,6 @@ class ImageMonitor:
|
||||
self.handler = None
|
||||
self.monitored_directories = []
|
||||
self.logger = get_logger("prompt_manager.image_monitor")
|
||||
self.review_recorder: Optional[Callable[[str], str]] = None
|
||||
|
||||
def set_review_recorder(self, recorder: Optional[Callable[[str], str]]) -> None:
|
||||
"""Record ambiguous outputs through ``recorder`` (None stops recording).
|
||||
|
||||
Applies to the running handler and to every handler started later.
|
||||
"""
|
||||
self.review_recorder = recorder
|
||||
if self.handler is not None:
|
||||
self.handler.review_recorder = recorder
|
||||
|
||||
def start_monitoring(self, output_directories: Optional[list] = None):
|
||||
"""
|
||||
@@ -702,7 +665,6 @@ class ImageMonitor:
|
||||
# Only a started observer becomes self.observer, so a failed start never
|
||||
# leaves the monitor stuck "already running" with nothing to stop.
|
||||
handler = ImageGenerationHandler(self.db_manager, self.prompt_tracker)
|
||||
handler.review_recorder = self.review_recorder
|
||||
observer = Observer()
|
||||
scheduled = []
|
||||
for output_dir in valid_directories:
|
||||
|
||||
+16
-46
@@ -46,10 +46,6 @@ DEFAULT_MAX_BYTES = 512 * 2**20
|
||||
DEFAULT_MAX_WORKERS = 2
|
||||
DEFAULT_IDLE_SECONDS = 900
|
||||
DEFAULT_HARD_SECONDS = 1800
|
||||
# Background recording (Needs review from the scan and the monitor) may use
|
||||
# every worker and capture slot but the last one, which stays free for an
|
||||
# interactive Tag dialog or Apply; it waits this long for room before failing.
|
||||
BACKGROUND_WAIT_SECONDS = 2.0
|
||||
MAX_INFERENCE_SIDE = 4096
|
||||
MAX_INFERENCE_PIXELS = 16_777_216
|
||||
MAX_RELATIVE_LENGTH = 1024
|
||||
@@ -84,7 +80,7 @@ class CaptureError(Exception):
|
||||
|
||||
|
||||
class CaptureCapacity(CaptureError):
|
||||
"""Count, byte or worker admission failed (interactive: nonblocking)."""
|
||||
"""Count, byte or worker admission failed (nonblocking)."""
|
||||
|
||||
code = "review-capacity"
|
||||
|
||||
@@ -224,8 +220,6 @@ class CaptureStore:
|
||||
# mkdtemp creates the directory with owner-only permissions.
|
||||
self.directory = tempfile.mkdtemp(prefix=_DIRECTORY_PREFIX, dir=parent)
|
||||
self._lock = threading.Lock()
|
||||
# Signalled whenever a worker or a capture slot frees up.
|
||||
self._room = threading.Condition(self._lock)
|
||||
self._entries: dict[str, _Entry] = {}
|
||||
self._completed_bytes = 0
|
||||
self._pending = 0
|
||||
@@ -244,14 +238,8 @@ class CaptureStore:
|
||||
epoch: str,
|
||||
scope: str,
|
||||
root_index: int = 0,
|
||||
background: bool = False,
|
||||
) -> Capture:
|
||||
"""Copy and hash one rooted media file into a private capture.
|
||||
|
||||
A ``background`` capture never takes the last worker or capture slot
|
||||
and waits up to ``BACKGROUND_WAIT_SECONDS`` for room; an interactive
|
||||
one is admitted or refused at once.
|
||||
"""
|
||||
"""Copy and hash one rooted media file into a private capture."""
|
||||
if isinstance(root_index, bool) or not isinstance(root_index, int):
|
||||
raise CaptureDenied("Invalid gallery root")
|
||||
if root_index < 0:
|
||||
@@ -262,7 +250,7 @@ class CaptureStore:
|
||||
parts, extension = _relative_parts(relative)
|
||||
candidate, locator = _resolve(root, parts)
|
||||
before = self._pre_stat(candidate)
|
||||
self._admit(before.st_size, reserve=True, background=background)
|
||||
self._admit(before.st_size, reserve=True)
|
||||
try:
|
||||
capture = self._copy(
|
||||
candidate, locator, extension, before, root_index, epoch, scope
|
||||
@@ -398,52 +386,35 @@ class CaptureStore:
|
||||
|
||||
# -- admission -------------------------------------------------------------
|
||||
|
||||
def _admit(self, size: int, *, reserve: bool, background: bool = False) -> None:
|
||||
def _admit(self, size: int, *, reserve: bool) -> None:
|
||||
try:
|
||||
self._admit_locked(size, reserve=reserve, background=background)
|
||||
self._admit_locked(size, reserve=reserve)
|
||||
finally:
|
||||
self._flush()
|
||||
|
||||
def _admit_locked(self, size: int, *, reserve: bool, background: bool) -> None:
|
||||
deadline = time.monotonic() + BACKGROUND_WAIT_SECONDS
|
||||
def _admit_locked(self, size: int, *, reserve: bool) -> None:
|
||||
with self._lock:
|
||||
while True:
|
||||
if self._closed:
|
||||
raise RuntimeError("The capture store is closed")
|
||||
self._sweep_locked(self._clock())
|
||||
try:
|
||||
self._check_room_locked(size, reserve, background)
|
||||
break
|
||||
except CaptureCapacity:
|
||||
remaining = deadline - time.monotonic()
|
||||
if not background or remaining <= 0:
|
||||
raise
|
||||
self._room.wait(remaining)
|
||||
if self._closed:
|
||||
raise RuntimeError("The capture store is closed")
|
||||
self._sweep_locked(self._clock())
|
||||
if self._workers >= self.max_workers:
|
||||
raise CaptureCapacity("Too many captures are in progress")
|
||||
if reserve:
|
||||
if len(self._entries) + self._pending >= self.max_captures:
|
||||
raise CaptureCapacity("Too many captures are held")
|
||||
reserved = self._completed_bytes + self._pending_bytes
|
||||
if reserved + size > self.max_bytes:
|
||||
raise CaptureCapacity("Capture byte budget exhausted")
|
||||
self._pending += 1
|
||||
self._pending_bytes += size
|
||||
self._workers += 1
|
||||
|
||||
def _check_room_locked(self, size: int, reserve: bool, background: bool) -> None:
|
||||
"""Raise ``CaptureCapacity`` unless one more capture or verify fits."""
|
||||
spare = 1 if background else 0
|
||||
if self._workers + spare >= self.max_workers:
|
||||
raise CaptureCapacity("Too many captures are in progress")
|
||||
if reserve:
|
||||
held = len(self._entries) + self._pending
|
||||
if held + spare >= self.max_captures:
|
||||
raise CaptureCapacity("Too many captures are held")
|
||||
reserved = self._completed_bytes + self._pending_bytes
|
||||
if reserved + size > self.max_bytes:
|
||||
raise CaptureCapacity("Capture byte budget exhausted")
|
||||
|
||||
def _release_admission(self, size: int, *, reserve: bool) -> None:
|
||||
with self._lock:
|
||||
self._workers -= 1
|
||||
if reserve:
|
||||
self._pending -= 1
|
||||
self._pending_bytes -= size
|
||||
self._room.notify_all()
|
||||
last_out = self._closed and self._pending == 0 and self._workers == 0
|
||||
if last_out:
|
||||
self._remove_directory()
|
||||
@@ -486,7 +457,6 @@ class CaptureStore:
|
||||
if self._entries.pop(entry.capture.token, None) is not None:
|
||||
self._completed_bytes -= entry.capture.size
|
||||
self._stale.append(entry.capture.path)
|
||||
self._room.notify_all()
|
||||
|
||||
def _flush(self) -> None:
|
||||
"""Unlink files released under the lock, without holding it (slow I/O)."""
|
||||
|
||||
@@ -39,12 +39,6 @@ from .pair_execution import ExecutionEvidenceError, read_pair_execution
|
||||
|
||||
_LOGGER = logging.getLogger("prompt_manager.prompt_pair_resolver")
|
||||
|
||||
# The ingestion resolver's version, recorded in pair-link provenance and on
|
||||
# Needs review items. Bump it when a change would resolve the same metadata
|
||||
# otherwise; a pending item stays applicable while its chosen candidate is
|
||||
# still offered by the captured bytes.
|
||||
RESOLVER_VERSION = 1
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class OutputAdapter:
|
||||
|
||||
@@ -1,299 +0,0 @@
|
||||
"""Capture-bound Needs review items for ambiguous files (#249 E2).
|
||||
|
||||
The output scan and the live monitor hand every file the ingestion resolver
|
||||
finds ``ambiguous`` to :func:`record_ambiguous_file`. It captures the file
|
||||
(the bounded private copy of ``utils.media_capture``, hashed while copied),
|
||||
reads the metadata again from the capture, resolves it with the default
|
||||
(ingestion) profile and stores only an ``ambiguous`` result, bound to the
|
||||
capture's byte record. The evidence a reviewer chooses from is therefore the
|
||||
evidence of exactly those bytes. An ambiguous result is stored only when it
|
||||
is ``reviewable``: at least two candidates carry a resolved prompt text. With
|
||||
none the file could not be traced; with exactly one, the untraced branches
|
||||
may still have made the file, so that text is not certain and the file stays
|
||||
unresolved (the resolver never flattens unknown semantics into a guess).
|
||||
Nothing here links a file, loads a model or decodes pixels; the capture is
|
||||
released before the database write.
|
||||
|
||||
Evidence is small on purpose (startup validates every row): the resolver's
|
||||
reason and, per candidate sampler, ``{sampler_id, state, reason, positive,
|
||||
negative}`` where each side is ``{text, storage_text, provenance}``. A
|
||||
candidate's ``hash`` is the SHA-256 of its canonical JSON without the hash.
|
||||
More than ``MAX_CANDIDATES`` candidates or evidence over the 8 MiB bound make
|
||||
a ``limited`` item without candidates (a result without any candidate is not
|
||||
``reviewable`` and is never stored).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
from ..database.library_commands import LibraryCommandError
|
||||
from ..database.review_schema import (
|
||||
MAX_CANDIDATES,
|
||||
MIN_CHOOSABLE,
|
||||
canonical_evidence,
|
||||
choosable,
|
||||
)
|
||||
from .media_capture import CaptureCapacity, CaptureError
|
||||
from .prompt_inspector import MetadataReadError, read_inspection_file
|
||||
from .prompt_pair_resolver import RESOLVER_VERSION, resolve_prompt_pair
|
||||
except ImportError: # imported with the repo root on sys.path
|
||||
from database.library_commands import LibraryCommandError
|
||||
from database.review_schema import (
|
||||
MAX_CANDIDATES,
|
||||
MIN_CHOOSABLE,
|
||||
canonical_evidence,
|
||||
choosable,
|
||||
)
|
||||
from utils.media_capture import CaptureCapacity, CaptureError
|
||||
from utils.prompt_inspector import MetadataReadError, read_inspection_file
|
||||
from utils.prompt_pair_resolver import RESOLVER_VERSION, resolve_prompt_pair
|
||||
|
||||
logger = logging.getLogger("prompt_manager.review_capture")
|
||||
|
||||
# Capture scope label: review captures are never handed to a client.
|
||||
REVIEW_SCOPE = "needs-review"
|
||||
# Nested ambiguous stages are flattened at most this deep.
|
||||
MAX_CANDIDATE_DEPTH = 8
|
||||
|
||||
|
||||
def record_ambiguous_file(db, store, root, root_index: int, relative: str) -> str:
|
||||
"""Persist the ambiguous evidence of one rooted file.
|
||||
|
||||
Returns ``recorded`` (new item with candidates), ``limited`` (new item
|
||||
without candidates), ``duplicate`` (these bytes already carry this
|
||||
evidence, or the file's current stat is the verified fingerprint of
|
||||
bytes an item cites, so it is not even copied, and that item is still
|
||||
pending), ``decided`` (as ``duplicate``, but the item carries a decision,
|
||||
so the file is not in the queue), ``untraced`` (ambiguous in its captured
|
||||
bytes, but not ``reviewable``: nothing to choose, nothing stored) or
|
||||
``skipped`` (the file
|
||||
changed, vanished, is not ambiguous in its captured bytes, or storage was
|
||||
restored meanwhile).
|
||||
``CaptureCapacity`` and database errors propagate to the caller, which
|
||||
logs and counts them.
|
||||
"""
|
||||
spelled_path, queued = _queued_spelling(db, root, relative)
|
||||
if queued is not None:
|
||||
return _already_queued(db, *queued)
|
||||
epoch = db.get_execution_epoch()
|
||||
try:
|
||||
# Background: the last worker stays free for an interactive request.
|
||||
capture = store.capture(
|
||||
root,
|
||||
relative,
|
||||
epoch=epoch,
|
||||
scope=REVIEW_SCOPE,
|
||||
root_index=root_index,
|
||||
background=True,
|
||||
)
|
||||
except CaptureCapacity:
|
||||
raise
|
||||
except CaptureError as error:
|
||||
logger.debug("Review capture skipped (%s): %s", error.code, error)
|
||||
return "skipped"
|
||||
try:
|
||||
captured = captured_review(store, capture)
|
||||
finally:
|
||||
store.release(capture.token)
|
||||
resolution = captured[1] if captured is not None else None
|
||||
if resolution is None or resolution.get("state") != "ambiguous":
|
||||
return "skipped"
|
||||
if not reviewable(resolution):
|
||||
return "untraced"
|
||||
evidence, count, limited = review_evidence(resolution)
|
||||
try:
|
||||
inserted = db.record_review_item(
|
||||
locator=capture.locator,
|
||||
spelled_path=spelled_path,
|
||||
sha256=capture.sha256,
|
||||
size=capture.size,
|
||||
mtime_ns=capture.mtime_ns,
|
||||
epoch=capture.epoch,
|
||||
resolver_version=RESOLVER_VERSION,
|
||||
evidence=evidence,
|
||||
candidate_count=count,
|
||||
limited=limited,
|
||||
)
|
||||
except LibraryCommandError:
|
||||
return "skipped"
|
||||
if not inserted:
|
||||
return _already_queued(db, capture.locator, capture.size, capture.mtime_ns)
|
||||
return "limited" if limited else "recorded"
|
||||
|
||||
|
||||
def _already_queued(db, locator: str, size: int, mtime_ns: int) -> str:
|
||||
"""``duplicate`` while the existing item is pending, else ``decided``."""
|
||||
if db.review_pending_at(locator, size, mtime_ns, RESOLVER_VERSION):
|
||||
return "duplicate"
|
||||
return "decided"
|
||||
|
||||
|
||||
def _queued_spelling(db, root, relative: str) -> tuple[str, tuple | None]:
|
||||
"""The file's spelling, and ``(locator, size, mtime_ns)`` when queued.
|
||||
|
||||
The second value is None unless the file's current stat is already
|
||||
queued.
|
||||
|
||||
The spelling is the capture's own candidate path (the real root joined
|
||||
with the listed relative path), so it folds to the capture's locator. A
|
||||
file whose size and ``mtime_ns`` equal the verified fingerprint of bytes
|
||||
an item already cites is not copied or hashed again (maintainer ruling:
|
||||
the rescan stays cheap and the file stays in the default scope); its
|
||||
items take the current spelling. A stat failure leaves the decision to
|
||||
the capture.
|
||||
"""
|
||||
spelled = os.path.join(os.path.realpath(os.fspath(root)), *relative.split("/"))
|
||||
try:
|
||||
facts = os.lstat(spelled)
|
||||
except OSError:
|
||||
return spelled, None
|
||||
locator = os.path.normcase(spelled)
|
||||
if db.review_fingerprint_queued(
|
||||
locator, facts.st_size, facts.st_mtime_ns, RESOLVER_VERSION
|
||||
):
|
||||
# Not copied again, but a case-only rename keeps the stat: record
|
||||
# the current spelling so the item's target still opens.
|
||||
db.refresh_review_spelling(locator, spelled)
|
||||
return spelled, (locator, facts.st_size, facts.st_mtime_ns)
|
||||
return spelled, None
|
||||
|
||||
|
||||
def captured_review(store, capture) -> tuple[dict, dict] | None:
|
||||
"""``(metadata, resolution)`` of the capture's own bytes, else None.
|
||||
|
||||
``metadata`` is what the scan hands the ingester: prompt and workflow
|
||||
JSON-decoded and ``file_info`` (size, dimensions, format) from the
|
||||
capture. None when the capture is gone or carries no prompt metadata.
|
||||
"""
|
||||
if store.acquire(capture.token) is None:
|
||||
return None
|
||||
try:
|
||||
try:
|
||||
read = read_inspection_file(Path(capture.path))
|
||||
except MetadataReadError:
|
||||
return None
|
||||
finally:
|
||||
store.release_use(capture.token)
|
||||
canonical = dict(read.metadata)
|
||||
for key in ("prompt", "workflow"):
|
||||
if isinstance(canonical.get(key), str):
|
||||
try:
|
||||
canonical[key] = json.loads(canonical[key])
|
||||
except (ValueError, TypeError, RecursionError):
|
||||
# Malformed metadata stays as text for the resolver to reject.
|
||||
pass
|
||||
if not canonical.get("prompt") and not canonical.get("parameters"):
|
||||
return None
|
||||
info = {"size": capture.size}
|
||||
if read.width is not None and read.height is not None:
|
||||
info["dimensions"] = [read.width, read.height]
|
||||
if read.format is not None:
|
||||
info["format"] = read.format
|
||||
canonical["file_info"] = info
|
||||
return canonical, resolve_prompt_pair(canonical)
|
||||
|
||||
|
||||
def reviewable(resolution: Any) -> bool:
|
||||
"""True when an ambiguous result offers a choice: ``MIN_CHOOSABLE`` of its
|
||||
selectable candidates (nested ambiguous stages flattened) are choosable.
|
||||
|
||||
Counted before the evidence is bounded, so a ``limited`` item (more than
|
||||
``MAX_CANDIDATES`` candidates) with prompts to choose is still queued.
|
||||
"""
|
||||
if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous":
|
||||
return False
|
||||
count = 0
|
||||
for candidate in _flattened(resolution.get("candidates"), 0):
|
||||
if choosable(candidate):
|
||||
count += 1
|
||||
if count >= MIN_CHOOSABLE:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def review_evidence(resolution: dict) -> tuple[dict, int, bool]:
|
||||
"""``(evidence, candidate_count, limited)`` for one ambiguous result."""
|
||||
base = {
|
||||
"resolver_version": RESOLVER_VERSION,
|
||||
"reason": _text(resolution.get("reason")),
|
||||
}
|
||||
candidates: list[dict] = []
|
||||
seen: set[str] = set()
|
||||
total = 0
|
||||
for candidate in _flattened(resolution.get("candidates"), 0):
|
||||
total += 1
|
||||
try:
|
||||
projected = _projection(candidate)
|
||||
digest = hashlib.sha256(
|
||||
canonical_evidence(projected).encode("utf-8")
|
||||
).hexdigest()
|
||||
except (ValueError, TypeError, KeyError):
|
||||
return _limited(base, total), 0, True
|
||||
if digest in seen:
|
||||
total -= 1
|
||||
continue
|
||||
seen.add(digest)
|
||||
candidates.append({**projected, "hash": digest})
|
||||
if not candidates or len(candidates) > MAX_CANDIDATES:
|
||||
return _limited(base, total), 0, True
|
||||
evidence = {**base, "candidates": candidates}
|
||||
try:
|
||||
canonical_evidence(evidence)
|
||||
except ValueError:
|
||||
return _limited(base, total), 0, True
|
||||
return evidence, len(candidates), False
|
||||
|
||||
|
||||
def _limited(base: dict, total: int) -> dict:
|
||||
return {**base, "limited": True, "candidate_total": total, "candidates": []}
|
||||
|
||||
|
||||
def _flattened(candidates: Any, depth: int):
|
||||
"""Selectable candidates; a nested ambiguous stage yields its own."""
|
||||
if not isinstance(candidates, list):
|
||||
return
|
||||
for candidate in candidates:
|
||||
if not isinstance(candidate, dict):
|
||||
continue
|
||||
nested = candidate.get("candidates")
|
||||
if (
|
||||
candidate.get("state") == "ambiguous"
|
||||
and isinstance(nested, list)
|
||||
and nested
|
||||
and depth < MAX_CANDIDATE_DEPTH
|
||||
):
|
||||
yield from _flattened(nested, depth + 1)
|
||||
else:
|
||||
yield candidate
|
||||
|
||||
|
||||
def _projection(candidate: dict) -> dict:
|
||||
return {
|
||||
"sampler_id": candidate.get("sampler_id"),
|
||||
"state": _text(candidate.get("state")),
|
||||
"reason": _text(candidate.get("reason")),
|
||||
"positive": _side(candidate.get("positive")),
|
||||
"negative": _side(candidate.get("negative")),
|
||||
}
|
||||
|
||||
|
||||
def _side(side: Any) -> dict | None:
|
||||
if not isinstance(side, dict):
|
||||
return None
|
||||
text = side["text"]
|
||||
return {
|
||||
"text": text,
|
||||
"storage_text": side.get("storage_text", text),
|
||||
"provenance": side.get("provenance"),
|
||||
}
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
+13
-87
@@ -1,7 +1,4 @@
|
||||
class GalleryManager {
|
||||
// A Cancel on a thumbnail job another tab or System started (#249 System S5, review I1).
|
||||
static THUMBNAILS_STOPPED_FOLLOWING = 'Stopped following thumbnail generation started elsewhere; it keeps running in Activity';
|
||||
|
||||
constructor() {
|
||||
this.images = [];
|
||||
this.currentPage = 1;
|
||||
@@ -1914,54 +1911,6 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
}
|
||||
}
|
||||
|
||||
/** Ask the server to stop thumbnail job `jobId` at its next file boundary (#249 System S5). */
|
||||
async cancelThumbnailJob(jobId) {
|
||||
try {
|
||||
await this.api.post(`/prompt_manager/jobs/${encodeURIComponent(jobId)}/cancel`, {});
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.warn('The thumbnail job could not be cancelled:', error);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Cancel for one thumbnail stream (#249 System S5, review I1). The stream's first frame, `job {id, started}`,
|
||||
* names the job it follows and whether it started it. `request()` stops that job by its id, and only when this
|
||||
* stream started it: a job another tab or System started keeps running, and the stream just stops following.
|
||||
* A request before the `job` frame keeps the stream open until the frame arrives; `ended()` (the stream closed
|
||||
* or refused without naming a job) leaves nothing to cancel.
|
||||
*/
|
||||
thumbnailCanceller(eventSource) {
|
||||
let job = null;
|
||||
let wanted = false;
|
||||
let done = false;
|
||||
const apply = () => {
|
||||
if (done || !wanted || !job) return;
|
||||
done = true;
|
||||
eventSource.close();
|
||||
if (job.started) this.cancelThumbnailJob(job.id);
|
||||
};
|
||||
return {
|
||||
identify(raw) {
|
||||
if (job) return;
|
||||
let data = null;
|
||||
try { data = JSON.parse(raw); } catch (error) { return; }
|
||||
if (typeof data?.id !== 'string' || !data.id) return;
|
||||
job = { id: data.id, started: data.started === true };
|
||||
apply();
|
||||
},
|
||||
request() { wanted = true; apply(); },
|
||||
/** True once the stream named a job this page did not start. */
|
||||
get following() { return job !== null && !job.started; },
|
||||
ended() {
|
||||
if (done || !wanted || job) return;
|
||||
done = true;
|
||||
eventSource.close();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async generateThumbnailsWithProgress(options) {
|
||||
const progressBar = document.getElementById('progressBar');
|
||||
const progressText = document.getElementById('progressText');
|
||||
@@ -1983,19 +1932,13 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
let cancelled = false;
|
||||
let resultData = null;
|
||||
const canceller = this.thumbnailCanceller(eventSource);
|
||||
eventSource.addEventListener('job', event => canceller.identify(event.data));
|
||||
|
||||
// Handle cancellation. The work is an Activity job that outlives this stream, so closing it no
|
||||
// longer stops anything: ask the server to stop this stream's job by its id (review I1).
|
||||
// Handle cancellation
|
||||
const cancelHandler = () => {
|
||||
if (cancelled) return;
|
||||
cancelled = true;
|
||||
cancelBtn.removeEventListener('click', cancelHandler);
|
||||
eventSource.close();
|
||||
cancelBtn.classList.add('hidden');
|
||||
const following = canceller.following;
|
||||
canceller.request();
|
||||
reject(new Error(following ? GalleryManager.THUMBNAILS_STOPPED_FOLLOWING : 'Thumbnail generation cancelled by user'));
|
||||
reject(new Error('Thumbnail generation cancelled by user'));
|
||||
};
|
||||
cancelBtn.addEventListener('click', cancelHandler);
|
||||
|
||||
@@ -2070,7 +2013,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
// Handle completion
|
||||
eventSource.addEventListener('complete', (event) => {
|
||||
if (cancelled) { canceller.ended(); return; }
|
||||
if (cancelled) return;
|
||||
resultData = JSON.parse(event.data);
|
||||
|
||||
// Update final progress
|
||||
@@ -2118,7 +2061,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
// Handle errors
|
||||
eventSource.addEventListener('error', (event) => {
|
||||
if (cancelled) { canceller.ended(); return; }
|
||||
if (cancelled) return;
|
||||
const data = JSON.parse(event.data);
|
||||
|
||||
// Clean up
|
||||
@@ -2131,7 +2074,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
// Handle EventSource errors
|
||||
eventSource.onerror = (error) => {
|
||||
if (cancelled) { canceller.ended(); return; }
|
||||
if (cancelled) return;
|
||||
console.error('EventSource error:', error);
|
||||
console.error('EventSource readyState:', eventSource.readyState);
|
||||
|
||||
@@ -2154,7 +2097,6 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
// Stream ended without a complete/error frame (close() does not trigger this)
|
||||
eventSource.onclose = () => {
|
||||
if (cancelled) canceller.ended();
|
||||
if (cancelled || resultData) return;
|
||||
cancelBtn.removeEventListener('click', cancelHandler);
|
||||
cancelBtn.classList.add('hidden');
|
||||
@@ -2750,15 +2692,12 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
statusMessages.scrollTop = statusMessages.scrollHeight;
|
||||
};
|
||||
|
||||
// Handle cancel: the stream's own job stops at its next file boundary (#249 System S5, review I1)
|
||||
let cancelStream = null;
|
||||
// Handle cancel
|
||||
cancelBtn.addEventListener('click', () => {
|
||||
if (isCancelled) return;
|
||||
isCancelled = true;
|
||||
addStatusMessage('Cancellation requested...', 'warning');
|
||||
cancelBtn.disabled = true;
|
||||
cancelBtn.textContent = 'Cancelling...';
|
||||
cancelStream?.();
|
||||
});
|
||||
|
||||
// Handle close (only appears when done)
|
||||
@@ -2782,8 +2721,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
progressDetails,
|
||||
progressETA,
|
||||
addStatusMessage,
|
||||
isCancelled: () => isCancelled,
|
||||
onCancel: handler => { cancelStream = handler; }
|
||||
isCancelled: () => isCancelled
|
||||
});
|
||||
|
||||
// Show completion
|
||||
@@ -2803,14 +2741,10 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
if (isCancelled) {
|
||||
addStatusMessage(error.message, 'warning');
|
||||
} else {
|
||||
console.error('Thumbnail generation failed:', error);
|
||||
addStatusMessage(`Generation failed: ${error.message}`, 'error');
|
||||
progressBar.classList.add('bg-pm-error');
|
||||
progressBar.classList.remove('bg-pm-accent');
|
||||
}
|
||||
console.error('Thumbnail generation failed:', error);
|
||||
addStatusMessage(`Generation failed: ${error.message}`, 'error');
|
||||
progressBar.classList.add('bg-pm-error');
|
||||
progressBar.classList.remove('bg-pm-accent');
|
||||
} finally {
|
||||
// Show close button and hide cancel
|
||||
cancelBtn.classList.add('hidden');
|
||||
@@ -2828,7 +2762,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
}, 100);
|
||||
}
|
||||
} async generateThumbnailsWithProgressInModal(uiElements) {
|
||||
const { progressText, progressPercent, progressBar, progressDetails, progressETA, addStatusMessage, isCancelled, onCancel } = uiElements;
|
||||
const { progressText, progressPercent, progressBar, progressDetails, progressETA, addStatusMessage, isCancelled } = uiElements;
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
addStatusMessage('Starting thumbnail generation...');
|
||||
@@ -2842,14 +2776,6 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
const eventSource = SseStream.connect(`/prompt_manager/images/generate-thumbnails/progress?quality=${encodeURIComponent(quality)}`, { method: 'POST' });
|
||||
|
||||
let resultData = null;
|
||||
const canceller = this.thumbnailCanceller(eventSource);
|
||||
eventSource.addEventListener('job', event => canceller.identify(event.data));
|
||||
// Cancel stops this stream's job by id, keeps what it wrote, and settles the dialog.
|
||||
onCancel?.(() => {
|
||||
const following = canceller.following;
|
||||
canceller.request();
|
||||
reject(new Error(following ? GalleryManager.THUMBNAILS_STOPPED_FOLLOWING : 'Thumbnail generation cancelled; previews already written are kept'));
|
||||
});
|
||||
|
||||
// Handle scanning/status events
|
||||
eventSource.addEventListener('status', (event) => {
|
||||
|
||||
@@ -3,8 +3,7 @@
|
||||
* concept's classes.
|
||||
*
|
||||
* - Header: the page heading names the job; its line is the Activity item's (D54), and its actions are Download report
|
||||
* (the job's JSON report, D59) and, while the job runs, "Cancel scan" (danger). D87: an ended scan that recorded
|
||||
* Needs review items adds the concept's **Review metadata** after Download report.
|
||||
* (the job's JSON report, D59) and, while the job runs, "Cancel scan" (danger).
|
||||
* - Progress (`section.surface`): `<progress>`, the server's "Processing file N of M · about X min left", and the
|
||||
* outcome counters as `dl.stat-row` tiles: Added to Library (green), Already known, Needs review (warning), No usable
|
||||
* metadata, Errors (red).
|
||||
@@ -14,11 +13,7 @@
|
||||
* - Log: `details.term` "Scan log", collapsed, holding the raw timestamped lines in `pre[tabindex=0]`.
|
||||
* - D89: the tiles, the outcome list, the openable rows and the log and cancel labels are the job kind's (`jobView`):
|
||||
* an image tagging job shows Tagged, Unchanged, Skipped and Errors, opens Tagged and Unchanged rows, and reads
|
||||
* "Tagging log" and "Cancel tagging".
|
||||
* - D111: the kind's descriptor (activity-kinds.mjs `viewOf`) decides them for every kind: a thumbnail job counts
|
||||
* Generated, Skipped and Errors; a kind without groups (the health check, a held legacy-page operation) shows no
|
||||
* tiles and no Results, never another kind's zeros. A job without a known total has an indeterminate `<progress>`;
|
||||
* only a job the server can stop shows Cancel.
|
||||
* "Tagging log" and "Cancel tagging"; another kind keeps the scan's tiles as "Job log" and "Cancel job".
|
||||
*
|
||||
* The summary follows the Activity store (one SSE stream per running job); the listed rows and the log are read from
|
||||
* `GET /prompt_manager/jobs/<id>` at most once a second while the job runs, and once more when it ends.
|
||||
@@ -27,9 +22,8 @@ import { createButton } from '../shared/toolkit/core.js';
|
||||
import { pageHeading } from './heading.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
import { parseJob, JOB_VIEWS, JOBS_URL, LOST_DETAIL } from './activity-store.mjs';
|
||||
import { viewOf, canCancel, showProgress } from './activity-kinds.mjs';
|
||||
import { jobDetail, CANCEL_FAILED_TOAST, createReviewMetadataButton, hasReviewItems } from './activity.mjs';
|
||||
import { parseJob, jobView, JOB_VIEWS, JOBS_URL, LOST_DETAIL } from './activity-store.mjs';
|
||||
import { jobDetail, CANCEL_FAILED_TOAST } from './activity.mjs';
|
||||
import { downloadJobReport } from './activity-scan.mjs';
|
||||
import { createMediaViewer } from './media-viewer.mjs';
|
||||
import { createInspectionController } from './inspection-state.mjs';
|
||||
@@ -76,10 +70,7 @@ export function parseDetail(raw) {
|
||||
/** The progress line: the server's while running, then what the job got through. */
|
||||
export function progressLine(job) {
|
||||
const files = `${number(job.processed)} of ${number(job.total)} ${noun(job.total, 'file')}`;
|
||||
// D111: a job that never counted files (a held legacy job, the health check) says only what the server says.
|
||||
const counted = job.total > 0;
|
||||
if (job.state === 'running') return job.status_line || (counted ? `${job.progress}%` : '');
|
||||
if (!counted && (job.state === 'finished' || job.state === 'cancelled')) return job.status_line;
|
||||
if (job.state === 'running') return job.status_line || `${job.progress}%`;
|
||||
if (job.state === 'finished') return `${files} processed`;
|
||||
if (job.state === 'cancelled') return `Cancelled after ${files}`;
|
||||
if (job.state === 'lost') return LOST_DETAIL;
|
||||
@@ -105,7 +96,7 @@ export function filterRows(rows, filter) {
|
||||
|
||||
/** The first outcome with files, so a view opens on what happened; the first outcome when nothing has yet. */
|
||||
export function defaultOutcome(counters, outcomes = JOB_VIEWS.scan.outcomes) {
|
||||
return (outcomes.find(([group]) => (counters[group] ?? 0) > 0) ?? outcomes[0])?.[0] ?? null;
|
||||
return (outcomes.find(([group]) => (counters[group] ?? 0) > 0) ?? outcomes[0])[0];
|
||||
}
|
||||
|
||||
function section(title, id) {
|
||||
@@ -124,7 +115,6 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
const parts = pageHeading(main);
|
||||
const download = createButton('Download report');
|
||||
const cancel = createButton('Cancel scan', 'danger');
|
||||
const review = createReviewMetadataButton(scope); // D87
|
||||
|
||||
const progressPart = section('Progress', 'job-progress-title');
|
||||
const status = element('span', '', 'small');
|
||||
@@ -187,10 +177,9 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
const viewer = createMediaViewer({ api, container: main, parentScope: scope, inspectionController: inspection,
|
||||
sender: createPromptSender({ scope }), onSelect: image => inspection.select(image) });
|
||||
|
||||
/** The tiles and the outcome list of the job's kind, built once (D89, D111: only the groups it counts). */
|
||||
function layout(job) {
|
||||
view = viewOf(job);
|
||||
if (!view.tiles.length) stats.remove();
|
||||
/** The tiles and the outcome list of the job's kind, built once. */
|
||||
function layout(kind) {
|
||||
view = jobView(kind); // D89
|
||||
for (const [group, label, tone] of view.tiles) {
|
||||
const tile = element('div', undefined, tone ? `stat ${tone}` : 'stat');
|
||||
const value = element('dd', '0');
|
||||
@@ -212,11 +201,10 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
function showBody(job) {
|
||||
if (mounted) return;
|
||||
mounted = true;
|
||||
if (!view) layout(job);
|
||||
if (!view) layout(job.kind);
|
||||
missing.remove();
|
||||
parts.actions.append(download, cancel);
|
||||
// A kind without outcome groups has no Results to browse (D111).
|
||||
main.append(...(view.outcomes.length ? [progressPart.node, resultsPart.node, log] : [progressPart.node, log]));
|
||||
main.append(progressPart.node, resultsPart.node, log);
|
||||
selected ??= defaultOutcome(job.counters, view.outcomes);
|
||||
}
|
||||
|
||||
@@ -252,7 +240,6 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
}
|
||||
|
||||
function renderRows(job) {
|
||||
if (!view.outcomes.length) return;
|
||||
const rows = detail?.outcomes[selected] ?? [];
|
||||
const openable = view.openable.has(selected);
|
||||
const wanted = filterRows(rows.map((row, index) => ({ ...row, index })), input.value);
|
||||
@@ -287,10 +274,8 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
const lost = job.state === 'lost';
|
||||
if (lost && document.activeElement === download) parts.h1.focus();
|
||||
download.disabled = lost;
|
||||
if (hasReviewItems(job)) { if (!review.isConnected) download.after(review); } // D87
|
||||
else review.remove();
|
||||
// Truth rule: a job that cannot be cancelled (a held legacy-page job) shows no Cancel.
|
||||
if (canCancel(job)) {
|
||||
if (job.state === 'running' && job.cancellable) {
|
||||
if (!cancel.isConnected) parts.actions.append(cancel);
|
||||
} else if (cancel.isConnected) {
|
||||
// A focused Cancel that goes away hands its focus to the job's heading, never to the page body.
|
||||
@@ -298,8 +283,8 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
cancel.remove();
|
||||
}
|
||||
status.textContent = progressLine(job);
|
||||
// D111: no value while the total is unknown; a job that ended with nothing counted shows how it ended.
|
||||
showProgress(bar, job.total > 0 || job.state !== 'finished' ? job : { ...job, progress: 100 });
|
||||
bar.value = job.progress;
|
||||
bar.textContent = `${job.progress}%`;
|
||||
bar.setAttribute('aria-label', scan ? 'Scan progress' : `${job.title} progress`);
|
||||
for (const [group, value] of tileValues) value.textContent = number(job.counters[group] ?? 0);
|
||||
for (const [group, button] of buttons) {
|
||||
|
||||
@@ -1,189 +0,0 @@
|
||||
/**
|
||||
* Per-kind Activity descriptors (#249 System S5, D111): one table that says, for each job kind, what Activity shows.
|
||||
*
|
||||
* `kindOf(kind)` → `{pillVerb, startToast(job), finishToast(job), line(job), kept(job), scope(job), tiles, outcomes,
|
||||
* openable, log, cancelLabel}`:
|
||||
* - `pillVerb`: the pill's verb ("Scanning"); null shows the job's title.
|
||||
* - `startToast` / `finishToast`: what a toast says when this tab starts the job and when it ends.
|
||||
* - `line`: the counters as one line ("N added · M errors"); `kept`: what a cancelled job kept ("N added"); `scope`:
|
||||
* what the job covers ("2 output folders").
|
||||
* - `tiles` (`[group, label, tone]`), `outcomes` (`[group, label]`) and `openable` (rows that open the viewer, D57).
|
||||
* - `log` and `cancelLabel` (`cancel` is the same label under the job view's older name).
|
||||
*
|
||||
* `scan` is exactly the D56 view and `image-tags` the D89 one. `thumbnails` counts Generated (green), Skipped and
|
||||
* Errors (red). Any other kind (the health check, a held legacy-page operation, a kind this page does not know) gets
|
||||
* the generic descriptor: title-based copy and no tiles or outcomes, so it never shows another kind's zeros (S1 review
|
||||
* M6, S2 review M3). `viewOf(job)` keeps only the groups the job itself counts, so a hold (no groups) has none either.
|
||||
*
|
||||
* Progress is indeterminate while a job runs without a known total (`indeterminate`): the `<progress>` has no value
|
||||
* and the pill shows no percentage, never a made-up 0%. A job shows Cancel only while it runs and the server can stop
|
||||
* it (`canCancel`).
|
||||
*/
|
||||
import { noun, countOf } from './plural.mjs';
|
||||
|
||||
// Grouped as the server writes its status lines ("8,240 of 19,622"), whatever the browser's locale.
|
||||
const number = value => value.toLocaleString('en-US');
|
||||
const counted = (job, group, word) => `${number(job.counters?.[group] ?? 0)} ${word}`;
|
||||
const errorsOf = job => `${number(job.counters?.errors ?? 0)} ${noun(job.counters?.errors ?? 0, 'error')}`;
|
||||
const whole = value => Number.isSafeInteger(value) && value >= 0;
|
||||
const tiles = rows => Object.freeze(rows.map(Object.freeze));
|
||||
const groups = (...names) => Object.freeze(new Set(names));
|
||||
|
||||
/** "1,420 added". */
|
||||
export const addedOf = job => counted(job, 'added', 'added');
|
||||
/** "12 tagged" (D89). */
|
||||
export const taggedOf = job => counted(job, 'tagged', 'tagged');
|
||||
/** "3 generated" (D111). */
|
||||
export const generatedOf = job => counted(job, 'generated', 'generated');
|
||||
|
||||
/** "2 output folders": what a scan or a thumbnail job walks. */
|
||||
function foldersOf(job) {
|
||||
const folders = job.scope?.output_folders;
|
||||
return whole(folders) ? countOf(folders, 'output folder') : '';
|
||||
}
|
||||
|
||||
/** "4 images" (D89), grouped as the server writes its lines. */
|
||||
function imagesOf(job) {
|
||||
const images = job.scope?.images;
|
||||
return whole(images) ? `${number(images)} ${noun(images, 'image')}` : '';
|
||||
}
|
||||
|
||||
const titleStarted = job => `${job.title} started.`;
|
||||
|
||||
function titleFinished(job) {
|
||||
if (job.state === 'cancelled') return `${job.title} was cancelled.`;
|
||||
if (job.state === 'failed') return `${job.title} failed.`;
|
||||
return `${job.title} finished.`;
|
||||
}
|
||||
|
||||
/** "<Noun> finished · <line>", "<Noun> cancelled · <kept>", "<Noun> failed · <line>". */
|
||||
const namedFinish = (name, kept, line) => job => {
|
||||
if (job.state === 'cancelled') return `${name(job)} cancelled · ${kept(job)}`;
|
||||
return `${name(job)} ${job.state === 'failed' ? 'failed' : 'finished'} · ${line(job)}`;
|
||||
};
|
||||
|
||||
const nothing = () => '';
|
||||
const GENERIC = Object.freeze({
|
||||
pillVerb: null, startToast: titleStarted, finishToast: titleFinished, line: nothing, kept: nothing,
|
||||
scope: foldersOf, tiles: tiles([]), outcomes: tiles([]), openable: groups(), log: 'Job log', cancelLabel: 'Cancel job',
|
||||
cancel: 'Cancel job',
|
||||
});
|
||||
|
||||
const scanLine = job => `${addedOf(job)} · ${errorsOf(job)}`;
|
||||
const tagLine = job => `${taggedOf(job)} · ${errorsOf(job)}`;
|
||||
const thumbnailLine = job => `${generatedOf(job)} · ${errorsOf(job)}`;
|
||||
// D85: an image tagging job says which of its two modes started.
|
||||
const IMAGE_TAG_START_TOASTS = Object.freeze({ add: 'Tagging started.', auto: 'Auto-tagging started.' });
|
||||
|
||||
/** Every kind with a descriptor of its own. */
|
||||
export const KINDS = Object.freeze({
|
||||
__proto__: null,
|
||||
// D56: the scan's tiles and outcome list in the mockup's orders; what got added and what went wrong first.
|
||||
scan: Object.freeze({
|
||||
...GENERIC,
|
||||
pillVerb: 'Scanning',
|
||||
startToast: () => 'Scan started.',
|
||||
finishToast: namedFinish(() => 'Scan', addedOf, scanLine),
|
||||
line: scanLine,
|
||||
kept: addedOf,
|
||||
tiles: tiles([['added', 'Added to Library', 'added'], ['known', 'Already known', ''],
|
||||
['review', 'Needs review', 'warn'], ['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad']]),
|
||||
outcomes: tiles([['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
|
||||
['no_metadata', 'No usable metadata'], ['known', 'Already known']]),
|
||||
openable: groups('added', 'known'),
|
||||
log: 'Scan log',
|
||||
cancelLabel: 'Cancel scan',
|
||||
cancel: 'Cancel scan',
|
||||
}),
|
||||
// D89: Tagged (green), Unchanged, Skipped (warning) and Errors (red); Tagged and Unchanged rows open the viewer.
|
||||
'image-tags': Object.freeze({
|
||||
...GENERIC,
|
||||
pillVerb: 'Tagging',
|
||||
startToast: job => IMAGE_TAG_START_TOASTS[job.scope?.mode] ?? titleStarted(job),
|
||||
finishToast: namedFinish(job => job.title, taggedOf, tagLine),
|
||||
line: tagLine,
|
||||
kept: taggedOf,
|
||||
scope: imagesOf,
|
||||
tiles: tiles([['tagged', 'Tagged', 'added'], ['unchanged', 'Unchanged', ''], ['skipped', 'Skipped', 'warn'],
|
||||
['errors', 'Errors', 'bad']]),
|
||||
outcomes: tiles([['tagged', 'Tagged'], ['errors', 'Errors'], ['skipped', 'Skipped'], ['unchanged', 'Unchanged']]),
|
||||
openable: groups('tagged', 'unchanged'),
|
||||
log: 'Tagging log',
|
||||
cancelLabel: 'Cancel tagging',
|
||||
cancel: 'Cancel tagging',
|
||||
}),
|
||||
// D111: Generated (green), Skipped (already fresh) and Errors (red); a row with a preview opens the viewer.
|
||||
thumbnails: Object.freeze({
|
||||
...GENERIC,
|
||||
pillVerb: 'Generating thumbnails',
|
||||
finishToast: namedFinish(() => 'Thumbnails', generatedOf, thumbnailLine),
|
||||
line: thumbnailLine,
|
||||
kept: generatedOf,
|
||||
tiles: tiles([['generated', 'Generated', 'added'], ['skipped', 'Skipped', ''], ['errors', 'Errors', 'bad']]),
|
||||
outcomes: tiles([['generated', 'Generated'], ['errors', 'Errors'], ['skipped', 'Skipped']]),
|
||||
openable: groups('generated', 'skipped'),
|
||||
}),
|
||||
});
|
||||
|
||||
/** The descriptor of a job `kind`; any kind without one gets the generic descriptor. */
|
||||
export function kindOf(kind) {
|
||||
return (typeof kind === 'string' && KINDS[kind]) || GENERIC;
|
||||
}
|
||||
|
||||
const countsOf = job => job.counters && typeof job.counters === 'object' ? job.counters : {};
|
||||
|
||||
/**
|
||||
* A legacy-page operation held on the registry (D112): not cancellable and counting no outcome groups. Whatever its
|
||||
* kind (the gallery's thumbnail clear is a `thumbnails` hold), it reads as the generic job under its own title.
|
||||
*/
|
||||
export const held = job => job.cancellable === false && Object.keys(countsOf(job)).length === 0;
|
||||
|
||||
/**
|
||||
* The descriptor for one job: its kind's, keeping only the outcome groups the job counts; a held job's is the generic
|
||||
* one.
|
||||
*/
|
||||
export function viewOf(job) {
|
||||
if (held(job)) return GENERIC;
|
||||
const view = kindOf(job.kind);
|
||||
const counts = countsOf(job);
|
||||
const has = ([group]) => Object.hasOwn(counts, group);
|
||||
if (view.tiles.every(has) && view.outcomes.every(has)) return view;
|
||||
return Object.freeze({ ...view, tiles: tiles(view.tiles.filter(has)), outcomes: tiles(view.outcomes.filter(has)),
|
||||
openable: groups(...[...view.openable].filter(group => Object.hasOwn(counts, group))) });
|
||||
}
|
||||
|
||||
/** A running job whose total the server has not reported (yet) has no percentage. */
|
||||
export function indeterminate(job) {
|
||||
return job.state === 'running' && !(job.total > 0);
|
||||
}
|
||||
|
||||
/** Set a `<progress>` to the job's: its value and "N%", or no value at all while indeterminate. */
|
||||
export function showProgress(node, job) {
|
||||
if (indeterminate(job)) {
|
||||
node.removeAttribute('value');
|
||||
node.textContent = '';
|
||||
return;
|
||||
}
|
||||
node.value = job.progress;
|
||||
node.textContent = `${job.progress}%`;
|
||||
}
|
||||
|
||||
/** Truth rule: Cancel only while the job runs and the server can stop it (never on a held legacy job). */
|
||||
export function canCancel(job) {
|
||||
return job.state === 'running' && job.cancellable !== false;
|
||||
}
|
||||
|
||||
/** The pill's text: "Scanning · 42% · 8,240 / 19,622"; the file counts once the total is known, no "%" before. */
|
||||
export function pillParts(job) {
|
||||
const verb = viewOf(job).pillVerb ?? job.title;
|
||||
const label = indeterminate(job) ? verb : `${verb} · ${job.progress}%`;
|
||||
const files = job.total > 0 ? ` · ${number(job.processed)} / ${number(job.total)}` : '';
|
||||
return { label, files, text: label + files };
|
||||
}
|
||||
|
||||
/** The toast after this tab starts a job. */
|
||||
export const startToast = job => viewOf(job).startToast(job);
|
||||
/** The toast when a job ends. */
|
||||
export const finishToast = job => viewOf(job).finishToast(job);
|
||||
/** The counters as one line; '' for a kind without one. */
|
||||
export const outcomeLine = job => viewOf(job).line(job);
|
||||
@@ -12,9 +12,7 @@
|
||||
* the finish toast follows any job this store saw running when it ends.
|
||||
*/
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { KINDS, kindOf, finishToast, startToast } from './activity-kinds.mjs';
|
||||
|
||||
export { addedOf, taggedOf, finishToast, startToast, pillParts, outcomeLine } from './activity-kinds.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
|
||||
export const JOBS_URL = '/prompt_manager/jobs';
|
||||
export const BACKOFF_FIRST_MS = 1000;
|
||||
@@ -25,13 +23,51 @@ export const LOST_DETAIL = 'This job ended when ComfyUI restarted.';
|
||||
const SERVER_STATES = new Set(['running', 'finished', 'cancelled', 'failed']);
|
||||
const FINAL_STATES = new Set(['finished', 'cancelled', 'failed']);
|
||||
const JOB_ID = /^[0-9a-f]{32}$/;
|
||||
// The pill's verb for each job kind; a kind without one shows its title (D89: image tagging jobs read "Tagging").
|
||||
const PILL_VERBS = Object.freeze({ scan: 'Scanning', 'image-tags': 'Tagging' });
|
||||
const START_TOASTS = Object.freeze({ scan: 'Scan started.' });
|
||||
|
||||
/** The scan's job view (D56): its counter tiles `[group, label, tone]` and outcome list in the mockup's orders. */
|
||||
const SCAN_VIEW = Object.freeze({
|
||||
tiles: Object.freeze([
|
||||
['added', 'Added to Library', 'added'], ['known', 'Already known', ''], ['review', 'Needs review', 'warn'],
|
||||
['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad'],
|
||||
].map(Object.freeze)),
|
||||
// What got added and what went wrong first.
|
||||
outcomes: Object.freeze([
|
||||
['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
|
||||
['no_metadata', 'No usable metadata'], ['known', 'Already known'],
|
||||
].map(Object.freeze)),
|
||||
// Rows that open the file in the viewer (D57); the others are plain text.
|
||||
openable: Object.freeze(new Set(['added', 'known'])),
|
||||
log: 'Scan log',
|
||||
cancel: 'Cancel scan',
|
||||
});
|
||||
|
||||
/**
|
||||
* Each job kind's copy and view live in one table (activity-kinds.mjs, D111): the scan's (D56), image tagging's (D89),
|
||||
* thumbnails' and the generic one. The names below are that table's, kept for this store's importers.
|
||||
* Each job kind's view: its tiles, outcome list, openable rows, log label and cancel label. D89: an image tagging job
|
||||
* (`image-tags`) counts Tagged (green), Unchanged, Skipped (warning) and Errors (red), lists what got tagged and what
|
||||
* went wrong first, and opens Tagged and Unchanged rows in the viewer like the scan's (D57).
|
||||
*/
|
||||
export const JOB_VIEWS = KINDS;
|
||||
export const JOB_VIEWS = Object.freeze({
|
||||
scan: SCAN_VIEW,
|
||||
'image-tags': Object.freeze({
|
||||
tiles: Object.freeze([['tagged', 'Tagged', 'added'], ['unchanged', 'Unchanged', ''], ['skipped', 'Skipped', 'warn'],
|
||||
['errors', 'Errors', 'bad']].map(Object.freeze)),
|
||||
outcomes: Object.freeze([['tagged', 'Tagged'], ['errors', 'Errors'], ['skipped', 'Skipped'],
|
||||
['unchanged', 'Unchanged']].map(Object.freeze)),
|
||||
openable: Object.freeze(new Set(['tagged', 'unchanged'])),
|
||||
log: 'Tagging log',
|
||||
cancel: 'Cancel tagging',
|
||||
}),
|
||||
});
|
||||
// Another kind keeps the scan's tiles and outcomes under the generic labels.
|
||||
const GENERIC_VIEW = Object.freeze({ ...SCAN_VIEW, log: 'Job log', cancel: 'Cancel job' });
|
||||
|
||||
/** The view of a job `kind`; an unknown kind gets the generic one. */
|
||||
export const jobView = kindOf;
|
||||
export function jobView(kind) {
|
||||
return Object.hasOwn(JOB_VIEWS, kind) ? JOB_VIEWS[kind] : GENERIC_VIEW;
|
||||
}
|
||||
|
||||
/** The wait before the `attempt`-th re-read (0-based): 1 s, 2 s, 4 s … never more than 30 s. */
|
||||
export function backoffDelay(attempt) {
|
||||
@@ -72,6 +108,57 @@ export function parseJob(raw) {
|
||||
};
|
||||
}
|
||||
|
||||
// Grouped as the server writes its status lines ("8,240 of 19,622"), whatever the browser's locale.
|
||||
const number = value => value.toLocaleString('en-US');
|
||||
const errorsOf = job => `${number(job.counters.errors ?? 0)} ${noun(job.counters.errors ?? 0, 'error')}`;
|
||||
/** "1,420 added". */
|
||||
export const addedOf = job => `${number(job.counters.added ?? 0)} added`;
|
||||
|
||||
/**
|
||||
* The toast when a job ends: "Scan finished · N added · M errors", "Scan cancelled · N added"; an image tagging job
|
||||
* the same way with its own title and count (D89): "Auto-tag images finished · N tagged · M errors".
|
||||
*/
|
||||
export function finishToast(job) {
|
||||
if (job.kind === 'scan') {
|
||||
if (job.state === 'cancelled') return `Scan cancelled · ${addedOf(job)}`;
|
||||
if (job.state === 'failed') return `Scan failed · ${addedOf(job)} · ${errorsOf(job)}`;
|
||||
return `Scan finished · ${addedOf(job)} · ${errorsOf(job)}`;
|
||||
}
|
||||
if (job.kind === 'image-tags') {
|
||||
if (job.state === 'cancelled') return `${job.title} cancelled · ${taggedOf(job)}`;
|
||||
return `${job.title} ${job.state === 'failed' ? 'failed' : 'finished'} · ${outcomeLine(job)}`;
|
||||
}
|
||||
if (job.state === 'cancelled') return `${job.title} was cancelled.`;
|
||||
if (job.state === 'failed') return `${job.title} failed.`;
|
||||
return `${job.title} finished.`;
|
||||
}
|
||||
|
||||
// D85: an image tagging job says which of its two modes started.
|
||||
const IMAGE_TAG_START_TOASTS = Object.freeze({ add: 'Tagging started.', auto: 'Auto-tagging started.' });
|
||||
|
||||
/** The toast after this tab starts a job. */
|
||||
export function startToast(job) {
|
||||
if (job.kind === 'image-tags' && IMAGE_TAG_START_TOASTS[job.scope.mode]) return IMAGE_TAG_START_TOASTS[job.scope.mode];
|
||||
return START_TOASTS[job.kind] ?? `${job.title} started.`;
|
||||
}
|
||||
|
||||
/** The pill's text: "Scanning · 42% · 8,240 / 19,622"; the file counts only once the total is known. */
|
||||
export function pillParts(job) {
|
||||
const label = `${PILL_VERBS[job.kind] ?? job.title} · ${job.progress}%`;
|
||||
const files = job.total > 0 ? ` · ${number(job.processed)} / ${number(job.total)}` : '';
|
||||
return { label, files, text: label + files };
|
||||
}
|
||||
|
||||
/** "12 tagged" (D89). */
|
||||
export const taggedOf = job => `${number(job.counters.tagged ?? 0)} tagged`;
|
||||
|
||||
/** The counters as one line: "N added · M errors" for a scan, "N tagged · M errors" for image tagging (D89). */
|
||||
export function outcomeLine(job) {
|
||||
if (job.kind === 'scan') return `${addedOf(job)} · ${errorsOf(job)}`;
|
||||
if (job.kind === 'image-tags') return `${taggedOf(job)} · ${errorsOf(job)}`;
|
||||
return '';
|
||||
}
|
||||
|
||||
function defaultConnect(url) {
|
||||
const resolve = globalThis.ApiClient?.resolveUrl ?? (value => value);
|
||||
return new EventSource(resolve(url));
|
||||
|
||||
@@ -11,36 +11,19 @@
|
||||
* - D54: each `.activity-item` is a real job. Its chip names the state (green "Completed", or blue "Running",
|
||||
* "Cancelled", "Failed", "Ended"), its line says when and what, and a running job shows its `<progress>` and
|
||||
* "Cancel job" (no "Simulate completion").
|
||||
* - D87 (plan D64): an ended scan that recorded Needs review items carries the concept's **Review metadata**
|
||||
* (`go-review`, app.js a738) after its line, which opens `#/review`; its chip still names its state (D54).
|
||||
* - D89: an image tagging job's line names its images and, once ended, "N tagged · M errors".
|
||||
* - D111: what a line says (scope, counters, what a cancel kept) is the job kind's descriptor (activity-kinds.mjs); a
|
||||
* job without a known total shows an indeterminate `<progress>`, and only a job the server can stop shows Cancel.
|
||||
*/
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { navigateTo } from './state.mjs';
|
||||
import { pageHeading } from './heading.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { pillParts, outcomeLine, LOST_DETAIL } from './activity-store.mjs';
|
||||
import { viewOf, canCancel, showProgress, indeterminate } from './activity-kinds.mjs';
|
||||
import { countOf, noun } from './plural.mjs';
|
||||
import { pillParts, outcomeLine, addedOf, taggedOf, LOST_DETAIL } from './activity-store.mjs';
|
||||
|
||||
export const ACTIVITY_DESCRIPTION = 'Scans, tagging, downloads and maintenance share one history.';
|
||||
export const EMPTY_ACTIVITY = 'No jobs have run since ComfyUI started. Scans and other long operations appear here.';
|
||||
export const LOAD_FAILED = 'Activity could not be read. It is tried again shortly.';
|
||||
export const CANCEL_FAILED_TOAST = 'The job could not be cancelled. Try again.';
|
||||
|
||||
/** Whether an ended job recorded items for Needs review (a metadata scan's `scope.review_items`). */
|
||||
export function hasReviewItems(job) {
|
||||
const count = job?.scope?.review_items;
|
||||
return job?.state !== 'running' && Number.isSafeInteger(count) && count > 0;
|
||||
}
|
||||
|
||||
/** The concept's **Review metadata** (D87): opens Needs review. */
|
||||
export function createReviewMetadataButton(scope) {
|
||||
const button = createButton('Review metadata');
|
||||
scope.listen(button, 'click', () => navigateTo('#/review'));
|
||||
return button;
|
||||
}
|
||||
const CHIPS = Object.freeze({
|
||||
running: ['blue', 'Running'], finished: ['green', 'Completed'], cancelled: ['blue', 'Cancelled'],
|
||||
failed: ['blue', 'Failed'], lost: ['blue', 'Ended'],
|
||||
@@ -62,9 +45,20 @@ export function clockTime(seconds) {
|
||||
|
||||
const join = parts => parts.filter(Boolean).join(' · ');
|
||||
|
||||
/** What a job covers ("2 output folders", "4 images") and what a cancelled one kept ("N added"), by kind (D111). */
|
||||
const scopeLine = job => viewOf(job).scope(job);
|
||||
const keptOf = job => viewOf(job).kept(job);
|
||||
const whole = value => Number.isSafeInteger(value) && value >= 0;
|
||||
|
||||
/** What a job covers: "2 output folders" for a scan, "4 images" for image tagging (D89). */
|
||||
function scopeLine(job) {
|
||||
// D89: grouped as the server writes its lines ("19,622 images").
|
||||
const images = job.scope?.images;
|
||||
if (job.kind === 'image-tags') return whole(images) ? `${images.toLocaleString('en-US')} ${noun(images, 'image')}` : '';
|
||||
const folders = job.scope?.output_folders;
|
||||
return whole(folders) ? countOf(folders, 'output folder') : '';
|
||||
}
|
||||
|
||||
/** What a cancelled job kept: "N added" for a scan, "N tagged" for image tagging (D89). */
|
||||
const KEPT = Object.freeze({ __proto__: null, scan: addedOf, 'image-tags': taggedOf }); // D89
|
||||
const keptOf = job => KEPT[job.kind]?.(job) ?? '';
|
||||
|
||||
/** The item's line under its title, by state. */
|
||||
export function jobDetail(job) {
|
||||
@@ -110,10 +104,7 @@ export function mountActivityTopbar({ store, scope, doc = document }) {
|
||||
const parts = pillParts(job);
|
||||
label.textContent = parts.label;
|
||||
files.textContent = parts.files;
|
||||
// D111 (S5 review M5): no total, no fill: the bar sweeps instead of showing an empty determinate 0%.
|
||||
const unknown = indeterminate(job);
|
||||
bar.classList.toggle('indeterminate', unknown);
|
||||
fill.style.width = unknown ? '' : `${job.progress}%`;
|
||||
fill.style.width = `${job.progress}%`;
|
||||
pill.href = `#/activity/${job.id}`;
|
||||
pill.setAttribute('aria-label', parts.text);
|
||||
};
|
||||
@@ -138,7 +129,6 @@ function itemFor(job, scope) {
|
||||
const actions = element('div', undefined, 'row-actions');
|
||||
const cancel = createButton('Cancel job');
|
||||
actions.append(cancel);
|
||||
const review = createReviewMetadataButton(scope); // D87
|
||||
item.append(heading, detail, progress, actions);
|
||||
scope.listen(cancel, 'click', async () => {
|
||||
if (cancel.getAttribute('aria-disabled') === 'true') return;
|
||||
@@ -161,22 +151,21 @@ function itemFor(job, scope) {
|
||||
// parentNode, not isConnected: on the first render the item is not in the page yet, so a job that had already
|
||||
// ended when Activity opened kept its Cancel job (and a running one's progress moved after its actions).
|
||||
if (running) {
|
||||
showProgress(progress, next); // D111: no value while the total is unknown
|
||||
progress.value = next.progress;
|
||||
progress.textContent = `${next.progress}%`;
|
||||
progress.setAttribute('aria-label', `${next.title} progress`);
|
||||
if (!progress.parentNode) item.insertBefore(progress, actions.parentNode ? actions : null);
|
||||
} else {
|
||||
progress.remove();
|
||||
}
|
||||
// Truth rule: a job that cannot be cancelled (a held legacy-page job) shows no Cancel job.
|
||||
if (canCancel(next)) {
|
||||
if (running && next.cancellable) {
|
||||
if (!actions.parentNode) item.append(actions);
|
||||
} else if (actions.parentNode) {
|
||||
// A focused Cancel job that goes away hands its focus to the item's title link, never to the page body.
|
||||
if (actions.contains(document.activeElement)) link.focus();
|
||||
actions.remove();
|
||||
}
|
||||
if (hasReviewItems(next)) { if (!review.isConnected) detail.after(review); } // D87
|
||||
else review.remove();
|
||||
};
|
||||
return item;
|
||||
}
|
||||
|
||||
@@ -1,81 +0,0 @@
|
||||
/**
|
||||
* The concept media card (docs/concepts/workspace/app.js `mediaCard` a396–399), shared by Images, every
|
||||
* Library-embedded gallery and the Needs review queue: `article.media-card` > `.media-art` (the `.image-open` button
|
||||
* holding the media, then `.image-overlays`) and `.media-info` (title, "W × H · EXT · Seed N", then
|
||||
* `.image-card-actions` with Info and Tag). #249 D7–D11, D19.
|
||||
*/
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { cardTitle, cardSubtitle, imageKey, mediaRatio } from './images-state.mjs';
|
||||
import { createWorkspaceIcon } from './icons.mjs';
|
||||
import { createTagButton } from './image-tagging.mjs';
|
||||
import { paintStar } from './images-favorites.mjs';
|
||||
|
||||
function el(tag, text, className) {
|
||||
const node = document.createElement(tag);
|
||||
if (text !== undefined) node.textContent = text;
|
||||
if (className) node.className = className;
|
||||
return node;
|
||||
}
|
||||
|
||||
/** The card's media: a video's first frame when it has no thumbnail, else the thumbnail or the image. */
|
||||
export function cardMedia(image, lazy = false) {
|
||||
const { resolveUrl } = window.ApiClient;
|
||||
const video = image.is_video && !image.thumbnail_url;
|
||||
const node = el(video ? 'video' : 'img');
|
||||
node.src = resolveUrl(image.thumbnail_url || image.url);
|
||||
if (video) { node.muted = true; node.playsInline = true; node.preload = 'metadata'; }
|
||||
else {
|
||||
// The concept card image: alt is the card title, with the listing's intrinsic size when known.
|
||||
node.alt = cardTitle(image); node.decoding = 'async'; if (lazy) node.loading = 'lazy';
|
||||
if ([image.width, image.height].every(value => Number.isFinite(value) && value > 0)) {
|
||||
node.width = image.width; node.height = image.height;
|
||||
}
|
||||
}
|
||||
return node;
|
||||
}
|
||||
|
||||
/**
|
||||
* One card for `image`. `checked` (a boolean) adds the D8 selection checkbox, `favoriteBusy` paints a pending star
|
||||
* (D7, shown wherever the image reports `favorite`), `media: false` leaves the open button empty for a file that
|
||||
* cannot be served, and `showInfo: false` hides the title and file line (D97).
|
||||
*/
|
||||
export function createImageCard(image, { checked = null, favoriteBusy = false, media = true, showInfo = true } = {}) {
|
||||
const card = el('article', undefined, 'media-card');
|
||||
card.dataset.imageKey = imageKey(image); card.setAttribute('role', 'listitem');
|
||||
card.style.setProperty('--media-ratio', mediaRatio(image));
|
||||
const art = el('div', undefined, 'media-art');
|
||||
const open = createButton('', 'image-open');
|
||||
open.setAttribute('aria-label', `Open ${image.filename}`);
|
||||
if (media) open.append(cardMedia(image, true));
|
||||
// D8: the selection checkbox (Images only: Library-embedded media has no selection bar), then D7: the star,
|
||||
// wherever the listing reports favorites (Images and Library-embedded media).
|
||||
const overlays = el('div', undefined, 'image-overlays');
|
||||
if (typeof checked === 'boolean') {
|
||||
const label = el('label'), box = el('input');
|
||||
box.type = 'checkbox'; box.checked = checked;
|
||||
box.setAttribute('aria-label', `Select ${image.filename}`);
|
||||
label.append(box);
|
||||
overlays.append(label);
|
||||
}
|
||||
if (typeof image.favorite === 'boolean') {
|
||||
const star = createButton('', 'star-button');
|
||||
star.setAttribute('aria-label', `Favorite ${image.filename}`);
|
||||
paintStar(star, image.favorite, favoriteBusy);
|
||||
overlays.append(star);
|
||||
}
|
||||
art.append(open, overlays);
|
||||
const info = el('div', undefined, 'media-info');
|
||||
const text = el('div');
|
||||
const title = el('h3', cardTitle(image)); if (image.path) title.title = image.path;
|
||||
text.append(title, el('p', cardSubtitle(image)));
|
||||
// D97: `showInfo` false (System › Appearance) hides the title and file line; Info and Tag stay.
|
||||
if (!showInfo) text.hidden = true;
|
||||
const actions = el('div', undefined, 'image-card-actions');
|
||||
const inspectButton = createButton('', 'icon-btn');
|
||||
inspectButton.append(createWorkspaceIcon('info'));
|
||||
inspectButton.setAttribute('aria-label', `Inspect ${image.filename}`);
|
||||
actions.append(inspectButton, createTagButton(image));
|
||||
info.append(text, actions);
|
||||
card.append(art, info);
|
||||
return card;
|
||||
}
|
||||
@@ -1,6 +1,5 @@
|
||||
import { createScope, createButton } from '../shared/toolkit/core.js';
|
||||
import { createImageStream, withItem, imageKey, countLine, captionDate, scopeLabel, activeRefinements, initialImagesView } from './images-state.mjs';
|
||||
import { createImageCard } from './image-card.mjs';
|
||||
import { createImageStream, withItem, imageKey, countLine, captionDate, scopeLabel, activeRefinements, cardTitle, cardSubtitle, mediaRatio, initialImagesView } from './images-state.mjs';
|
||||
import { layoutImages, visibleImages, gridMetrics, stacksCardInfo } from './image-layout.mjs';
|
||||
import { mountFileInspector } from './file-inspector.mjs';
|
||||
import { createInspectionController } from './inspection-state.mjs';
|
||||
@@ -17,7 +16,7 @@ import { openMetadataScan } from './activity-scan.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { imagesFilters, imagesViewState } from './saved-views.mjs';
|
||||
import { displayPreferences } from './preferences.mjs';
|
||||
import { onImageTagsSaved } from './image-tagging.mjs';
|
||||
import { createTagButton, onImageTagsSaved } from './image-tagging.mjs';
|
||||
import { followImageTagSaves, openCardTagging } from './images-tagging.mjs';
|
||||
import { favoriteBusy, paintStar, receiveFavorites, setFavoriteVisitTarget, toggleFavorite } from './images-favorites.mjs';
|
||||
export { onFavoriteChange } from './images-favorites.mjs';
|
||||
@@ -110,6 +109,7 @@ export function mountImages(main, api, parentScope, {embedded = false, fetchPage
|
||||
parentScope.signal.addEventListener('abort', dispose, {once:true});
|
||||
scope.cleanup(() => parentScope.signal.removeEventListener('abort', dispose));
|
||||
if (parentScope.signal.aborted) { scope.dispose(); return {dispose}; }
|
||||
const { resolveUrl } = window.ApiClient;
|
||||
const sender = createPromptSender({scope});
|
||||
let detailInspector;
|
||||
// System › Appearance: the display defaults this visit starts from (preferences.mjs); null without gallery-settings.js.
|
||||
@@ -271,6 +271,20 @@ export function mountImages(main, api, parentScope, {embedded = false, fetchPage
|
||||
}});
|
||||
scope.cleanup(() => mediaViewer.dispose());
|
||||
|
||||
function mediaElement(image, lazy = false) {
|
||||
const video = image.is_video && !image.thumbnail_url;
|
||||
const node = el(video ? 'video' : 'img');
|
||||
node.src = resolveUrl(image.thumbnail_url || image.url);
|
||||
if (video) { node.muted = true; node.playsInline = true; node.preload = 'metadata'; }
|
||||
else {
|
||||
// The concept card image: alt is the card title, with the listing's intrinsic size when known.
|
||||
node.alt = cardTitle(image); node.decoding = 'async'; if (lazy) node.loading = 'lazy';
|
||||
if ([image.width, image.height].every(value => Number.isFinite(value) && value > 0)) {
|
||||
node.width = image.width; node.height = image.height;
|
||||
}
|
||||
}
|
||||
return node;
|
||||
}
|
||||
function releaseVideo(video) {
|
||||
video.pause(); video.removeAttribute('src'); video.load();
|
||||
}
|
||||
@@ -292,12 +306,46 @@ export function mountImages(main, api, parentScope, {embedded = false, fetchPage
|
||||
if (!image) return;
|
||||
mediaViewer.open({items:state.items,selectedKey:imageKey(image),trigger:document.activeElement});
|
||||
}
|
||||
/** The concept media card (image-card.mjs): the image opens the viewer, Info inspects (D7–D11, D19). */
|
||||
/** The concept media card (app.js mediaCard): the image opens the viewer, Info inspects (D7–D11, D19). */
|
||||
function cardFor(image) {
|
||||
const key = imageKey(image);
|
||||
const card = createImageCard(image, {checked:bar ? bar.isChecked(key) : null, favoriteBusy:favoriteBusy(key),
|
||||
// D97: "Show image details on cards" off hides the title and file line; Info and Tag stay.
|
||||
showInfo:preferences?.showImageInfo !== false});
|
||||
const card = el('article', undefined, 'media-card');
|
||||
card.dataset.imageKey = key; card.setAttribute('role', 'listitem');
|
||||
card.style.setProperty('--media-ratio', mediaRatio(image));
|
||||
const art = el('div', undefined, 'media-art');
|
||||
const open = createButton('', 'image-open');
|
||||
open.setAttribute('aria-label', `Open ${image.filename}`);
|
||||
open.append(mediaElement(image, true));
|
||||
// D8: the selection checkbox (Images only: Library-embedded media has no selection bar), then D7: the star,
|
||||
// wherever the listing reports favorites (Images and Library-embedded media).
|
||||
const overlays = el('div', undefined, 'image-overlays');
|
||||
if (bar) {
|
||||
const label = el('label'), box = el('input');
|
||||
box.type = 'checkbox'; box.checked = bar.isChecked(key);
|
||||
box.setAttribute('aria-label', `Select ${image.filename}`);
|
||||
label.append(box);
|
||||
overlays.append(label);
|
||||
}
|
||||
if (typeof image.favorite === 'boolean') {
|
||||
const star = createButton('', 'star-button');
|
||||
star.setAttribute('aria-label', `Favorite ${image.filename}`);
|
||||
paintStar(star, image.favorite, favoriteBusy(key));
|
||||
overlays.append(star);
|
||||
}
|
||||
art.append(open, overlays);
|
||||
const info = el('div', undefined, 'media-info');
|
||||
const text = el('div');
|
||||
const title = el('h3', cardTitle(image)); title.title = image.path;
|
||||
text.append(title, el('p', cardSubtitle(image)));
|
||||
// D97: "Show image details on cards" off hides the title and file line; Info and Tag stay.
|
||||
if (preferences?.showImageInfo === false) text.hidden = true;
|
||||
const actions = el('div', undefined, 'image-card-actions');
|
||||
const inspectButton = createButton('', 'icon-btn');
|
||||
inspectButton.append(createWorkspaceIcon('info'));
|
||||
inspectButton.setAttribute('aria-label', `Inspect ${image.filename}`);
|
||||
actions.append(inspectButton, createTagButton(image));
|
||||
info.append(text, actions);
|
||||
card.append(art, info);
|
||||
markSelected(card, key === selection);
|
||||
return card;
|
||||
}
|
||||
|
||||
+10
-38
@@ -1,7 +1,7 @@
|
||||
import '../api-client.js';
|
||||
import { createButton, createScope } from '../shared/toolkit/core.js';
|
||||
import { routes, mountRouter, summarizeStats, routeTitle, loadNavCounts,
|
||||
LIBRARY_WRITE_COUNTS, NAV_COUNT_ROUTES, FAVORITE_WRITE_COUNTS, FAVORITE_COUNT_DELAY_MS, REVIEW_WRITE_COUNTS } from './state.mjs';
|
||||
LIBRARY_WRITE_COUNTS, NAV_COUNT_ROUTES, FAVORITE_WRITE_COUNTS, FAVORITE_COUNT_DELAY_MS } from './state.mjs';
|
||||
import { mountWorkspaceTheme } from './theme.mjs';
|
||||
import { mountImages, onFavoriteChange } from './images.mjs';
|
||||
import { mountLibrary, mountNewPromptAction, onLibraryWrite } from './library.mjs';
|
||||
@@ -15,7 +15,6 @@ import { createActivityStore } from './activity-store.mjs';
|
||||
import { mountActivityPage, mountActivityTopbar } from './activity.mjs';
|
||||
import { mountActivityJobView } from './activity-job-view.mjs';
|
||||
import { mountInspect } from './inspect.mjs';
|
||||
import { mountReview } from './review.mjs';
|
||||
import { createPromptSender } from './prompt-sender.mjs';
|
||||
import { mountSystem } from './system.mjs';
|
||||
|
||||
@@ -86,14 +85,13 @@ function mountWorkspace() {
|
||||
}
|
||||
};
|
||||
// SAVED VIEWS (#249 D16): the concept's saved-nav rows (search icon, name, no count), loaded with the shell and
|
||||
// re-read after every change. /* D49 */ no "PROPOSED" badge. The first rows are the concept's Favorites (star icon,
|
||||
// count, D51) and Needs review (review icon, the exact pending count, ruling R5), kept across every re-read so
|
||||
// their counts stay.
|
||||
const [favoritesLink, reviewLink] = navLinks([['favorites', 'star'], ['review', 'review']]);
|
||||
savedNav.replaceChildren(favoritesLink, reviewLink);
|
||||
// re-read after every change. /* D49 */ no "PROPOSED" badge; /* D50 */ no "Needs review" row. The first row is
|
||||
// the concept's Favorites (star icon, count), kept across every re-read so its count stays (D51).
|
||||
const [favoritesLink] = navLinks([['favorites', 'star']]);
|
||||
savedNav.replaceChildren(favoritesLink);
|
||||
const savedViews = createSavedViews(api, scope.signal);
|
||||
scope.cleanup(savedViews.subscribe(({ views }) => {
|
||||
savedNav.replaceChildren(favoritesLink, reviewLink, ...views.map(view => {
|
||||
savedNav.replaceChildren(favoritesLink, ...views.map(view => {
|
||||
const link = element('a');
|
||||
link.append(createWorkspaceIcon('search'), document.createTextNode(view.name));
|
||||
link.href = savedViewHash(view.id);
|
||||
@@ -115,21 +113,13 @@ function mountWorkspace() {
|
||||
countRequests.set(id, request);
|
||||
// Each badge lands as its own read settles; a count that failed to refresh is removed, never left stale.
|
||||
loadNavCounts(api, request.signal, (_, count) => {
|
||||
if (!request.signal.aborted) showCount(id, count);
|
||||
if (request.signal.aborted) return;
|
||||
const link = sidebar.querySelector(`nav [data-route="${id}"]`);
|
||||
link?.querySelector('.nav-count')?.remove();
|
||||
if (link && Number.isSafeInteger(count)) link.append(element('span', count.toLocaleString(), 'nav-count'));
|
||||
}, [id]);
|
||||
}
|
||||
}
|
||||
function showCount(id, count) {
|
||||
const link = sidebar.querySelector(`nav [data-route="${id}"]`);
|
||||
link?.querySelector('.nav-count')?.remove();
|
||||
if (link && Number.isSafeInteger(count)) link.append(element('span', count.toLocaleString(), 'nav-count'));
|
||||
}
|
||||
/** A count a page has just read itself: it replaces the badge, and an older read in flight no longer lands. */
|
||||
function setCount(id, count) {
|
||||
countRequests.get(id)?.abort();
|
||||
countRequests.delete(id);
|
||||
showCount(id, count);
|
||||
}
|
||||
scope.cleanup(() => { for (const request of countRequests.values()) request.abort(); });
|
||||
// A confirmed Library or vocabulary write cannot change the Images count, so only Library and Tags are re-read.
|
||||
scope.cleanup(onLibraryWrite(() => refreshCounts(LIBRARY_WRITE_COUNTS)));
|
||||
@@ -141,17 +131,6 @@ function mountWorkspace() {
|
||||
favoriteCountTimer = setTimeout(() => refreshCounts(FAVORITE_WRITE_COUNTS), FAVORITE_COUNT_DELAY_MS);
|
||||
}));
|
||||
scope.cleanup(() => clearTimeout(favoriteCountTimer));
|
||||
// A metadata scan that ends may have added items to Needs review: re-read its count once per ended scan.
|
||||
const runningScans = new Set();
|
||||
scope.cleanup(activity.subscribe(() => {
|
||||
let ended = false;
|
||||
for (const job of activity.jobs()) {
|
||||
if (job.kind !== 'scan') continue;
|
||||
if (job.state === 'running') runningScans.add(job.id);
|
||||
else if (runningScans.delete(job.id)) ended = true;
|
||||
}
|
||||
if (ended) refreshCounts(REVIEW_WRITE_COUNTS);
|
||||
}));
|
||||
refreshCounts(NAV_COUNT_ROUTES);
|
||||
let viewScope;
|
||||
let firstRender = true;
|
||||
@@ -282,13 +261,6 @@ function mountWorkspace() {
|
||||
else mountActivityPage(main, { store: activity, scope: view });
|
||||
return;
|
||||
}
|
||||
if (route.id === 'review') {
|
||||
// Needs review (ruling R5): a recorded decision re-reads the row's count, and the open page's total is the
|
||||
// count (F3: a change the page hears of costs no count read).
|
||||
mountReview(main, { api, scope: view, onDecision: () => refreshCounts(REVIEW_WRITE_COUNTS),
|
||||
onTotal: total => setCount('review', total) });
|
||||
return;
|
||||
}
|
||||
if (route.id === 'system') {
|
||||
// The concept's System (app.js renderSystem): its heading description, section nav and Save row.
|
||||
description.textContent = route.description;
|
||||
|
||||
@@ -1,449 +0,0 @@
|
||||
/**
|
||||
* Needs review state (#249 PR F): the pending queue, the selected item's sampler paths and one explicit decision, over
|
||||
* the review routes (`py/api/review.py`).
|
||||
*
|
||||
* - The queue is read 40 at a time, oldest first; `total` is the exact pending count. The cursor is bound to the
|
||||
* storage epoch only, so Load more keeps working while new items arrive.
|
||||
* - Selecting an item reads its detail: one "Sampler <id> · <positive> / <negative>" option per candidate, unusable
|
||||
* ones disabled, the first workable one chosen, then "Leave unresolved" (rulings R2/R3).
|
||||
* - Apply takes a fresh capture of the file (`observe`), then posts the decision; the server releases a capture it
|
||||
* used. A refused decision releases the capture and keeps the selection with the server's sentence as the warning;
|
||||
* a restored storage or a changed queue reads the queue again. A request that got no answer keeps its capture and
|
||||
* its exact body, and the next Apply of the same choice (`retry()`) resends that body once.
|
||||
* - Leave for later writes nothing. Tagging the image (the shared Tag dialog) never changes the item.
|
||||
* - `refresh()` re-reads the queue quietly when the server says the queue changed (F2: the review change stream:
|
||||
* a scan or the monitor added items, another tab decided one). The open item, its choice and an unanswered decision
|
||||
* stay; when the open item has left the queue the next one opens with a note. Refreshes asked for while one runs
|
||||
* collapse into one more read. Every re-read keeps the loaded depth (F3): it reads page after page until it holds as
|
||||
* many items as were shown, so Load more pages stay and the page does not jump.
|
||||
* - `sync(version)` is the stream's `ready`: it re-reads only when the queue read last is older than the version the
|
||||
* stream names (`{library_epoch, review_revision}`, the pair every list read answers), so a visit reads its queue
|
||||
* once (F3). An unknown version re-reads.
|
||||
* - A change heard while a decision is in flight is owed: it is read once the decision is refused or unanswered (a
|
||||
* recorded one leaves the page). Load more asked for during a re-read waits for it and continues from the queue it
|
||||
* read; a re-read started during Load more takes the click over and reads that page too.
|
||||
* - Apply fences on the epoch of the detail it decides: an observe under another storage releases its capture and
|
||||
* reads the queue and the item again. An unanswered decision that a re-read shows landed finishes as recorded.
|
||||
*/
|
||||
import { noun } from './plural.mjs';
|
||||
|
||||
export const REVIEW_URL = '/prompt_manager/review/items';
|
||||
const OBSERVE_URL = '/prompt_manager/image-annotations/observe';
|
||||
const CAPTURE_URL = '/prompt_manager/image-annotations/capture/';
|
||||
export const PAGE_LIMIT = 40;
|
||||
export const UNRESOLVED = 'unresolved';
|
||||
|
||||
export const COPY = Object.freeze({
|
||||
title: 'Metadata review',
|
||||
description: 'Uncertain evidence stays visible until you decide.',
|
||||
notice: 'Multiple sampler paths were found. The existing image links were preserved. Choose the correct context or leave them unresolved.',
|
||||
// D86 (plan D63, ruling R6): the empty queue's sentence.
|
||||
empty: 'Ambiguous images are added when a metadata scan or a new generation finds more than one sampler path.',
|
||||
panelTitle: 'Resolve the association',
|
||||
leaveUnresolved: 'Leave unresolved',
|
||||
// The states the concept lacks are text only (Global Constraints); D119.
|
||||
loading: 'Loading the review queue…',
|
||||
loadFailed: 'The review queue could not be read.',
|
||||
detailLoading: 'Loading sampler paths…',
|
||||
detailFailed: 'This item could not be read. Select it again to retry.',
|
||||
moreFailed: 'More items could not be loaded. Try again.',
|
||||
limited: 'This file has too many sampler paths to list. You can leave it unresolved.',
|
||||
unavailable: 'This file is not under a current image source, so it cannot be opened. You can leave it unresolved.',
|
||||
lost: 'The decision may not have been saved. Apply again to send the same decision once more.',
|
||||
storageChanged: 'Prompt storage was restored; reload the review queue.',
|
||||
// D119 (F2): the open item left the queue in a re-read, and a failed re-read.
|
||||
decidedElsewhere: name => `${name} was decided elsewhere, so the next image is open.`,
|
||||
refreshFailed: 'The review queue could not be read again. Reload the page to see the latest images.',
|
||||
failed: 'The decision could not be saved. Try again.',
|
||||
applied: 'Association accepted.',
|
||||
leftUnresolved: 'Left unresolved.',
|
||||
});
|
||||
|
||||
// Answers after which the queue itself is out of date: read it again.
|
||||
const RELOAD = new Set(['storage-changed', 'review-conflict', 'review-missing']);
|
||||
|
||||
/** The notice heading: "No images need a decision", "1 image needs a decision", "3 images need a decision". */
|
||||
export function noticeTitle(count) {
|
||||
if (!count) return 'No images need a decision';
|
||||
return `${count.toLocaleString('en-US')} ${noun(count, 'image')} ${count === 1 ? 'needs' : 'need'} a decision`;
|
||||
}
|
||||
|
||||
/** One Sampler context option's text. */
|
||||
export function optionLabel(candidate) {
|
||||
return `Sampler ${candidate.sampler_id} · ${candidate.label}`;
|
||||
}
|
||||
|
||||
const workable = candidate => candidate?.state === 'resolved' && typeof candidate.positive?.text === 'string' &&
|
||||
typeof candidate.negative?.text === 'string';
|
||||
|
||||
/** The Sampler context options of a detail: its candidates (unusable ones disabled), then Leave unresolved. */
|
||||
export function contextOptions(detail) {
|
||||
const usable = detail.target !== null && !detail.unavailable;
|
||||
const options = (Array.isArray(detail.candidates) ? detail.candidates : []).map(candidate => Object.freeze({
|
||||
value: candidate.hash, label: optionLabel(candidate), disabled: !usable || !workable(candidate), candidate,
|
||||
}));
|
||||
options.push(Object.freeze({ value: UNRESOLVED, label: COPY.leaveUnresolved, disabled: false, candidate: null }));
|
||||
return Object.freeze(options);
|
||||
}
|
||||
|
||||
/** The prompt blocks of the chosen option: its two role texts, or the ambiguous status for Leave unresolved. */
|
||||
function promptsOf(option) {
|
||||
if (!option?.candidate) return { positive: { status: 'ambiguous' }, negative: { status: 'ambiguous' } };
|
||||
const { positive, negative } = option.candidate;
|
||||
return { positive: { status: 'resolved', text: positive.text }, negative: { status: 'resolved', text: negative.text } };
|
||||
}
|
||||
|
||||
function warningOf(detail) {
|
||||
if (detail.unavailable || detail.target === null) return COPY.unavailable;
|
||||
if (detail.limited) return COPY.limited;
|
||||
return '';
|
||||
}
|
||||
|
||||
const isAbort = error => error?.name === 'AbortError';
|
||||
// No answer at all (the network, or a body that was not the server's JSON): the decision may have been saved.
|
||||
const isLost = error => !isAbort(error) && !error?.code && !(error?.status >= 400 && error?.status < 500);
|
||||
|
||||
const initial = () => ({
|
||||
status: 'idle', epoch: null, total: 0, items: Object.freeze([]), nextCursor: null, more: 'idle',
|
||||
selectedId: null, detail: null, detailStatus: 'idle', options: Object.freeze([]), choice: null, prompts: null,
|
||||
phase: 'idle', warning: '', canRetry: false, toast: null,
|
||||
});
|
||||
|
||||
/**
|
||||
* @param {{api: {get: Function, post: Function, del: Function}}} options
|
||||
* @returns the controller: `state`, `subscribe(listener)`, `load()`, `refresh()`, `loadMore()`, `select(id)`, `choose(value)`,
|
||||
* `sync(version)`, `apply()`, `retry()`, `leave()`, `dispose()`.
|
||||
*/
|
||||
export function createReviewController({ api }) {
|
||||
let state = Object.freeze(initial());
|
||||
const listeners = new Set();
|
||||
const lifetime = new AbortController();
|
||||
let listRead = null, detailRead = null, pending = null, disposed = false;
|
||||
let refreshing = null, refreshAgain = false;
|
||||
// A change heard while a decision was in flight (I1); read when the phase is idle again.
|
||||
let owedRefresh = false;
|
||||
// Load more's own read, the one waiting for a re-read, and a click a re-read took over (I2).
|
||||
let moreRead = null, moreTask = null, moreOwed = false, listing = null;
|
||||
// What the first read owes once it answers: nothing, a re-read ('always'), or a re-read unless it is this version.
|
||||
let afterLoad = null;
|
||||
// The version of the queue read last: {epoch, revision}.
|
||||
let shown = null;
|
||||
|
||||
function set(update) {
|
||||
if (disposed) return;
|
||||
state = Object.freeze({ ...state, ...update });
|
||||
for (const listener of [...listeners]) listener(state);
|
||||
}
|
||||
const signalOf = controller => AbortSignal.any ? AbortSignal.any([lifetime.signal, controller.signal]) : controller.signal;
|
||||
|
||||
function release(token) {
|
||||
if (!token) return;
|
||||
api.del(`${CAPTURE_URL}${encodeURIComponent(token)}`, { keepalive: true }).catch(() => {});
|
||||
}
|
||||
function dropPending() {
|
||||
release(pending?.body.capture_token);
|
||||
pending = null;
|
||||
}
|
||||
|
||||
const isVersion = version => typeof version?.library_epoch === 'string' && Number.isSafeInteger(version?.review_revision);
|
||||
const current = version => isVersion(version) && shown !== null && shown.epoch === version.library_epoch &&
|
||||
shown.revision >= version.review_revision;
|
||||
|
||||
/** One list page; `cursor` continues after the last item already read. */
|
||||
function readPage(read, cursor) {
|
||||
const query = new URLSearchParams({ limit: String(PAGE_LIMIT) });
|
||||
if (cursor) query.set('cursor', cursor);
|
||||
return api.get(`${REVIEW_URL}?${query}`, { signal: signalOf(read), cache: 'no-store' });
|
||||
}
|
||||
|
||||
/**
|
||||
* The queue down to `depth` items: the first page, then the next ones while fewer than `depth` are held. A restore
|
||||
* met after the first page (its cursor names the old storage) reads once more from the first page.
|
||||
*/
|
||||
async function readQueue(read, depth) {
|
||||
for (let attempt = 0; ; attempt += 1) {
|
||||
const first = await readPage(read, null);
|
||||
const items = [...(first.items ?? [])];
|
||||
let { total } = first;
|
||||
let next = first.next_cursor ?? null;
|
||||
try {
|
||||
while (next && items.length < depth) {
|
||||
const page = await readPage(read, next);
|
||||
const known = new Set(items.map(entry => entry.id));
|
||||
items.push(...(page.items ?? []).filter(entry => !known.has(entry.id)));
|
||||
total = page.total;
|
||||
next = page.next_cursor ?? null;
|
||||
}
|
||||
} catch (error) {
|
||||
if (attempt === 0 && RELOAD.has(error?.code)) continue;
|
||||
throw error;
|
||||
}
|
||||
return { first, items, total, next };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the queue as deep as it is shown; `stale` re-reads the selected item's detail too (its revision or epoch
|
||||
* moved). `quiet` (a refresh) keeps the page as it is while reading and after a failed read.
|
||||
*/
|
||||
function load(options) {
|
||||
const run = listing = loadQueue(options);
|
||||
run.finally(() => { if (listing === run) listing = null; });
|
||||
return run;
|
||||
}
|
||||
|
||||
async function loadQueue({ keepWarning = false, stale = false, quiet = false } = {}) {
|
||||
listRead?.abort();
|
||||
if (moreRead) { moreRead.abort(); moreRead = null; moreOwed = true; }
|
||||
const read = listRead = new AbortController();
|
||||
if (!quiet) set({ status: state.items.length ? state.status : 'loading', ...(keepWarning ? {} : { warning: '' }) });
|
||||
let readOk = false;
|
||||
try {
|
||||
// One page, or as many as Load more had shown (and one page more for a click this read took over).
|
||||
const depth = state.items.length + (moreOwed ? PAGE_LIMIT : 0);
|
||||
const queue = await readQueue(read, depth > PAGE_LIMIT ? depth : 0);
|
||||
if (read !== listRead || disposed) return;
|
||||
moreOwed = false;
|
||||
// M3: another storage (a restore) makes the open item's detail stale too.
|
||||
if (state.epoch !== null && queue.first.library_epoch !== state.epoch) stale = true;
|
||||
const items = Object.freeze(queue.items.map(entry => Object.freeze({ ...entry })));
|
||||
const left = state.items.find(entry => entry.id === state.selectedId);
|
||||
shown = Number.isSafeInteger(queue.first.review_revision)
|
||||
? { epoch: queue.first.library_epoch, revision: queue.first.review_revision } : null;
|
||||
set({ status: 'ready', epoch: queue.first.library_epoch, total: queue.total, items, nextCursor: queue.next,
|
||||
more: 'idle' });
|
||||
readOk = true;
|
||||
const keep = items.find(entry => entry.id === state.selectedId);
|
||||
// A re-read keeps the open item as it is (even a failed read of it, which has its own Retry).
|
||||
if (keep && !stale && (quiet || state.detail)) return;
|
||||
if (quiet && left && !keep && pending?.id === left.id) {
|
||||
// M5: the decision that got no answer was recorded (the server used its capture).
|
||||
const choice = pending.body.choice;
|
||||
pending = null;
|
||||
set({ phase: 'done', warning: '', canRetry: false,
|
||||
toast: choice === UNRESOLVED ? COPY.leftUnresolved : COPY.applied });
|
||||
} else if (quiet && left && !keep && items.length) {
|
||||
set({ warning: COPY.decidedElsewhere(left.filename) });
|
||||
await select(items[0].id, { keepWarning: true });
|
||||
} else if (items.length) await select((keep ?? items[0]).id, { keepWarning });
|
||||
else set({ selectedId: null, detail: null, detailStatus: 'idle', options: Object.freeze([]), choice: null, prompts: null });
|
||||
} catch (error) {
|
||||
if (read !== listRead || disposed || isAbort(error)) return;
|
||||
// M7: a click this read took over can be made again.
|
||||
moreOwed = false;
|
||||
if (quiet && state.status === 'ready') set({ warning: COPY.refreshFailed, more: 'idle' });
|
||||
else set({ status: 'error', more: 'idle' });
|
||||
} finally {
|
||||
// A change heard while the first read ran: that read may predate it, so read once more (unless the
|
||||
// stream's ready named a version this read already holds).
|
||||
const owed = afterLoad;
|
||||
if (readOk && owed && !disposed) {
|
||||
afterLoad = null;
|
||||
if (owed === 'always' || !current(owed)) refresh();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** The stream's `ready`: re-read only when the queue shown is older than `version`. */
|
||||
function sync(version) {
|
||||
if (disposed) return Promise.resolve();
|
||||
if (state.status === 'loading') {
|
||||
// A second version, or a change, while the first read runs: no longer one version to compare.
|
||||
afterLoad = afterLoad === null && isVersion(version) ? version : 'always';
|
||||
return Promise.resolve();
|
||||
}
|
||||
if (state.status === 'ready' && current(version)) return Promise.resolve();
|
||||
return refresh();
|
||||
}
|
||||
|
||||
/** Re-read the queue after a change elsewhere; calls made while one runs collapse into one more read. */
|
||||
function refresh() {
|
||||
if (disposed) return Promise.resolve();
|
||||
// Before the queue has been read there is nothing to keep: the first read (or Retry) reads the latest.
|
||||
if (state.status !== 'ready') {
|
||||
if (state.status === 'loading') afterLoad = 'always';
|
||||
return Promise.resolve();
|
||||
}
|
||||
if (refreshing) {
|
||||
refreshAgain = true;
|
||||
return refreshing;
|
||||
}
|
||||
refreshing = (async () => {
|
||||
do {
|
||||
refreshAgain = false;
|
||||
// A decision in flight finishes first: the page leaves when it is recorded, and a refused or
|
||||
// unanswered one reads the owed change (I1).
|
||||
if (state.phase === 'applying') { owedRefresh = true; return; }
|
||||
await load({ keepWarning: true, quiet: true });
|
||||
} while (refreshAgain && !disposed);
|
||||
})().finally(() => { refreshing = null; });
|
||||
return refreshing;
|
||||
}
|
||||
|
||||
function loadMore() {
|
||||
if (moreTask) return moreTask;
|
||||
if (!state.nextCursor || state.more === 'loading') return Promise.resolve();
|
||||
set({ more: 'loading' });
|
||||
moreTask = readMore().finally(() => { moreTask = null; });
|
||||
return moreTask;
|
||||
}
|
||||
|
||||
async function readMore() {
|
||||
// I2: a re-read in flight finishes first; the next page continues after the queue it read.
|
||||
while ((refreshing || listing) && !disposed) await (refreshing ?? listing);
|
||||
if (disposed || !state.nextCursor || state.status !== 'ready') {
|
||||
if (state.more === 'loading') set({ more: 'idle' });
|
||||
return;
|
||||
}
|
||||
if (state.more !== 'loading') set({ more: 'loading' });
|
||||
const read = moreRead = new AbortController();
|
||||
try {
|
||||
const query = new URLSearchParams({ limit: String(PAGE_LIMIT), cursor: state.nextCursor });
|
||||
const page = await api.get(`${REVIEW_URL}?${query}`, { signal: signalOf(read), cache: 'no-store' });
|
||||
if (read !== moreRead || disposed) return;
|
||||
moreRead = null;
|
||||
const known = new Set(state.items.map(entry => entry.id));
|
||||
const added = (page.items ?? []).filter(entry => !known.has(entry.id)).map(entry => Object.freeze({ ...entry }));
|
||||
set({ items: Object.freeze([...state.items, ...added]), total: page.total, nextCursor: page.next_cursor ?? null,
|
||||
more: 'idle' });
|
||||
} catch (error) {
|
||||
if (read !== moreRead || disposed || isAbort(error)) return;
|
||||
moreRead = null;
|
||||
if (RELOAD.has(error?.code)) { set({ more: 'idle' }); await load({ stale: true }); return; }
|
||||
set({ more: 'error' });
|
||||
}
|
||||
}
|
||||
|
||||
async function select(id, { keepWarning = false } = {}) {
|
||||
if (!state.items.some(entry => entry.id === id) || state.phase === 'applying') return;
|
||||
detailRead?.abort();
|
||||
dropPending();
|
||||
const read = detailRead = new AbortController();
|
||||
set({ selectedId: id, detail: null, detailStatus: 'loading', options: Object.freeze([]), choice: null, prompts: null,
|
||||
canRetry: false, ...(keepWarning ? {} : { warning: '' }) });
|
||||
try {
|
||||
const detail = await api.get(`${REVIEW_URL}/${encodeURIComponent(id)}`, { signal: signalOf(read), cache: 'no-store' });
|
||||
if (read !== detailRead || disposed) return;
|
||||
const options = contextOptions(detail);
|
||||
const chosen = options.find(option => !option.disabled);
|
||||
set({ detail: Object.freeze(detail), detailStatus: 'ready', options, choice: chosen.value, prompts: promptsOf(chosen),
|
||||
warning: (keepWarning && state.warning) || warningOf(detail) });
|
||||
} catch (error) {
|
||||
if (read !== detailRead || disposed || isAbort(error)) return;
|
||||
if (RELOAD.has(error?.code)) { set({ warning: error.message }); await load({ keepWarning: true, stale: true }); return; }
|
||||
set({ detailStatus: 'error', warning: COPY.detailFailed });
|
||||
}
|
||||
}
|
||||
|
||||
function choose(value) {
|
||||
const option = state.options.find(entry => entry.value === value);
|
||||
if (!option || option.disabled || state.phase === 'applying') return;
|
||||
// Another choice is another decision: an unanswered one is no longer resent.
|
||||
if (pending && pending.body.choice !== value) { dropPending(); set({ canRetry: false }); }
|
||||
set({ choice: value, prompts: promptsOf(option) });
|
||||
}
|
||||
|
||||
/** The phase is idle again after a refused or unanswered decision: read a change heard meanwhile (I1). */
|
||||
function idleAgain(reloading) {
|
||||
const owed = owedRefresh;
|
||||
owedRefresh = false;
|
||||
if (owed && !reloading && !disposed) refresh();
|
||||
}
|
||||
|
||||
/** Send `body`; true once the decision is recorded. */
|
||||
async function send(body) {
|
||||
const id = state.selectedId;
|
||||
const choice = body.choice;
|
||||
set({ phase: 'applying', canRetry: false });
|
||||
try {
|
||||
const result = await api.post(`${REVIEW_URL}/${encodeURIComponent(id)}/decision`, body, { signal: lifetime.signal });
|
||||
pending = null;
|
||||
set({ phase: 'done', total: result.total, warning: '',
|
||||
toast: choice === UNRESOLVED ? COPY.leftUnresolved : COPY.applied });
|
||||
return true;
|
||||
} catch (error) {
|
||||
// Leaving the page mid-request: the server may never see it, so the capture is released here (a release
|
||||
// of a capture the server already used is a harmless 404).
|
||||
if (disposed || isAbort(error)) { release(body.capture_token); return false; }
|
||||
if (isLost(error)) {
|
||||
pending = { id, body };
|
||||
set({ phase: 'idle', warning: COPY.lost, canRetry: true });
|
||||
idleAgain(false);
|
||||
return false;
|
||||
}
|
||||
pending = null;
|
||||
release(body.capture_token);
|
||||
set({ phase: 'idle', warning: error?.message || COPY.failed });
|
||||
idleAgain(RELOAD.has(error?.code));
|
||||
if (RELOAD.has(error?.code)) await load({ keepWarning: true, stale: true });
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function apply() {
|
||||
const { detail, choice } = state;
|
||||
if (state.phase !== 'idle' || state.detailStatus !== 'ready' || !detail || !choice) return false;
|
||||
// After an unanswered request, Apply resends that same decision once (its capture and body).
|
||||
if (pending?.id === state.selectedId && pending.body.choice === choice) return retry();
|
||||
dropPending();
|
||||
set({ phase: 'applying', warning: '', canRetry: false });
|
||||
let capture = null;
|
||||
if (detail.target) {
|
||||
let observed;
|
||||
try {
|
||||
// M4: not aborted on leave, so the capture it answers can be released.
|
||||
observed = await api.post(OBSERVE_URL, { target: detail.target });
|
||||
} catch (error) {
|
||||
if (disposed || isAbort(error)) return false;
|
||||
set({ phase: 'idle', warning: error?.message || COPY.failed });
|
||||
idleAgain(RELOAD.has(error?.code));
|
||||
if (RELOAD.has(error?.code)) await load({ keepWarning: true, stale: true });
|
||||
return false;
|
||||
}
|
||||
capture = observed.capture;
|
||||
if (disposed) { release(capture?.token); return false; }
|
||||
// M3: the file was observed under another storage than the detail was read from.
|
||||
if (observed.library_epoch !== detail.library_epoch) {
|
||||
release(capture?.token);
|
||||
set({ phase: 'idle', warning: COPY.storageChanged });
|
||||
idleAgain(true);
|
||||
await load({ keepWarning: true, stale: true });
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return send({
|
||||
expected_epoch: detail.library_epoch, expected_revision: detail.revision, capture_token: capture?.token ?? null,
|
||||
sha256: capture?.sha256 ?? detail.sha256, dependency_fingerprint: detail.dependency_fingerprint, choice,
|
||||
});
|
||||
}
|
||||
|
||||
/** Resend the unanswered decision once, with its own capture; false when there is nothing to resend. */
|
||||
async function retry() {
|
||||
if (!pending || state.phase !== 'idle' || pending.id !== state.selectedId) return false;
|
||||
const { body } = pending;
|
||||
pending = null;
|
||||
return send(body);
|
||||
}
|
||||
|
||||
function leave() {
|
||||
if (state.phase === 'applying') return;
|
||||
dropPending();
|
||||
set({ phase: 'done', toast: null, canRetry: false });
|
||||
}
|
||||
|
||||
return {
|
||||
get state() { return state; },
|
||||
subscribe(listener) {
|
||||
listeners.add(listener);
|
||||
return () => listeners.delete(listener);
|
||||
},
|
||||
load: () => load(), refresh, sync, loadMore, select: id => select(id), choose, apply, retry, leave,
|
||||
dispose() {
|
||||
if (disposed) return;
|
||||
dropPending();
|
||||
disposed = true;
|
||||
lifetime.abort();
|
||||
listeners.clear();
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -1,347 +0,0 @@
|
||||
/**
|
||||
* Needs review (#249 PR F): the concept's Metadata review page (docs/concepts/workspace/app.js `renderReview`
|
||||
* a749–760; style.css s407–415, s796–814 and their breakpoints) over the review controller (review-state.mjs).
|
||||
*
|
||||
* - Heading "Metadata review", its line, and **Open activity** (`#/activity`).
|
||||
* - `.notice`: "N images need a decision" and the concept's sentence; with an empty queue, D86 (plan D63, ruling R6)
|
||||
* "No images need a decision" and the scan/new-generation sentence, and nothing else.
|
||||
* - `.gallery-layout`: the queue's concept media cards (image-card.mjs; the image opens the shared viewer, Info selects
|
||||
* the item, Tag opens the shared Tag image dialog with the review scope note), then `section.surface` "Resolve the
|
||||
* association": filename, Tag image, Sampler context, the chosen path's prompt blocks and the actions Leave for later
|
||||
* and Apply association. Both return to Images (ruling R7).
|
||||
* - States the concept lacks (loading, a failed read, a refused or unanswered decision, a limited or unavailable file)
|
||||
* are text: `p.warning-note` in the panel, or the page's status line (D119).
|
||||
* - The open page re-reads its queue on each `change` of the review change stream (`/prompt_manager/review/events`,
|
||||
* F2): items a scan or the monitor added, and decisions made in other tabs. Each `ready` (the first one, and one
|
||||
* after the browser reconnects a dropped stream) names the queue's version, and the page re-reads only when the
|
||||
* queue it shows is older (F3: one read per visit). A re-read keeps the depth Load more had reached. Nothing polls.
|
||||
* - The stream lives only while the tab is visible: a hidden tab closes it and a visible one opens it again, so
|
||||
* background tabs hold no connection (I4); the reopened stream's `ready` reads only when the queue changed meanwhile.
|
||||
* A stream the browser gave up on (a 503 or a proxy error on its reconnect) is opened again after 3 s, doubling to
|
||||
* 30 s; while the stream is down the page says so in a quiet status line (D120, I3).
|
||||
* - A re-read refreshes each card whose media, target or availability changed (M6). Cards show the Images listing's
|
||||
* thumbnail when the server knows one (M10).
|
||||
* - The SAVED VIEWS row's count follows the total the page shows (`onTotal`), so a change costs no count read.
|
||||
* - D119: the queue cards carry no selection checkbox and no star (this page has no selection bar, and the queue
|
||||
* carries no favorite state); more than 40 items add the Images page's `.infinite-continuation` Load more after the
|
||||
* layout; the toasts "Association accepted." and "Left unresolved." stand in for the concept's prototype toast.
|
||||
*/
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { createReviewController, COPY, noticeTitle } from './review-state.mjs';
|
||||
import { createImageCard } from './image-card.mjs';
|
||||
import { isTagButton, openImageTagging } from './image-tagging.mjs';
|
||||
import { mountPromptBlocks } from './prompt-blocks.mjs';
|
||||
import { createPromptSender } from './prompt-sender.mjs';
|
||||
import { createMediaViewer } from './media-viewer.mjs';
|
||||
import { createInspectionController } from './inspection-state.mjs';
|
||||
import { createWorkspaceIcon } from './icons.mjs';
|
||||
import { field } from './workspace-dialogs.mjs';
|
||||
import { pageHeading } from './heading.mjs';
|
||||
import { navigateTo } from './state.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { imageKey } from './images-state.mjs';
|
||||
|
||||
export const REVIEW_EVENTS_URL = '/prompt_manager/review/events';
|
||||
// I3: reopening a stream the browser closed for good waits this long, doubling up to the maximum.
|
||||
export const REOPEN_FIRST_MS = 3000;
|
||||
export const REOPEN_MAX_MS = 30000;
|
||||
// D120: the page's line while the change stream is down (the concept has no such state).
|
||||
export const NOT_LIVE = 'Live updates are paused: changes made elsewhere appear once the connection returns.';
|
||||
const openEvents = url => new EventSource(window.ApiClient.resolveUrl(url));
|
||||
|
||||
function el(tag, text, className) {
|
||||
const node = document.createElement(tag);
|
||||
if (text !== undefined) node.textContent = text;
|
||||
if (className) node.className = className;
|
||||
return node;
|
||||
}
|
||||
function setText(node, text) {
|
||||
if (node.textContent !== text) node.textContent = text;
|
||||
}
|
||||
function setDisabled(button, disabled) {
|
||||
if (disabled) button.setAttribute('aria-disabled', 'true');
|
||||
else button.removeAttribute('aria-disabled');
|
||||
}
|
||||
const isDisabled = button => button.getAttribute('aria-disabled') === 'true';
|
||||
|
||||
/** A queue item as the card, the viewer and the Tag dialog read a listed image. */
|
||||
export function imageOf(item) {
|
||||
const target = item.target ?? {};
|
||||
return { filename: item.filename, root_index: target.root_index, root_scope: target.root_scope, path: target.path,
|
||||
url: item.url, thumbnail_url: item.thumbnail_url ?? null, is_video: item.is_video === true,
|
||||
media_type: item.is_video ? 'video' : 'image' };
|
||||
}
|
||||
|
||||
/** True when a re-read changed what a card shows or can open (M6). */
|
||||
const cardChanged = (before, after) => before.url !== after.url || before.thumbnail_url !== after.thumbnail_url ||
|
||||
before.unavailable !== after.unavailable || JSON.stringify(before.target) !== JSON.stringify(after.target);
|
||||
|
||||
/** The queue version a `ready` event names, or null (the page then re-reads). */
|
||||
export function readyVersion(event) {
|
||||
try {
|
||||
const version = JSON.parse(event?.data ?? 'null');
|
||||
return version && typeof version === 'object' ? version : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Mount the page under the shell's heading. `onDecision()` runs after a recorded decision and `onTotal(total)` each
|
||||
* time the pending total the page shows changes (the shell's Needs review count); `controller` and `connect` (the
|
||||
* change stream's EventSource) are injectable for tests.
|
||||
*/
|
||||
export function mountReview(main, { api, scope, onDecision = () => {}, onTotal,
|
||||
connect = openEvents, controller = createReviewController({ api }) }) {
|
||||
scope.cleanup(() => controller.dispose());
|
||||
const heading = pageHeading(main);
|
||||
heading.h1.textContent = COPY.title;
|
||||
heading.description.textContent = COPY.description;
|
||||
const openActivity = createButton('Open activity');
|
||||
heading.actions.append(openActivity);
|
||||
scope.listen(openActivity, 'click', () => navigateTo('#/activity'));
|
||||
|
||||
const status = el('p', COPY.loading, 'empty-state');
|
||||
status.setAttribute('role', 'status');
|
||||
const retryLoad = createButton('Retry');
|
||||
const notice = el('div', undefined, 'notice');
|
||||
const noticeCopy = el('div');
|
||||
const noticeTitleNode = el('h2');
|
||||
const noticeText = el('p');
|
||||
noticeCopy.append(noticeTitleNode, noticeText);
|
||||
notice.append(noticeCopy);
|
||||
|
||||
const layout = el('div', undefined, 'gallery-layout');
|
||||
const grid = el('div', undefined, 'image-grid review-queue');
|
||||
grid.setAttribute('role', 'list');
|
||||
grid.setAttribute('aria-label', 'Review queue');
|
||||
const panel = el('section', undefined, 'surface review-panel');
|
||||
const panelTitle = el('h2', COPY.panelTitle);
|
||||
panelTitle.id = 'review-panel-title';
|
||||
panel.setAttribute('aria-labelledby', panelTitle.id);
|
||||
const filename = el('p', '', 'small');
|
||||
const tagRow = el('div', undefined, 'row-actions spaced-15');
|
||||
const tagButton = createButton('');
|
||||
tagButton.append(createWorkspaceIcon('tagImage'), ' Tag image');
|
||||
tagRow.append(tagButton);
|
||||
const select = el('select');
|
||||
const contextField = field('Sampler context', select, 'review-context');
|
||||
const warning = el('p', '', 'warning-note'); // D119: the states the concept lacks
|
||||
warning.setAttribute('role', 'status');
|
||||
const blocks = el('div');
|
||||
const actions = el('div', undefined, 'dialog-actions');
|
||||
const later = createButton('Leave for later');
|
||||
const apply = createButton('Apply association', 'ca-primary');
|
||||
actions.append(later, apply);
|
||||
panel.append(panelTitle, filename, tagRow, contextField, warning, blocks, actions);
|
||||
layout.append(grid, panel);
|
||||
|
||||
const live = el('p', NOT_LIVE, 'small review-live'); // D120: the change stream is down
|
||||
live.setAttribute('role', 'status');
|
||||
live.hidden = true;
|
||||
heading.description.after(live);
|
||||
|
||||
const continuation = el('div', undefined, 'infinite-continuation'); // D119: Load more past 40 items
|
||||
const moreText = el('p', '', 'small');
|
||||
const more = createButton('Load more images');
|
||||
continuation.append(moreText, more);
|
||||
|
||||
const sender = createPromptSender({ scope });
|
||||
const prompts = mountPromptBlocks(blocks, { prompts: null, sender, parentScope: scope, includePairAction: false,
|
||||
onRetry: () => { if (controller.state.selectedId) controller.select(controller.state.selectedId); } });
|
||||
const inspection = createInspectionController({ api });
|
||||
scope.cleanup(() => inspection.dispose());
|
||||
const viewer = createMediaViewer({ api, container: main, parentScope: scope, inspectionController: inspection,
|
||||
sender, onSelect: image => inspection.select(image) });
|
||||
|
||||
const cards = new Map();
|
||||
let shownItems = null, shownSelected, shownOptions = null, shownPrompts, shownDetail;
|
||||
|
||||
function createCard(item) {
|
||||
// D119: no selection checkbox and no star on a queue card.
|
||||
const card = createImageCard(imageOf(item), { media: Boolean(item.url) });
|
||||
card.dataset.reviewId = item.id;
|
||||
card.item = item;
|
||||
if (!item.url) setDisabled(card.querySelector('.image-open'), true);
|
||||
if (!item.target) setDisabled(card.querySelector('.image-card-actions button:last-child'), true);
|
||||
cards.set(item.id, card);
|
||||
return card;
|
||||
}
|
||||
|
||||
function renderCards(state) {
|
||||
if (state.items !== shownItems) {
|
||||
shownItems = state.items;
|
||||
const ids = new Set(state.items.map(item => item.id));
|
||||
for (const [id, card] of cards) if (!ids.has(id)) { card.remove(); cards.delete(id); }
|
||||
state.items.forEach((item, index) => {
|
||||
let card = cards.get(item.id);
|
||||
if (card && cardChanged(card.item, item)) {
|
||||
// M6: a root came or went: the card's media, Open and Tag follow the re-read.
|
||||
const fresh = createCard(item);
|
||||
card.replaceWith(fresh);
|
||||
card = fresh;
|
||||
} else if (card) card.item = item;
|
||||
if (!card) card = createCard(item);
|
||||
if (grid.children[index] !== card) grid.insertBefore(card, grid.children[index] ?? null);
|
||||
});
|
||||
shownSelected = undefined;
|
||||
}
|
||||
if (state.selectedId !== shownSelected) {
|
||||
shownSelected = state.selectedId;
|
||||
for (const [id, card] of cards) card.classList.toggle('selected', id === state.selectedId);
|
||||
}
|
||||
}
|
||||
|
||||
function renderPanel(state) {
|
||||
const item = state.items.find(entry => entry.id === state.selectedId);
|
||||
setText(filename, item?.filename ?? '');
|
||||
setDisabled(tagButton, !item?.target || state.phase === 'applying');
|
||||
if (state.options !== shownOptions) {
|
||||
shownOptions = state.options;
|
||||
select.replaceChildren(...state.options.map(option => {
|
||||
const node = el('option', option.label);
|
||||
node.value = option.value;
|
||||
node.disabled = option.disabled;
|
||||
return node;
|
||||
}));
|
||||
}
|
||||
if (state.choice !== null && select.value !== state.choice) select.value = state.choice;
|
||||
select.disabled = state.detailStatus !== 'ready' || state.phase === 'applying';
|
||||
setText(warning, state.warning);
|
||||
warning.hidden = !state.warning;
|
||||
if (state.prompts !== shownPrompts || state.detailStatus !== shownDetail) {
|
||||
shownPrompts = state.prompts;
|
||||
shownDetail = state.detailStatus;
|
||||
if (state.detailStatus === 'ready') prompts.setState({ prompts: state.prompts });
|
||||
else prompts.setState({ loading: state.detailStatus === 'loading', error: state.detailStatus === 'error' });
|
||||
}
|
||||
setDisabled(apply, state.detailStatus !== 'ready' || state.phase !== 'idle');
|
||||
setDisabled(later, state.phase === 'applying');
|
||||
}
|
||||
|
||||
function render(state) {
|
||||
if (scope.signal.aborted) return;
|
||||
if (state.phase === 'done') return finish(state);
|
||||
const ready = state.status === 'ready';
|
||||
if (!ready) {
|
||||
setText(status, state.status === 'error' ? COPY.loadFailed : COPY.loading);
|
||||
status.setAttribute('role', state.status === 'error' ? 'alert' : 'status');
|
||||
if (!status.isConnected) main.append(status);
|
||||
if (state.status === 'error' && !retryLoad.isConnected) main.append(retryLoad);
|
||||
if (state.status !== 'error') retryLoad.remove();
|
||||
if (state.status === 'error' || !state.items.length) { notice.remove(); layout.remove(); continuation.remove(); }
|
||||
return;
|
||||
}
|
||||
status.remove();
|
||||
retryLoad.remove();
|
||||
setText(noticeTitleNode, noticeTitle(state.total));
|
||||
setText(noticeText, state.total ? COPY.notice : COPY.empty); // D86
|
||||
if (!notice.isConnected) main.append(notice);
|
||||
if (!state.items.length) { layout.remove(); continuation.remove(); return; }
|
||||
if (!layout.isConnected) notice.after(layout);
|
||||
renderCards(state);
|
||||
renderPanel(state);
|
||||
if (state.nextCursor) {
|
||||
setText(moreText, `${state.items.length.toLocaleString('en-US')} of ${state.total.toLocaleString('en-US')} images`);
|
||||
setDisabled(more, state.more === 'loading');
|
||||
if (state.more === 'error') setText(moreText, COPY.moreFailed);
|
||||
if (!continuation.isConnected) layout.after(continuation);
|
||||
} else continuation.remove();
|
||||
}
|
||||
|
||||
let finished = false;
|
||||
function finish(state) {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
if (state.toast) {
|
||||
showWorkspaceToast(state.toast);
|
||||
onDecision();
|
||||
}
|
||||
// Ruling R7: Apply association and Leave for later both return to Images.
|
||||
navigateTo('#/images');
|
||||
}
|
||||
|
||||
scope.listen(retryLoad, 'click', () => controller.load());
|
||||
scope.listen(more, 'click', () => { if (!isDisabled(more)) controller.loadMore(); });
|
||||
scope.listen(select, 'change', () => controller.choose(select.value));
|
||||
scope.listen(apply, 'click', () => { if (!isDisabled(apply)) controller.apply(); });
|
||||
scope.listen(later, 'click', () => { if (!isDisabled(later)) controller.leave(); });
|
||||
scope.listen(tagButton, 'click', () => {
|
||||
const item = controller.state.items.find(entry => entry.id === controller.state.selectedId);
|
||||
if (isDisabled(tagButton) || !item?.target) return;
|
||||
openImageTagging({ api, scope, image: imageOf(item), source: 'review', trigger: tagButton });
|
||||
});
|
||||
scope.listen(grid, 'click', event => {
|
||||
const card = event.target.closest('article[data-review-id]');
|
||||
if (!card) return;
|
||||
const { item } = card;
|
||||
const button = event.target.closest('button');
|
||||
if (!button || isDisabled(button)) return;
|
||||
if (isTagButton(button)) {
|
||||
openImageTagging({ api, scope, image: imageOf(item), source: 'review', trigger: button });
|
||||
return;
|
||||
}
|
||||
if (button.matches('.image-open')) {
|
||||
const items = controller.state.items.filter(entry => entry.url).map(imageOf);
|
||||
viewer.open({ items, selectedKey: imageKey(imageOf(item)), trigger: button });
|
||||
return;
|
||||
}
|
||||
// Info selects the item, as the concept's `inspect` does on this page (app.js a1244–1248).
|
||||
controller.select(item.id);
|
||||
});
|
||||
|
||||
let shownTotal = null;
|
||||
const reportTotal = state => {
|
||||
if (state.status !== 'ready' || state.phase === 'done' || state.total === shownTotal) return;
|
||||
shownTotal = state.total;
|
||||
onTotal?.(state.total);
|
||||
};
|
||||
scope.cleanup(controller.subscribe(render));
|
||||
scope.cleanup(controller.subscribe(reportTotal));
|
||||
render(controller.state);
|
||||
controller.load();
|
||||
connectEvents({ scope, connect, controller, live });
|
||||
return { controller };
|
||||
}
|
||||
|
||||
/**
|
||||
* The change stream while the page is mounted: open while the tab is visible (I4), reopened with a bounded backoff
|
||||
* after the browser gives up on it (I3), with `live` shown while it is down.
|
||||
*/
|
||||
function connectEvents({ scope, connect, controller, live }) {
|
||||
let events = null, timer = null, delay = REOPEN_FIRST_MS;
|
||||
const setLive = up => { live.hidden = up; };
|
||||
function close() {
|
||||
clearTimeout(timer);
|
||||
timer = null;
|
||||
events?.close();
|
||||
events = null;
|
||||
}
|
||||
function open() {
|
||||
if (events || scope.signal.aborted || document.hidden) return;
|
||||
const source = events = connect(REVIEW_EVENTS_URL);
|
||||
source.addEventListener('ready', event => {
|
||||
if (source !== events || scope.signal.aborted) return;
|
||||
delay = REOPEN_FIRST_MS;
|
||||
setLive(true);
|
||||
controller.sync(readyVersion(event));
|
||||
});
|
||||
source.addEventListener('change', () => { if (source === events && !scope.signal.aborted) controller.refresh(); });
|
||||
source.addEventListener('error', () => {
|
||||
if (source !== events || scope.signal.aborted) return;
|
||||
setLive(false);
|
||||
// The browser retries a dropped connection itself; one it closed for good is reopened here.
|
||||
if (source.readyState !== 2) return; // EventSource.CLOSED
|
||||
events = null;
|
||||
source.close();
|
||||
timer = setTimeout(() => { timer = null; open(); }, delay);
|
||||
delay = Math.min(delay * 2, REOPEN_MAX_MS);
|
||||
});
|
||||
}
|
||||
scope.listen(document, 'visibilitychange', () => {
|
||||
if (document.hidden) close();
|
||||
else open();
|
||||
});
|
||||
scope.cleanup(close);
|
||||
open();
|
||||
}
|
||||
@@ -12,10 +12,6 @@ export const routes = [
|
||||
{ id: 'overview', label: 'Overview' },
|
||||
// SAVED VIEWS' first row (#249 D51): Images filtered to favorites. No href, so Overview lists no card for it.
|
||||
{ id: 'favorites', label: 'Favorites' },
|
||||
// SAVED VIEWS' Needs review row (ruling R5): the concept's Metadata review page (app.js renderReview 749–760; its
|
||||
// breadcrumb is "Review", app.js render 1175–1188),
|
||||
// the queue of ambiguous sampler evidence. No href: no Overview card.
|
||||
{ id: 'review', label: 'Needs review', title: 'Review' },
|
||||
// The concept's Activity (app.js renderActivity 732–738): this server session's jobs. No href: no Overview card.
|
||||
{ id: 'activity', label: 'Activity' },
|
||||
// Native System (#249 System S2): `#/system/<section>`. Q12: Overview's card opens its Overview section (`native`:
|
||||
@@ -25,7 +21,7 @@ export const routes = [
|
||||
description: 'Configuration, maintenance and recovery—with browsing kept in the workspace.', action: 'Open System' },
|
||||
];
|
||||
|
||||
const NATIVE_ROUTES = new Set(['images', 'library', 'tags', 'inspect', 'overview', 'saved', 'favorites', 'activity', 'review', 'system']);
|
||||
const NATIVE_ROUTES = new Set(['images', 'library', 'tags', 'inspect', 'overview', 'saved', 'favorites', 'activity', 'system']);
|
||||
|
||||
/**
|
||||
* System's sections in the concept's order (app.js `systemSections` 635–644); the concept's `general` is production's
|
||||
@@ -92,8 +88,6 @@ const NAV_COUNT_SOURCES = Object.freeze([
|
||||
['favorites', '/prompt_manager/images/output?mode=workspace&favorites=1&limit=1'],
|
||||
['library', '/prompt_manager/library?limit=1'],
|
||||
['tags', '/prompt_manager/vocabulary/tags?limit=1'],
|
||||
// Needs review (ruling R5): the queue's `total` is the exact pending count.
|
||||
['review', '/prompt_manager/review/items?limit=1'],
|
||||
]);
|
||||
|
||||
/** Route id -> count for the sidebar links; null when a page is missing or its total is not a count. */
|
||||
@@ -108,7 +102,7 @@ export function summarizeNavCounts(pages = {}) {
|
||||
* Read the counts; a failed read yields null for its route only. `onCount(id, count)` runs as each read settles, so a
|
||||
* slow listing (the Images output scan) never holds back the cheap Library and Tags counts.
|
||||
*/
|
||||
/** The sidebar routes that carry a count (WORKSPACE, plus SAVED VIEWS' Favorites and Needs review). */
|
||||
/** The sidebar routes that carry a count (WORKSPACE, plus SAVED VIEWS' Favorites). */
|
||||
export const NAV_COUNT_ROUTES = Object.freeze(NAV_COUNT_SOURCES.map(([id]) => id));
|
||||
|
||||
/** The counts a confirmed Library or vocabulary write can change; the Images listing cannot. */
|
||||
@@ -116,8 +110,6 @@ export const LIBRARY_WRITE_COUNTS = Object.freeze(['library', 'tags']);
|
||||
|
||||
/** The count a confirmed favorite change can change. */
|
||||
export const FAVORITE_WRITE_COUNTS = Object.freeze(['favorites']);
|
||||
/** The count a review decision or a finished metadata scan can change. */
|
||||
export const REVIEW_WRITE_COUNTS = Object.freeze(['review']);
|
||||
// Trailing delay that coalesces quick star toggles into one Favorites count read (#249 M1).
|
||||
export const FAVORITE_COUNT_DELAY_MS = 300;
|
||||
|
||||
|
||||
@@ -7,7 +7,6 @@
|
||||
* N issues / Incomplete); the review, backup, thumbnail and sources rows say what the server reports, and nothing
|
||||
* reads 0 before the read answers: a load shows a status line, a failure an alert and Retry. The thumbnail row is
|
||||
* its own read of `/images/thumbnails/status` (a walk of every source): "Checking thumbnails…" until it answers.
|
||||
* D99: its Review thumbnails opens the same "Thumbnail maintenance" dialog as Sources (thumbnail-review.mjs).
|
||||
* - D96: the Statistics dialog lists the real runs and average rating per category, then "Library totals".
|
||||
* - D95: the "Health report" lists every check the server ran, Missing files per image source, with a chip per real
|
||||
* result and when it was checked. The check runs from Maintenance's notice ("Run health checks", D123) as a
|
||||
@@ -21,7 +20,6 @@ import { closeWorkspaceDialog, openContentDialog } from './workspace-dialogs.mjs
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
import { operationRow } from './system-controls.mjs';
|
||||
import { openThumbnailReview } from './thumbnail-review.mjs';
|
||||
|
||||
export const OVERVIEW_URL = '/prompt_manager/system/overview';
|
||||
export const STATISTICS_URL = '/prompt_manager/system/statistics';
|
||||
@@ -319,20 +317,22 @@ function surface(heading, rows) {
|
||||
}
|
||||
|
||||
/** The Overview's nodes, and its thumbnail row (painted by its own read). */
|
||||
function overviewNodes(overview, { scope, navigate, openStatistics, openThumbnails }) {
|
||||
function overviewNodes(overview, { scope, navigate, openStatistics }) {
|
||||
const go = section => () => navigate(`#/system/${section}`);
|
||||
const review = operationRow({ scope, title: ambiguousTitle(overview.needs_review),
|
||||
description: Number.isSafeInteger(overview.needs_review?.pending)
|
||||
? 'Review metadata before assigning a positive / negative pair.' : NOT_AVAILABLE,
|
||||
label: 'Review metadata', onClick: () => navigate('#/review') });
|
||||
label: 'Review metadata', onClick: () => {} });
|
||||
// D93 (Q6): the review queue (Needs review) is not part of the workspace yet.
|
||||
review.querySelector('button').setAttribute('aria-disabled', 'true');
|
||||
const [sourcesTitle, sourcesDescription] = sourcesCopy(overview.sources);
|
||||
const thumbnails = operationRow({ scope, title: THUMBNAILS_LOADING, description: 'Generate only missing previews when necessary.',
|
||||
label: 'Review thumbnails', onClick: openThumbnails });
|
||||
label: 'Review thumbnails', onClick: go('sources') });
|
||||
const health = surface('Library health', [
|
||||
review,
|
||||
operationRow({ scope, title: 'Backup status', description: backupText(overview.backup), label: 'View backups',
|
||||
onClick: go('backups') }),
|
||||
// D99: Review thumbnails opens the thumbnail dialog (the concept's row starts a sample job directly).
|
||||
// Until the thumbnail review dialog (S5), Review thumbnails opens Sources, where thumbnail generation lives.
|
||||
thumbnails,
|
||||
]);
|
||||
const preferences = surface('Workspace preferences', [
|
||||
@@ -352,7 +352,7 @@ function overviewNodes(overview, { scope, navigate, openStatistics, openThumbnai
|
||||
* answer replaces in place (the concept's metrics and columns sit directly in `.settings-layout`). A read that a newer
|
||||
* one superseded, or one answering after the section left, paints nothing.
|
||||
*/
|
||||
export function mountOverview(container, { api, activity, scope, navigate = navigateTo, notify = showWorkspaceToast,
|
||||
export function mountOverview(container, { api, scope, navigate = navigateTo, notify = showWorkspaceToast,
|
||||
latestOnly = window.ApiClient.latestOnly, STALE = window.ApiClient.STALE }) {
|
||||
const overviewRead = latestOnly({ lifetime: scope.signal });
|
||||
const statisticsRead = latestOnly({ lifetime: scope.signal });
|
||||
@@ -394,8 +394,7 @@ export function mountOverview(container, { api, activity, scope, navigate = navi
|
||||
try {
|
||||
const overview = await overviewRead(signal => api.get(OVERVIEW_URL, { signal, cache: 'no-store' }));
|
||||
if (overview === STALE || scope.signal.aborted) return;
|
||||
const { nodes, thumbnails } = overviewNodes(overview, { scope, navigate, openStatistics,
|
||||
openThumbnails: trigger => openThumbnailReview({ api, store: activity, scope, trigger, navigate }) });
|
||||
const { nodes, thumbnails } = overviewNodes(overview, { scope, navigate, openStatistics });
|
||||
thumbnailRow = thumbnails;
|
||||
sources = overview.sources;
|
||||
swap(nodes);
|
||||
|
||||
@@ -14,8 +14,8 @@
|
||||
* it to the host draft of `gallery_root_paths`; Remove takes one out. Both wait for the one Save row.
|
||||
* - "Watch for new outputs" (`monitoring_enabled`) and "Worker threads" (`worker_threads`, a range up to the server's
|
||||
* `cpu_count`, D113) are host drafts too. "Enrich existing metadata" reviews a scan of every source.
|
||||
* - "Thumbnail generation → Review options" opens "Thumbnail maintenance" (thumbnail-review.mjs, S5 Task 14), which
|
||||
* starts the thumbnail Activity job.
|
||||
* - "Thumbnail generation → Review options" opens the existing gallery's thumbnail tool until its review dialog (S5),
|
||||
* the D94 hand-off for the one capability not native yet.
|
||||
*
|
||||
* D98: per-source Remove; an added source reads "not saved yet" until Save; the line under the list says when
|
||||
* ComfyUI's output folder is used instead; "restart pending" for a source the running monitor does not watch yet; the
|
||||
@@ -28,7 +28,6 @@
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { closeWorkspaceDialog, field, openContentDialog } from './workspace-dialogs.mjs';
|
||||
import { openMetadataScan } from './activity-scan.mjs';
|
||||
import { openThumbnailReview } from './thumbnail-review.mjs';
|
||||
import { settingRow, operationRow } from './system-controls.mjs';
|
||||
import { HOST_UNAVAILABLE } from './system-appearance.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
@@ -37,6 +36,7 @@ import { noun } from './plural.mjs';
|
||||
const SETTINGS_URL = '/prompt_manager/settings';
|
||||
const VALIDATE_URL = '/prompt_manager/settings/validate-source';
|
||||
const RESCAN_URL = '/prompt_manager/gallery/rescan';
|
||||
const GALLERY_URL = '/prompt_manager/gallery';
|
||||
const ROOTS_KEY = 'gallery_root_paths';
|
||||
|
||||
export const NOTICE_KEY = 'pm_update_notice_dismissed';
|
||||
@@ -222,7 +222,8 @@ const controlOf = (row, id) => row.querySelector(`[data-setting="${id}"]`);
|
||||
* Activity store (scans start through it), `drafts` the System drafts.
|
||||
*/
|
||||
export function mountSources(container, { api, activity, scope, drafts, notify = showWorkspaceToast, storage = defaultStorage,
|
||||
latestOnly = window.ApiClient.latestOnly, STALE = window.ApiClient.STALE }) {
|
||||
openWindow = (url) => window.open(url, '_blank', 'noopener'), latestOnly = window.ApiClient.latestOnly,
|
||||
STALE = window.ApiClient.STALE }) {
|
||||
const statusRead = latestOnly({ lifetime: scope.signal });
|
||||
// undefined while read, null when the read failed, else GET /settings' `sources`.
|
||||
let sources;
|
||||
@@ -269,8 +270,9 @@ export function mountSources(container, { api, activity, scope, drafts, notify =
|
||||
surface.append(monitoring, workers,
|
||||
operationRow({ scope, title: 'Enrich existing metadata', description: 'Resolve both prompts, deduplicate recipes, preserve uncertain links.',
|
||||
label: 'Review scan', onClick: scanAll }),
|
||||
// D94: the gallery keeps thumbnail generation until its review dialog (S5).
|
||||
operationRow({ scope, title: 'Thumbnail generation', description: 'Generate missing previews; rebuilding all is a separate explicit option.',
|
||||
label: 'Review options', onClick: trigger => openThumbnailReview({ api, store: activity, scope, trigger }) }),
|
||||
label: 'Review options', onClick: () => openWindow(window.ApiClient.resolveUrl(GALLERY_URL)) }),
|
||||
// D98: the legacy gallery's Refresh, which rereads every source's file list.
|
||||
operationRow({ scope, title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.',
|
||||
label: 'Refresh', onClick: refresh }));
|
||||
|
||||
@@ -1,172 +0,0 @@
|
||||
/**
|
||||
* Thumbnail review (#249 System S5 Task 14): the concept's "Thumbnail maintenance" dialog (`thumbnail-review`, app.js
|
||||
* 1433–1437), opened by Sources' "Thumbnail generation → Review options" and Overview's "Review thumbnails". It starts
|
||||
* the `thumbnails` Activity job (S5 Task 12) with the chosen options.
|
||||
*
|
||||
* - Scope: "Missing thumbnails only" (`missing`, the default) or "Rebuild all (explicit opt-in)" (`rebuild`).
|
||||
* - Quality: "Medium · optimized" / "High · original" / "Low · fastest" (`medium` / `high` / `low`), preselected from
|
||||
* Appearance's Image loading quality (`gallerySettings.imageQuality`, the same three values; Q3).
|
||||
* - Preview size limit: the longest edge, a whole number from 128 to 2048 (concept default 512). A size outside that
|
||||
* marks the field `aria-invalid`, says so in the plan line and sends nothing; Start reports the field's validity.
|
||||
* - D99: a `p.small[role=status]` line before the actions says what the server's plan counts for the chosen scope
|
||||
* (`GET /images/thumbnails/plan`, read-only): "N previews to generate · M already up to date." or, for a rebuild,
|
||||
* "Every preview is regenerated: N files.", plus "· R more on the next run" past the per-run cap. Each scope is read
|
||||
* once per dialog (the size and quality change no count). The primary is "Start" (the concept's "Simulate thumbnail
|
||||
* job"): it closes the dialog and starts the job through the Activity store, which toasts that it started; the job
|
||||
* opens in Activity, where its progress lives. A start refused because another job holds the slot toasts the D57
|
||||
* busy link toast and stays.
|
||||
*/
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { closeWorkspaceDialog, field, openContentDialog } from './workspace-dialogs.mjs';
|
||||
import { busyToast } from './activity-scan.mjs';
|
||||
import { readPreferences } from './preferences.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { navigateTo } from './state.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
|
||||
export const THUMBNAIL_PLAN_URL = '/prompt_manager/images/thumbnails/plan';
|
||||
export const THUMBNAILS_KIND = 'thumbnails';
|
||||
export const SCOPE_OPTIONS = Object.freeze([['missing', 'Missing thumbnails only'], ['rebuild', 'Rebuild all (explicit opt-in)']]);
|
||||
export const QUALITY_OPTIONS = Object.freeze([['medium', 'Medium · optimized'], ['high', 'High · original'], ['low', 'Low · fastest']]);
|
||||
export const SIZE_LIMIT = Object.freeze({ min: 128, max: 2048, value: 512 });
|
||||
export const PLAN_LOADING = 'Counting previews…';
|
||||
export const PLAN_FAILED = 'The previews could not be counted.';
|
||||
export const SIZE_INVALID = `Enter a size from ${SIZE_LIMIT.min} to ${SIZE_LIMIT.max} pixels.`;
|
||||
export const START_FAILED = 'Thumbnail generation could not start. Try again.';
|
||||
|
||||
const QUALITIES = new Set(QUALITY_OPTIONS.map(([value]) => value));
|
||||
const count = value => value.toLocaleString('en-US');
|
||||
const isCount = value => Number.isSafeInteger(value) && value >= 0;
|
||||
|
||||
/** The preview size for the field's text: a whole number of pixels in range, else null. */
|
||||
export function previewSize(text) {
|
||||
if (typeof text !== 'string' || !/^\d{1,5}$/.test(text)) return null;
|
||||
const size = Number(text);
|
||||
return size >= SIZE_LIMIT.min && size <= SIZE_LIMIT.max ? size : null;
|
||||
}
|
||||
|
||||
/** Q3: Image loading quality preselects the dialog's quality; anything it does not know is Medium. */
|
||||
export function preselectedQuality(imageQuality) {
|
||||
return QUALITIES.has(imageQuality) ? imageQuality : 'medium';
|
||||
}
|
||||
|
||||
/** D99: the plan line for the server's `{mode, targets, fresh, remaining}`; an answer without counts failed. */
|
||||
export function planText(plan) {
|
||||
if (!plan || !['missing', 'rebuild'].includes(plan.mode) || ![plan.targets, plan.fresh, plan.remaining].every(isCount)) {
|
||||
return PLAN_FAILED;
|
||||
}
|
||||
const more = plan.remaining ? ` · ${count(plan.remaining)} more on the next run` : '';
|
||||
if (plan.mode === 'rebuild') return `Every preview is regenerated: ${count(plan.targets)} ${noun(plan.targets, 'file')}${more}.`;
|
||||
return `${count(plan.targets)} ${noun(plan.targets, 'preview')} to generate · ${count(plan.fresh)} already up to date${more}.`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Start the thumbnail job with `options` (`{mode, size, quality}`) and open its view; a refused or failed start says
|
||||
* why and stays. Nothing is said once the section that started it is gone.
|
||||
*/
|
||||
export async function startThumbnails({ store, scope, options, notify = showWorkspaceToast, navigate = navigateTo }) {
|
||||
try {
|
||||
const result = await store.start(THUMBNAILS_KIND, options);
|
||||
if (scope.signal.aborted) return;
|
||||
if (result.busy !== undefined) {
|
||||
notify(busyToast(result.busy));
|
||||
return;
|
||||
}
|
||||
navigate(`#/activity/${result.job.id}`);
|
||||
} catch {
|
||||
if (!scope.signal.aborted) notify(START_FAILED);
|
||||
}
|
||||
}
|
||||
|
||||
function el(tag, text, className) {
|
||||
const node = document.createElement(tag);
|
||||
if (text !== undefined) node.textContent = text;
|
||||
if (className) node.className = className;
|
||||
return node;
|
||||
}
|
||||
|
||||
function select(options, value) {
|
||||
const node = el('select');
|
||||
node.append(...options.map(([optionValue, label]) => {
|
||||
const option = el('option', label);
|
||||
option.value = optionValue;
|
||||
return option;
|
||||
}));
|
||||
node.value = value;
|
||||
return node;
|
||||
}
|
||||
|
||||
const defaultQuality = () => {
|
||||
try {
|
||||
return readPreferences().values.imageQuality;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Open "Thumbnail maintenance" on the section's `scope`. `store` is the page's Activity store; `trigger` gets focus
|
||||
* back when the dialog closes.
|
||||
*/
|
||||
export function openThumbnailReview({ api, store, scope, trigger = null, quality = defaultQuality(), notify = showWorkspaceToast,
|
||||
navigate = navigateTo }) {
|
||||
const mode = select(SCOPE_OPTIONS, 'missing');
|
||||
const encoder = select(QUALITY_OPTIONS, preselectedQuality(quality));
|
||||
const size = el('input');
|
||||
size.type = 'number';
|
||||
size.value = String(SIZE_LIMIT.value);
|
||||
size.min = String(SIZE_LIMIT.min);
|
||||
size.max = String(SIZE_LIMIT.max);
|
||||
const line = el('p', PLAN_LOADING, 'small');
|
||||
line.setAttribute('role', 'status');
|
||||
const cancel = createButton('Cancel');
|
||||
const start = createButton('Start', 'ca-primary');
|
||||
const actions = el('div', undefined, 'dialog-actions');
|
||||
actions.append(cancel, start);
|
||||
|
||||
// The plan per scope: a promise of its answer (null when it failed); the size and quality change no count.
|
||||
const plans = new Map();
|
||||
let open = true;
|
||||
const chosenSize = () => previewSize(size.value);
|
||||
const validate = () => {
|
||||
const valid = chosenSize() !== null;
|
||||
size.setCustomValidity(valid ? '' : SIZE_INVALID);
|
||||
if (valid) size.removeAttribute('aria-invalid');
|
||||
else size.setAttribute('aria-invalid', 'true');
|
||||
return valid;
|
||||
};
|
||||
const readPlan = chosen => {
|
||||
const query = new URLSearchParams({ mode: chosen, size: String(chosenSize()), quality: encoder.value });
|
||||
return api.get(`${THUMBNAIL_PLAN_URL}?${query}`, { signal: scope.signal, cache: 'no-store' }).catch(() => null);
|
||||
};
|
||||
async function paint() {
|
||||
if (!validate()) { line.textContent = SIZE_INVALID; return; }
|
||||
const chosen = mode.value;
|
||||
if (!plans.has(chosen)) plans.set(chosen, readPlan(chosen));
|
||||
const pending = plans.get(chosen);
|
||||
line.textContent = PLAN_LOADING;
|
||||
const plan = await pending;
|
||||
// A later choice painted its own line meanwhile, or the dialog closed.
|
||||
if (!open || scope.signal.aborted || mode.value !== chosen || !validate()) return;
|
||||
line.textContent = planText(plan);
|
||||
}
|
||||
|
||||
scope.listen(mode, 'change', paint);
|
||||
scope.listen(size, 'input', paint);
|
||||
scope.listen(cancel, 'click', () => closeWorkspaceDialog());
|
||||
scope.listen(start, 'click', () => {
|
||||
if (!validate()) {
|
||||
size.reportValidity();
|
||||
size.focus();
|
||||
return;
|
||||
}
|
||||
const options = { mode: mode.value, size: chosenSize(), quality: encoder.value };
|
||||
closeWorkspaceDialog();
|
||||
startThumbnails({ store, scope, options, notify, navigate });
|
||||
});
|
||||
openContentDialog({ scope, title: 'Thumbnail maintenance', trigger,
|
||||
nodes: [field('Scope', mode, 'thumb-mode'), field('Quality', encoder, 'thumb-quality'),
|
||||
field('Preview size limit', size, 'thumb-limit'), line, actions],
|
||||
onClose: () => { open = false; } });
|
||||
paint();
|
||||
}
|
||||
@@ -24,10 +24,6 @@ main:focus-visible { outline: 2px solid var(--ca-focus); outline-offset: 4px; }
|
||||
.activity-pill:hover { border-color: var(--ca-accent); /* D53 */ }
|
||||
.activity-pill-bar { display: block; inline-size: 70px; block-size: 5px; border-radius: 3px; background: var(--line); overflow: hidden; /* D53 */ }
|
||||
.activity-pill-bar i { display: block; block-size: 100%; inline-size: 0; background: var(--ca-accent); /* D53 */ }
|
||||
/* D111 (S5): a running job without a known total sweeps a short segment; reduced motion shows a dimmed full bar. */
|
||||
.activity-pill-bar.indeterminate i { inline-size: 40%; animation: activity-pill-sweep 1.4s ease-in-out infinite; /* D111 */ }
|
||||
@keyframes activity-pill-sweep { from { transform: translateX(-100%); } to { transform: translateX(250%); } }
|
||||
@media (prefers-reduced-motion: reduce) { .activity-pill-bar.indeterminate i { inline-size: 100%; opacity: .45; animation: none; /* D111 */ } }
|
||||
@media (pointer: coarse) { .activity-pill:not([hidden]) { min-block-size: 40px; /* D53 */ } }
|
||||
@media (max-width: 700px) {
|
||||
/* D52: with the Activity button the full-width row can be wider than a zoomed phone, so its items wrap. */
|
||||
@@ -398,17 +394,6 @@ pre.prompt-text { font: inherit; font-size: 12px; line-height: 1.75; margin-bloc
|
||||
.ca-tools #tag-suggestions .chip.tag-suggestion > button { min-block-size: 40px; min-inline-size: 40px; }
|
||||
}
|
||||
|
||||
/* Needs review (#249 PR F): the concept's Metadata review page (app.js a749–760) in its own classes. */
|
||||
/* #249 D3: the review queue is not virtualized, so its cards stay in the concept grid's flow. */
|
||||
.image-grid.review-queue > .media-card { position: static; /* D3 */ }
|
||||
/* #249 D20: the prompt blocks' mount point is transparent, so the chosen path's sections sit in the panel as the
|
||||
concept's do. */
|
||||
.review-panel > .prompt-inspector { display: contents; /* D20 */ }
|
||||
/* D119: a file that cannot be served (no current root) has an inert, empty media button that keeps the concept's
|
||||
1.33 media area (app.js `.image-open img`), so its card lines up with the queue's other cards. */
|
||||
.review-queue .image-open[aria-disabled="true"] { cursor: not-allowed; aspect-ratio: 1.33; /* D119 */ }
|
||||
/* D119: a long file name in the panel wraps inside the panel, where the concept's sample name never overflows. */
|
||||
.review-panel > p.small { min-inline-size: 0; overflow-wrap: anywhere; /* D119 */ }
|
||||
/* Auto-tag selected images (#249 PR D, D85/D90): the Tag image dialog's model panel, then How to apply, a radio group
|
||||
in the concept's `.field` row whose radios keep their own size (the concept's `.field input` is a full-width text
|
||||
field) beside their label, as the concept's own radio rows (s1160 `.compare-item label`). */
|
||||
|
||||
Reference in New Issue
Block a user