Compare commits

..
Author SHA1 Message Date
Vito Sansevero 566f6de23d Merge remote-tracking branch 'origin/main' into test/tagging-panel-settle 2026-10-23 02:36:11 -07:00
Vito Sansevero a288a427e3 Merge remote-tracking branch 'origin/main' into test/tagging-panel-settle 2026-10-23 00:52:03 -07:00
Vito Sansevero a6143db75e test(annotations): capture the details panel after it has loaded
"Tag opens the dialog without selecting the card" read the panel while it
still said "Loading file details…", then compared it with the same card's
finished details. On a slow WebKit runner the read raced the load.
2026-10-22 20:25:09 -07:00
150 changed files with 693 additions and 25099 deletions
+14 -792
View File
@@ -50,13 +50,6 @@ stdin open and the server (`--exit-on-stdin-eof`) shuts down cleanly, removing i
temp root, when that pipe closes, so a worker killed without teardown leaves no
server behind. A test that fails gets the stderr its server wrote during the test
attached to the report as `server-stderr` (an aiohttp traceback behind a 500).
A `/__test__/.../reset` still running after 2 s writes a report to the job log,
passing test or not: every thread's stack taken while it waits, the default
executor's threads and queue, the open connections and WAL size, then the total
and how long each executor call queued and ran. Its lines start with
`pm-slow-reset| ` (`tests/browser/slow_reset_watch.py`). The report never reads a
connection another thread is using: one that is busy inside SQLite would block
that read with the GIL held and freeze the whole server.
The coverage run can be split the way CI splits it, and each shard keeps the raw
V8 files it writes under `coverage/js/raw`:
@@ -121,13 +114,6 @@ npm run --prefix tools/vendor test:comfyui
CI performs this setup automatically. `PM_COMFY_URL` can select another
`http://127.0.0.1:PORT/` address (default port 8189). This is an owned managed
server: a running host is never reused. Do not point it at your live checkout.
One port serves one run. The launcher refuses a port that is already in use and
reports ready only when `/system_stats` names its own base directory, so two runs
started together on 8189 fail at startup ("already in use" or "answered by a
ComfyUI host this run did not start") instead of one run testing the other's
host and losing it mid-test (`ECONNREFUSED`, `ERR_CONNECTION_REFUSED`, Network
Error). Every test checks the answering host again. Agents sharing a machine
give each run its own `PM_COMFY_URL` port.
The launcher verifies the upstream SHA and clean tracked files, snapshots only
tracked PromptManager files, and creates temporary user/input/output/database/
configuration/log directories. Personal untracked files are excluded. It runs
@@ -392,8 +378,7 @@ with source `zeroed-conditioning`. `easy ifElse` with a literal boolean follows
branch only; with a linked boolean both branches are traced, equal texts resolve and
different texts make the sampler `ambiguous` with one candidate per choice (at most
32, else `work-limit`). ConditioningCombine/Concat/Average resolve only when every
part has the same text, otherwise `unresolved` with `composite` naming the node
(but see the added-conditioning floor below for a later stage).
part has the same text, otherwise `unresolved` with `composite` naming the node.
Results that depend on a forced choice are never memoized. A `FaceDetailer`
(`detailer-stage`) is a sampler stage: the samplers reached through its `image`
ancestry are compared with it. Equal texts resolve with the detailer as the sampler;
@@ -404,65 +389,9 @@ A sampler's latent ancestry is walked through every link input of non-sampler
nodes within the same visit and depth budgets: with no earlier sampler it is a
single stage whatever the origin (VAEEncode, EmptyLatentBatch, EmptyFlux2LatentImage,
EmptyHunyuanLatentVideo, SetLatentNoiseMask, InpaintModelConditioning or
WanImageToVideo slot 2). Pass-through lineage (#304 follow-up, maintainer option B,
2026-10-23): the table's `lineage` section lists the outputs whose image or latent
continues one input's (`VAEDecode`/`VAEDecodeTiled` `samples`, `VAEEncode`/
`VAEEncodeTiled` `pixels`, `ImageUpscaleWithModel`, `ImageScale`, `ImageScaleBy`,
`ImageScaleToTotalPixels` `image`, `LatentUpscale`/`LatentUpscaleBy` `samples`,
cg-image-filter `Image Filter` slot 0 `images` and slot 1 `latents`; a selection of
the same batch), plus Reroutes. Lineage describes no prompt, so its classes are not
claimed by a prompt category. When the latent walk finds exactly one earlier sampler
and the latent descends from that sampler's output slot 0 only through lineage
nodes (a hires-fix or refiner chain, img2img from an in-graph decode, a direct
latent link), the two stages are compared like a detailer and its upstream stage:
equal role texts resolve with the later sampler named, different texts are
`ambiguous` `multistage-differs` with one candidate per stage, and an unreadable
earlier stage makes the chain `unresolved` with that stage's reason. Unanchored, the
earlier sampler is judged inside the chain when its output reaches no other file
output. A size or resolution node reading the same image beside the lineage (a
`ResolutionCalculator` feeding `ImageScale` width and height) is allowed, since the
walk then still finds only that sampler. Any other earlier sampler, or one reached
only through an unlisted image node, stays `unresolved` "Multistage sampler ancestry
is ambiguous". Added-conditioning floor: a later stage whose own positive fails only
as `composite` keeps its earlier stages' shared pair ("A later stage adds
conditioning to the base prompt", naming the earlier stage's sampler) when that
positive reaches, through Reroutes and conditioning pass-throughs, a composite the
table marks `additive` (ConditioningCombine, ConditioningConcat; never
ConditioningAverage, which can weight a part away), one of its parts is exactly the
earlier stages' positive text, and its negative text equals theirs. This is the
three-stage Flux run whose UltimateSDUpscale combines the PromptManager prompt with
a style-trigger encoder: legacy linked the PromptManager text and every stage is
conditioned on it. The positive is always labelled (maintainer ruling): its source is
`added-conditioning` (D126; with a D125 omission too,
`runtime-parts-omitted-and-added-conditioning`) and its provenance names the
composite (`added_by`), whatever the added text is; no length heuristic. A combine without the base text, a different negative, or
different earlier stages stay as they were (`composite` or `multistage-differs`).
An image-edit batch (several samplers with distinct prompts, one shared saver
prefix) stays `ambiguous`, and text generated at run time stays untraceable
(`tests/test_resolver_pass_through.py` on synthetic shapes from
`tests/passthrough_graphs.py`; the browser port is compared on the named
`select-*` chain cases and `passthrough_fuzz_cases`). Ingestion keeps the default
profile's historical latent rule (D1), so the admin scan still reaches these files
through its legacy-parser fallback. Run-time PromptManager parts (maintainer option
A, 2026-10-23): LoraManager's `TriggerWord Toggle` is a `runtime` class (its output is
the LoRA's trigger words as filtered at run time). A PromptManager (or
PromptManagerPair) `prepend_text` or `append_text` linked directly to a `runtime`
node is left out: the text is the node's own, combined with any other literal or
traced part exactly as before, the side's source is `runtime-parts-omitted`, and a
resolved pair's reason is "Text added at run time, such as trigger words, is not
included". The prompt block shows that line under the text (D125; the evidence
badge stays "Workflow inferred"). The label follows the text through encoders,
concatenations and chained PromptManagers: a side is labelled when any node on its
traced path left a part out, and its provenance lists those PromptManager nodes
(`omitted_parts`). Comparisons include what a text leaves out, so a prompt that left
out a part never equals a complete one (a branch with the toggle and one without stay
ambiguous; such stages are `multistage-differs`). Only classes in the table's
`runtime` list are left out, never another `runtime-value` source (a JoinStringMulti
with a linked list flag stays `runtime-value`). The toggle's output is never computed. A run-time node further upstream (inside a
concatenation), a run-time main `text`, or an unknown feeder keep their outcome
(`runtime-value` or `unsupported-node`). This is what legacy linked for the June
hires-fix runs (`TestRunTimePromptManagerParts`, the `select-runtime-*` parity
cases); an unlisted node with sampler-shaped inputs (positive, negative, guider,
WanImageToVideo slot 2). An earlier sampler (img2img from a decode, hires fix) stays
`unresolved` "Multistage sampler ancestry is ambiguous" until stage texts are
compared; an unlisted node with sampler-shaped inputs (positive, negative, guider,
latent_image, a pipe, video embeds, custom-sampler parts, or a `model` link together
with conditioning, neg_conditioning, samples or latent) is `unsupported-node`
naming it, here and in a detailer's image ancestry; a latent that is not a link or
@@ -627,9 +556,8 @@ Recorded shell deviations:
`a saved view row matches the concept's` saves one view in both documents and
compares the rows; `tests/browser/workspace-saved-views.spec.mjs` covers saving,
reopening, the defaults that replace rejected values, and failed reads and saves.
- **D50** (retired by #249 PR F, ruling R5): SAVED VIEWS now lists the concept's
"Needs review" row after Favorites, with its count; the comparison maps nothing out
for it (only the concept's sample count "3" reads as the fixture's "0").
- **D50**: no "Needs review" row in SAVED VIEWS until the tag-review slice (maintainer
ruling, 2026-10-21). The comparison maps only that row out.
- **D51** (maintainer ruling, 2026-10-21, beyond the concept): SAVED VIEWS starts with
the concept's Favorites row (star icon, name, count). Its count is the Favorites
listing's own `limit=1` total (`/images/output?mode=workspace&favorites=1`), follows
@@ -1749,20 +1677,8 @@ mapped differences"):
A video replaced while still starting (a source, but neither metadata nor an error yet)
stays where it is, inert and `aria-hidden`, with the new preview inserted after it, and
the session's revoke of its URL waits; once it reports metadata or an error (or after
15 s) it is released as above and only then is its URL revoked. For that moment (the
time to metadata, or at most 15 s for a video that never settles) the old video and the
new preview both show, the new one below it: a known, brief overlap. Stacking them in one
slot instead hung again, because it changes the video's own box: the old video
absolutely positioned in its own box hung 5 of 120 pages (8 of 120 with `opacity: 0`),
and a one-cell grid 4 of 120. Only `opacity: 0` on the video, left in the flow, did not
hang (0 of 120), and alone it would leave an empty hole above the new preview. A route
change within the workspace waits the same way: the shell renders the next route once a
video still starting on Inspect settles (`leaveGuard`; the latest route wins). Route
changes at `loadstart` + 0–7 ms hung 4–5 of 120 pages before and 0 of 120 after.
A real page leave cannot wait. With no page code releasing anything, navigating
away hung 8 of 120 pages and reload 22 of 120 (3 and 8 of 120 with the page's own
release), so that WebKit teardown stays a known limitation; dispose still releases at
once. Cause, found with native stacks of the hung WPE web process in the
15 s) it is released as above and only then is its URL revoked. Page leave and dispose
release it at once. Cause, found with native stacks of the hung WPE web process in the
pinned image: WebKit (GStreamer) deadlocks when a video leaves the page or the viewport
before its pipeline has started. The main thread changes the pipeline state and waits
for the source's streaming thread, which is itself waiting on the main thread in a
@@ -1773,9 +1689,8 @@ mapped differences"):
`loadedmetadata` hung 0 of 180. Specs: `replacing a video that just started loading
never blocks the page`, and `a video replaced while still starting is reset once it
settles`, which now runs on WebKit too (15 fresh pages, replaced at `loadstart` + 0–4
ms), and `leaving for another route while a video starts waits for it to settle`. The
viewer's own video teardown was checked the same way (`closing the viewer on a video
that just started never blocks the page`, 15 fresh pages on WebKit) and needed no
ms). The viewer's own video teardown was checked the same way (`closing the viewer on a
video that just started never blocks the page`, 15 fresh pages on WebKit) and needed no
change.
- **D63** (status line): one `p.small.spaced-16` in `#local-preview`, for states the
concept never shows: "Drop one file at a time.", "Folders cannot be inspected. Choose
@@ -1881,518 +1796,6 @@ npm run --prefix tools/vendor test:workflows -- inspect-concept.spec.mjs workspa
npm run --prefix tools/vendor test:comfyui -- inspect-sender.spec.mjs
```
### Needs review page and recorded deviations (#249 PR F)
`#/review` is the concept's Metadata review page (app.js `renderReview` a749–760) over
the review routes (`web/js/workspace/review.mjs`, `review-state.mjs`): breadcrumb and
document title "Review" (the concept's), heading "Metadata review" with its line and
**Open activity** (`#/activity`), the `.notice` "N images need a decision" with the
concept's sentence, then `.gallery-layout`: the queue's concept media cards
(`image-card.mjs`, shared with Images; the image opens the shared viewer, Info selects
the item, Tag opens the shared Tag image dialog with the review scope note) and
`section.surface` "Resolve the association" (filename, Tag image, Sampler context,
the chosen path's prompt blocks, Leave for later and primary Apply association). The
first item is selected on load, as the concept's first image is. Sampler context lists
"Sampler <id> · <positive> / <negative>" per candidate (the server's label), disables
unusable ones, preselects the first workable one and ends with "Leave unresolved"
(rulings R2/R3); "Leave unresolved" shows the ambiguous status in both blocks. Apply
takes a fresh capture (`observe`) and posts the decision; Leave unresolved on a file
under no current root posts without a capture. Both actions return to Images (R7); a
recorded decision re-reads the row's count, as does every metadata scan that ends. A
refused decision keeps the selection and shows the server's sentence (the
association conflict's explanation included) as `p.warning-note` in the panel; a
restored storage or a changed queue reads the queue again; a request with no answer
keeps its capture and body, and the next Apply of the same choice resends it once.
The SAVED VIEWS row "Needs review" (review icon) follows Favorites with the exact
pending count (`GET /prompt_manager/review/items?limit=1`, its `total`).
- **D86** (plan D63, ruling R6): an empty queue shows only the notice, "No images need a
decision" and "Ambiguous images are added when a metadata scan or a new generation
finds more than one sampler path."
- **D87** (plan D64): an ended metadata scan whose `scope.review_items` is above 0
carries the concept's **Review metadata** (app.js a738 `go-review`): in its Activity
item after the item's line, and in its job view's heading after Download report. The
item's chip still names its state (D54), where the concept's says "Needs attention".
- **D119**: the queue cards have no selection checkbox and
no star (the page has no selection bar and the queue carries no favorite state); a
file under no current root has an empty, `aria-disabled` image button and Tag; more
than 40 items add the Images page's `.infinite-continuation` ("40 of 45 images",
Load more images) after the layout; loading and a failed read are the page's status
line (with Retry); a limited or unavailable file and the decision states are text in
the panel's `p.warning-note`; the toasts "Association accepted." and "Left
unresolved." replace the concept's prototype toast. The concept's "Sample review
queue" chip and "sample" wording are prototype copy (D29). When a re-read (below)
finds the open item gone, the next one opens with "<file> was decided elsewhere, so
the next image is open."; a failed re-read keeps the queue and says "The review queue
could not be read again. Reload the page to see the latest images." A queue card's
title is its file name and its line the extension only (an undecided item names no
prompt, and D11 omits the size and seed a queue item does not carry); its image's alt
is the file name; the panel's Copy and Send buttons carry no prompt title in their
names; the Tag image dialog opened from the queue heads with the file name. An
unavailable file's empty media button keeps the concept's 1.33 media area, so its
card lines up with the others. A long file name in the panel's `p.small` wraps inside
the panel (`overflow-wrap: anywhere`), where the concept's short sample name never
overflows. A queue card shows the Images listing's thumbnail when the server knows one
(`thumbnail_url`, the same URL the Images page uses), else the original; a re-read
rebuilds a card whose media, target or availability changed (a root came or went), so
Open and Tag follow it. When an unanswered decision's item leaves the queue in a
re-read, the decision was recorded: the page finishes as a recorded decision does (its
toast, back to Images), not with "decided elsewhere".
- **D120** (the concept has no live-update state): while the
change stream is down, the heading carries the quiet status line "Live updates are
paused: changes made elsewhere appear once the connection returns." (`p.small
.review-live`, `role="status"`); the stream's next `ready` hides it.
**Live queue (F2).** An open Needs review page holds one server-sent event stream,
`GET /prompt_manager/review/events` (`py/api/review_events.py`): `event: ready` when it
opens, then `event: change` after each committed queue change (an item the scan or the
monitor recorded, a spelling refresh, a decision from any tab; `database/review_changes.py`
notifies after the commit, from whichever thread wrote it) and after a restore. `change`
carries no data. `ready` carries the queue's version `{library_epoch, review_revision}`,
read after the stream subscribed, and every list read answers the same pair
(`review_revision` is the review index revision, which every queue change moves; a
restore moves the epoch). **One read per visit (F3):** the page re-reads on `ready` only
when the queue it shows is older than that version (an unknown version, `{}` when it
cannot be read, re-reads); a `ready` heard while the first read runs is compared once that
read answers. The test server counts a stream as open only once it has sent `ready`, so a
browser test that changes the queue after that count is a `change`, never part of the
version. The page re-reads on every `change`, keeping the open item, its choice and an
unanswered decision; re-reads asked for while one runs collapse into one more. A change
heard while a decision is in flight is owed: a refused or unanswered decision reads it
once the panel is idle again (a recorded one leaves the page). Load more asked for during
a re-read waits for it and continues after the queue it read; a re-read started during
Load more takes that click over and reads one page deeper, and a failed re-read leaves
Load more clickable. Apply fences on the epoch of the detail it decides: an observe under
another storage releases its capture and reads the queue and the item again, and any
re-read under another epoch reads the open item again. **Every re-read keeps the loaded depth (F3):** it
reads page after page (40 each, the route's limit) until it holds as many items as Load
more had shown, so the cards stay and the scroll position does not move; a restore met
after the first page reads once more from the first page. The row's count follows the
total the page shows (no count read per change). Nothing polls: a `: keepalive` comment
every 25 s only lets the server drop a closed tab's subscription. The server sends the
first change at once and collapses the notices of the next 0.4 s into one trailing
`change` (`COALESCE_SECONDS`), so a busy scan costs an open page one re-read per 0.4 s at
most. Shutdown (`on_shutdown`) ends every stream and answers a later one 503. A
reconnect the browser gives up on (a 503, or a proxy's error while ComfyUI restarts) is
opened again by the page after 3 s, doubling to 30 s, with D120's line shown while
the stream is down; its `ready` re-syncs the queue. The stream lives only while the tab is
visible: a hidden tab closes it and a visible one opens it again, and that `ready` reads
only when the queue changed meanwhile. A duplicate recording and a refused decision change
nothing and send nothing. Residual: each visible Needs review page holds one HTTP/1.1
connection (browsers allow six per origin, shared with ComfyUI's editor), so several
side-by-side visible windows on the page, during a metadata scan's job stream, can still
queue the origin's other requests; a re-read of a deep queue costs one request per 40
items shown.
Pinned by `tests/test_api_review_events.py`, `tests/js/workspace-review-state.test.js`
("refresh …", "a refresh keeps the loaded depth …", "ready naming …") and
`tests/browser/workspace-review.spec.mjs` ("an open page shows …", "a visit reads its
queue once …", "a live re-read keeps the loaded depth and the scroll position", "a stream
the browser gives up on …", "a hidden tab closes its stream …", "a re-read refreshes a
card …"); the controller's owed, Load more and epoch cases are "I1: …", "I2: …", "M3:
…", "M5: …" and "M7: …" in `workspace-review-state.test.js`, and the coalescing is "a
burst of changes …" in `test_api_review_events.py`.
Leaving the page mid-decision releases that decision's capture (the server may never see
the request), and leaving during the observe releases the capture it then answers
("leaving during the observe …").
**Concept comparison (F3).** `tests/browser/review-concept.spec.mjs` measures the whole
page (`reviewInventory` in `tests/browser/tagging-canonical-comparison.mjs`: every shown
box below `main` with its place, fonts, edges and layout, a leaf's text, a button's name,
a select's options) in the concept at `#review` and in production on the fixture's
concept-shaped queue (`POST /__test__/review/reset {items: 3, concept: true}`: the
concept's first three image names and sizes, and the concept's first prompt on the open
item), and `mapConceptReview` applies only: D29 (no "Sample review queue" chip, no
"sample" in the panel heading and Apply), D119 (above), D3 (the grid's
`position: relative`), production's hooks and accessible names (`review-queue`,
`review-panel`, the list roles, the region names) and sample data (the candidate options
except Leave unresolved; widths and heights that follow from the changed copy, moved by
exactly that much). It then matches at 1600×1100 and 390×844 in both themes on all four
projects; the gallery layout follows the concept at 1600, 1200, 900 and 390px (310px,
275px with a 15px gap, 280px, one column); the SAVED VIEWS rows match with Needs review
after Favorites; the queue's Tag image dialog matches the concept's needs-review dialog
(`needs-review-tagging.png`) under the C4 mapping; and the comparison is shown to notice
a dropped notice border and reordered actions. The paired captures (populated queue in
both themes and widths, conflict, limited item, unavailable file, empty queue) are test
artifacts; no screenshot baseline is added. The comparison found one gap, the collapsed
media area of an unavailable file's card, fixed above (`workspace-review.spec.mjs` "a
limited item and an unavailable file …" pins it).
- **D3** and **D20** apply to the queue grid (its cards stay in the grid's flow; it is
not virtualized) and to the panel's prompt blocks (their mount point is transparent).
Tests: `tests/browser/review-concept.spec.mjs` (concept comparison),
`tests/js/workspace-review-state.test.js` (controller),
`tests/js/workspace-state.test.js` (route and count),
`tests/browser/workspace-review.spec.mjs` (queue, count, apply, leave, Tag, empty
queue, Load more, Activity's Review metadata; items recorded by the real recorder in
`tests/browser/workspace_review_fixture.py`), and the SAVED VIEWS rows in
`shell-concept.spec.mjs` and `workspace-saved-views.spec.mjs`.
### Native image tagging and Needs review (#249 PRs A–F)
One place for the whole image-tagging slice; the details live in the sections named.
- **Routes.** Tagging: `POST /prompt_manager/image-annotations/observe`, `GET`/`DELETE
/prompt_manager/image-annotations/capture/<token>`, `POST
/prompt_manager/image-annotations/suggest`, `PUT /prompt_manager/image-annotations/tags`
("Tag image dialog and recorded deviations"). Needs review: `GET
/prompt_manager/review/items?limit=1..40&cursor=` (answers `library_epoch`,
`review_revision`, `total`), `GET /prompt_manager/review/items/<id>`, `POST
/prompt_manager/review/items/<id>/decision` and the change stream `GET
/prompt_manager/review/events` ("Needs review decisions and queue reads", "Needs review
page and recorded deviations"). Every legacy route keeps its payload; the only legacy
change is `409 model-busy` from the legacy Auto-tag routes while another model
operation holds the shared gate.
- **S1 (fingerprint-current reads, ruled 2026-10-22).** The Images Tags filter, search and
the inspector/viewer chips use a byte record only while the file's `(st_size,
st_mtime_ns)` equals its verified fingerprint; writes hash the bytes. Residual risk
under "Image annotations schema".
- **Capture limits.** 256 MiB per file, 8 completed captures, 512 MiB reserved, 2
capture/verify workers, 900 s idle, 1800 s hard; the browser fixtures release every
capture on reset, and leaving the Needs review page mid-decision releases its capture.
- **The record-changing job.** Bulk Add tags and Auto-tag selected run as the
`image-tags` Activity job (PR D); like the metadata scan it is a record-changing job, so
a start while another runs is refused (`busy`). Needs review items come from that scan
and from the live monitor, and a decision changes links only when compatible
(`association-conflict` otherwise, with zero writes).
- **No downloads.** No automated test downloads or loads a model: the browser server
injects `TestModel` (its `download_model` raises), native routes and jobs never
download, and every tagging spec asserts no `/autotag/download/` request.
- **Restart.** PRs A, B, D and E and PR F (the review change stream, the queue version)
change Python: the maintainer restarts ComfyUI once after merging. HTML, CSS and JS
changes need only a browser refresh.
### Native System (#249 System)
`#/system/<section>` is a native workspace route rendered by `web/js/workspace/system.mjs`
on the concept's `renderSystem` markup (app.js a698–706; `concept.css` 709–895). `#/system`
is Overview under the sidebar's own hash; `#/system/overview|appearance|sources|
integrations|tagging|maintenance|backups|diagnostics` select a section, and an unknown
child (the concept's `#/system/general`) is replaced by `#/system/overview`. The
production hub `#/overview` is unchanged (D28); its System card opens
`#/system/overview` (Q12). Switching sections keeps the shell, the drafts and focus on
the pressed section button and disposes the previous section's scope, so its reads,
listeners and dialogs never outlive it.
Drafts (`system-state.mjs`) live for the page session (Q10). Save writes the dirty
browser fields, then the host fields (`POST /prompt_manager/settings` with
`expected_revision`, #220), then the runtime fields. A failed domain keeps its drafts and
highlights their controls; `settings-changed` is a conflict and keeps the draft. System
reads the host values on mount and again after a conflict; until a read returned a
revision, a host save fails without a request, so it never sends a blind
`expected_revision: null` (which S1 merges into the file on disk). Save stays disabled
until a section has an unsaved change.
Appearance (`system-appearance.mjs`, S3) is the concept's "Browsing & display" surface
on real values. Its browser rows are this browser's display defaults in the legacy
gallery's own `gallerySettings` object (`web/js/workspace/preferences.mjs`, Q3): Default
layout and Thumbnail size (`workspaceLayout` grid/list/masonry, `workspaceDensity`
comfortable/large/compact, new keys), Load metadata on selection (`autoLoadMetadata`),
Cache metadata in this browser (`cacheMetadata`), Image loading quality (`imageQuality`:
Optimized→`medium`, Original→`high`, Fast→`low`; the thumbnail dialog's preselected quality
once S5 lands), the three video rows (`videoAutoplay`, `videoMute`, `videoLoop`) and Show
file paths (`showFilePaths`). A write re-reads the stored object and writes back only the
changed keys, every other key (legacy paging, thumbnail state, unknown keys) as stored;
the legacy gallery's Save also keeps the keys it does not edit, so the two pages stay in
sync. The workspace never reads the legacy paging keys (`defaultLimit`, `gridColumns`,
`infiniteScroll`; README continuous-browsing ruling). A browser that refuses storage
(denied, or a full quota that leaves the stored bytes as they were) keeps the change for
this tab only and the status says "Saved for this tab only; browser storage is
unavailable.", never "Saved in this browser.". Another tab's change (a `storage` event)
repaints the clean controls; a draft it overtook stays, highlighted, with the conflict
status. Open from ComfyUI (New tab→`newtab`, Popup window→`popup`) and Node search
auto-hide (seconds) (a whole number 0–300; anything else is highlighted and never sent)
are the server's `webui_display_mode` and `result_timeout`, saved through
`POST /prompt_manager/settings` with the revision. The consumers read the defaults on
their next visit, never the current view: a new Images visit starts from the layout and
size (the last visit keeps its own, a saved view opens at Grid and Comfortable as
before); with metadata off a selection reads nothing until "Load metadata"; with the
cache off selecting the shown file again re-reads it (on, the one current record is
reused); paths off leaves out only the Path row of File & provenance; the viewer reads
the video defaults.
Sources & scans (`system-sources.mjs`, S4) is the concept's "sources" body on the
server's sources. `GET /prompt_manager/settings` reports them (public paths only):
`sources.configured` (each `gallery_root_paths` entry with `valid`, `code` and the
server's `reason`), `sources.effective` (what is scanned and served, with its
`root_index` and `watch`: `watching` only when the image monitor's observer is alive,
scheduled that folder and monitoring is on, `stopping` when it still watches it but
monitoring was switched off, `manual` when monitoring is off or no monitor runs, else
`restart_pending`), `sources.auto_detected`, `monitor` and `settings.monitoring_enabled`.
`POST /prompt_manager/settings/validate-source {path}` checks one folder and writes
nothing. Every caller (validate-source, the settings save, config.json at load) refuses
a source through the same checks (`sources.source_problem`): its form first, then
containment decided as text (`normcase(normpath)` against the allowed folders' own
spellings), before anything resolves, stats or opens the path, so a UNC share, the
`\\?\` and `\\.\` namespaces, a device or any folder outside the allowed ones never
reaches the filesystem or the network (no SMB request, no existence or timing oracle).
Only then is the path resolved and contained again, which catches a symlink out. The
filesystem root and the home directory answer `outside-allowed` with the same reason as
any outside path. ComfyUI folders outside its base directory (`--output-directory`)
are path anchors too, so their sources keep a relative public path
(`renders/portraits`) that round-trips and never collides by folder name. A save whose
list holds a source the server cannot use is a 400 `invalid-source` naming its public
path (`source`, as text: a refused path is never resolved to name it), its `index` and
`reason`, and saves nothing. A save that changes the sources (compared as folders, not
spellings) holds the registry as `records` ("Source change"), so it is refused with
409 `busy` while a scan runs. `POST /prompt_manager/jobs/scan {scope, roots, paths}`
scans only the given effective `root_index`es (strict ints, at most 64), each with the
public path the page read for it; an index past the list or one that now names another
folder is 409 `sources-changed` and starts nothing (the page toasts the server's
reason and reads the sources again); the legacy `/scan` is unchanged. The section
lists one row per source ("<path> · watching | watching until restart | manual scan |
restart pending" with "Scan changes" for a watched source, else "Scan folder"), which
opens the shared "Review metadata scan" for that one source ("1 output folder") and
starts the session job over it; the job opens in Activity, and the toast only says it
started. Enrich
existing metadata and the notice's Review scan review every source. The notice shows
until this browser dismissed the 3.2.0 update notice (`pm_update_notice_dismissed`,
the legacy admin page's key); Later dismisses it without scanning, and Review scan
dismisses it as the legacy Scan does. "+ Add source" asks validate-source, then adds
the folder to the host draft of `gallery_root_paths`; Remove takes one out; "Watch for
new outputs" (`monitoring_enabled`) and Worker threads (`worker_threads`) are host
drafts too, and Save writes them with the revision. A save the server refuses because a
scan runs toasts "Another job is running: <title>." with a link to that job (D57) and
keeps the drafts; a save refused for one source says the server's message as the status
and marks that row. After a save the source list is read again; a monitoring change
waits for a restart (the running observer keeps watching until then, and the status
says "Restart ComfyUI to apply: Watch for new outputs."). Thumbnail generation's
Review options opens the legacy gallery in a new tab (D94) until its review dialog (S5).
A list that cannot be read shows "Sources could not be loaded. Your saved data has not
changed." and Retry, never an empty list.
Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per
visit through `latestOnly()` on the section scope; a late answer after the section left
paints nothing. Until it answers, the section shows "Loading the overview…"; a failure
shows "The overview could not be loaded. Your saved data has not changed." and Retry,
never zeros. The thumbnail row is a separate read of `/images/thumbnails/status` (a
walk of every source), so the other values never wait for it: "Checking thumbnails…"
until it answers. The health check (`POST /prompt_manager/system/health`) is a `read`
registry job of kind `health`: one at a time, beside a scan, not cancellable and with
no file outcomes. The page tracks the started job in the Activity store (its event
stream; the store marks it lost if the server stops listing it), never by polling. A press while one runs (another tab) waits
for that one. The last report lives for this server session (`GET
/prompt_manager/system/health`) and describes the database it was taken from: a
restore that replaces the database forgets it, and a check that started before the
swap is not kept. The check's own connection opens a `mode=ro` URI that keeps a Windows
UNC path (`file:////server/share/prompts.db`; pathlib's `file://server/share/...` is a
remote authority SQLite refuses). A check that raises fails only itself, and the
missing-files check reads its rows inside the snapshot but stats the files after it
ended, so a slow source never pins the WAL. Statistics
read `GET /prompt_manager/system/statistics` (`by_category`, the legacy `/stats` totals
and `rated_images`).
Read-only standard for a check or preview (S2's health check; S6 and S8 previews use
the same): the database and `-wal` bytes are unchanged, the `-shm` size is unchanged,
the connection's `total_changes` is 0 at close and the data reads back unchanged. The
`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no
reader can keep it byte-identical (`test_health_check_never_writes`).
Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121–D128 assigned):
- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No
unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server
configuration." / "Applied until the server restarts.", "Some changes were not saved.
Review the highlighted settings.", "Saved for this tab only; browser storage is
unavailable." and "Settings changed elsewhere. Reload this section to see them."
"Save preferences" is `aria-disabled` (opacity .55, `not-allowed`) while clean or
saving; a setting the last save could not save gets a red outline. CSS under D92.
- **D93** (Overview): real values. The metrics are the Library's eligible positives
and active saved negatives and the indexed `generated_images` rows (Q2). "Database
integrity" is "Not checked" until a check ran this session, then "N issues" (checks
that found issues, even beside a check that could not run), "Passed" or "Incomplete".
The review row is "N ambiguous image links" (the Needs review queue's pending count,
`needs_review.pending`) with the concept's line; its Review metadata action opens
`#/review` (the concept's `go-review`). A count
that is not an integer reads "Ambiguous image links" / "Not available yet." (Q6).
"Backup status" reads "No retained backup record." until verified backups exist (S8).
The thumbnail row is "Checking thumbnails…" until its own read answers, then "No
missing thumbnails", "N missing thumbnails" (the pending count of
`/images/thumbnails/status`) or "Thumbnail status unavailable"; Review thumbnails
opens Sources until the thumbnail dialog (S5). "One display configuration" says "Grid
and density settings are shared across views." (no paging). The sources row is
"N output sources" with "Monitoring and enrichment share one source list.", prefixed
by "M of N configured sources available." or "None configured is available; ComfyUI's
output folder is used." when that is the case.
- **D94** (all sections): a section not delivered yet shows the hand-off panel "Your
current tools are ready" with a link to each legacy tool that still owns it (the
plan's legacy coverage table); retired section by section, gone after S12.
- **D95** (health report): the concept's `health-check` dialog titled "Health report",
with no sample note. It lists every check in the server's order: Integrity check,
Foreign keys & pair references, Canonical hashes, Pair uniqueness (added), Missing
files, Ambiguous metadata, Consistency (added). Missing files has one row per image
source when there are several ("Missing files · output"). Each chip is the real
result: "Passed", a count with what was found ("2 broken references", "1 link needs
review", the concept's "N links need review" and "N images need review"; "100+
problems found" when SQLite's integrity check stopped at its limit), "Not
available" (an unavailable source), "Not available yet" (Ambiguous metadata, Q6) or
"Could not check". A `p.small` after the list says when it was checked and whether
every check passed, some could not run, or issues need review. Close / Review missing
links (to Maintenance, its hand-off until S7). Nothing is changed by a check.
- **D96** (Statistics): "Usage statistics" without the sample note: the concept's
"Runs by category" table (Category / Runs / Avg. rating) with the real runs and
average rating of eligible parents per category ("Uncategorized", "Not rated"),
then a second table "Library totals" (Total / Count: Prompts, Tags, Images from the
legacy statistics, and Rated images: indexed images whose linked prompt is rated).
The Overview row says "Runs and ratings by category, with library totals." because
that is what the dialog shows (the concept's own dialog shows only "Runs by
category"); per-prompt run counts, prompt tags and dated runs are recorded, but no
table of them exists yet. A failed read toasts "Statistics could not
be loaded. Your saved data has not changed."
- **D121** (production hub): the Overview hub's System card reads the concept's System
heading and "Open System"; the concept has no hub card.
- **D122** (shell): the section nav's `aria-label` is "System sections"; the concept's
literal label is its prototype action name "system-tab sections".
- **D123** (Maintenance): the concept's Maintenance notice
("Check first. Review changes. Then apply." + Run health checks) is delivered ahead of
the rest of Maintenance, above its D94 hand-off, as the native place that runs the
health check. A press shows "Checking…" (`aria-disabled`) and opens the report; a
failed check toasts "The health check could not finish. Nothing was changed."
- **D97** (Appearance): added rows for the legacy gallery settings the concept leaves
out, after its own rows: "Show image details on cards" (`showImageInfo`; off hides a
card's title and file line, Info and Tag stay at the end of the row), "Collapse viewer
details by default" (`sidebarCollapsedByDefault`; the viewer opens with
`#viewer-details` hidden and a "Show details"/"Hide details" toggle with
`aria-expanded`, the stage taking the whole width), and the operation "Reset display
defaults" ("Restore this browser’s display defaults, then save. Other settings are
kept.", button "Reset"), which drafts the gallery's defaults for the browser rows only.
The node search timeout shows the server's real value, not the sample 30. CSS under D97.
- **D98** (Sources): each configured source's row has a plain Remove after its scan
button; nothing changes until Save. A source added this session reads "<path> · not
saved yet" with Remove only until Save; a configured source the server cannot use
reads "<path> · <server reason>" with Remove only. A line under the list says when the
list is not the whole story: "No source is configured. After saving, ComfyUI’s output
folder is used." (the last one removed), "No source is configured; ComfyUI’s output
folder is used." and "No configured source is available; ComfyUI’s output folder is
used." (auto-detected rows, listed without Remove). "restart pending" is the third
watch state. The operation "Refresh files" ("Read every source again to find files
added or removed outside ComfyUI.", button "Refresh") closes the surface: it calls the
legacy gallery's `/gallery/rescan` and toasts "Files refreshed. N files in M output
folders." or "Files could not be refreshed. Try again.". The concept's "sample path"
note is prototype copy and is dropped.
- **D113** (Sources): the Worker threads range ends at the server's `cpu_count` (the
concept's fixed `max=16`) and shows the server's real `worker_threads` (the concept's
sample 4).
- **D127** (Sources): the concept's `add-source` `simpleInput` with production
copy: the field is "Folder path" (the concept's "(sample only)" dropped), it starts
empty (not the demo "/images/new-collection"), and the action is "Add" (not "Apply in
prototype"); Enter in the field adds too. `add source dialog matches the concept`
(system-concept.spec) compares the open dialog with these three mapped, and `add
source comparison detects mutations` proves the mapping is the only difference. A
folder the server refuses keeps the dialog open with the field
`aria-invalid` and the server's reason in a `p.small[role=alert]` above the actions;
a folder the server names as one already listed says "This folder is already a
source.", and a check that cannot finish says "The folder could not be checked. Try
again."; neither adds anything.
The concept dialog does no check, so `invalid source shows the server reason`
(workspace-system-sources.spec) pins it.
- **D128** (Sources): states the concept's sample rows never show. A source the
running monitor still watches after "Watch for new outputs" was saved off reads
"<path> · watching until restart" (still "Scan changes"), and the save's status adds
"Restart ComfyUI to apply: Watch for new outputs." from `restart_required_fields`. A
configured source the effective list holds under another spelling reads "<path> ·
same folder as another source" with only Remove. A save refused for one source
(`invalid-source`) marks that row (`aria-invalid`, outlined like a D92 setting) and
the status is the server's "<path> can't be used: <reason>. Remove it or reconnect it,
then save." ("Remove it, then save." when reconnecting cannot help). Pinned by
`Watch for new outputs saves monitoring_enabled; a restart applies it` and `a save
refused for one unusable source names it and marks its row`
(workspace-system-sources.spec) and the system-state and system-sources node tests.
- **D124** (inspector): with "Load metadata on selection" off, the Images panel and
the viewer's details show "Metadata loads when you ask for it." and a "Load metadata"
button instead of reading the file. The concept has no such state, so no concept
comparison renders it: `metadata off shows an explicit load and loads once on request`
(workspace-system.spec) pins it instead.
- **D125** (prompt blocks, maintainer option A): a resolved prompt whose source is
`runtime-parts-omitted` (a PromptManager `prepend_text` or `append_text` fed directly
by a listed run-time node, such as LoraManager's TriggerWord Toggle, was left out)
shows the muted line "Text added at run time, such as trigger words, is not
included." under its text. The evidence badge stays "Workflow inferred"; the line is
never sent or copied. The concept has no such state (`a prompt that left out run-time
trigger words says so under its text (D125)` in `tests/js/prompt-blocks.test.js`).
- **D126** (prompt blocks, maintainer ruling on the added-conditioning floor): a
resolved positive whose source is `added-conditioning` (a later stage combined more
conditioning with the base prompt, as UltimateSDUpscale's ConditioningCombine of the
PromptManager and a style trigger) shows "Conditioning added at a later stage is not
included." the same way. A prompt with both shows both lines, D125 first (source
`runtime-parts-omitted-and-added-conditioning`). Always labelled, with no length or
"short trigger" heuristic (`a prompt a later stage added conditioning to says so, and
both lines show together (D126)`).
Appearance scope and states. The Default layout and Thumbnail size defaults start the
Images page only (a new visit, not the current view or a saved view); the embedded
galleries (a prompt history's Shared gallery and its All images / Repeat run dialogs)
have no toolbar, so they keep Grid/Comfortable, while the metadata, cache and card
defaults apply there too. A host row whose `/settings` read failed stays disabled with
"Server settings could not be read."; a rejected timeout says "Some changes were not
saved. Review the highlighted settings." A display default another page saved under one
of this tab's drafts flags that draft with the conflict status, whether it arrived while
System was mounted (a `storage` event) or while it was not (checked again on mount, and
before a write); the draft is never written over the other page's value without that
status first. A change kept for this tab only is stored by the next write storage accepts.
Open question (saved-views follow-up, M8): a saved view stores no layout or size and
always opens at Grid/Comfortable, even when the Default layout is Masonry. Should saved
views store a layout, or follow the default?
`tests/browser/system-concept.spec.mjs` compares the shell and Overview with the concept
(dark and light, 1600×1100 and 390×844), mapping only the deviations above; unmapped,
the difference is exactly the D92, D93, D96 and D122 paths. Each surface's `top` is
measured, which pins the concept's `.surface + .surface` 18px offset that concept.css
also applies inside `.two-col` (copied as is). `statistics and health dialogs match
the concept` maps D95 from the concept's own five rows (labels verbatim): it inserts
Pair uniqueness after Canonical hashes and Consistency last, splits Missing files per
source when there are several, replaces only the chips with the fixture's expected
results, and asserts the full unmapped path list; it maps D96 and compares the
Maintenance notice unchanged. `system comparison detects mutations` removes a row,
renames a metric label, swaps the `--line` token, recolours a chip, renames a check,
removes the totals table and renames a caption. `appearance matches the concept` maps
only D97 and the real timeout (unmapped, exactly those paths differ), and `appearance
comparison detects mutations` removes a row, renames a label and an option and swaps
the `--line` token. `sources matches the concept with|without the notice` (both
themes and sizes; the fixture's monitor watches the first source, as the concept's
first sample is watched, and the concept's Later removes its notice) maps the real
sources for the two samples, D98's Remove and Refresh files, and D113; unmapped,
exactly those paths differ. `sources comparison detects mutations` removes the notice
and a source row, renames a label and the Add source button and swaps the `--line`
token. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`,
`{concept,production}-{statistics,health}-{dark,light}.png`,
`{concept,production}-appearance-{dark,light}-{1600,390}.png`,
`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`) are test artifacts; there
are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts,
the health check and metric 4, a failed check, the thumbnail row against the status
endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the
route, a load failure with Retry and late answers after leaving, and Appearance: new
visits versus the current view and a saved view, the embedded gallery keeping Grid,
metadata off and on request, the cache (counted up to a request barrier), hidden paths,
the video defaults, the host fields with their revision, a stale one and a failed read,
reload and a second tab, a second tab's save while System was closed, storage refused,
the D97 rows and reset. `tests/browser/workspace-system-sources.spec.mjs` covers the
notice's Later and Review scan, a scoped scan from one source into Activity, Add source
with its server check and Save, a refused folder, Remove down to auto-detect, an absent
configured source, the three watch states, the monitoring and worker rows, Refresh
files, the thumbnail hand-off, a save refused by a running scan and a failed read. Its hooks are
`tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected
failures, an absent source, a deleted linked file, unthumbnailed files, a fake image
monitor) on the
concept-shaped fixture, with delays through the shared request holds; reset also restores
the server's host settings and its `config.json`, which must be the one inside the server's
temp root (`tests/test_workspace_system_fixture.py`).
```sh
python3 -m unittest tests.test_database_health tests.test_system_api tests.test_library \
tests.test_workspace_system_fixture -v
node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-controls.test.js \
tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js \
tests/js/workspace-system-appearance.test.js tests/js/workspace-preferences.test.js \
tests/js/workspace-inspection-state.test.js tests/js/workspace-images-state.test.js \
tests/js/workspace-system-sources.test.js
python3 -m unittest tests.test_settings_sources tests.test_scan_scope -v
npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \
workspace-system-sources.spec.mjs \
--project chromium --project touch --project firefox --project webkit
```
## Workspace data fields for the concept pages (#249)
These fields are additive. The legacy `/prompt_manager/images/output` listing (no
@@ -2434,24 +1837,8 @@ and skip the parse.
integers past 2**53 as their exact digits, so the card and the inspector read
the same digits. The parse runs after the file handle closes. A record over
2 MiB (UTF-8), a parser error, a recursion error, a placeholder, or a
non-integral, boolean or non-finite seed gives null. When the prompt graph's
sampler stages read different seed inputs and the inspector's resolution names
one sampler for a resolved pair (a detailer or pass-through chain, whose facts
are the named stage's, D19), the card compares that stage's seed (read by the
same parser, the other samplers hidden from it) with its own and shows no seed
only when they differ. A seed that agrees, or one the named stage does not make
known, is kept, as on main (a saved sampler beside a preview-only one keeps its
seed; an ambiguous chain, whose facts merge and fill from the parser, keeps it).
Only such graphs are resolved for the card, with `INSPECTION_PROFILE`
(`test_a_chain_the_inspector_names_one_stage_of_shows_no_differing_seed`,
`test_a_seed_that_agrees_with_the_named_stage_is_kept`,
`test_a_named_ambiguous_chain_keeps_the_parser_seed`; a literal named seed needs no
second parse, and stages sharing one linked seed skip the check. Measured on a
200-card page of about 54 KB graphs, half multi-stage with per-stage seeds: 618 ms
cold against main's 387 ms, about 1.2 ms more per such card, paid once per file
through the header-facts cache below;
before this rule a KSampler then FaceDetailer chain with two seeds broke the
contract below). A card never shows a seed that differs from the inspector's
non-integral, boolean or non-finite seed gives null. A card never shows a seed
that differs from the inspector's
(`test_card_seed_equals_inspector_seed_when_both_present`, which `JSON.parse`s
the listing in Node and compares `String(seed)` with the inspector's seed,
including one past 2**53). The card shows no seed for A1111 `parameters` files
@@ -2567,167 +1954,6 @@ inspection's own stable stat (`tests/test_api_workspace_images.py`
browser fixtures clear `media_image_tags`, `media_annotation_bytes` and
`media_annotation_files` on reset and never the metadata row.
Review queue schema (#249, `database/review_schema.py`,
`tests/test_review_schema.py`): the group `media_review_metadata` (one row:
schema version, `review_index_revision`), `media_review_items` (one ambiguous
resolver result per `(byte_id, resolver_version, evidence_hash)`, citing an
annotation byte record `ON DELETE RESTRICT`, with the file's recorded
`spelled_path` (the locator is case-folded on Windows, a listing target is
not), 0..16 candidates, `limited`,
`state` pending/selected/unresolved and a revision) and `media_review_decisions`
(one row per item decision revision; `selected` exactly when a
`candidate_hash` is present; `pair_id`/`image_id` navigation is set to NULL,
never cascaded, when the pair or image is deleted, through
`idx_media_review_decisions_pair`/`_image`). Evidence is canonical JSON
(`canonical_evidence`: one object, sorted keys, no whitespace, ASCII only, no
NaN/infinity, at most 8 MiB) and `evidence_hash` is its SHA-256. The group is
valid only beside a valid annotation group (a review group without annotation
tables is refused, catalog only, before the annotation install), installs like
the annotation group (optional in old backups, required once current under
`require_library_state`, SAVEPOINT install validated before and after), and
validation re-canonicalizes every evidence document in one streamed pass. A
startup runs that pass once (`test_startup_validates_stored_evidence_once`; the
install trusts the annotation group it has just validated, and the SQL row
predicates check type and size only, because the streamed pass proves the
rest): 2.5 s to open a database of 40,000 items with 4 KiB of evidence each,
was 7.7 s with two passes. Restore and `verify_database_file` keep the full
validation, annotation group included. The browser fixtures clear
`media_review_decisions` and `media_review_items` before the annotation tables,
never the metadata row.
Needs review items (#249, `utils/review_capture.py`, `database/review.py`,
`tests/test_review.py`, `tests/test_review_outcome.py`,
`tests/test_review_choosable.py`): a file is queued only when its final outcome
is ambiguous and unlinked and there is something to choose. Only scans produce
items; the monitor links new outputs and clears pending items it resolves
(maintainer rulings, 2026-10-23). The output scan (Activity's
Scan for metadata and the legacy `/scan`) hands `record_ambiguous_file` a file
the ingestion resolver finds `ambiguous` only when the browser-parser fallback
then links nothing (status `ambiguous`) and no saved row links the file
already; an ambiguous file a saved row links is reported `unchanged` ("Already
linked to its prompt"), not Needs review. A file the fallback links, an exact
pair the ingester stores, and `conflict` and `unresolved` files are not
queued. Neither is an ambiguous file with nothing to choose: fewer than two of
its candidates are `resolved` with non-blank positive text (`reviewable`). With
none the file could not be traced; with exactly one, the untraced branches may
still have made the file, so that text is not certain either. Both report
`unresolved` ("could not be traced"), are never linked, and their pending,
undecided items whose own evidence offers no choice (a limited item counts as
a choice when it holds at least two candidates) leave the queue on the next
scan or monitor pass (`settle_untraced_review_items`). The maintainer's
second live check found 39 items whose 11 candidates were all "unresolved". The queue heals itself: after each batch, the scan removes the pending
items of every listed file (read or skipped as known) that a saved row now
links, and the monitor does the same for each ambiguous output it links
(`settle_linked_review_items`; the review index revision moves). Only an
undecided item goes (`pending`, revision 0, no decision row): a decision is
never deleted or changed, so no schema state was added. The monitor never
queues an output it linked; an unlinked file passed to `record_review_file`
follows the scan's rule (a file the fallback would link is `skipped`). The
maintainer's live check of #284/#288 found the queue at 10,232 against a review
group of 4,677: every fallback-linked ambiguous file had been queued. It captures
the file (scope `needs-review`), reads its metadata again from the capture,
resolves it with the default profile and stores only an ambiguous result,
bound to the capture's byte record (created or refreshed like an observe) and
fenced by the capture's storage epoch. Evidence is the resolver reason plus one
`{sampler_id, state, reason, positive, negative, hash}` per selectable
candidate (nested ambiguous stages flattened, duplicates dropped; each side
`{text, storage_text, provenance}`; `hash` = SHA-256 of the canonical candidate
without it). More than 16 candidates, or evidence over 8 MiB, is a `limited`
item without candidates. Recording is idempotent per `(byte_id,
resolver_version, evidence_hash)` and never touches an existing item or its
decisions; changed bytes add a new pending item and keep the old one. The
capture is released before the write; no model is loaded and no pixel is
decoded. Scan statuses, counters, outcome rows and the legacy SSE events are
unchanged apart from the already-linked and nothing-to-choose cases above; the job's `scope` gains
`review_items` (this scan's files that are in the pending queue after it, new
or already there: the review group's ambiguous rows, less files a capture could
not read or a decision already settled), `review_added` (new items),
`review_settled` (items removed because their file is linked now),
`review_untraced` (items removed because they offer nothing to choose) and
`review_errors`. The log ends with `? N images added to Needs review`,
`? N images already in Needs review`, `= N images left Needs review · now
linked`, `= N images left Needs review · nothing to choose` and `! N images could not be added to Needs review` (failures, e.g.
capture capacity), each only when N > 0. Recording captures in the background: it never takes the store's
last worker or capture slot, which stay free for an interactive Tag dialog or
Apply, and waits up to 2 s for room before counting the file as not added
(`test_background_recording_leaves_a_worker_for_the_tag_dialog`). The monitor's
pass runs after its own link, best effort: a failure is logged and the link
stays. Recording follows the registry's `records`
admission (#289): the scan's recording runs under its own `records` job, and
the monitor's (one file, not a job) is refused while a `records` or `guarded`
job or hold runs. It then records nothing, logs that the file waits for the
next scan, and answers `deferred`; the next "new" scan reads the ambiguous file
again (`test_monitor_recording_is_refused_beside_record_changing_jobs`). A new item moves `review_index_revision`. A
rescan does not copy or hash an ambiguous file whose current size and
`mtime_ns` equal the verified fingerprint of bytes an item already cites (a
stat compare); the file still stays in the default "new" scope, because it is
not a known image. The skip and a duplicate insert record the current spelling
of the locator on its items (`refresh_review_spelling`, the review index
revision moves, item revisions do not), so a case-only rename on Windows or
macOS keeps its item applicable. Residual risks of the skip, like S1's: an
in-place rewrite that keeps both size and `mtime_ns` is never re-queued (Apply
still re-hashes, so it cannot link those bytes), and a file that is deleted or
moved leaves its pending item counted in `total` (its target then formats as
`unavailable` or fails to open, and Leave unresolved clears it).
Needs review decisions and queue reads (#249, `py/api/review.py`,
`database/review.py`, `database/review_decisions.py`,
`tests/test_api_review.py`): `GET /prompt_manager/review/items?limit=1..40&cursor=`
answers `{success, library_epoch, total, items, next_cursor}` with pending items
oldest first (each item adds the card's `url`, the Images listing's
`/prompt_manager/images/serve/<path>?root=<i>&root_scope=<scope>` formatted without a
stat, null when the file is under no current root, and `is_video` from its extension); `total` is the exact pending count. The cursor is an HMAC keyset
on `(created_at, id)` bound to the storage epoch only (ruling M3, so "load more"
keeps working while the monitor adds an item per generation): a malformed one
is 400, a well-formed one this process cannot verify (tampered, or opened
before a ComfyUI restart: the secret is per process) is 409 `review-conflict`
with the current `total`, and one from before a restore is 409
`storage-changed`. A decision or a new item leaves an open cursor valid: a
decided item drops out of later pages and new items (stamped `utc_now_iso()`,
microseconds) append after it. Residual risk: an item recorded after a wall
clock step backwards sorts before an open cursor and shows only on reload.
Targets are formatted from `spelled_path` against the current roots (the
roots' real paths are the only file access); a file under no current root is
`unavailable` with a null target. Every read and decision lists the roots and
resolves real paths on a worker thread, never on the event loop. `GET .../items/{id}` adds `candidates`
(`hash`, `sampler_id`, `state` resolved/unresolved, `reason`, `label` "<positive
title> / <negative title>" from the saved prompt's Library title else the
first line, and the role texts), `dependency_fingerprint` (SHA-256 of the
file's legacy rows under every spelling of the same physical file, positive
hashes and pair links with their negative hashes, as the ingester reads them) and `sha256`; a detail over 8 MiB, or a
limited item, has no candidates. `POST .../items/{id}/decision
{expected_epoch, expected_revision, capture_token, sha256, dependency_fingerprint,
choice}` with a candidate hash (Apply) requires a capture of the item's file
whose bytes are the item's, re-hashes the current file (`file-changed`
otherwise, also for a fresh capture of new bytes), re-reads the capture and
requires that its bytes are still ambiguous and still offer the chosen
candidate (`review-conflict` otherwise; a resolver change that rewords the
reason or reorders candidates leaves the item applicable, and
`RESOLVER_VERSION` in `utils/prompt_pair_resolver.py` is the one version
ingestion provenance and review share), then in one `_pair_transaction`
checks the epoch, the item revision and state (`review-conflict` with the
current item for a repeat), the re-hashed digest, the dependency fingerprint
and the public ingester's compatible-only rules through
`_ingest_resolved_pair_in` (any refusal is 409 `association-conflict` with
zero writes, explained inline; Apply never reparents a link). Physical
identity: the roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, `subst` or a mapped
drive spells the same file otherwise. Every legacy row whose spelling ends with
the file's name (the link summaries' unindexed suffix lookup) is resolved by
`normcase(realpath)` against the locator, else `samefile`; two rows of the
file refuse the choice, one is enriched under the spelling it was linked
with, and a same-name row the request did not classify is a changed
association (`TestPhysicalIdentity`, symlinked root and a portable fake
junction), and writes
the link with `kind:"user-selection"` provenance (decision id, reviewed digest,
candidate hash, resolver version, role provenance), the decision row, the
item's new state and revision and `review_index_revision`. `"unresolved"`
records the decision only and accepts `capture_token: null`. A successful
decision releases its capture. No decision changes run or usage counts or
calls an execution writer; a Tag save never changes a review item. Errors carry
fixed sentences: `review-failed` / `decision-failed` 500s never echo exception
text or paths.
Capture store (#249, `utils/media_capture.py`, `tests/test_media_capture.py`):
one explicit action captures one rooted media file into an owner-only
`prompt-manager-captures-*` temp directory, streaming it in 1 MiB chunks while
@@ -2744,12 +1970,8 @@ directories are `CaptureDenied`; the locator is
`normcase(join(realpath(root), relative))`, so a symlinked root shares the
listing's identity. Fixed limits: 256 MiB per file (stat and stream), 8
completed captures, 512 MiB reserved, 2 capture/verify workers, 900 s idle,
1800 s hard; interactive admission is nonblocking (`CaptureCapacity`), a
`background=True` capture (Needs review recording) never takes the last worker
or capture slot and waits up to `BACKGROUND_WAIT_SECONDS` (2 s) for room, and
lifetimes are tested with an injected clock. The host's on_cleanup close is
final: a recording still in flight gets a capture error instead of creating a
new temp directory after it. An acquired capture outlives expiry and release
1800 s hard; admission is nonblocking (`CaptureCapacity`) and lifetimes are
tested with an injected clock. An acquired capture outlives expiry and release
until its last use ends. Header-only dimensions (4096 px per side,
16,777,216 px) and video mark Generate unsupported; `verify` re-hashes the
current file without a copy. Every handle closes before unlink (a patched
-18
View File
@@ -91,9 +91,6 @@ try:
if host_app is not None and hasattr(host_app, "on_cleanup"):
host_app.on_cleanup.append(api.cleanup_history_reads)
host_app.on_cleanup.append(api.cleanup_annotation_captures)
if host_app is not None and hasattr(host_app, "on_shutdown"):
# Open Needs review change streams end before the server waits on them.
host_app.on_shutdown.append(api.close_review_events)
except Exception as e:
# Log to internal logging system without console spam
@@ -125,21 +122,6 @@ try:
_global_prompt_tracker = get_prompt_tracker(_global_db)
_global_image_monitor = get_image_monitor(_global_db, _global_prompt_tracker)
# The Needs review pass for ambiguous new outputs runs through the API's
# capture store (the one the scan and the Tag dialog use): a linked output
# is never queued and its pending items leave the queue. It stops before
# that store is closed at shutdown.
_review_api = globals().get("api")
if _review_api is not None:
_global_image_monitor.set_review_recorder(_review_api.record_review_file)
_review_host = getattr(config.server_instance, "app", None)
if _review_host is not None and hasattr(_review_host, "on_cleanup"):
async def _stop_review_recording(app, monitor=_global_image_monitor):
monitor.set_review_recorder(None)
_review_host.on_cleanup.insert(0, _stop_review_recording)
# Start monitoring
_global_image_monitor.start_monitoring()
_init_logger.info("Global image monitoring system started successfully")
-354
View File
@@ -1,354 +0,0 @@
"""The read-only health check behind System › Overview and Maintenance.
``run_health_checks`` opens its own ``mode=ro`` connection, reads one
snapshot and closes the connection in ``finally``; it never writes. Each
check reports ``passed``, ``issues`` (with an exact ``count``; the
integrity check stops at ``INTEGRITY_LIMIT`` problems and then says
``capped``),
``unavailable`` (it could not be checked here, so ``count`` is None, or
``missing_files`` checked only some sources and found nothing) or ``failed``
(it raised; the other checks still run). ``missing_files`` also reports each
source and whether the check was ``partial``; it reads the image rows inside
the snapshot and checks their files after the snapshot ended, so a slow source
never holds the WAL. At most ``MAX_DETAIL`` detail strings are returned per
check, and none of them carries a server path.
"""
import datetime
import logging
import ntpath
import os
import sqlite3
from pathlib import PurePosixPath, PureWindowsPath
from urllib.parse import quote
try:
from ..utils.hashing import generate_prompt_hash
from .models import normalize_image_path
except ImportError:
from utils.hashing import generate_prompt_hash
from database.models import normalize_image_path
logger = logging.getLogger("prompt_manager.database.health")
MAX_DETAIL = 10 # detail strings returned per check
INTEGRITY_LIMIT = 100 # PRAGMA integrity_check(N): problems SQLite reports
BATCH_ROWS = 100 # image rows read per fetch
REVIEW_UNAVAILABLE = "Needs review is not available yet."
CHECK_FAILED = "The check could not finish."
# Check ids and the labels of the concept's health report, in its order;
# pair uniqueness and consistency are the added rows (D63, now D95).
CHECKS = (
("integrity", "Integrity check"),
("foreign_keys", "Foreign keys & pair references"),
("canonical_hashes", "Canonical hashes"),
("pair_uniqueness", "Pair uniqueness"),
("missing_files", "Missing files"),
("ambiguous_metadata", "Ambiguous metadata"),
("consistency", "Consistency"),
)
_DATABASE_CHECKS = (
"integrity",
"foreign_keys",
"canonical_hashes",
"pair_uniqueness",
"missing_files",
"consistency",
)
def iter_consistency_issues(conn):
"""The legacy consistency issues: tag and image rows of a missing prompt."""
for ref in conn.execute(
"SELECT pt.prompt_id FROM prompt_tags pt "
"LEFT JOIN prompts p ON pt.prompt_id = p.id WHERE p.id IS NULL"
):
yield f"prompt_tags entry references non-existent prompt {ref[0]}"
for ref in conn.execute(
"SELECT gi.id, gi.prompt_id FROM generated_images gi "
"LEFT JOIN prompts p ON gi.prompt_id = p.id WHERE p.id IS NULL"
):
yield f"Image {ref[0]} references non-existent prompt {ref[1]}"
def _result(count, detail, *, status=None):
return {
"status": status or ("issues" if count else "passed"),
"count": count,
"detail": detail[:MAX_DETAIL],
}
def _counted(lines):
"""(count, first MAX_DETAIL lines) of an iterable, consumed once."""
count, detail = 0, []
for line in lines:
count += 1
if len(detail) < MAX_DETAIL:
detail.append(line)
return _result(count, detail)
def readonly_uri(db_path, *, windows=None):
"""The ``mode=ro`` SQLite URI of *db_path*, on this platform or the given one.
pathlib spells a UNC path ``\\\\server\\share\\x.db`` as
``file://server/share/x.db``, a remote authority SQLite refuses to open.
With an empty authority (``file:////server/share/x.db``) SQLite hands
Windows the ``//server/share`` path, which it opens as the UNC path.
"""
if windows is None:
windows = os.name == "nt"
db_path = os.path.abspath(db_path)
if not windows:
return PurePosixPath(db_path).as_uri() + "?mode=ro"
path = PureWindowsPath(db_path)
if ntpath.splitdrive(str(path))[0].startswith("\\\\"):
return "file:" + quote("//" + path.as_posix()) + "?mode=ro"
return path.as_uri() + "?mode=ro"
def _integrity(conn, _roots):
rows = [
row[0] for row in conn.execute(f"PRAGMA integrity_check({INTEGRITY_LIMIT})")
]
if rows == ["ok"]:
return {**_result(0, []), "capped": False}
# SQLite stops listing at the limit: there may be more problems than that.
return {**_result(len(rows), rows), "capped": len(rows) >= INTEGRITY_LIMIT}
def _foreign_keys(conn, _roots):
return _counted(
f"{table} row {rowid} references a missing {parent} row"
for table, rowid, parent, _fkid in conn.execute("PRAGMA foreign_key_check")
)
def _prompt_hash(text):
return generate_prompt_hash(text) if isinstance(text, str) else None
def _canonical_hashes(conn, _roots):
conn.create_function("pm_prompt_hash", 1, _prompt_hash, deterministic=True)
return _counted(
f"Prompt {row[0]}: the stored hash does not match its text"
for row in conn.execute(
"SELECT id FROM prompts WHERE hash IS NOT pm_prompt_hash(text) ORDER BY id"
)
)
def _pair_uniqueness(conn, _roots):
groups = conn.execute(
"SELECT positive_prompt_id,negative_prompt_id,COUNT(*) FROM prompt_pairs "
"GROUP BY positive_prompt_id,negative_prompt_id HAVING COUNT(*)>1 "
"ORDER BY positive_prompt_id,negative_prompt_id"
).fetchall()
detail = [
f"Positive prompt {positive} with "
+ ("no negative" if negative is None else f"negative {negative}")
+ f": {rows} pair rows"
for positive, negative, rows in groups
]
# The surplus rows: what keeping one row per pair would leave out.
return _result(sum(rows - 1 for _p, _n, rows in groups), detail)
def _consistency(conn, _roots):
return _counted(iter_consistency_issues(conn))
def _readable(root):
return os.path.isdir(root) and os.access(root, os.R_OK | os.X_OK)
def _root_prefix(root):
return normalize_image_path(root).rstrip("\\/") + os.sep
def _source_name(root, index):
"""A source's folder name for the report (never its server path)."""
return os.path.basename(str(root).rstrip("\\/")) or f"Source {index + 1}"
def _owner(path, prefixes):
"""Index of the longest source prefix *path* lies under, or None."""
owner, longest = None, -1
for index, prefix in enumerate(prefixes):
if path.startswith(prefix) and len(prefix) > longest:
owner, longest = index, len(prefix)
return owner
def _image_owner(image_path, file_path, prefixes):
spelled = file_path or normalize_image_path(image_path)
owner = _owner(spelled, prefixes)
if owner is None:
owner = _owner(normalize_image_path(os.path.realpath(image_path)), prefixes)
return owner
def _unavailable_line(unavailable, *, partial):
noun = "source is" if unavailable == 1 else "sources are"
effect = "its files were" if unavailable == 1 else "their files were"
if not partial:
effect = "missing files were"
return f"{unavailable} image {noun} unavailable, so {effect} not counted."
def _missing_files(conn, roots):
"""Links whose file is gone, counted per source (only readable ones).
Each source reports ``checked`` with its own count, or ``unavailable``
(an unmounted drive) with no count: its links are never turned into a
count, while the readable sources still are. The check's ``count`` is the
readable sources' total (None when none is readable) and ``partial`` says
some sources were not checked. Links outside every source are named in the
detail, not counted.
Inside the snapshot this only reads the image rows; it returns the step
that checks their files, which runs after the snapshot ended.
"""
if not roots:
result = _result(None, ["No image source is available."], status="unavailable")
return {**result, "partial": False, "sources": []}
rows = []
cursor = conn.execute(
"SELECT id,image_path,filename,file_path FROM generated_images ORDER BY id"
)
while batch := cursor.fetchmany(BATCH_ROWS):
rows.extend(batch)
return lambda: _check_files(rows, roots)
def _check_files(rows, roots):
"""The ``missing_files`` result of the image *rows*, from the file system."""
readable = [_readable(root) for root in roots]
sources = [
{
"name": _source_name(root, index),
"status": "checked" if ok else "unavailable",
"count": 0 if ok else None,
}
for index, (root, ok) in enumerate(zip(roots, readable))
]
unavailable = readable.count(False)
if not any(readable):
result = _result(
None, [_unavailable_line(unavailable, partial=False)], status="unavailable"
)
return {**result, "partial": False, "sources": sources}
prefixes = [_root_prefix(root) for root in roots]
outside, missing = 0, []
for image_id, image_path, filename, file_path in rows:
owner = _image_owner(image_path, file_path, prefixes)
if owner is None:
outside += 1
elif readable[owner] and not os.path.exists(image_path):
sources[owner]["count"] += 1
if len(missing) < MAX_DETAIL:
name = filename or os.path.basename(image_path)
missing.append(f"Image {image_id}: {name} is missing")
count = sum(source["count"] or 0 for source in sources)
detail = [_unavailable_line(unavailable, partial=True)] if unavailable else []
detail += missing
if outside:
links = "link" if outside == 1 else "links"
verb = "was" if outside == 1 else "were"
detail.append(
f"{outside} {links} outside the image sources {verb} not checked."
)
status = None if count or not unavailable else "unavailable"
result = _result(count, detail, status=status)
return {**result, "partial": bool(unavailable), "sources": sources}
def _ambiguous(review_count):
if review_count is None:
return _result(None, [REVIEW_UNAVAILABLE], status="unavailable")
return _result(review_count, [])
def overall_status(checks) -> str:
"""``passed`` only when every check passed; any check not run is incomplete."""
statuses = {check["status"] for check in checks}
if statuses & {"unavailable", "failed"} or any(
check.get("partial") for check in checks
):
return "incomplete"
return "issues" if "issues" in statuses else "passed"
def _failed():
return _result(None, [CHECK_FAILED], status="failed")
def _run_check(check_id, step, *args):
"""One check's result, or None (reported as failed) when it raised."""
try:
return step(*args)
except Exception: # any error fails this check only; the others still run
logger.exception(f"Health check {check_id} failed")
return None
def _run_database_checks(db_path, roots):
"""Each database check's result (None when it raised), from one snapshot.
A check may return a step instead of its result: the step runs after the
snapshot ended and the connection closed (``missing_files``' file checks).
"""
results = dict.fromkeys(_DATABASE_CHECKS)
try:
conn = sqlite3.connect(readonly_uri(db_path), uri=True)
except sqlite3.Error:
logger.exception("Health check could not open the database")
return results
try:
conn.execute("BEGIN") # one read snapshot for every check
for check_id in _DATABASE_CHECKS:
runner = globals()["_" + check_id] # looked up per run (tests patch it)
results[check_id] = _run_check(check_id, runner, conn, roots)
except sqlite3.Error:
logger.exception("Health check could not read the database")
finally:
conn.close()
for check_id, result in results.items():
if callable(result):
results[check_id] = _run_check(check_id, result)
return results
class HealthOperations:
"""Mixin for PromptDatabase: the read-only System health check."""
def run_health_checks(self, roots, *, review_count=None) -> dict:
"""Run every check and return a ``HealthReport``.
Args:
roots: The image source directories to judge missing files under:
the effective sources plus any configured source that is not
available (reported as an unavailable source of ``missing_files``).
review_count: The Needs-review pending count, or None while no
review queue exists (``ambiguous_metadata`` is then unavailable).
"""
if review_count is not None and (
type(review_count) is not int or review_count < 0
):
raise ValueError("review_count must be a non-negative integer or None")
roots = [str(root) for root in roots]
database = _run_database_checks(self.model.db_path, roots)
checks = []
for check_id, label in CHECKS:
if check_id == "ambiguous_metadata":
result = _ambiguous(review_count)
else:
result = database[check_id] or _failed()
checks.append({"id": check_id, "label": label, **result})
return {
"success": True,
"checked_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"status": overall_status(checks),
"checks": checks,
}
-54
View File
@@ -628,57 +628,3 @@ class LibraryOperations:
"kind": "prompt-manager-library",
"positives": [owners[key] for key in order],
}
def library_counts(self) -> dict:
"""System › Overview counts from one snapshot, writing nothing.
``positive_prompts`` and ``negative_variants`` are the unfiltered
Library's ``total`` and ``variant_total``: eligible parents and active
pairs (every pair's positive is eligible). ``images`` counts indexed
``generated_images`` rows, not files on disk. An image has no rating
of its own (it shows its linked prompt's), so ``rated_images`` counts
the indexed images whose linked prompt is rated. ``review_pending`` is
the Needs review queue's pending total.
"""
with self.library_snapshot() as (conn, _epoch, _ceiling):
row = conn.execute(
"SELECT (SELECT COUNT(*) FROM prompts p WHERE " + _ELIGIBLE + "),"
"(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0),"
"(SELECT COUNT(*) FROM generated_images),"
"(SELECT COUNT(*) FROM generated_images gi JOIN prompts r "
"ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL),"
"(SELECT COUNT(*) FROM media_review_items WHERE state='pending')"
).fetchone()
return {
"positive_prompts": row[0],
"negative_variants": row[1],
"images": row[2],
"rated_images": row[3],
"review_pending": row[4],
}
def usage_by_category(self) -> list[dict]:
"""Runs and average rating of the Library's parents, per category.
Only eligible parents count, so a negative-only prompt's runs (which
repeat its pairs' runs) are never added twice. A blank category groups
with no category as ``None``. Unrated parents are left out of the
average, which is ``None`` when no parent is rated. Most runs first.
"""
with self.library_snapshot() as (conn, _epoch, _ceiling):
rows = conn.execute(
"SELECT * FROM (SELECT NULLIF(TRIM(p.category),'') AS category,"
"COALESCE(SUM(p.run_count),0) AS runs,AVG(p.rating) AS rating "
"FROM prompts p WHERE " + _ELIGIBLE + " GROUP BY 1) "
"ORDER BY runs DESC,category IS NULL,category"
).fetchall()
return [
{
"category": row["category"],
"runs": row["runs"],
"average_rating": (
None if row["rating"] is None else round(row["rating"], 2)
),
}
for row in rows
]
+3 -20
View File
@@ -18,11 +18,6 @@ from typing import Optional, Set, Tuple
from .library_schema import library_schema_error, migrate_library_state
from .image_favorites import image_favorites_schema_error, migrate_image_favorites
from .annotation_schema import annotation_schema_error, migrate_annotation_state
from .review_schema import (
migrate_review_state,
review_schema_error,
review_without_annotations_error,
)
from .saved_views import migrate_saved_views, saved_views_schema_error
from .execution_history_schema import (
history_schema_error,
@@ -308,14 +303,7 @@ class PromptModel:
migrate_library_state(conn)
migrate_image_favorites(conn)
migrate_saved_views(conn)
# A review group without its annotation group is refused before the
# annotation install could make it look valid (catalog only; the rows
# are validated once, in migrate_review_state).
review_error = review_without_annotations_error(conn)
if review_error:
raise sqlite3.DatabaseError(review_error)
migrate_annotation_state(conn)
migrate_review_state(conn)
def _migrate_prompt_pairs(self, conn: sqlite3.Connection) -> None:
conn.execute("SAVEPOINT prompt_pair_schema")
@@ -1035,9 +1023,9 @@ class PromptModel:
Args:
path: File to inspect
require_library_state: Also require the tables every current
database has (Library state, image favorites, saved views,
image annotations and the review queue), as a migrated
restore stage must; old backups may lack them.
database has (Library state, image favorites, saved views
and image annotations), as a migrated restore stage must;
old backups may lack them.
Returns:
(ok, reason): reason is "ok" when the file is acceptable, otherwise
@@ -1107,11 +1095,6 @@ class PromptModel:
)
if annotation_error:
return False, annotation_error
review_error = review_schema_error(
conn, allow_missing=not require_library_state
)
if review_error:
return False, review_error
return True, "ok"
except sqlite3.Error as e:
return False, f"Invalid SQLite database: {e}"
+31 -20
View File
@@ -4,7 +4,6 @@ Database operations for KikoTextEncode prompt storage and retrieval.
import sqlite3
import json
import math
import datetime
import os
from typing import Optional, List, Dict, Any, Set, Tuple
@@ -14,8 +13,6 @@ from .pairs import PairDependencyError, PairOperations
from .library import LibraryOperations, titled
from .image_favorites import ImageFavoriteOperations
from .annotations import AnnotationOperations
from .review import ReviewOperations
from .review_decisions import ReviewDecisionOperations
from .saved_views import SavedViewOperations
from .library_commands import LibraryCommandOperations
from .vocabulary import VocabularyOperations
@@ -23,13 +20,11 @@ from .execution_history import ExecutionHistoryOperations
from .execution_contexts import ExecutionContextOperations, LIVE_HINT_CTES
from .execution_outputs import ExecutionOutputOperations
from .history_reads import HistoryReadOperations
from .health import HealthOperations, iter_consistency_issues
# Import logging system
try:
from ..utils.logging_config import get_logger
from ..utils.hashing import generate_prompt_hash
from ..utils.json_strict import dumps_strict
except ImportError:
import sys
@@ -37,7 +32,6 @@ except ImportError:
sys.path.insert(0, current_dir)
from utils.logging_config import get_logger
from utils.hashing import generate_prompt_hash
from utils.json_strict import dumps_strict
# Subquery to fetch tags from junction table, embedded in SELECT statements
@@ -347,10 +341,7 @@ class PromptDatabase(
HistoryReadOperations,
ImageFavoriteOperations,
AnnotationOperations,
ReviewOperations,
ReviewDecisionOperations,
SavedViewOperations,
HealthOperations,
):
"""Database operations class for managing prompts."""
@@ -1704,9 +1695,9 @@ class PromptDatabase(
width,
height,
image_format,
dumps_strict(metadata.get("workflow", {})),
dumps_strict(metadata.get("prompt", {})),
dumps_strict(metadata.get("parameters", {})),
json.dumps(metadata.get("workflow", {})),
json.dumps(metadata.get("prompt", {})),
json.dumps(metadata.get("parameters", {})),
),
)
conn.commit()
@@ -1911,13 +1902,13 @@ class PromptDatabase(
obj: The object to clean (dict, list, or scalar)
Returns:
Cleaned object with NaN and Infinity values replaced by None
Cleaned object with NaN values replaced by None
"""
if isinstance(obj, dict):
return {key: self._clean_nan_values(value) for key, value in obj.items()}
elif isinstance(obj, list):
return [self._clean_nan_values(item) for item in obj]
elif isinstance(obj, float) and not math.isfinite(obj):
elif isinstance(obj, float) and str(obj) == "nan":
return None
else:
return obj
@@ -2579,13 +2570,33 @@ class PromptDatabase(
return cursor.rowcount
def check_consistency(self) -> List[str]:
"""Run consistency checks and return a list of issue descriptions.
The same queries back the System health check's Consistency row
(database/health.py).
"""
"""Run consistency checks and return a list of issue descriptions."""
issues: List[str] = []
with self.model.get_connection() as conn:
return list(iter_consistency_issues(conn))
# Check for orphaned prompt_tags entries
cursor = conn.execute("""
SELECT pt.prompt_id, pt.tag_id FROM prompt_tags pt
LEFT JOIN prompts p ON pt.prompt_id = p.id
WHERE p.id IS NULL
""")
for ref in cursor.fetchall():
issues.append(
"prompt_tags entry references non-existent prompt "
f"{ref['prompt_id']}"
)
# Check for orphaned image entries
cursor = conn.execute("""
SELECT gi.id, gi.prompt_id FROM generated_images gi
LEFT JOIN prompts p ON gi.prompt_id = p.id
WHERE p.id IS NULL
""")
for ref in cursor.fetchall():
issues.append(
f"Image {ref['id']} references non-existent prompt "
f"{ref['prompt_id']}"
)
return issues
def _image_row_to_dict(self, row: sqlite3.Row) -> Dict[str, Any]:
"""
+111 -139
View File
@@ -1,5 +1,6 @@
"""Atomic metadata enrichment for a single known generation context."""
import json
import os
from typing import Any, Dict
@@ -8,12 +9,8 @@ from .operations import _coerce_file_info
try:
from ..utils.hashing import generate_prompt_hash
from ..utils.json_strict import dumps_strict
from ..utils.prompt_pair_resolver import RESOLVER_VERSION
except ImportError:
from utils.hashing import generate_prompt_hash
from utils.json_strict import dumps_strict
from utils.prompt_pair_resolver import RESOLVER_VERSION
def ingest_resolved_pair(
@@ -22,143 +19,118 @@ def ingest_resolved_pair(
state = resolution.get("state", "unresolved")
if state != "resolved":
return {"status": state, "pairs_added": 0, "links_added": 0}
with db._pair_transaction() as conn:
return _ingest_resolved_pair_in(conn, db, image_path, metadata, resolution)
def _conflict() -> Dict[str, Any]:
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
def _ingest_resolved_pair_in(
conn: Any,
db: Any,
image_path: str,
metadata: Dict[str, Any],
resolution: Dict[str, Any],
*,
provenance: Dict[str, Any] | None = None,
) -> Dict[str, Any]:
"""``ingest_resolved_pair`` inside the caller's open ``_pair_transaction``.
The compatible-only rules are the public ingester's: an image with more
than one exact row, another positive, or a prior pair with another
negative is a ``conflict`` and nothing is written. The caller owns the
transaction, so its own later failure rolls the link back too.
``provenance`` replaces the link's default resolver provenance (Needs
review records the user's selection there).
"""
positive, negative = resolution["positive"], resolution["negative"]
positive_text = positive.get("storage_text", positive["text"])
negative_text = negative.get("storage_text", negative["text"])
normalized = normalize_image_path(image_path)
if provenance is None:
provenance = {
"resolver_version": RESOLVER_VERSION,
"positive": positive,
"negative": negative,
}
encoded = dumps_strict(provenance, sort_keys=True, separators=(",", ":"))
images = conn.execute(
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=?",
(normalized, image_path),
).fetchall()
if len(images) > 1 or (
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
):
return _conflict()
existing = images[0] if images else None
if existing:
prior = conn.execute(
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
(existing["id"],),
).fetchall()
expected = (
generate_prompt_hash(negative_text) if negative_text.strip() else None
)
if any(row["negative_hash"] != expected for row in prior):
return _conflict()
prior_pair = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE p.hash=? AND n.hash IS ?",
(
generate_prompt_hash(positive_text),
generate_prompt_hash(negative_text) if negative_text.strip() else None,
),
).fetchone()
pair_id = db._save_pair(conn, positive_text, negative_text)
pair = conn.execute("SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)).fetchone()
pairs_added = int(prior_pair is None)
file_size, width, height, image_format = _coerce_file_info(metadata)
# ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON.
snapshots = {
column: dumps_strict(metadata[key], sort_keys=True)
for column, key in (
("workflow_data", "workflow"),
("prompt_metadata", "prompt"),
("parameters", "parameters"),
)
if key in metadata
}
info = {
"file_size": file_size,
"width": width,
"height": height,
"format": image_format,
}
snapshots.update({key: value for key, value in info.items() if value is not None})
changed = False
if existing:
image_id = existing["id"]
updates = {
key: value for key, value in snapshots.items() if existing[key] != value
}
if updates:
conn.execute(
"UPDATE generated_images SET "
+ ",".join(f"{key}=?" for key in updates)
+ " WHERE id=?",
[*updates.values(), image_id],
)
changed = True
else:
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
values = [
pair["positive_prompt_id"],
image_path,
os.path.basename(image_path),
normalized,
*snapshots.values(),
]
image_id = conn.execute(
"INSERT INTO generated_images("
+ ",".join(columns)
+ ") VALUES("
+ ",".join("?" for _ in columns)
+ ")",
values,
).lastrowid
linked = conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
"AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()
db._link_image_to_pair(conn, pair_id, image_id, provenance)
status = (
"added"
if existing is None
else "enriched" if changed or not linked else "unchanged"
provenance = {"resolver_version": 1, "positive": positive, "negative": negative}
encoded = json.dumps(
provenance, sort_keys=True, separators=(",", ":"), allow_nan=False
)
return {
"status": status,
"pairs_added": pairs_added,
"links_added": int(not linked),
"pair_id": pair_id,
"image_id": image_id,
}
with db._pair_transaction() as conn:
images = conn.execute(
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=?",
(normalized, image_path),
).fetchall()
if len(images) > 1 or (
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
existing = images[0] if images else None
if existing:
prior = conn.execute(
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
(existing["id"],),
).fetchall()
expected = (
generate_prompt_hash(negative_text) if negative_text.strip() else None
)
if any(row["negative_hash"] != expected for row in prior):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
prior_pair = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE p.hash=? AND n.hash IS ?",
(
generate_prompt_hash(positive_text),
generate_prompt_hash(negative_text) if negative_text.strip() else None,
),
).fetchone()
pair_id = db._save_pair(conn, positive_text, negative_text)
pair = conn.execute(
"SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)
).fetchone()
pairs_added = int(prior_pair is None)
file_size, width, height, image_format = _coerce_file_info(metadata)
snapshots = {
column: json.dumps(metadata[key], sort_keys=True, allow_nan=False)
for column, key in (
("workflow_data", "workflow"),
("prompt_metadata", "prompt"),
("parameters", "parameters"),
)
if key in metadata
}
info = {
"file_size": file_size,
"width": width,
"height": height,
"format": image_format,
}
snapshots.update(
{key: value for key, value in info.items() if value is not None}
)
changed = False
if existing:
image_id = existing["id"]
updates = {
key: value for key, value in snapshots.items() if existing[key] != value
}
if updates:
conn.execute(
"UPDATE generated_images SET "
+ ",".join(f"{key}=?" for key in updates)
+ " WHERE id=?",
[*updates.values(), image_id],
)
changed = True
else:
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
values = [
pair["positive_prompt_id"],
image_path,
os.path.basename(image_path),
normalized,
*snapshots.values(),
]
image_id = conn.execute(
"INSERT INTO generated_images("
+ ",".join(columns)
+ ") VALUES("
+ ",".join("?" for _ in columns)
+ ")",
values,
).lastrowid
linked = conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
"AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()
db._link_image_to_pair(conn, pair_id, image_id, provenance)
status = (
"added"
if existing is None
else "enriched" if changed or not linked else "unchanged"
)
return {
"status": status,
"pairs_added": pairs_added,
"links_added": int(not linked),
"pair_id": pair_id,
"image_id": image_id,
}
-561
View File
@@ -1,561 +0,0 @@
"""Needs review items (#249): persist and read capture-bound ambiguous evidence.
An item is one ambiguous resolver result for one annotation byte record (the
whole-file SHA-256 of one physical file at one locator), keyed by
``(byte_id, resolver_version, evidence_hash)``. Recording is idempotent: the
same evidence for the same bytes is a duplicate and never touches an existing
item, its state, revision or decisions. Changed bytes are a new byte record
and therefore a new pending item; the old one stays.
Callers pass verified facts (the capture's digest, size and ``mtime_ns``);
nothing here reads the filesystem. Writes run in one ``_pair_transaction``
fenced by the storage epoch the capture was taken under.
"""
import hashlib
import json
import os
import re
import sqlite3
from typing import Iterator
try:
from ..utils.hashing import generate_prompt_hash
except ImportError:
from utils.hashing import generate_prompt_hash
from .annotation_schema import (
BYTES_TABLE,
FILES_TABLE,
METADATA_TABLE as ANNOTATION_METADATA_TABLE,
)
from .annotations import _new_id, _validate_locator, _validate_target
from .models import normalize_image_path, utc_now_iso
from .review_changes import notify_review_change
from .review_schema import (
DECISIONS_TABLE,
ITEMS_TABLE,
MAX_CANDIDATES,
MAX_SPELLED_PATH_BYTES,
METADATA_TABLE,
canonical_evidence,
evidence_digest,
evidence_offers_choice,
)
# Only this module's table constants are interpolated into the SQL below.
_INSERT_ITEM = (
f"INSERT OR IGNORE INTO {ITEMS_TABLE}(id,byte_id,spelled_path," # nosec B608
"resolver_version,evidence_hash,evidence,candidate_count,limited,state,"
"revision,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?,'pending',0,?,?)"
)
_BUMP_REVIEW_INDEX = (
f"UPDATE {METADATA_TABLE} SET " # nosec B608
"review_index_revision=review_index_revision+1 WHERE singleton_id=1"
)
# A later spelling of the same locator (a case-only rename on a case-folding
# filesystem) replaces the recorded one, so targets keep formatting.
_REFRESH_SPELLING = (
f"UPDATE {ITEMS_TABLE} SET spelled_path=? " # nosec B608
f"WHERE spelled_path<>? AND byte_id IN (SELECT b.id FROM {BYTES_TABLE} b "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE f.locator=?)"
)
_REVIEW_INDEX = (
f"SELECT review_index_revision FROM {METADATA_TABLE} " # nosec B608
"WHERE singleton_id=1"
)
_BUMP_IMAGE_INDEX = (
f"UPDATE {ANNOTATION_METADATA_TABLE} SET " # nosec B608
"image_index_revision=image_index_revision+1 WHERE singleton_id=1"
)
_ITEMS_OF_LOCATOR = (
"SELECT i.id,i.byte_id,i.resolver_version,i.evidence_hash," # nosec B608
"i.evidence,i.candidate_count,i.limited,i.state,i.revision,i.created_at,"
f"i.updated_at,b.sha256 FROM {FILES_TABLE} f "
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? ORDER BY i.created_at,i.id"
)
_QUEUED = (
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
"LIMIT 1"
)
_PENDING_AT = (
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
"AND i.state='pending' LIMIT 1"
)
# Pending items of one file: the candidates a link settles.
_PENDING_OF_LOCATOR = (
f"SELECT i.id FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND i.state='pending'"
)
# Pending items of one file with their evidence: an untraced file's settle.
_PENDING_EVIDENCE_OF_LOCATOR = (
f"SELECT i.id,i.evidence FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND i.state='pending'"
)
# Only an undecided item goes: a decision row (RESTRICT) is never deleted.
_SETTLE_ITEM = (
f"DELETE FROM {ITEMS_TABLE} WHERE id=? AND state='pending' " # nosec B608
f"AND revision=0 AND NOT EXISTS (SELECT 1 FROM {DECISIONS_TABLE} d "
f"WHERE d.item_id={ITEMS_TABLE}.id)"
)
_EPOCH = "SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id=1"
_PENDING_COUNT = (
f"SELECT COUNT(*) FROM {ITEMS_TABLE} WHERE state='pending'" # nosec B608
)
_QUEUE_COLUMNS = (
"id",
"state",
"revision",
"created_at",
"candidate_count",
"limited",
"spelled_path",
"locator",
)
_QUEUE_SELECT = (
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
f"i.limited,i.spelled_path,f.locator FROM {ITEMS_TABLE} i "
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id "
)
_QUEUE_FIRST = _QUEUE_SELECT + (
"WHERE i.state='pending' ORDER BY i.created_at,i.id LIMIT ?"
)
_QUEUE_AFTER = _QUEUE_SELECT + (
"WHERE i.state='pending' AND (i.created_at,i.id)>(?,?) "
"ORDER BY i.created_at,i.id LIMIT ?"
)
_DETAIL_COLUMNS = _QUEUE_COLUMNS + (
"byte_id",
"resolver_version",
"evidence_hash",
"evidence",
"updated_at",
"sha256",
)
_DETAIL = (
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
"i.limited,i.spelled_path,f.locator,i.byte_id,i.resolver_version,"
"i.evidence_hash,i.evidence,i.updated_at,b.sha256 "
f"FROM {ITEMS_TABLE} i JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE i.id=?"
)
# The legacy rows and pair links the ingester's compatibility rules read.
_DEPENDENT_IMAGES = (
"SELECT gi.id,p.hash,gi.image_path FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=? ORDER BY gi.id"
)
_DEPENDENT_LINKS = (
"SELECT DISTINCT ip.pair_id,n.hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE ip.image_id=? ORDER BY ip.pair_id"
)
# Rows whose spelling ends with the file's name: the candidates for another
# spelling of the same physical file (the link summaries' filename fallback).
_SAME_NAME_ROWS = (
"SELECT id,image_path FROM generated_images "
"WHERE image_path LIKE ? ESCAPE '\\' ORDER BY id"
)
_TITLES = (
"SELECT p.hash,s.title FROM prompts p "
"JOIN prompt_library_state s ON s.prompt_id=p.id "
"WHERE p.hash IN ({marks}) AND s.title IS NOT NULL ORDER BY p.id"
)
_ITEM_ID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
_ITEM_COLUMNS = (
"id",
"byte_id",
"resolver_version",
"evidence_hash",
"evidence",
"candidate_count",
"limited",
"state",
"revision",
"created_at",
"updated_at",
"sha256",
)
def _validate_spelling(spelled_path: object, locator: str) -> None:
if (
not isinstance(spelled_path, str)
or not spelled_path
or len(spelled_path.encode("utf-8", "surrogatepass")) > MAX_SPELLED_PATH_BYTES
or os.path.normcase(spelled_path) != locator
):
raise ValueError("spelled_path must be the file's spelling of its locator")
def dependent_images(conn: sqlite3.Connection, spellings) -> list:
"""``(id, image_path, positive_hash)`` of every legacy row of the file.
``spellings`` are the paths the file is linked under (the Apply path and
every alias of the same physical file); each is matched as the ingester
matches one path. Rows are distinct and ordered by id.
"""
found = {}
for spelled in dict.fromkeys(spellings):
for row in conn.execute(
_DEPENDENT_IMAGES, (normalize_image_path(spelled), spelled)
):
found[row[0]] = (row[0], row[2], row[1])
return [found[key] for key in sorted(found)]
def same_name_rows(conn: sqlite3.Connection, name: str) -> list:
"""``(id, image_path)`` of rows whose spelling may end with ``name``."""
# Imported here: operations composes this module's mixin.
from .operations import escape_like
return [
(row[0], row[1])
for row in conn.execute(_SAME_NAME_ROWS, (f"%{escape_like(name)}",))
]
def dependency_fingerprint(
conn: sqlite3.Connection, image_path: str, aliases=()
) -> str:
"""SHA-256 of what Apply's compatibility depends on for ``image_path``.
The exact legacy rows of the file under ``image_path`` and every alias
spelling of it, with their positive hashes, and each row's pair links
with the negative hash, read as the ingester reads them. Unrelated prompt
changes (tags, titles) do not move it.
"""
rows = []
for image_id, _spelled, positive_hash in dependent_images(
conn, (image_path, *aliases)
):
links = conn.execute(_DEPENDENT_LINKS, (image_id,)).fetchall()
rows.append([image_id, positive_hash, [list(link) for link in links]])
text = canonical_evidence({"v": 1, "images": rows})
return hashlib.sha256(text.encode("utf-8")).hexdigest()
class ReviewOperations:
"""PromptDatabase mixin: insert and read Needs review items."""
def record_review_item(
self,
*,
locator: str,
spelled_path: str,
sha256: str,
size: int,
mtime_ns: int,
epoch: str,
resolver_version: int,
evidence: dict,
candidate_count: int,
limited: bool,
) -> bool:
"""Store one ambiguous result for these exact bytes; True when new.
The byte record is created (or its verified fingerprint refreshed)
like an observe. ``spelled_path`` is the file's own spelling (it must
fold to ``locator``); targets are formatted from it. Raises
``LibraryCommandError("storage-changed")`` when the storage epoch is
no longer ``epoch`` (a restore since the capture) and ``ValueError``
for invalid facts or evidence.
"""
_validate_target(locator, sha256, size, mtime_ns)
_validate_spelling(spelled_path, locator)
if type(resolver_version) is not int or resolver_version < 1:
raise ValueError("resolver_version must be a positive integer")
if (
type(candidate_count) is not int
or not 0 <= candidate_count <= MAX_CANDIDATES
):
raise ValueError(f"candidate_count must be between 0 and {MAX_CANDIDATES}")
if not isinstance(limited, bool) or (limited and candidate_count):
raise ValueError("A limited item carries no candidates")
text = canonical_evidence(evidence)
evidence_hash = evidence_digest(text)
with self._pair_transaction() as conn:
self._library_epoch(conn, epoch)
now = utc_now_iso()
file_id = self._annotation_owner(conn, locator, now)
byte_id, _revision, created, moved = self._annotation_byte(
conn, file_id, sha256, size, mtime_ns, now
)
if created or moved:
conn.execute(_BUMP_IMAGE_INDEX)
cursor = conn.execute(
_INSERT_ITEM,
(
_new_id(),
byte_id,
spelled_path,
resolver_version,
evidence_hash,
text,
candidate_count,
int(limited),
now,
now,
),
)
inserted = cursor.rowcount > 0
refreshed = _refresh_spelling(conn, locator, spelled_path)
if inserted or refreshed:
conn.execute(_BUMP_REVIEW_INDEX)
if inserted or refreshed:
notify_review_change(self.model.db_path)
return inserted
def refresh_review_spelling(self, locator: str, spelled_path: str) -> bool:
"""Record ``spelled_path`` as the spelling of every item at ``locator``.
The rescan's skip calls it for a file it does not copy again: a
case-only rename keeps size and ``mtime_ns``, and a target formatted
from the old spelling would be refused. True when a spelling changed
(the review index revision moves; item revisions do not, so an open
decision stays valid).
"""
_validate_locator(locator)
_validate_spelling(spelled_path, locator)
with self._pair_transaction() as conn:
refreshed = _refresh_spelling(conn, locator, spelled_path)
if refreshed:
conn.execute(_BUMP_REVIEW_INDEX)
if refreshed:
notify_review_change(self.model.db_path)
return refreshed
def review_index_revision(self) -> int:
"""The queue's revision (moved by every queue change)."""
row = self.model.get_connection().execute(_REVIEW_INDEX).fetchone()
if row is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
return row[0]
def review_queue_version(self) -> dict:
"""``{library_epoch, review_revision}`` of the queue, from one snapshot.
The pair a list read answers: equal pairs mean the same queue (a
restore moves the epoch, every queue change moves the revision).
"""
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
revision = conn.execute(_REVIEW_INDEX).fetchone()
if epoch is None or revision is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
return {"library_epoch": epoch[0], "review_revision": revision[0]}
def review_items_for(self, locator: str) -> list:
"""Every item of the file at ``locator``, oldest first, evidence parsed."""
_validate_locator(locator)
return list(self._iter_review_items(locator))
def _iter_review_items(self, locator: str) -> Iterator[dict]:
conn = self.model.get_connection()
for row in conn.execute(_ITEMS_OF_LOCATOR, (locator,)):
item = dict(zip(_ITEM_COLUMNS, tuple(row)))
item["evidence"] = json.loads(item["evidence"])
item["limited"] = bool(item["limited"])
yield item
# -- queue reads (#249 E3) -------------------------------------------------
def review_fingerprint_queued(
self, locator: str, size: int, mtime_ns: int, resolver_version: int
) -> bool:
"""True when an item cites a byte record verified at this exact stat.
A rescan uses it to skip the copy and hash of a file it already
queued: the same ``(size, mtime_ns)`` the bytes were verified with.
"""
row = (
self.model.get_connection()
.execute(_QUEUED, (locator, size, mtime_ns, resolver_version))
.fetchone()
)
return row is not None
def review_pending_at(
self, locator: str, size: int, mtime_ns: int, resolver_version: int
) -> bool:
"""True when a *pending* item cites bytes verified at this exact stat.
``review_fingerprint_queued`` also matches decided items; a file whose
only item is decided is not in the queue.
"""
row = (
self.model.get_connection()
.execute(_PENDING_AT, (locator, size, mtime_ns, resolver_version))
.fetchone()
)
return row is not None
def review_file_linked(self, spellings) -> bool:
"""True when a saved row links the file under any of ``spellings``."""
with self._annotation_read() as conn:
return bool(dependent_images(conn, spellings))
def settle_linked_review_items(self, files) -> int:
"""Remove the pending, undecided items of files that are now linked.
``files`` holds ``(locator, spellings)`` per file. An item leaves the
queue only when it is still ``pending`` at revision 0 with no decision
row and a saved row links its file under one of ``spellings``; a
decided item is never touched. Files without a pending item cost one
indexed read and open no write transaction. Returns how many items
were removed (the review index revision moves when any was).
"""
files = [(locator, tuple(spellings)) for locator, spellings in files]
for locator, _spellings in files:
_validate_locator(locator)
conn = self.model.get_connection()
candidates = [
(locator, spellings)
for locator, spellings in files
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
]
if not candidates:
return 0
removed = 0
with self._pair_transaction() as conn:
for locator, spellings in candidates:
if not dependent_images(conn, spellings):
continue
for (item_id,) in conn.execute(
_PENDING_OF_LOCATOR, (locator,)
).fetchall():
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
if removed:
conn.execute(_BUMP_REVIEW_INDEX)
if removed:
notify_review_change(self.model.db_path)
return removed
def settle_untraced_review_items(self, locators) -> int:
"""Remove the pending, undecided items that offer nothing to choose.
``locators`` are files the caller found ambiguous without a choice
(fewer than two candidates carry a prompt). Of their items, one leaves
the queue only when its own stored evidence offers no choice either
(``evidence_offers_choice``) and it is still ``pending`` at revision 0
with no decision row, the same safe delete as
``settle_linked_review_items``. Files without a pending item cost one
indexed read and open no write transaction. Returns how many items
were removed (the review index revision moves when any was).
"""
locators = list(locators)
for locator in locators:
_validate_locator(locator)
conn = self.model.get_connection()
pending = [
locator
for locator in locators
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
]
if not pending:
return 0
removed = 0
with self._pair_transaction() as conn:
for locator in pending:
rows = conn.execute(_PENDING_EVIDENCE_OF_LOCATOR, (locator,)).fetchall()
for item_id, evidence in rows:
if evidence_offers_choice(json.loads(evidence)):
continue
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
if removed:
conn.execute(_BUMP_REVIEW_INDEX)
return removed
def review_queue_page(self, *, limit: int, after: tuple | None = None) -> dict:
"""One page of pending items, oldest first, from one snapshot.
``after`` is the ``(created_at, id)`` keyset of the last item already
shown. Returns ``{epoch, revision, total, items, more}``; ``total`` is
the exact pending count and ``items`` carry their ``spelled_path`` and
``locator`` (server-side only).
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be a positive integer")
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
revision = conn.execute(_REVIEW_INDEX).fetchone()
if epoch is None or revision is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
total = conn.execute(_PENDING_COUNT).fetchone()[0]
if after is None:
rows = conn.execute(_QUEUE_FIRST, (limit + 1,)).fetchall()
else:
rows = conn.execute(_QUEUE_AFTER, (*after, limit + 1)).fetchall()
items = [_queue_row(row, _QUEUE_COLUMNS) for row in rows[:limit]]
return {
"epoch": epoch[0],
"revision": revision[0],
"total": total,
"items": items,
"more": len(rows) > limit,
}
def review_item(self, item_id: str) -> dict | None:
"""One item with its parsed evidence and the pending total, or None."""
if not isinstance(item_id, str) or _ITEM_ID.fullmatch(item_id) is None:
return None
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
if epoch is None:
raise sqlite3.DatabaseError("Missing execution history metadata")
row = conn.execute(_DETAIL, (item_id,)).fetchone()
if row is None:
return None
item = _queue_row(row, _DETAIL_COLUMNS)
item["evidence"] = json.loads(item["evidence"])
item["epoch"] = epoch[0]
item["total"] = conn.execute(_PENDING_COUNT).fetchone()[0]
return item
def review_dependency_fingerprint(self, image_path: str, aliases=()) -> str:
with self._annotation_read() as conn:
return dependency_fingerprint(conn, image_path, aliases)
def review_same_name_rows(self, name: str) -> list:
"""``(id, image_path)`` of legacy rows that may spell a file ``name``."""
with self._annotation_read() as conn:
return same_name_rows(conn, name)
def stored_prompt_titles(self, texts) -> dict:
"""Library titles of saved prompts by text (dedup hash), when set."""
hashes = {generate_prompt_hash(text): text for text in texts if text.strip()}
if not hashes:
return {}
marks = ",".join("?" * len(hashes))
found: dict[str, tuple] = {}
conn = self.model.get_connection()
for prompt_hash, title in conn.execute(
_TITLES.format(marks=marks), list(hashes)
):
text = hashes[prompt_hash]
found[text] = found.get(text, ()) + (title,)
return found
def _refresh_spelling(conn: sqlite3.Connection, locator: str, spelled: str) -> bool:
return conn.execute(_REFRESH_SPELLING, (spelled, spelled, locator)).rowcount > 0
def _queue_row(row, columns) -> dict:
item = dict(zip(columns, tuple(row)))
item["limited"] = bool(item["limited"])
return item
-64
View File
@@ -1,64 +0,0 @@
"""Needs review queue change notices (#249 PR F2).
Every committed change to the pending queue (an item recorded by the scan or
the monitor, a spelling refresh, a decision) calls ``notify_review_change``
after its transaction, from whichever thread wrote it. The review event
stream (``py/api/review_events.py``) subscribes, so an open Needs review page
re-reads its queue when it changes instead of polling. Observers are keyed by
the database file, like ``subscribe_history_restore``; an observer that fails
is logged and never fails the write.
"""
import os
import threading
from typing import Callable
try:
from ..utils.logging_config import get_logger
except ImportError:
from utils.logging_config import get_logger
_logger = get_logger("prompt_manager.database.review_changes")
_lock = threading.Lock()
_observers: dict[str, dict[object, Callable[[], None]]] = {}
def _identity(db_path: str) -> str:
return os.path.normcase(os.path.realpath(db_path))
def subscribe_review_changes(
db_path: str, callback: Callable[[], None]
) -> Callable[[], None]:
"""Call ``callback()`` after each committed queue change; return the unsubscribe."""
identity = _identity(db_path)
token = object()
with _lock:
_observers.setdefault(identity, {})[token] = callback
def unsubscribe() -> None:
with _lock:
observers = _observers.get(identity)
if observers is not None:
observers.pop(token, None)
if not observers:
_observers.pop(identity, None)
return unsubscribe
def observer_count(db_path: str) -> int:
"""How many observers watch ``db_path`` (tests and diagnostics)."""
with _lock:
return len(_observers.get(_identity(db_path), {}))
def notify_review_change(db_path: str) -> None:
"""Tell every observer of ``db_path``; call it after the commit."""
with _lock:
callbacks = tuple(_observers.get(_identity(db_path), {}).values())
for callback in callbacks:
try:
callback()
except Exception:
_logger.warning("Needs review change observer failed", exc_info=True)
-291
View File
@@ -1,291 +0,0 @@
"""Needs review decisions (#249 E3): explicit, auditable and compatible-only.
A decision is one row of ``media_review_decisions`` for one item revision.
``selected`` links the chosen candidate's pair to the file through the public
ingester's compatible-only rules (``_ingest_resolved_pair_in``) in the same
``_pair_transaction`` as the decision row, the item's new state and revision
and the review index revision; any conflict or later failure rolls all of it
back. ``unresolved`` records the decision and links nothing. No decision
counts a run, touches usage or calls an execution writer.
Callers verify the file first (the capture must be the item's bytes and the
current file must still hash to them); nothing here reads the filesystem.
"""
import os
import sqlite3
from .annotation_schema import BYTES_TABLE
from .annotations import _new_id
from .library_commands import LibraryCommandError
from .models import utc_now_iso
from .review import (
_BUMP_REVIEW_INDEX,
_DETAIL,
_DETAIL_COLUMNS,
_PENDING_COUNT,
_queue_row,
dependency_fingerprint,
dependent_images,
same_name_rows,
)
from .review_changes import notify_review_change
from .review_schema import DECISIONS_TABLE, ITEMS_TABLE, canonical_evidence
DECISION_KIND = "user-selection"
_INSERT_DECISION = (
f"INSERT INTO {DECISIONS_TABLE}(id,item_id,decision_revision," # nosec B608
"created_at,outcome,candidate_hash,evidence,pair_id,image_id) "
"VALUES(?,?,?,?,?,?,?,?,?)"
)
_ADVANCE_ITEM = (
f"UPDATE {ITEMS_TABLE} SET state=?,revision=revision+1," # nosec B608
"updated_at=? WHERE id=? AND revision=?"
)
_ITEM_BYTES = (
f"SELECT b.sha256 FROM {ITEMS_TABLE} i " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id WHERE i.id=?"
)
class ReviewCommandError(LibraryCommandError):
"""A decision lost a race or would rewrite an association.
``code`` is ``review-conflict`` (the item was decided or changed since it
was read), ``association-conflict`` (the choice is not compatible with
the file's saved links, or those links changed), ``file-changed`` (the
item is not about these bytes) or ``review-missing``. ``item`` is the
current item read inside the transaction, ``total`` the pending count and
``fingerprint`` the current dependency fingerprint, when known;
``changed`` marks an association conflict caused by links that changed
since the reviewer read them (rather than an incompatible choice).
"""
def __init__(
self, code, epoch, *, item=None, total=None, fingerprint=None, changed=False
):
super().__init__(code, epoch)
self.item = item
self.total = total
self.fingerprint = fingerprint
self.changed = changed
# The public view of ``item``, set by the API when it formats one.
self.view = None
class _Rollback(Exception):
"""Raised inside the transaction so the context manager rolls back."""
def __init__(self, error: ReviewCommandError):
super().__init__(error.code)
self.error = error
class ReviewDecisionOperations:
"""PromptDatabase mixin: record one decision on one Needs review item."""
def decide_review_item(
self,
*,
item_id: str,
expected_epoch: str,
expected_revision: int,
sha256: str,
outcome: str,
candidate: dict | None = None,
image_path: str | None = None,
metadata: dict | None = None,
expected_fingerprint: str | None = None,
aliases: tuple = (),
known_rows: frozenset | None = None,
) -> dict:
"""Record the decision; return ``{item, total, pair}``.
``selected`` needs the evidence ``candidate`` (with its ``hash``), the
``image_path`` the scan and the monitor link under, the captured
``metadata`` and the ``expected_fingerprint`` the reviewer saw.
``aliases`` are the other spellings legacy rows of the same physical
file use (a symlinked, junctioned or mapped root) and ``known_rows``
the ids of every same-name row the caller classified; a same-name row
it did not see is a changed association.
Raises ``LibraryCommandError("storage-changed")`` after a restore and
:class:`ReviewCommandError` for every refused decision, with zero
writes either way.
"""
if outcome not in ("selected", "unresolved"):
raise ValueError("outcome must be selected or unresolved")
if outcome == "selected" and (
not isinstance(candidate, dict) or not image_path
):
raise ValueError("A selected decision needs a candidate and a file")
try:
with self._pair_transaction() as conn:
decided = self._decide_in(
conn,
item_id=item_id,
expected_epoch=expected_epoch,
expected_revision=expected_revision,
sha256=sha256,
outcome=outcome,
candidate=candidate,
image_path=image_path,
metadata=metadata or {},
expected_fingerprint=expected_fingerprint,
aliases=tuple(aliases),
known_rows=known_rows,
)
except _Rollback as rollback:
raise rollback.error from None
# After the commit: open Needs review pages re-read their queue.
notify_review_change(self.model.db_path)
return decided
def _decide_in(self, conn: sqlite3.Connection, **command) -> dict:
epoch = self._library_epoch(conn, command["expected_epoch"])
item_id = command["item_id"]
current = _current_item(conn, item_id)
if current is None:
raise _Rollback(ReviewCommandError("review-missing", epoch))
total = conn.execute(_PENDING_COUNT).fetchone()[0]
if (
current["state"] != "pending"
or current["revision"] != command["expected_revision"]
):
raise _Rollback(
ReviewCommandError("review-conflict", epoch, item=current, total=total)
)
if conn.execute(_ITEM_BYTES, (item_id,)).fetchone()[0] != command["sha256"]:
raise _Rollback(ReviewCommandError("file-changed", epoch))
decision_id = _new_id()
pair = None
candidate = command["candidate"] if command["outcome"] == "selected" else None
if candidate is not None:
pair = self._apply_candidate(
conn, epoch, current, decision_id, total, **command
)
evidence = {
"kind": DECISION_KIND,
"decision_id": decision_id,
"outcome": command["outcome"],
"reviewed_digest": command["sha256"],
"item_evidence_hash": current["evidence_hash"],
"resolver_version": current["resolver_version"],
"candidate": candidate,
"dependency_fingerprint": command["expected_fingerprint"],
"pair": pair,
}
now = utc_now_iso()
self._insert_review_decision(
conn,
(
decision_id,
item_id,
current["revision"] + 1,
now,
command["outcome"],
candidate["hash"] if candidate is not None else None,
canonical_evidence(evidence),
pair["id"] if pair else None,
pair["image_id"] if pair else None,
),
)
conn.execute(
_ADVANCE_ITEM, (command["outcome"], now, item_id, current["revision"])
)
conn.execute(_BUMP_REVIEW_INDEX)
return {
"item": _current_item(conn, item_id),
"total": conn.execute(_PENDING_COUNT).fetchone()[0],
"pair": pair,
"decision_id": decision_id,
}
def _apply_candidate(
self, conn, epoch, current, decision_id, total, **command
) -> dict:
"""Link the candidate's pair, compatible-only; raise on any conflict."""
# Imported here: pair_ingestion imports operations, which composes this.
from .pair_ingestion import _ingest_resolved_pair_in
image_path = command["image_path"]
aliases = command["aliases"]
fingerprint = dependency_fingerprint(conn, image_path, aliases)
known = command["known_rows"]
unseen = known is not None and any(
row_id not in known
for row_id, _path in same_name_rows(conn, os.path.basename(image_path))
)
if unseen or fingerprint != command["expected_fingerprint"]:
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
changed=True,
)
)
rows = dependent_images(conn, (image_path, *aliases))
if len(rows) > 1:
# One file linked under several spellings: no single row to enrich.
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
)
)
if rows:
# Enrich the one row of this file under the spelling it was
# linked with, so the ingester's compatibility rules see it.
image_path = rows[0][1]
candidate = command["candidate"]
positive, negative = candidate["positive"], candidate["negative"]
provenance = {
"kind": DECISION_KIND,
"decision_id": decision_id,
"reviewed_digest": command["sha256"],
"candidate_hash": candidate["hash"],
"resolver_version": current["resolver_version"],
"positive": positive,
"negative": negative,
}
result = _ingest_resolved_pair_in(
conn,
self,
image_path,
command["metadata"],
{"state": "resolved", "positive": positive, "negative": negative},
provenance=provenance,
)
if result["status"] == "conflict":
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
)
)
return {
"id": result["pair_id"],
"image_id": result["image_id"],
"status": result["status"],
}
@staticmethod
def _insert_review_decision(conn: sqlite3.Connection, values: tuple) -> None:
conn.execute(_INSERT_DECISION, values)
def _current_item(conn: sqlite3.Connection, item_id: str) -> dict | None:
row = conn.execute(_DETAIL, (item_id,)).fetchone()
if row is None:
return None
item = _queue_row(row, _DETAIL_COLUMNS)
item.pop("evidence")
return item
-507
View File
@@ -1,507 +0,0 @@
"""Needs review schema group (#249): capture-bound ambiguous items and decisions.
The group installs like ``image_annotations``: optional in old backups,
required once a database is current, and valid only beside a valid image
annotation group, because every item cites one annotation byte record. It owns
- ``media_review_metadata``: one row, the group's schema version and the
review index revision (moved by every queue change);
- ``media_review_items``: one ambiguous resolver result for one byte record,
keyed by ``(byte_id, resolver_version, evidence_hash)``, with the file's
spelling when it was recorded (the locator is case-folded on Windows, a
target is not), its canonical evidence, candidate count, ``limited`` flag,
state and revision;
- ``media_review_decisions``: the auditable decision history of an item, one
row per decision revision, with optional navigation to the pair and image
the decision linked (nulled, never cascaded, when those are deleted).
Evidence is canonical JSON (:func:`canonical_evidence`): one object, sorted
keys, no whitespace, ASCII only (so characters, bytes and the 8 MiB bound
agree and lone surrogates survive), no NaN or infinity. ``evidence_hash`` is
the SHA-256 of the item's evidence text.
Installation never backfills anything and never touches the filesystem.
Validation reads only: catalog objects compared after whitespace compaction,
columns and foreign keys through the PRAGMAs, persisted values in single
bounded SQL passes, then one streamed pass that re-canonicalizes each evidence
document (one row in memory at a time).
"""
import hashlib
import json
import re
import sqlite3
from typing import Any, Optional
from .annotation_schema import BYTES_TABLE
from .annotation_schema import TABLES as ANNOTATION_TABLES
from .annotation_schema import _catalog as _annotation_catalog
from .annotation_schema import annotation_schema_error
try:
from ..utils.json_strict import dumps_strict
except ImportError:
from utils.json_strict import dumps_strict
SCHEMA_VERSION = 1
METADATA_TABLE = "media_review_metadata"
ITEMS_TABLE = "media_review_items"
DECISIONS_TABLE = "media_review_decisions"
TABLES = (METADATA_TABLE, ITEMS_TABLE, DECISIONS_TABLE)
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
MAX_CANDIDATES = 16
# The file's own spelling when it was recorded (bytes of UTF-8 text). Locators
# are case-folded on Windows, so a target is formatted from this instead.
MAX_SPELLED_PATH_BYTES = 16384
ITEM_STATES = ("pending", "selected", "unresolved")
DECISION_OUTCOMES = ("selected", "unresolved")
_TIMESTAMP_BYTES = 64
_HEX64 = "length({c})=64 AND {c} NOT GLOB '*[^0-9a-f]*'"
_EVIDENCE_CHECK = f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES})"
_TABLE_DDL = {
METADATA_TABLE: (
f"CREATE TABLE {METADATA_TABLE} (singleton_id INTEGER PRIMARY KEY "
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
f"CHECK(schema_version={SCHEMA_VERSION}), "
"review_index_revision INTEGER NOT NULL DEFAULT 0 "
"CHECK(review_index_revision>=0))"
),
ITEMS_TABLE: (
f"CREATE TABLE {ITEMS_TABLE} (id TEXT PRIMARY KEY CHECK(length(id)=36), "
f"byte_id TEXT NOT NULL REFERENCES {BYTES_TABLE}(id) ON DELETE RESTRICT, "
"spelled_path TEXT NOT NULL CHECK(length(CAST(spelled_path AS BLOB)) "
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}), "
"resolver_version INTEGER NOT NULL, evidence_hash TEXT NOT NULL "
f"CHECK({_HEX64.format(c='evidence_hash')}), "
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
"candidate_count INTEGER NOT NULL "
f"CHECK(candidate_count BETWEEN 0 AND {MAX_CANDIDATES}), "
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), "
"state TEXT NOT NULL CHECK(state IN ('pending','selected','unresolved')), "
"revision INTEGER NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, "
"updated_at TEXT NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
),
DECISIONS_TABLE: (
f"CREATE TABLE {DECISIONS_TABLE} (id TEXT PRIMARY KEY "
"CHECK(length(id)=36), item_id TEXT NOT NULL "
f"REFERENCES {ITEMS_TABLE}(id) ON DELETE RESTRICT, "
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
f"({_HEX64.format(c='candidate_hash')})), "
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
"UNIQUE(item_id, decision_revision), "
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
),
}
_INDEX_DDL = {
# The queue reads pending items oldest first with a stable tiebreak.
"idx_media_review_items_queue": (
"CREATE INDEX idx_media_review_items_queue ON "
f"{ITEMS_TABLE}(state, created_at, id)"
),
# Deleting a pair or an image nulls decision navigation by index, not scan.
"idx_media_review_decisions_pair": (
"CREATE INDEX idx_media_review_decisions_pair ON " f"{DECISIONS_TABLE}(pair_id)"
),
"idx_media_review_decisions_image": (
"CREATE INDEX idx_media_review_decisions_image ON "
f"{DECISIONS_TABLE}(image_id)"
),
}
# ``sqlite_autoindex_<table>_<n>``: the TEXT primary key and the UNIQUE key.
_AUTOINDEXES = {ITEMS_TABLE: 2, DECISIONS_TABLE: 2}
# (name, declared type, not null, default, primary key position)
_COLUMNS = {
METADATA_TABLE: (
("singleton_id", "INTEGER", 0, None, 1),
("schema_version", "INTEGER", 1, None, 0),
("review_index_revision", "INTEGER", 1, "0", 0),
),
ITEMS_TABLE: (
("id", "TEXT", 0, None, 1),
("byte_id", "TEXT", 1, None, 0),
("spelled_path", "TEXT", 1, None, 0),
("resolver_version", "INTEGER", 1, None, 0),
("evidence_hash", "TEXT", 1, None, 0),
("evidence", "TEXT", 1, None, 0),
("candidate_count", "INTEGER", 1, None, 0),
("limited", "INTEGER", 1, None, 0),
("state", "TEXT", 1, None, 0),
("revision", "INTEGER", 1, None, 0),
("created_at", "TEXT", 1, None, 0),
("updated_at", "TEXT", 1, None, 0),
),
DECISIONS_TABLE: (
("id", "TEXT", 0, None, 1),
("item_id", "TEXT", 1, None, 0),
("decision_revision", "INTEGER", 1, None, 0),
("created_at", "TEXT", 1, None, 0),
("outcome", "TEXT", 1, None, 0),
("candidate_hash", "TEXT", 0, None, 0),
("evidence", "TEXT", 1, None, 0),
("pair_id", "INTEGER", 0, None, 0),
("image_id", "INTEGER", 0, None, 0),
),
}
# (column, parent table, parent column, on delete)
_FOREIGN_KEYS = {
METADATA_TABLE: set(),
ITEMS_TABLE: {("byte_id", BYTES_TABLE, "id", "RESTRICT")},
DECISIONS_TABLE: {
("item_id", ITEMS_TABLE, "id", "RESTRICT"),
("pair_id", "prompt_pairs", "id", "SET NULL"),
("image_id", "generated_images", "id", "SET NULL"),
},
}
class _NonCanonical(ValueError):
pass
def _reject_constant(name: str) -> Any:
raise _NonCanonical(f"JSON constant {name} is not allowed")
# Built once: validation re-encodes every stored document at startup.
_ENCODER = json.JSONEncoder(
sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False
)
_DECODER = json.JSONDecoder(parse_constant=_reject_constant)
def canonical_evidence(value: Any) -> str:
"""Serialize one evidence object canonically; ``ValueError`` when invalid.
The text is a JSON object with sorted keys, no whitespace, ASCII only and
no NaN/infinity, at most ``MAX_EVIDENCE_BYTES`` bytes. A non-finite number
(ComfyUI stamps ``NaN`` into graphs) is written as ``null``, the store's
policy everywhere (``utils.json_strict.dumps_strict``).
"""
if not isinstance(value, dict):
raise ValueError("Review evidence must be a JSON object")
try:
try:
text = _ENCODER.encode(value)
except ValueError as error:
if "Out of range float" not in str(error):
raise
text = dumps_strict(
value, sort_keys=True, separators=(",", ":"), ensure_ascii=True
)
except (TypeError, RecursionError) as error:
raise ValueError(f"Review evidence is not serializable: {error}") from None
if len(text) > MAX_EVIDENCE_BYTES:
raise ValueError("Review evidence is larger than 8 MiB")
return text
def choosable(candidate: Any) -> bool:
"""True when a candidate offers a prompt to choose: it is ``resolved`` and
its positive side carries non-blank text. The page disables any other
candidate, so it is never something to choose.
"""
if not isinstance(candidate, dict) or candidate.get("state") != "resolved":
return False
positive = candidate.get("positive")
if not isinstance(positive, dict):
return False
text = positive.get("text")
return isinstance(text, str) and bool(text.strip())
# An item is worth a decision only when at least this many candidates offer
# a prompt; with fewer there is nothing to choose between (maintainer test of
# #284/#288: 39 items whose candidates were all "unresolved").
MIN_CHOOSABLE = 2
def evidence_offers_choice(evidence: Any) -> bool:
"""False when stored evidence provably offers nothing to choose.
Unlimited evidence must list ``MIN_CHOOSABLE`` choosable candidates. A
limited item stores none, only ``candidate_total``; it may offer a choice
(more than ``MAX_CANDIDATES`` prompts) unless that total is too small.
"""
if not isinstance(evidence, dict):
return False
if evidence.get("limited"):
total = evidence.get("candidate_total")
return type(total) is int and total >= MIN_CHOOSABLE
candidates = evidence.get("candidates")
if not isinstance(candidates, list):
return False
return sum(1 for c in candidates if choosable(c)) >= MIN_CHOOSABLE
def evidence_digest(text: str) -> str:
"""SHA-256 of canonical evidence text (ASCII, hence UTF-8 identical)."""
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def _is_canonical(text: Any) -> bool:
if not isinstance(text, str):
return False
try:
value = _DECODER.decode(text)
return canonical_evidence(value) == text
except (ValueError, RecursionError):
return False
def _compact(sql: str) -> str:
return re.sub(r"\s+", "", sql).lower()
def _uuid(column: str) -> str:
"""SQL: *column* is a lowercase hyphenated 36-character UUID string."""
return (
f"typeof({column}) = 'text' AND length({column}) = 36 "
f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' "
f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' "
f"AND length(replace({column}, '-', '')) = 32 "
f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'"
)
def _timestamp(column: str) -> str:
return (
f"typeof({column}) = 'text' "
f"AND length(CAST({column} AS BLOB)) <= {_TIMESTAMP_BYTES}"
)
def _non_negative(column: str) -> str:
return f"typeof({column}) = 'integer' AND {column} >= 0"
def _hex64(column: str) -> str:
return f"typeof({column}) = 'text' AND " + _HEX64.format(c=column)
def _evidence(column: str) -> str:
# Type and size only: the streamed pass below proves the text canonical
# (which implies printable ASCII), so a pattern scan here would read every
# document a second time.
return (
f"typeof({column}) = 'text' "
f"AND length(CAST({column} AS BLOB)) BETWEEN 2 AND {MAX_EVIDENCE_BYTES}"
)
def _nullable_integer(column: str) -> str:
return f"({column} IS NULL OR typeof({column}) = 'integer')"
def _in(column: str, values: tuple) -> str:
return f"typeof({column}) = 'text' AND {column} IN ({repr(values)[1:-1]})"
# Every row of a table must satisfy its predicate; one pass each, LIMIT 1.
_ROW_PREDICATES = {
ITEMS_TABLE: " AND ".join(
(
_uuid("id"),
_uuid("byte_id"),
"typeof(spelled_path) = 'text' AND length(CAST(spelled_path AS BLOB)) "
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}",
"typeof(resolver_version) = 'integer'",
_hex64("evidence_hash"),
_evidence("evidence"),
_non_negative("candidate_count"),
f"candidate_count <= {MAX_CANDIDATES}",
"typeof(limited) = 'integer' AND limited IN (0, 1)",
_in("state", ITEM_STATES),
_non_negative("revision"),
_timestamp("created_at"),
_timestamp("updated_at"),
)
),
DECISIONS_TABLE: " AND ".join(
(
_uuid("id"),
_uuid("item_id"),
_non_negative("decision_revision"),
_timestamp("created_at"),
_in("outcome", DECISION_OUTCOMES),
"(CASE WHEN outcome = 'selected' THEN "
f"{_hex64('candidate_hash')} ELSE candidate_hash IS NULL END)",
_evidence("evidence"),
_nullable_integer("pair_id"),
_nullable_integer("image_id"),
)
),
}
def _expected_catalog() -> set:
objects = {("table", name, _compact(sql)) for name, sql in _TABLE_DDL.items()}
objects |= {("index", name, _compact(sql)) for name, sql in _INDEX_DDL.items()}
objects |= {
("index", f"sqlite_autoindex_{table}_{n}", "")
for table, count in _AUTOINDEXES.items()
for n in range(1, count + 1)
}
return objects
def _catalog(conn: sqlite3.Connection) -> list:
names = TABLES + tuple(_INDEX_DDL)
marks = ",".join("?" * len(names))
table_marks = ",".join("?" * len(TABLES))
return conn.execute( # only "?" placeholders are interpolated
"SELECT type,name,sql FROM sqlite_master "
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
names + TABLES,
).fetchall()
def _structure_error(conn: sqlite3.Connection) -> Optional[str]:
for table, expected in _COLUMNS.items():
columns = tuple(
(row[1], row[2].upper(), row[3], row[4], row[5])
for row in conn.execute(f"PRAGMA table_info({table})")
)
if columns != expected:
return f"Review queue table {table} has invalid columns"
foreign_keys = {
(row[3], row[2], row[4], row[6])
for row in conn.execute(f"PRAGMA foreign_key_list({table})")
}
if foreign_keys != _FOREIGN_KEYS[table]:
return f"Review queue table {table} has invalid foreign keys"
if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
return f"Review queue table {table} contains foreign key violations"
return None
def _evidence_error(conn: sqlite3.Connection) -> Optional[str]:
"""Stream every evidence document once; one row is held at a time."""
for evidence_hash, text in conn.execute(
f"SELECT evidence_hash, evidence FROM {ITEMS_TABLE}" # nosec B608
):
if not _is_canonical(text) or evidence_digest(text) != evidence_hash:
return f"Review queue table {ITEMS_TABLE} contains invalid evidence"
for (text,) in conn.execute(
f"SELECT evidence FROM {DECISIONS_TABLE}" # nosec B608
):
if not _is_canonical(text):
return f"Review queue table {DECISIONS_TABLE} contains invalid evidence"
return None
def _value_error(conn: sqlite3.Connection) -> Optional[str]:
# Only this module's constants are interpolated into the SQL below.
rows = conn.execute(
"SELECT COUNT(*), MIN(typeof(singleton_id) = 'integer' " # nosec B608
"AND singleton_id = 1 AND typeof(schema_version) = 'integer' "
f"AND schema_version = {SCHEMA_VERSION} "
f"AND {_non_negative('review_index_revision')}) FROM {METADATA_TABLE}"
).fetchone()
if rows[0] != 1 or rows[1] != 1:
return "Review queue metadata is invalid"
for table, predicate in _ROW_PREDICATES.items():
if conn.execute(
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1" # nosec B608
).fetchone():
return f"Review queue table {table} contains invalid values"
return _evidence_error(conn)
def review_without_annotations_error(conn: sqlite3.Connection) -> Optional[str]:
"""Catalog only: a review group present without its annotation tables.
Startup asks this before the annotation group installs, which would make
a review group copied without it look valid. It reads no rows; the full
validation runs once in :func:`migrate_review_state`.
"""
if not _catalog(conn):
return None
present = {
name for kind, name, _sql in _annotation_catalog(conn) if kind == "table"
}
if present != set(ANNOTATION_TABLES):
return (
"Review queue schema requires a valid image annotation schema: "
"Missing image annotation schema"
)
return None
def review_schema_error(
conn: sqlite3.Connection,
*,
allow_missing: bool = True,
annotations_valid: bool = False,
) -> Optional[str]:
"""Reject partial or unsupported review schemas and invalid rows.
Reads only: nothing is ever repaired. A database without any of the
group's objects is acceptable for backups that predate it unless
``allow_missing`` is false. A present group requires a present, valid
image annotation group; ``annotations_valid`` says the caller has just
validated it (startup's annotation install), so only its presence is
checked again.
"""
objects = _catalog(conn)
if not objects:
return None if allow_missing else "Missing review queue schema"
if annotations_valid:
annotation_error = review_without_annotations_error(conn)
else:
annotation_error = annotation_schema_error(conn, allow_missing=False)
if annotation_error:
annotation_error = (
"Review queue schema requires a valid image annotation schema: "
+ annotation_error
)
if annotation_error:
return annotation_error
present = {name for kind, name, _sql in objects if kind == "table"}
if present != set(TABLES):
return "Incomplete review queue schema"
actual = {(kind, name, _compact(sql or "")) for kind, name, sql in objects}
if actual != _expected_catalog():
return "Review queue has an invalid schema definition"
return _structure_error(conn) or _value_error(conn)
def migrate_review_state(conn: sqlite3.Connection) -> None:
"""Install the group atomically; validate before and after creating it.
Runs after ``migrate_annotation_state`` (which has just validated the
annotation group): the group is only valid beside the annotation group it
cites. A present group is validated once, evidence included.
"""
conn.execute("SAVEPOINT media_review_schema")
try:
installed = bool(_catalog(conn))
error = review_schema_error(conn, annotations_valid=True)
if error:
raise sqlite3.DatabaseError(error)
if installed:
# Already complete and valid: nothing to create, and the evidence
# pass is the expensive part of validation, so it runs once.
conn.execute("RELEASE media_review_schema")
return
for sql in _TABLE_DDL.values():
conn.execute(sql.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
for sql in _INDEX_DDL.values():
conn.execute(sql.replace("INDEX", "INDEX IF NOT EXISTS", 1))
conn.execute(
f"INSERT OR IGNORE INTO {METADATA_TABLE} "
"(singleton_id, schema_version, review_index_revision) VALUES (1, ?, 0)",
(SCHEMA_VERSION,),
)
error = review_schema_error(conn, allow_missing=False, annotations_valid=True)
if error:
raise sqlite3.DatabaseError(error)
conn.execute("RELEASE media_review_schema")
except BaseException:
conn.execute("ROLLBACK TO media_review_schema")
conn.execute("RELEASE media_review_schema")
raise
+7 -38
View File
@@ -8,10 +8,6 @@ Split into domain-specific mixins for maintainability:
- AutotagRoutesMixin: auto-tagging model management and tagging
- JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py)
- ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs
- ReviewRoutesMixin: the Needs review queue, item detail and decisions
- ReviewRecordingMixin: Needs review recording for the scan and the monitor
- ReviewEventsMixin: the Needs review change stream an open page re-reads on
- SystemRoutesMixin: System health check, Overview and usage statistics
"""
import asyncio
@@ -19,7 +15,6 @@ import datetime
import functools
import gzip as gzip_module
import json
import math
import ntpath
import os
import re
@@ -33,9 +28,6 @@ from .pairs import PairRoutesMixin
from .library import LibraryRoutesMixin
from .vocabulary import VocabularyRoutesMixin
from .annotations import AnnotationRoutesMixin
from .review import ReviewRoutesMixin
from .review_recording import ReviewRecordingMixin
from .review_events import ReviewEventsMixin
from .saved_views import SavedViewRoutesMixin
from .history import HistoryRoutesMixin
from .images import ImageRoutesMixin
@@ -43,7 +35,6 @@ from .admin import AdminRoutesMixin
from .settings_routes import SettingsRoutesMixin
from .job_routes import JobRoutesMixin
from .image_tag_jobs import ImageTagJobsMixin
from .system import SystemRoutesMixin
from .jobs import JobRegistry
from .logging_routes import LoggingRoutesMixin
from .autotag_routes import AutotagRoutesMixin
@@ -270,9 +261,6 @@ class PromptManagerAPI(
LibraryRoutesMixin,
VocabularyRoutesMixin,
AnnotationRoutesMixin,
ReviewRoutesMixin,
ReviewRecordingMixin,
ReviewEventsMixin,
SavedViewRoutesMixin,
HistoryRoutesMixin,
PromptRoutesMixin,
@@ -281,7 +269,6 @@ class PromptManagerAPI(
SettingsRoutesMixin,
JobRoutesMixin,
ImageTagJobsMixin,
SystemRoutesMixin,
LoggingRoutesMixin,
AutotagRoutesMixin,
LoraIntegrationMixin,
@@ -313,8 +300,6 @@ class PromptManagerAPI(
# created on first use with their restore subscription.
self._captures = None
self._captures_unsubscribe = None
# Set by close_annotation_captures (shutdown): no store is recreated.
self._captures_closed = False
self._cached_output_dir = None # Lazy-cached by _find_comfyui_output_dir()
self._html_cache = {} # Cached HTML file contents keyed by path
self._gallery_cache = {} # dict: path_str -> (files, timestamp)
@@ -340,11 +325,6 @@ class PromptManagerAPI(
self._scan_job = None
# The current duplicate scan (a py/api/jobs.Job outside the registry).
self._duplicate_job = None
# The last System health report this session and the database
# generation it describes, bumped when a restore replaces self.db
# (py/api/system.py).
self._health_report = None
self._db_generation = 0
# Run cleanup on initialization to remove any existing duplicates
try:
@@ -495,8 +475,6 @@ class PromptManagerAPI(
self._register_library_routes(routes)
self._register_vocabulary_routes(routes)
self._register_annotation_routes(routes)
self._register_review_routes(routes)
self._register_review_event_routes(routes)
self._register_saved_view_routes(routes)
self._register_history_routes(routes)
self._register_prompt_routes(routes)
@@ -504,7 +482,6 @@ class PromptManagerAPI(
self._register_admin_routes(routes)
self._register_settings_routes(routes)
self._register_job_routes(routes)
self._register_system_routes(routes)
self._register_logging_routes(routes)
self._register_autotag_routes(routes)
self._register_lora_routes(routes)
@@ -623,12 +600,12 @@ class PromptManagerAPI(
return prompts
def _clean_nan_recursive(self, obj):
"""Recursively replace NaN and Infinity in nested data with None."""
"""Recursively clean NaN values from nested data structures."""
if isinstance(obj, dict):
return {key: self._clean_nan_recursive(value) for key, value in obj.items()}
elif isinstance(obj, list):
return [self._clean_nan_recursive(item) for item in obj]
elif isinstance(obj, float) and not math.isfinite(obj):
elif isinstance(obj, float) and str(obj) == "nan":
return None
else:
return obj
@@ -751,13 +728,6 @@ class PromptManagerAPI(
Returns:
List[Path]: Validated directory paths, possibly empty.
"""
return self._output_dirs_with_origin()[0]
def _output_dirs_with_origin(self):
"""``(dirs, auto_detected)``: the ``_get_all_output_dirs`` list, and
whether it comes from detection or ComfyUI's output rather than from
a valid configured root (False when there is no directory at all).
"""
from ..config import GalleryConfig
configured = list(GalleryConfig.MONITORING_DIRECTORIES)
@@ -776,16 +746,15 @@ class PromptManagerAPI(
output_dirs.append(Path(real))
if output_dirs:
return output_dirs, False
return output_dirs
if configured:
# Every configured root was rejected: never let auto-detection
# resurrect it, serve ComfyUI's own output directory instead.
fallback_dirs = self._comfyui_output_dir_list()
else:
fallback = self._find_comfyui_output_dir()
fallback_dirs = [Path(fallback)] if fallback else []
return fallback_dirs, bool(fallback_dirs)
return self._comfyui_output_dir_list()
fallback = self._find_comfyui_output_dir()
return [Path(fallback)] if fallback else []
def _comfyui_output_dir_list(self):
"""ComfyUI's output directory as a one-element list, or [] if unknown."""
+32 -132
View File
@@ -4,7 +4,6 @@ import asyncio
import contextlib
from contextlib import closing
import datetime
import functools
import hashlib
import json
import math
@@ -42,10 +41,9 @@ try:
PLACEHOLDERS,
GenerationParamsError,
extract_generation_params,
parse_page_json,
parse_js_json,
)
from ...utils.prompt_pair_resolver import resolve_prompt_pair
from ...utils.review_capture import reviewable
except ImportError:
from database.operations import PromptDatabase
from database.pair_ingestion import ingest_resolved_pair
@@ -53,10 +51,9 @@ except ImportError:
PLACEHOLDERS,
GenerationParamsError,
extract_generation_params,
parse_page_json,
parse_js_json,
)
from utils.prompt_pair_resolver import resolve_prompt_pair
from utils.review_capture import reviewable
try:
@@ -214,30 +211,6 @@ def _time_left(done, total, batches, elapsed):
return f" · about {minutes} min left"
def new_review_counts():
"""The scan's Needs review tally: ``added`` new items, ``queued`` files
in the queue after this scan (new or already there), ``settled`` items
removed because their file is linked now, ``untraced`` items removed
because they offer nothing to choose, and ``errors``."""
return {"added": 0, "queued": 0, "settled": 0, "untraced": 0, "errors": 0}
def _review_scope(review):
"""The scan job's ``scope`` keys for its Needs review tally.
``review_items`` is the number of this scan's files that are in the
pending queue after it: the review group's ambiguous files, less those a
capture could not read or a decision already settled.
"""
return {
"review_items": review["queued"],
"review_added": review["added"],
"review_settled": review["settled"],
"review_untraced": review["untraced"],
"review_errors": review["errors"],
}
def _scan_status_line(done, total, batches, elapsed):
""" "Processing file N of M", with an estimate once enough batches ran."""
line = f"Processing file {done:,} of {total:,}"
@@ -831,9 +804,6 @@ class AdminRoutesMixin:
result = await self._run_in_executor(
self._delete_duplicate_images_sync, image_paths, output_dir
)
if result["deleted_count"]:
# The cached listing would keep (and count) the deleted files.
self.invalidate_gallery_cache()
response_data = {
"success": True,
@@ -1301,7 +1271,6 @@ class AdminRoutesMixin:
self.logger.info(f"Removed {len(removed)} old safety backups")
previous = self.db
self.db = PromptDatabase(model.db_path)
self._database_replaced() # a health report described the old one
# Its connections would otherwise linger (a file lock on Windows)
previous.close_all()
prompt_count = self.db.model.get_database_info().get("total_prompts", 0)
@@ -1371,34 +1340,28 @@ class AdminRoutesMixin:
}
)
def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE, roots=None, output_dirs=None):
def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE):
"""Register the scan job synchronously; its runner starts on the loop.
``files`` is the scope (``SCAN_SCOPES``). ``roots`` selects sources by
their index in ``output_dirs`` (the effective sources the caller
resolved); None scans every source, resolved when the job starts.
Raises JobBusy while another record-changing job runs.
``files`` is the scope (``SCAN_SCOPES``). Raises JobBusy while another
record-changing job runs.
"""
if files not in SCAN_SCOPES:
raise ValueError(f"Unknown scan scope: {files!r}")
if roots is not None and (
output_dirs is None or not roots or max(roots) >= len(output_dirs)
):
raise ValueError("A scoped scan needs the sources its roots index")
job = self._jobs.start(
SCAN_JOB_KIND,
SCAN_JOB_TITLE,
record_changing=True,
scope={"files": files, "roots": None if roots is None else list(roots)},
runner=functools.partial(self._run_scan_job, output_dirs=output_dirs),
scope={"files": files},
runner=self._run_scan_job,
groups=SCAN_GROUP_NAMES,
)
self._scan_job = job
return job
async def _run_scan_job(self, job, output_dirs=None):
async def _run_scan_job(self, job):
final = None
async for payload in self._scan_images_events(job, output_dirs):
async for payload in self._scan_images_events(job):
job.publish(payload)
final = payload.get("type")
state = SCAN_FINAL_STATES.get(final, "finished")
@@ -1433,25 +1396,16 @@ class AdminRoutesMixin:
)
return response
async def _scan_images_events(self, job, output_dirs=None):
"""Yield progress payloads while scanning the output directories.
async def _scan_images_events(self, job):
"""Yield progress payloads while scanning every configured output directory.
Every effective source is walked, or only those ``job.scope["roots"]``
selects (indexes into ``output_dirs``). The payloads are the legacy SSE
events, unchanged; ``job`` also gets its status line, totals and one
outcome per processed file. Rows keep each source's listing index.
The payloads are the legacy SSE events, unchanged; ``job`` also gets
its status line, totals and one outcome per processed file.
"""
try:
self.logger.info("Starting image scan operation")
if output_dirs is None:
output_dirs = self._get_all_output_dirs()
selected = job.scope.get("roots")
walked = (
output_dirs
if selected is None
else [output_dirs[index] for index in selected]
)
if not walked:
output_dirs = self._get_all_output_dirs()
if not output_dirs:
self.logger.error("No output directories found")
message = (
"No output directories found. "
@@ -1461,18 +1415,18 @@ class AdminRoutesMixin:
yield {"type": "error", "message": message}
return
dir_names = [self._public_path(d) for d in walked]
folders = _plural(len(walked), "output folder")
job.scope["output_folders"] = len(walked)
dir_names = [self._public_path(d) for d in output_dirs]
folders = _plural(len(output_dirs), "output folder")
job.scope["output_folders"] = len(output_dirs)
scope = SCAN_SCOPE_LOG[job.scope.get("files", "all")]
job.log_line(f"Scan started · {folders} · {scope}")
job.update(status_line=f"Looking for media files in {folders}")
yield self._scan_progress(
0, f"Scanning {len(walked)} directory(ies) for media files..."
0, f"Scanning {len(output_dirs)} directory(ies) for media files..."
)
media_files = await self._run_in_executor(
self._collect_output_media_sync, walked
self._collect_output_media_sync, output_dirs
)
counts = dict.fromkeys(SCAN_COUNT_KEYS, 0)
job.update(
@@ -1487,13 +1441,10 @@ class AdminRoutesMixin:
known = set()
if media_files and job.scope.get("files") != "all":
known = await self._known_scan_files(job, media_files)
review = new_review_counts()
job.scope.update(_review_scope(review))
async for event in self._scan_batches(
media_files, counts, job, roots, known, review
media_files, counts, job, roots, known
):
yield event
self._log_review_items(job, review)
self.logger.info(
f"Scan completed: processed={counts['processed']}, "
@@ -1530,16 +1481,12 @@ class AdminRoutesMixin:
sizes = {str(media_file): _file_size(media_file) for media_file in media_files}
return self.db.get_known_resolved_paths(sizes)
async def _scan_batches(
self, media_files, counts, job, roots=(), known=(), review=None
):
async def _scan_batches(self, media_files, counts, job, roots=(), known=()):
"""Process ``media_files`` in executor batches, updating ``counts`` in place.
Records each file's outcome in ``job`` and yields one progress event
per batch. Files in ``known`` (path strings) are not read: they count
as processed and unchanged and are recorded as skipped. ``review``
counts the Needs review items the batches record (``job.scope``
``review_items`` / ``review_errors`` follow it after every batch).
as processed and unchanged and are recorded as skipped.
"""
total = len(media_files)
started = time.monotonic()
@@ -1562,16 +1509,9 @@ class AdminRoutesMixin:
batch_results,
roots=roots,
outcomes=outcomes,
review=review,
)
for key in counts:
counts[key] += batch_counts[key]
if review is not None:
# After the batch's links: a file linked now leaves the queue.
await self._run_in_executor(
self._settle_scanned_reviews, batch, roots, review
)
job.scope.update(_review_scope(review))
for group, row, log_line in outcomes:
job.record(group, row, log_line)
@@ -1621,20 +1561,12 @@ class AdminRoutesMixin:
return map_parallel(extract, file_batch, PromptManagerConfig.WORKER_THREADS)
def _ingest_scan_batch_sync(
self, batch_results, roots=(), outcomes=None, review=None
):
def _ingest_scan_batch_sync(self, batch_results, roots=(), outcomes=None):
"""Store the prompts found in a metadata batch (blocking DB work).
Returns the legacy counts. When ``outcomes`` is a list, every processed
file appends its ``(group, row, log_line)`` for the scan job, with the
path relative to its root in ``roots`` (see ``_scan_roots``). A file
whose final outcome is ambiguous and unlinked (the resolver found it
ambiguous and the fallback did not link it) is recorded for Needs
review (best effort, counted in ``review`` when given); one a saved
row already links is reported ``unchanged`` and never queued. An
ambiguous file with nothing to choose is reported ``unresolved`` and
its pending items without a choice leave the queue.
path relative to its root in ``roots`` (see ``_scan_roots``).
"""
counts = dict.fromkeys(SCAN_COUNT_KEYS, 0)
@@ -1646,27 +1578,13 @@ class AdminRoutesMixin:
if self._scan_job and self._scan_job.cancelled:
break
counts["processed"] += 1
ambiguous, untraced = [], []
try:
outcome = self._ingest_scanned_file(
media_file,
metadata,
detailed=True,
ambiguous=ambiguous,
untraced=untraced,
)
outcome = self._ingest_scanned_file(media_file, metadata, detailed=True)
except Exception as e:
self.logger.error(f"Error processing {media_file.name}: {e}")
counts["failed"] += 1
note(media_file, "error")
continue
if ambiguous and outcome == "ambiguous":
if self._review_file_linked(media_file, roots):
outcome = "unchanged" # already linked: not a review case
else:
self._record_scanned_review(media_file, roots, review)
elif untraced and outcome == "unresolved":
self._settle_untraced_reviews(untraced, roots, review)
if outcome is None:
note(media_file, None)
continue
@@ -1684,14 +1602,11 @@ class AdminRoutesMixin:
@staticmethod
def _graph_prompt_text(metadata):
"""The positive prompt the browser parser traces from the graphs, or None.
Records parse as the legacy pages parsed them (NaN reads as null).
"""
"""The positive prompt the browser parser traces from the graphs, or None."""
records = {}
for key in ("prompt", "workflow"):
value = metadata.get(key)
records[key] = parse_page_json(value) if isinstance(value, str) else value
records[key] = parse_js_json(value) if isinstance(value, str) else value
try:
positive = extract_generation_params(records)["positivePrompt"]
except GenerationParamsError:
@@ -1700,21 +1615,13 @@ class AdminRoutesMixin:
return None
return positive.strip() or None
def _ingest_scanned_file(
self, media_file, metadata, detailed=False, ambiguous=None, untraced=None
):
def _ingest_scanned_file(self, media_file, metadata, detailed=False):
"""Enrich exact pairs; retain legacy positive-only import for absent pairs.
Returns the file's scan status, or None without prompt metadata. With
``detailed``, a stored or linked file returns a dict instead: its
``status``, the prompt's display ``title`` and, for a resolved pair,
the pair counts. When the resolver finds the file ambiguous (whether
or not the browser parser then links it) and ``ambiguous`` is a list,
``media_file`` is appended to it; only a file that then returns
``"ambiguous"`` (nothing linked it) is a Needs review case. An
ambiguous result with nothing to choose (``reviewable`` is False: fewer
than two candidates carry a prompt) that nothing links returns
``"unresolved"`` instead, and ``media_file`` goes to ``untraced``.
the pair counts.
"""
if not metadata:
return None
@@ -1730,13 +1637,6 @@ class AdminRoutesMixin:
return None
_with_file_size(canonical, media_file)
resolution = resolve_prompt_pair(canonical)
# Ambiguous without a choice: unresolved unless something links it.
unchoosable = resolution["state"] == "ambiguous" and not reviewable(resolution)
if unchoosable:
if untraced is not None:
untraced.append(media_file)
elif resolution["state"] == "ambiguous" and ambiguous is not None:
ambiguous.append(media_file)
if resolution["state"] == "resolved":
result = ingest_resolved_pair(
self.db, str(media_file), canonical, resolution
@@ -1756,9 +1656,9 @@ class AdminRoutesMixin:
# Graphs without a recognized sampler keep the legacy extractor below.
prompt_text = self._graph_prompt_text(metadata)
if not prompt_text:
return "unresolved" if unchoosable else resolution["state"]
return resolution["state"]
elif resolution["state"] == "ambiguous":
return "unresolved" if unchoosable else "ambiguous"
return "ambiguous"
elif resolution["state"] == "unresolved" and not isinstance(
canonical.get("parameters"), str
):
+13 -35
View File
@@ -13,7 +13,6 @@ import datetime
import functools
import os
import re
import threading
from pathlib import Path
from aiohttp import web
@@ -74,9 +73,6 @@ _HEX64 = re.compile(r"[0-9a-f]{64}")
_UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
# secrets.token_urlsafe(32) is 43 characters; the bound only rejects junk.
_TOKEN = re.compile(r"[A-Za-z0-9_-]{1,128}")
# The store is created on first use by the event loop, the scan's executor or
# the monitor's worker (Needs review), so its creation is serialized.
_CAPTURE_STORE_LOCK = threading.Lock()
_PREVIEW_TYPES = {
".png": "image/png",
@@ -263,46 +259,31 @@ class AnnotationRoutesMixin:
Created lazily so an API object that never observes a file (most tests,
a server that never opens the Tag dialog) owns no temp directory. A
restore of the database dooms every capture (``invalidate_all``).
Thread-safe: Needs review recording calls it off the event loop.
Raises ``CaptureError`` after ``close_annotation_captures``.
"""
with _CAPTURE_STORE_LOCK:
if getattr(self, "_captures_closed", False):
# A recording still in flight at shutdown must not create a
# new temp directory after the cleanup removed the old one.
raise CaptureError("The capture store is closed")
if self._captures is None:
self._captures = CaptureStore()
if self._captures_unsubscribe is None:
self._captures_unsubscribe = subscribe_history_restore(
self.db.model.db_path, self._invalidate_captures
)
return self._captures
if self._captures is None:
self._captures = CaptureStore()
if self._captures_unsubscribe is None:
self._captures_unsubscribe = subscribe_history_restore(
self.db.model.db_path, self._invalidate_captures
)
return self._captures
def _invalidate_captures(self) -> None:
store = self._captures
if store is not None:
store.invalidate_all()
def close_annotation_captures(self, *, final: bool = False) -> None:
"""Remove every capture file and the owned directory. Idempotent.
Taken under the store lock. ``final`` (the shutdown cleanup) also makes
``_capture_store`` refuse to create another store, so a recording
still in flight cannot leak a new temp directory after it.
"""
with _CAPTURE_STORE_LOCK:
if final:
self._captures_closed = True
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
store, self._captures = self._captures, None
def close_annotation_captures(self) -> None:
"""Remove every capture file and the owned directory. Idempotent."""
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
if unsubscribe is not None:
unsubscribe()
store, self._captures = self._captures, None
if store is not None:
store.close()
async def cleanup_annotation_captures(self, app: web.Application) -> None:
self.close_annotation_captures(final=True)
self.close_annotation_captures()
# -- shared request steps --------------------------------------------------
@@ -348,10 +329,7 @@ class AnnotationRoutesMixin:
root, error = self._scoped_root(index, scope)
if error is not None:
return error
try:
store = self._capture_store()
except CaptureError as exc: # the server is shutting down
return _capture_error(exc)
store = self._capture_store()
try:
epoch = await self._run_in_executor(self.db.get_execution_epoch)
capture = await _settled(
+29 -30
View File
@@ -3,7 +3,6 @@
import asyncio
import base64
import binascii
import functools
import hashlib
import hmac
import json
@@ -38,7 +37,6 @@ try:
from ...database.models import normalize_image_path
from ...utils.card_seed import card_seed, header_params, parsed_positive
from ...utils.parallel import map_parallel
from ...utils.json_strict import dumps_strict
from ...utils.inspection_response import (
encode_inspection_response,
library_metadata,
@@ -58,7 +56,6 @@ except ImportError:
from database.models import normalize_image_path
from utils.card_seed import card_seed, header_params, parsed_positive
from utils.parallel import map_parallel
from utils.json_strict import dumps_strict
from utils.inspection_response import (
encode_inspection_response,
library_metadata,
@@ -671,25 +668,6 @@ def _iter_media_files(output_path, max_files):
yield root_path / name
def media_thumbnail_url(output_path, rel_path, root_index, *, root_scope=None):
"""The Images listing's thumbnail URL for *rel_path* under *output_path*,
or None when no thumbnail exists (one blocking stat). The Needs review
queue cards use the same URL (#249 F, M10)."""
rel_path = Path(rel_path)
extension = rel_path.suffix.lower()
thumb_rel = (
f"thumbnails/{rel_path.with_suffix('').as_posix()}_thumb"
f"{'.jpg' if extension in VIDEO_EXTENSIONS else extension}"
)
thumbnail_url = thumbnail_url_for(thumb_rel, Path(output_path) / thumb_rel)
if thumbnail_url is None:
return None
thumbnail_url += f"&root={root_index}"
if root_scope is not None:
thumbnail_url += f"&root_scope={root_scope}"
return thumbnail_url
def _output_image_entry(
media_path, output_path, root_index, public_root=None, *, root_scope=None
):
@@ -705,13 +683,19 @@ def _output_image_entry(
extension = media_path.suffix.lower()
is_video = extension in VIDEO_EXTENSIONS
thumbnail_url = media_thumbnail_url(
output_path, rel_path, root_index, root_scope=root_scope
thumb_rel = (
f"thumbnails/{rel_path.with_suffix('').as_posix()}_thumb"
f"{'.jpg' if is_video else extension}"
)
thumbnail_url = thumbnail_url_for(thumb_rel, output_path / thumb_rel)
if thumbnail_url is not None:
thumbnail_url += f"&root={root_index}"
encoded_path = urllib.parse.quote(rel_path.as_posix(), safe="/")
url = f"/prompt_manager/images/serve/{encoded_path}?root={root_index}"
if root_scope is not None:
url += f"&root_scope={root_scope}"
if thumbnail_url is not None:
thumbnail_url += f"&root_scope={root_scope}"
entry = {
"id": hashlib.sha1(
@@ -998,12 +982,27 @@ class ImageRoutesMixin:
# Add url and thumbnail_url so the frontend can serve them
cleaned_images = self._present_images(cleaned_images)
# Rows an older version stored may still hold NaN/Infinity: answer
# strict JSON, and leave prompt text that spells "NaN" untouched.
return web.json_response(
{"success": True, "images": cleaned_images},
dumps=functools.partial(dumps_strict, default=str),
)
# Additional fallback: convert to JSON string and clean NaN values manually
try:
response_data = {"success": True, "images": cleaned_images}
# Convert to JSON string
json_str = json.dumps(response_data, default=str)
# Clean any remaining NaN values with regex
json_str = re.sub(r":\s*NaN", ": null", json_str)
json_str = re.sub(r"\[\s*NaN\s*\]", "[null]", json_str)
json_str = re.sub(r",\s*NaN\s*,", ", null,", json_str)
json_str = re.sub(r",\s*NaN\s*\]", ", null]", json_str)
json_str = re.sub(r"\[\s*NaN\s*,", "[null,", json_str)
# Parse back to verify it's valid JSON
cleaned_data = json.loads(json_str)
return web.json_response(cleaned_data)
except Exception as json_error:
self.logger.error(f"JSON cleaning error: {json_error}")
# Fallback to original response
return web.json_response({"success": True, "images": cleaned_images})
except Exception as e:
self.logger.error(f"Get prompt images error: {e}")
+17 -93
View File
@@ -18,74 +18,29 @@ def _not_found():
return web.json_response({"success": False, "error": "Job not found"}, status=404)
MAX_SCAN_ROOTS = 64
_SCOPE_ERROR = 'The scan scope must be "new" or "all"'
_ROOTS_ERROR = (
f"roots must be a non-empty list of up to {MAX_SCAN_ROOTS} source indexes, "
"with paths naming the public path of each"
)
SOURCES_CHANGED_ERROR = "The sources changed since they were loaded. Reload them."
def _bad_request(message):
return web.json_response({"success": False, "error": message}, status=400)
def _bad_request(message, code=None):
body = {"success": False, "error": message}
if code is not None:
body["code"] = code
return web.json_response(body, status=400)
async def _scan_scope(request):
"""The requested scan scope ("new" or "all"), or None when the body is invalid.
def _scan_roots(body):
"""``(roots, error)``: ``{index: public path}`` of the selected sources, or None.
Absent ``roots`` means every source. When present it is a non-empty list
of at most MAX_SCAN_ROOTS non-negative ints (never bools or floats), and
``paths`` the same-length list of the public path the caller read for
each (GET /settings), so a stale index never scans another folder.
An empty body (also a chunked one with no bytes), or one without
``scope``, asks for the default.
"""
if "roots" not in body:
return None, None
raw = body["roots"]
paths = body.get("paths")
if not isinstance(raw, list) or not raw or len(raw) > MAX_SCAN_ROOTS:
return None, _ROOTS_ERROR
if not isinstance(paths, list) or len(paths) != len(raw):
return None, _ROOTS_ERROR
selected = {}
for value, path in zip(raw, paths):
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
return None, _ROOTS_ERROR
if not isinstance(path, str) or selected.get(value, path) != path:
return None, _ROOTS_ERROR
selected[value] = path
return dict(sorted(selected.items())), None
async def _scan_request(request):
"""``({files, roots}, None)`` for a scan start, or ``(None, message)``.
``files`` is the scope ("new" or "all"); ``roots`` the selected source
indexes or None for every source. An empty body (also a chunked one with
no bytes), or one without ``scope``, asks for the default scope.
"""
default = {"files": DEFAULT_SCAN_SCOPE, "roots": None}
if not request.can_read_body:
return default, None
return DEFAULT_SCAN_SCOPE
text = await request.text()
if not text.strip(): # e.g. a chunked POST that sent no bytes
return default, None
return DEFAULT_SCAN_SCOPE
try:
body = json.loads(text)
except ValueError:
return None, _SCOPE_ERROR
return None
if not isinstance(body, dict):
return None, _SCOPE_ERROR
files = body.get("scope", DEFAULT_SCAN_SCOPE)
if files not in SCAN_SCOPES:
return None, _SCOPE_ERROR
roots, error = _scan_roots(body)
if error:
return None, error
return {"files": files, "roots": roots}, None
return None
scope = body.get("scope", DEFAULT_SCAN_SCOPE)
return scope if scope in SCAN_SCOPES else None
def busy_response(busy: JobBusy):
@@ -109,13 +64,6 @@ def _summary_event(job):
class JobRoutesMixin:
"""Mixin providing the /prompt_manager/jobs endpoints."""
def _roots_still_listed(self, selected, output_dirs):
"""Whether each ``{index: public path}`` still names that source."""
return all(
index < len(output_dirs) and self._public_path(output_dirs[index]) == path
for index, path in selected.items()
)
def _register_job_routes(self, routes):
@routes.get("/prompt_manager/jobs")
async def list_jobs_route(request):
@@ -184,37 +132,13 @@ class JobRoutesMixin:
"""Start the metadata scan as a job (202), or 409 naming the busy job.
The body's ``scope`` is "new" (the default: new and unresolved files)
or "all". ``roots``, when given, selects sources by their index in
the effective sources (``_get_all_output_dirs``), and ``paths`` names
the public path read for each. An index past them, or one whose
source now has another public path (the list changed since it was
read), is a 409 ``sources-changed`` and starts nothing. The job walks
the sources resolved here, so a later source change cannot move it.
or "all"; anything else is a 400.
"""
scan, error = await _scan_request(request)
if error:
return _bad_request(error)
output_dirs = None
roots = None
if scan["roots"] is not None:
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
listed = await self._run_in_executor(
self._roots_still_listed, scan["roots"], output_dirs
)
if not listed:
return web.json_response(
{
"success": False,
"code": "sources-changed",
"error": SOURCES_CHANGED_ERROR,
},
status=409,
)
roots = list(scan["roots"])
files = await _scan_scope(request)
if files is None:
return _bad_request('The scan scope must be "new" or "all"')
try:
job = self._start_scan_job(
scan["files"], roots=roots, output_dirs=output_dirs
)
job = self._start_scan_job(files)
except JobBusy as busy:
return busy_response(busy)
return web.json_response({"success": True, "job": job.summary()}, status=202)
+3 -7
View File
@@ -88,9 +88,6 @@ DUPLICATE_CLEANUP_JOB_KIND = "duplicate_cleanup"
THUMBNAILS_JOB_KIND = "thumbnails"
LORA_IMPORT_JOB_KIND = "lora_import"
AUTOTAG_BATCH_JOB_KIND = "autotag"
# The System health check (py/api/system.py): a read-only ``read`` job with
# no file outcomes, so it may run beside a scan.
HEALTH_JOB_KIND = "health"
def resolve_admission(admission=None, record_changing=None) -> str:
@@ -435,8 +432,7 @@ class JobRegistry:
if same_kind is not None:
return same_kind
refused = _REFUSED_BESIDE[resolve_admission(admission, record_changing)]
# A snapshot: the monitor's worker thread asks too (Needs review).
running = [job for job in tuple(self._jobs.values()) if job.running]
running = [job for job in self._jobs.values() if job.running]
running.sort(key=lambda job: job._started_seq)
for job in running:
if job.admission in refused:
@@ -467,8 +463,8 @@ class JobRegistry:
return self._jobs.get(job_id)
def active(self, kind):
"""The running job of ``kind``, or None (safe off the event loop)."""
for job in tuple(self._jobs.values()):
"""The running job of ``kind``, or None."""
for job in self._jobs.values():
if job.kind == kind and job.running:
return job
return None
-706
View File
@@ -1,706 +0,0 @@
"""Needs review routes (#249 E3): the pending queue, item detail, decisions.
``GET /prompt_manager/review/items`` lists pending items oldest first behind
an HMAC keyset cursor bound to the storage epoch only (a restore makes an
old cursor stale; a decision or a new item does not, because the
``(created_at, id)`` keyset stays ordered when items leave or are appended);
``total`` is the exact pending count. ``GET .../items/{id}`` adds the
candidates and the dependency fingerprint. ``POST .../items/{id}/decision``
records one explicit decision: Apply links the chosen sampler's pair only
when the capture is the item's bytes, the current file still hashes to them,
the evidence they yield is unchanged and the ingester's compatible-only rules
allow it; Leave unresolved links nothing.
Targets are formatted from the recorded spelling against the current roots
(no file access beyond the roots' real paths); a file under no current root
is ``unavailable``. Responses carry no server path, no locator and no
exception text.
"""
import base64
import binascii
import hashlib
import hmac
import json
import os
import re
import urllib.parse
from aiohttp import web
from .annotations import _HEX64, _TOKEN, _UUID, _capture_error, _error
from .images import media_thumbnail_url
from .inspection_service import VIDEO_EXTENSIONS
from .prompts import bad_request
try:
from ...database.library import display_title
from ...database.library_commands import LibraryCommandError
from ...database.review_decisions import ReviewCommandError
from ...utils.media_capture import CaptureError
from ...utils.review_capture import captured_review, review_evidence
except ImportError:
from database.library import display_title
from database.library_commands import LibraryCommandError
from database.review_decisions import ReviewCommandError
from utils.media_capture import CaptureError
from utils.review_capture import captured_review, review_evidence
ITEMS_URL = "/prompt_manager/review/items"
DEFAULT_LIMIT = 40
MAX_LIMIT = 40
MAX_CURSOR = 4096
MAX_BODY_BYTES = 64 * 1024
# The serialized detail stays under this; past it the detail is ``limited``.
MAX_DETAIL_BYTES = 8 * 1024 * 1024
UNRESOLVED = "unresolved"
_DECISION_FIELDS = frozenset(
{
"expected_epoch",
"expected_revision",
"capture_token",
"sha256",
"dependency_fingerprint",
"choice",
}
)
_CURSOR_SECRET = os.urandom(32)
# The real path of a spelling; module-level so a test can fake a junction.
_realpath = os.path.realpath
_LIMIT = re.compile(r"[0-9]{1,3}")
_NO_STORE = {"Cache-Control": "no-store"}
_STATUS = {
"review-missing": 404,
"review-conflict": 409,
"association-conflict": 409,
"file-changed": 409,
"storage-changed": 409,
"source-changed": 409,
"capture-expired": 409,
}
_MESSAGES = {
"review-missing": "This review item no longer exists. Reload the queue.",
"review-conflict": (
"This item was decided or changed since it was opened. Reload the queue."
),
"association-conflict": (
"This image is already linked to another prompt or negative, so this "
"sampler path cannot be attached safely. The existing links were "
"preserved."
),
"dependencies-changed": (
"The image's saved links changed since this item was opened. Reopen it "
"to see the current association."
),
"file-changed": (
"The file changed since it was reviewed. A metadata scan adds its new "
"bytes as a new item."
),
"storage-changed": "Prompt storage was restored; reload the review queue.",
"source-changed": (
"The source changed. Refresh the image collection before reviewing this "
"file."
),
"capture-expired": "The preview expired. Open the item again.",
"other-file": "The preview belongs to another file. Open the item again.",
"evidence-changed": (
"This file's metadata no longer gives the evidence that was reviewed. "
"Reload the queue."
),
}
_DEFAULT_MESSAGE = "The decision could not be saved. Reload the queue."
def _coded(code: str, message_key: str | None = None, **extra) -> web.Response:
message = _MESSAGES.get(message_key or code, _DEFAULT_MESSAGE)
return _error(_STATUS.get(code, 409), code, message, **extra)
def _failed() -> web.Response:
return _error(500, "review-failed", "The review queue could not be read.")
def format_review_target(spelled_path, locator, real_roots, *, pathmod=os.path):
"""``(root_index, relative)`` of the first root that lists the file, or None.
``spelled_path`` is the file's recorded spelling, ``locator`` its folded
identity and ``real_roots`` ``(index, realpath)`` pairs. The relative
path keeps the recorded spelling (a Windows locator is case-folded, a
listing target is not) and must fold back to the locator.
"""
sep = pathmod.sep
for index, real_root in real_roots:
head = spelled_path[: len(real_root)]
if pathmod.normcase(head) != pathmod.normcase(real_root):
continue
rest = spelled_path[len(real_root) :]
if not real_root.endswith(sep):
if not rest.startswith(sep):
continue
rest = rest[1:]
parts = rest.split(sep)
if any(
part in ("", ".", "..") or "/" in part or "\\" in part or ":" in part
for part in parts
):
continue
if pathmod.normcase(pathmod.join(real_root, *parts)) != locator:
continue
return index, "/".join(parts)
return None
class _StaleCursor(ValueError):
"""A well-formed cursor this process cannot verify (e.g. after a restart)."""
def _encode_cursor(epoch: str, key: tuple) -> str:
payload = json.dumps(
{"v": 2, "e": epoch, "k": list(key)},
separators=(",", ":"),
ensure_ascii=True,
).encode()
signature = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
return base64.urlsafe_b64encode(signature + payload).decode().rstrip("=")
def _decode_cursor(token: str) -> tuple:
"""``(epoch, (created_at, id))`` of a cursor.
``ValueError`` when malformed; :class:`_StaleCursor` when well-formed but
signed by another secret (the per-process secret changes on every start,
so an open page's cursor is stale after a restart, not forged).
"""
try:
if len(token) > MAX_CURSOR:
raise ValueError()
raw = base64.b64decode(
token + "=" * (-len(token) % 4), altchars=b"-_", validate=True
)
signature, payload = raw[:32], raw[32:]
if len(signature) != 32 or not payload:
raise ValueError()
expected = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
if not hmac.compare_digest(signature, expected):
raise _StaleCursor("This page of the queue is out of date")
data = json.loads(payload)
key = data["k"]
if (
data["v"] != 2
or not isinstance(data["e"], str)
or not isinstance(key, list)
or len(key) != 2
or not all(isinstance(part, str) for part in key)
):
raise ValueError()
return data["e"], tuple(key)
except _StaleCursor:
raise
except (ValueError, KeyError, TypeError, UnicodeError, binascii.Error):
raise ValueError("Invalid review cursor; reload the queue") from None
def _limit(value):
if value is None:
return DEFAULT_LIMIT
if _LIMIT.fullmatch(value) is None or not 1 <= int(value) <= MAX_LIMIT:
return None
return int(value)
def _side_text(side) -> str | None:
if isinstance(side, dict) and isinstance(side.get("text"), str):
return side["text"]
return None
def _workable(candidate: dict) -> bool:
return (
candidate.get("state") == "resolved"
and _side_text(candidate.get("positive")) is not None
and _side_text(candidate.get("negative")) is not None
)
def _side_label(titles: dict, side, empty: str) -> str:
"""The Library title of the side's saved prompt, else its first line."""
text = _side_text(side)
if text is None:
return empty
stored = side.get("storage_text")
stored = stored if isinstance(stored, str) else text
return display_title(titles.get(stored, ()), text) or empty
class ReviewRoutesMixin:
"""The Needs review queue over E1's schema and E2's recorded items."""
def _register_review_routes(self, routes: web.RouteTableDef) -> None:
routes.get(ITEMS_URL)(self.list_review_items)
routes.get(ITEMS_URL + "/{item_id}")(self.get_review_item)
routes.post(ITEMS_URL + "/{item_id}/decision")(self.decide_review_item)
# -- shared ----------------------------------------------------------------
def _review_roots(self):
"""``(roots, real_roots)``: the current gallery roots and their realpaths.
Blocking (it lists the roots and resolves them): call it on a worker.
"""
roots = list(self._get_all_output_dirs())
return roots, [(i, _realpath(str(root))) for i, root in enumerate(roots)]
def _review_view(self, row: dict, roots, real_roots) -> dict:
"""The public ReviewItem of one stored row (no path, no locator)."""
found = format_review_target(row["spelled_path"], row["locator"], real_roots)
target = url = thumbnail_url = None
if found is not None:
index, relative = found
target = {
"root_index": index,
"root_scope": self._gallery_root_scope(roots[index], index),
"path": relative,
}
# The Images listing's media URL for the same file (formatting only).
url = (
"/prompt_manager/images/serve/"
f"{urllib.parse.quote(relative, safe='/')}?root={index}"
f"&root_scope={target['root_scope']}"
)
# The Images listing's thumbnail, when one exists (one stat).
thumbnail_url = media_thumbnail_url(
roots[index], relative, index, root_scope=target["root_scope"]
)
filename = os.path.basename(row["spelled_path"])
return {
"id": row["id"],
"state": row["state"],
"revision": row["revision"],
"created_at": row["created_at"],
"filename": filename,
"target": target,
"unavailable": target is None,
"candidate_count": row["candidate_count"],
"limited": row["limited"],
"url": url,
"thumbnail_url": thumbnail_url,
"is_video": os.path.splitext(filename)[1].lower() in VIDEO_EXTENSIONS,
}
@staticmethod
def _review_image_path(row: dict, roots, target) -> str:
"""The path the scan and the monitor link this file under."""
if target is None:
return row["spelled_path"]
root = roots[target["root_index"]]
return os.path.join(str(root), *target["path"].split("/"))
# -- queue -----------------------------------------------------------------
async def list_review_items(self, request: web.Request) -> web.Response:
"""GET ``?limit=1..40&cursor=``: one page of pending items."""
limit = _limit(request.query.get("limit"))
if limit is None:
return bad_request(f"limit must be an integer from 1 to {MAX_LIMIT}")
bound = None
cursor = request.query.get("cursor")
if cursor:
try:
bound = _decode_cursor(cursor)
except _StaleCursor:
return await self._stale_cursor()
except ValueError as exc:
return bad_request(str(exc))
try:
page, views = await self._run_in_executor(
self._review_page, limit, bound[1] if bound else None
)
except Exception:
self.logger.exception("Review queue read failed")
return _failed()
if bound is not None and bound[0] != page["epoch"]:
return _coded("storage-changed", library_epoch=page["epoch"])
items = page["items"]
next_cursor = None
if page["more"] and items:
last = items[-1]
next_cursor = _encode_cursor(
page["epoch"], (last["created_at"], last["id"])
)
return web.json_response(
{
"success": True,
"library_epoch": page["epoch"],
# F3: the queue version the change stream's ``ready`` names.
"review_revision": page["revision"],
"total": page["total"],
"items": views,
"next_cursor": next_cursor,
},
headers=_NO_STORE,
)
async def _stale_cursor(self) -> web.Response:
"""409 ``review-conflict`` with the current total: reload the queue."""
try:
page = await self._run_in_executor(self.db.review_queue_page, limit=1)
except Exception:
self.logger.exception("Review queue read failed")
return _failed()
return _coded(
"review-conflict", total=page["total"], library_epoch=page["epoch"]
)
def _review_page(self, limit: int, after):
"""Worker thread: one queue page and the public views of its items."""
page = self.db.review_queue_page(limit=limit, after=after)
roots, real_roots = self._review_roots()
return page, [
self._review_view(row, roots, real_roots) for row in page["items"]
]
# -- detail ----------------------------------------------------------------
async def get_review_item(self, request: web.Request) -> web.Response:
"""GET one item with its candidates and dependency fingerprint."""
try:
read = await self._run_in_executor(
self._review_detail, request.match_info["item_id"]
)
except Exception:
self.logger.exception("Review item read failed")
return _failed()
if read is None:
return _coded("review-missing")
item, view, fingerprint, candidates = read
detail = {
"success": True,
"library_epoch": item["epoch"],
**view,
"candidates": candidates,
"dependency_fingerprint": fingerprint,
"sha256": item["sha256"],
"total": item["total"],
}
if len(json.dumps(detail).encode("utf-8")) > MAX_DETAIL_BYTES:
detail = {**detail, "candidates": [], "limited": True}
return web.json_response(detail, headers=_NO_STORE)
def _review_detail(self, item_id: str):
"""Worker thread: ``(item, view, fingerprint, candidates)`` or None."""
item = self.db.review_item(item_id)
if item is None:
return None
roots, real_roots = self._review_roots()
view = self._review_view(item, roots, real_roots)
image_path = self._review_image_path(item, roots, view["target"])
aliases, _known = physical_spellings(self.db, image_path, item["locator"])
fingerprint = self.db.review_dependency_fingerprint(image_path, aliases)
return item, view, fingerprint, self._review_candidates(item)
def _review_candidates(self, item: dict) -> list:
"""The detail's candidates: role texts, a label and selectability."""
if item["limited"]:
return []
candidates = [
c for c in item["evidence"].get("candidates", []) if isinstance(c, dict)
]
texts = set()
for candidate in candidates:
for role in ("positive", "negative"):
side = candidate.get(role)
if isinstance(side, dict):
for key in ("text", "storage_text"):
if isinstance(side.get(key), str):
texts.add(side[key])
titles = self.db.stored_prompt_titles(texts)
rows = []
for candidate in candidates:
positive = _side_text(candidate.get("positive"))
negative = _side_text(candidate.get("negative"))
label = (
_side_label(titles, candidate.get("positive"), "No positive")
+ " / "
+ _side_label(titles, candidate.get("negative"), "No negative")
)
rows.append(
{
"hash": candidate.get("hash"),
"sampler_id": candidate.get("sampler_id"),
"state": "resolved" if _workable(candidate) else "unresolved",
"reason": candidate.get("reason") or "",
"label": label,
"positive": {"text": positive} if positive is not None else None,
"negative": {"text": negative} if negative is not None else None,
}
)
return rows
# -- decision --------------------------------------------------------------
async def decide_review_item(self, request: web.Request) -> web.Response:
"""POST one explicit decision on one pending item."""
data, error = await self._read_json_body(request, max_bytes=MAX_BODY_BYTES)
if error is not None:
return error
guards, error = _decision_guards(data)
if error is not None:
return error
try:
item = await self._run_in_executor(
self.db.review_item, request.match_info["item_id"]
)
except Exception:
self.logger.exception("Review item read failed")
return _failed()
if item is None:
return _coded("review-missing")
choice = guards["choice"]
candidate = None
if choice != UNRESOLVED:
candidate = next(
(
c
for c in item["evidence"].get("candidates", [])
if isinstance(c, dict) and c.get("hash") == choice
),
None,
)
if candidate is None or item["limited"]:
return bad_request("choice is not one of this item's candidates")
if not _workable(candidate):
return bad_request("This sampler path cannot be applied")
if guards["capture_token"] is None:
return bad_request("Apply needs a capture of the file")
verified = None
if guards["capture_token"] is not None:
verified, error = await self._verified_review_capture(
item, guards, need_metadata=candidate is not None
)
if error is not None:
return error
elif guards["sha256"] != item["sha256"]:
return bad_request("sha256 does not match the item")
return await self._record_decision(item, guards, candidate, verified)
async def _verified_review_capture(self, item, guards, *, need_metadata):
"""``((capture, facts, image_path, metadata), None)`` or ``(None, error)``.
The capture must be the item's file and bytes, under the expected
epoch and a current root; the current file must re-hash to the same
bytes; for Apply, the capture's metadata must still yield exactly the
reviewed evidence.
"""
store = self._captures
capture = store.get(guards["capture_token"]) if store is not None else None
if capture is None:
return None, _coded("capture-expired")
if capture.locator != item["locator"]:
return None, _coded("capture-expired", "other-file")
if guards["sha256"] != capture.sha256:
return None, bad_request("sha256 does not match the preview")
if capture.sha256 != item["sha256"]:
return None, _coded("file-changed")
if guards["expected_epoch"] != capture.epoch:
current = await self._run_in_executor(self.db.get_execution_epoch)
return None, _coded("storage-changed", library_epoch=current)
try:
root, error = await self._run_in_executor(
self._scoped_root, capture.root_index, capture.scope
)
if error is not None:
return None, error
read = await self._run_in_executor(
_verify_and_read, store, capture, item, root, need_metadata
)
except CaptureError as exc:
return None, _capture_error(exc)
if read is None:
return None, _coded("source-changed")
relative, facts, captured = read
if facts.locator != item["locator"]:
return None, _coded("capture-expired", "other-file")
if facts.sha256 != item["sha256"]:
return None, _coded("file-changed")
metadata = None
if need_metadata:
if captured is None or not _candidate_still_offered(
captured[1], guards["choice"]
):
return None, _coded("review-conflict", "evidence-changed")
metadata = captured[0]
image_path = os.path.join(str(root), *relative.split("/"))
return (capture, facts, image_path, metadata), None
def _decide_sync(self, item, guards, candidate, verified):
"""Worker thread: resolve the file's spellings, decide, build the view.
Returns ``(result, view)``; a refusal raises with ``exc.view`` set.
"""
_capture, facts, image_path, metadata = verified or (None, None, None, None)
aliases, known = (), None
if candidate is not None:
aliases, known = physical_spellings(self.db, image_path, item["locator"])
try:
result = self.db.decide_review_item(
item_id=item["id"],
expected_epoch=guards["expected_epoch"],
expected_revision=guards["expected_revision"],
# The bytes the re-hash proved, or the body's for Leave unresolved.
sha256=facts.sha256 if facts is not None else guards["sha256"],
outcome="selected" if candidate is not None else UNRESOLVED,
candidate=candidate,
image_path=image_path,
metadata=metadata,
expected_fingerprint=guards["dependency_fingerprint"],
aliases=aliases,
known_rows=known,
)
except ReviewCommandError as exc:
if exc.item is not None:
roots, real_roots = self._review_roots()
exc.view = self._review_view(exc.item, roots, real_roots)
raise
roots, real_roots = self._review_roots()
return result, self._review_view(result["item"], roots, real_roots)
async def _record_decision(self, item, guards, candidate, verified):
capture = verified[0] if verified else None
try:
result, view = await self._run_in_executor(
self._decide_sync, item, guards, candidate, verified
)
except ReviewCommandError as exc:
return self._decision_refused(exc)
except LibraryCommandError as exc:
return _coded(exc.code, library_epoch=exc.epoch)
except Exception:
self.logger.exception("Review decision failed")
return _error(500, "decision-failed", "The decision could not be saved.")
if capture is not None:
# The decision used the capture; the page returns to Images.
self._captures.release(capture.token)
body = {
"success": True,
"library_epoch": guards["expected_epoch"],
"item": view,
"total": result["total"],
}
if result["pair"] is not None:
body["pair"] = result["pair"]
return web.json_response(body, headers=_NO_STORE)
def _decision_refused(self, exc: ReviewCommandError) -> web.Response:
extra = {"library_epoch": exc.epoch}
if exc.total is not None:
extra["total"] = exc.total
if exc.view is not None:
extra["item"] = exc.view
message = None
if exc.fingerprint is not None:
extra["dependency_fingerprint"] = exc.fingerprint
if exc.code == "association-conflict" and exc.changed:
message = "dependencies-changed"
return _coded(exc.code, message, **extra)
def _decision_guards(data: dict):
"""The validated decision fields, or ``(None, error)``."""
if set(data) != _DECISION_FIELDS:
return None, bad_request(
"Request must contain exactly " + ", ".join(sorted(_DECISION_FIELDS))
)
epoch, revision = data["expected_epoch"], data["expected_revision"]
if not isinstance(epoch, str) or _UUID.fullmatch(epoch) is None:
return None, bad_request("expected_epoch must be a canonical UUID")
if type(revision) is not int or revision < 0:
return None, bad_request("expected_revision must be a non-negative integer")
token = data["capture_token"]
if token is not None and (
not isinstance(token, str) or _TOKEN.fullmatch(token) is None
):
return None, bad_request("Invalid capture token")
for name in ("sha256", "dependency_fingerprint"):
value = data[name]
if not isinstance(value, str) or _HEX64.fullmatch(value) is None:
return None, bad_request(
f"{name} must be 64 lowercase hexadecimal characters"
)
choice = data["choice"]
if not isinstance(choice, str) or (
choice != UNRESOLVED and _HEX64.fullmatch(choice) is None
):
return None, bad_request("choice must be a candidate hash or unresolved")
return {**data}, None
def _verify_and_read(store, capture, item, root, need_metadata: bool):
"""Worker thread: ``(relative, facts, captured)``, or None off the root.
Formats the item's target under the capture's root (resolving the root's
real path), re-hashes the current file, then reads the capture.
"""
found = format_review_target(
item["spelled_path"],
item["locator"],
[(capture.root_index, _realpath(str(root)))],
)
if found is None:
return None
relative = found[1]
facts = store.verify(root, relative)
captured = captured_review(store, capture) if need_metadata else None
return relative, facts, captured
def _same_file(spelled: str, image_path: str, locator: str) -> bool:
"""True when ``spelled`` reaches the physical file at ``locator``."""
try:
if os.path.normcase(_realpath(spelled)) == locator:
return True
return os.path.samefile(spelled, image_path)
except (OSError, ValueError):
return False
def physical_spellings(db, image_path: str, locator: str):
"""``(aliases, known_rows)`` for the legacy rows of one physical file.
The roots are real paths, while the monitor links a file under the
directory it watches, so a symlinked or junctioned root, a ``subst`` or a
mapped drive spells the same file otherwise. Every row whose spelling
ends with the file's name (the link summaries' fallback lookup) is a
candidate; ``aliases`` are the candidates that resolve to ``locator``
(``normcase(realpath)``, as the capture forms it) or are the same file,
and ``known_rows`` every candidate id, so a decision can refuse a
same-name row it did not classify. Blocking: call it on a worker.
"""
rows = db.review_same_name_rows(os.path.basename(image_path))
aliases = tuple(
spelled
for _row_id, spelled in rows
if isinstance(spelled, str)
and spelled != image_path
and _same_file(spelled, image_path, locator)
)
return tuple(dict.fromkeys(aliases)), frozenset(row_id for row_id, _ in rows)
def _candidate_still_offered(resolution: dict, choice: str) -> bool:
"""True when the captured bytes still offer the chosen candidate.
Only what the decision depends on is re-checked: the bytes are still
``ambiguous`` and the chosen candidate (its hash covers its sampler,
sides and provenance) is still among their candidates. A resolver change
that rewords the top-level reason or reorders candidates leaves a
pending item applicable.
"""
if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous":
return False
evidence, _count, limited = review_evidence(resolution)
if limited:
return False
return any(c.get("hash") == choice for c in evidence.get("candidates", ()))
-171
View File
@@ -1,171 +0,0 @@
"""Needs review change stream (#249 PR F2).
``GET /prompt_manager/review/events`` is a server-sent event stream for an
open Needs review page: ``event: ready`` when it opens, then
``event: change`` after each committed queue change (an item the scan or the
monitor recorded, a decision from any tab, a spelling refresh) and after a
restore. The first notice is sent at once; notices in the next
``COALESCE_SECONDS`` collapse into one trailing ``change`` (M9: a busy scan
records file after file, and each event costs every open page a re-read).
The page re-reads its queue on each ``change``; nothing polls. ``change``
carries no data (``{}``). ``ready`` carries the queue's version
``{library_epoch, review_revision}``, read after the stream subscribed, the
same pair a list read answers: a page whose first read already answered that
version skips the re-read (F3, one read per visit), and any later change is a
``change`` event. Nothing else about the queue is pushed.
A ``: keepalive`` comment every ``HEARTBEAT_SECONDS`` lets the server notice
a closed tab (its write fails) and drop the subscription; clients ignore
comments. Shutdown (``close_review_events``, an ``on_shutdown`` hook) ends
every stream, and a stream asked for afterwards is a 503.
"""
import asyncio
import json
from aiohttp import web
try:
from ...database.execution_history import subscribe_history_restore
from ...database.review_changes import subscribe_review_changes
except ImportError:
from database.execution_history import subscribe_history_restore
from database.review_changes import subscribe_review_changes
EVENTS_URL = "/prompt_manager/review/events"
HEARTBEAT_SECONDS = 25
# After a ``change``, the next one waits this long, collapsing the notices in between.
COALESCE_SECONDS = 0.4
_CHANGE = b"event: change\ndata: {}\n\n"
_KEEPALIVE = b": keepalive\n\n"
def _ready(version: dict) -> bytes:
data = json.dumps(version, separators=(",", ":"), sort_keys=True)
return f"event: ready\ndata: {data}\n\n".encode("utf-8")
class _Stream:
"""One open stream: a change flag set from any thread, and a close flag."""
def __init__(self, loop: asyncio.AbstractEventLoop) -> None:
self._loop = loop
self.changed = asyncio.Event()
self.closed = asyncio.Event()
self.ready = False
def wake(self) -> None:
"""Thread-safe: mark a change (the loop may already be gone)."""
try:
self._loop.call_soon_threadsafe(self.changed.set)
except RuntimeError: # the loop closed
pass
async def next(self, timeout: float) -> str:
"""``change``, ``closed`` or ``idle`` (nothing within ``timeout``)."""
waits = {
asyncio.ensure_future(self.changed.wait()): "change",
asyncio.ensure_future(self.closed.wait()): "closed",
}
try:
done, _ = await asyncio.wait(
waits, timeout=timeout, return_when=asyncio.FIRST_COMPLETED
)
finally:
for waiter in waits:
waiter.cancel()
if self.closed.is_set():
return "closed"
if done:
self.changed.clear()
return "change"
return "idle"
async def closed_within(self, timeout: float) -> bool:
"""Wait up to ``timeout``; true when the stream closed meanwhile."""
try:
await asyncio.wait_for(self.closed.wait(), timeout)
except asyncio.TimeoutError:
return False
return True
class ReviewEventsMixin:
"""The Needs review change stream."""
def _register_review_event_routes(self, routes: web.RouteTableDef) -> None:
routes.get(EVENTS_URL)(self.review_events)
def _review_streams(self) -> set:
streams = getattr(self, "_open_review_streams", None)
if streams is None:
streams = self._open_review_streams = set()
return streams
def review_stream_count(self) -> int:
"""How many change streams have sent ``ready`` and are open (tests and
diagnostics): a queue change after this count moved is a ``change``
event on them, never part of the version ``ready`` named."""
return sum(1 for stream in self._review_streams() if stream.ready)
async def close_review_events(self, app) -> None:
"""End every open stream; later requests get 503 (shutdown hook)."""
self._review_events_closed = True
for stream in list(self._review_streams()):
stream.closed.set()
async def _queue_version(self) -> dict:
"""The queue version for ``ready``, read after subscribing (so a change
after this read is an event); ``{}`` when it cannot be read, and the
page then re-reads."""
try:
return await self._run_in_executor(self.db.review_queue_version)
except Exception:
self.logger.warning("Needs review version read failed", exc_info=True)
return {}
async def review_events(self, request: web.Request) -> web.StreamResponse:
if getattr(self, "_review_events_closed", False):
return web.json_response(
{"success": False, "error": "The server is shutting down."},
status=503,
)
stream = _Stream(asyncio.get_running_loop())
db_path = self.db.model.db_path
unsubscribers = (
subscribe_review_changes(db_path, stream.wake),
subscribe_history_restore(db_path, stream.wake),
)
streams = self._review_streams()
streams.add(stream)
response = web.StreamResponse(
status=200,
headers={
"Content-Type": "text/event-stream",
"Cache-Control": "no-store",
"X-Accel-Buffering": "no",
},
)
try:
version = await self._queue_version()
await response.prepare(request)
await response.write(_ready(version))
stream.ready = True
while True:
outcome = await stream.next(HEARTBEAT_SECONDS)
if outcome == "closed":
break
if outcome == "idle":
await response.write(_KEEPALIVE)
continue
await response.write(_CHANGE)
if await stream.closed_within(COALESCE_SECONDS):
break
await response.write_eof()
except (ConnectionError, RuntimeError) as exc: # the tab went away
self.logger.debug(f"Needs review stream closed ({exc})")
finally:
streams.discard(stream)
for unsubscribe in unsubscribers:
unsubscribe()
return response
-241
View File
@@ -1,241 +0,0 @@
"""Needs review recording for the output scan and the live monitor (#249).
The rule (maintainer test of #284/#288): a file is queued only when its
final outcome is ambiguous and unlinked. The strict resolver must find it
``ambiguous``, the scan's legacy fallback (the browser parser,
``_graph_prompt_text``) must not link it, and no saved row may link it
already. A file the fallback links, an exact pair the ingester stored and an
already linked file are never queued. Neither is an ambiguous file with
nothing to choose: fewer than two of its candidates carry a prompt
(``utils.review_capture.reviewable``); the scan reports it unresolved ("could
not be traced").
The queue heals itself: a pending item whose file is now linked (by the
fallback, a later resolution or the monitor) is removed by the next scan
that lists the file or by the monitor's next pass over it, and so is a
pending item without a choice whose file is still ambiguous without one
(``settle_untraced_review_items``). Only an undecided item goes
(``pending``, revision 0, no decision row); a decision is never deleted or
changed (``settle_linked_review_items``).
The scan records through ``_record_scanned_review`` (best effort: a failure
is logged and counted); the monitor calls ``record_review_file`` from its
worker thread after its own link. Both go through
``utils.review_capture.record_ambiguous_file`` on the API's capture store.
Nothing here links a file.
Recording changes records, so it follows the registry's ``records``
admission (#289). The scan records under its own ``records`` job. The
monitor's pass lasts one file and is not a job: it is refused while a
``records`` or ``guarded`` job runs (``deferred``), and the next scan reads
the file again.
"""
import os
from pathlib import Path
from .admin import SCAN_LOG_PREFIXES, _plural, _scan_location, _scan_roots
# The kind the monitor's recording checks admission under (never registered).
REVIEW_RECORDING_KIND = "review_recording"
# record_ambiguous_file outcomes that leave the file in the pending queue.
QUEUED_OUTCOMES = ("recorded", "limited", "duplicate")
NEW_OUTCOMES = ("recorded", "limited")
try:
from ...utils.review_capture import record_ambiguous_file
except ImportError:
from utils.review_capture import record_ambiguous_file
# The scan log's reason for items that leave the queue with nothing to choose.
NOTHING_TO_CHOOSE = "nothing to choose"
class ReviewRecordingMixin:
"""Record the scan's and the monitor's ambiguous files for Needs review."""
def _record_scanned_review(self, media_file, roots, review=None):
"""Record one ambiguous, unlinked scanned file (logged and counted)."""
try:
outcome = self._record_review_sync(media_file, roots, review)
except Exception:
self.logger.warning(
f"{media_file.name} could not be added to Needs review",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return
if review is None:
return
if outcome in NEW_OUTCOMES:
review["added"] += 1
if outcome in QUEUED_OUTCOMES:
review["queued"] += 1
def _record_review_sync(self, media_file, roots, review=None):
"""``record_ambiguous_file`` for a file under one of ``roots`` (blocking).
An ``untraced`` file (its captured bytes offer nothing to choose) also
loses its pending items that offer nothing either.
"""
located = self._review_location(media_file, roots)
if located is None:
return "skipped"
root, index, relative, _spelled = located
outcome = record_ambiguous_file(
self.db, self._capture_store(), Path(root), index, relative
)
if outcome == "untraced":
self._settle_untraced_reviews([media_file], roots, review)
return outcome
@staticmethod
def _review_location(media_file, roots, real_roots=None):
"""``(root, root_index, relative, spelled)`` of a listed file, or None.
``spelled`` is the capture's spelling of the file (the real root joined
with the relative path); ``os.path.normcase`` of it is the locator
review items are stored under. ``real_roots`` caches each root's
``realpath`` across a batch.
"""
relative, _label, index = _scan_location(media_file, roots)
root = next((r for r, _l, i in roots if i == index), None)
if index is None or root is None:
return None
if real_roots is None:
real_roots = {}
real = real_roots.get(index)
if real is None:
real = real_roots[index] = os.path.realpath(os.fspath(root))
spelled = os.path.join(real, *relative.split("/"))
return root, index, relative, spelled
def _review_file_linked(self, media_file, roots):
"""True when a saved row links the scanned file (any spelling)."""
located = self._review_location(media_file, roots)
spellings = (str(media_file),) + ((located[3],) if located else ())
return self.db.review_file_linked(spellings)
def _settle_scanned_reviews(self, media_files, roots, review=None):
"""Remove the pending items of ``media_files`` that are now linked.
Blocking; best effort like recording (a failure is logged and
counted in ``review["errors"]``). Returns how many items left.
"""
real_roots = {}
files = []
for media_file in media_files:
located = self._review_location(media_file, roots, real_roots)
if located is not None:
spelled = located[3]
files.append((os.path.normcase(spelled), (str(media_file), spelled)))
try:
settled = self.db.settle_linked_review_items(files) if files else 0
except Exception:
self.logger.warning(
"Needs review items of linked files could not be removed",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return 0
if review is not None:
review["settled"] += settled
return settled
def _settle_untraced_reviews(self, media_files, roots, review=None):
"""Remove the pending items without a choice of untraced ``media_files``.
``media_files`` are files found ambiguous with fewer than two
candidates that carry a prompt. Blocking; best effort like the linked
settle (a failure is logged and counted in ``review["errors"]``).
Returns how many items left.
"""
locators = []
for media_file in media_files:
located = self._review_location(media_file, roots)
if located is not None:
locators.append(os.path.normcase(located[3]))
try:
removed = self.db.settle_untraced_review_items(locators) if locators else 0
except Exception:
self.logger.warning(
"Needs review items without a choice could not be removed",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return 0
if review is not None:
review["untraced"] += removed
return removed
def _fallback_links(self, metadata):
"""True when the scan's legacy fallback would link these records."""
return bool(metadata.get("prompt")) and bool(self._graph_prompt_text(metadata))
def record_review_file(self, image_path):
"""Apply the Needs review rule to one monitor output (its worker thread).
Maps ``image_path`` to the gallery root that lists it; a file under no
current root is ``skipped``. A linked file is never queued: its
pending items are removed and the answer is ``linked``. An unlinked
file the scan's fallback would link is ``skipped`` (the next scan
links it); any other file is recorded when it is ambiguous with a
choice, and is ``untraced`` (its pending items without a choice
removed) when it is ambiguous without one. While a
job runs that a ``records`` job may not run beside, nothing is written
and the answer is ``deferred``. Errors propagate: the monitor logs
them and keeps its link.
"""
busy = self._jobs.blocker(REVIEW_RECORDING_KIND, admission="records")
if busy is not None:
self.logger.info(
f"Needs review was not updated for "
f"{os.path.basename(str(image_path))} while {busy.title} runs; "
"the next scan reads it again"
)
return "deferred"
full = os.path.abspath(str(image_path))
located = Path(os.path.realpath(os.path.dirname(full))) / os.path.basename(full)
roots = _scan_roots(self._get_all_output_dirs())
if self._review_location(located, roots) is None:
return "skipped"
if self._review_file_linked(located, roots):
self._settle_scanned_reviews([located], roots)
return "linked"
if self._fallback_links(self._extract_comfyui_metadata(str(located))):
return "skipped"
return self._record_review_sync(located, roots)
@staticmethod
def _log_review_items(job, review):
"""The scan log's Needs review summary lines (none when nothing moved)."""
prefix = SCAN_LOG_PREFIXES["review"]
if review["added"]:
job.log_line(
f"{prefix} {_plural(review['added'], 'image')} added to Needs review"
)
already = review["queued"] - review["added"]
if already:
job.log_line(
f"{prefix} {_plural(already, 'image')} already in Needs review"
)
if review["settled"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['known']} "
f"{_plural(review['settled'], 'image')} left Needs review · now linked"
)
if review["untraced"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['known']} "
f"{_plural(review['untraced'], 'image')} left Needs review · "
f"{NOTHING_TO_CHOOSE}"
)
if review["errors"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['errors']} "
f"{_plural(review['errors'], 'image')} could not be added to "
"Needs review"
)
+31 -246
View File
@@ -4,15 +4,8 @@ A save validates the payload, commits it to config.json through the config
store and publishes it at runtime inside that commit; a save that cannot be
written changes nothing at runtime and says so. ``_config_commit_failure``
is shared with the LoRA integration's save.
GET also reports the image sources and the monitor (``sources.py``). A save
that changes the sources runs as a ``source_change`` registry hold, so it is
refused while a record-changing job (a scan) runs. ``monitoring_enabled``
takes effect at the next restart. ``POST /prompt_manager/settings/validate-source``
checks one path and writes nothing.
"""
import contextlib
import os
from aiohttp import web
@@ -22,10 +15,7 @@ from ..config_store import (
RevisionConflict,
get_config_store,
)
from . import sources
from .admin import _current_image_monitor
from .job_routes import busy_response
from .jobs import JobBusy
try:
from ...utils.validators import validate_result_timeout, validate_worker_threads
@@ -33,75 +23,13 @@ except ImportError:
from utils.validators import validate_result_timeout, validate_worker_threads
# Settings owned by POST /settings: config attribute -> its keys in config.json.
# Settings owned by POST /settings: config attribute -> (section, key).
_SETTINGS_FILE_KEYS = {
"RESULT_TIMEOUT": ("web_ui", "result_timeout"),
"WEBUI_DISPLAY_MODE": ("web_ui", "webui_display_mode"),
"INFINITE_SCROLL": ("web_ui", "infinite_scroll"),
"WORKER_THREADS": ("performance", "worker_threads"),
"MONITORING_ENABLED": ("gallery", "monitoring", "enabled"),
}
# Attributes that live on GalleryConfig; the others on PromptManagerConfig.
_GALLERY_SETTINGS = frozenset({"MONITORING_ENABLED"})
SOURCE_CHANGE_JOB_KIND = "source_change"
SOURCE_CHANGE_JOB_TITLE = "Source change"
def _source_identity(path):
"""The case-folded spelling two roots are the same directory by (Windows)."""
return os.path.normcase(path)
def _stat_identity(path):
"""``(st_dev, st_ino)`` of an existing directory, or None.
Catches case variants on a case-insensitive filesystem whose spelling
normcase keeps (macOS).
"""
try:
stat = os.stat(path)
except (OSError, ValueError):
return None
return (stat.st_dev, stat.st_ino) if stat.st_ino else None
# Refusals that a reconnected drive or folder can lift.
_RECONNECTABLE = frozenset({"not-found", "not-directory"})
def _root_key(root):
"""The directory a configured root names (resolved), else its spelling."""
from ..config import GalleryConfig
try:
return GalleryConfig.resolve_gallery_root(root)
except (OSError, TypeError, ValueError, AttributeError):
return root
def _request_error(message):
return {"code": "invalid-request", "error": message}
def _invalid_source(index, entry, code, reason):
"""The 400 body for the source at ``index`` that cannot be used.
Names it by its public spelling (computed as text: a refused path is
never resolved), so the page can mark that row and say why.
"""
source = sources.public_spelling(entry)
advice = (
"Remove it or reconnect it, then save."
if code in _RECONNECTABLE
else "Remove it, then save."
)
return {
"code": "invalid-source",
"source": source,
"index": index,
"reason": reason,
"error": f"{source} can't be used: {reason}. {advice}",
}
def _section(config, name):
@@ -119,8 +47,10 @@ def _settings_mutation(updates, new_roots):
def mutate(config):
for name, value in updates.items():
*sections, key = _SETTINGS_FILE_KEYS[name]
_set_nested(config, sections, key, value)
section, key = _SETTINGS_FILE_KEYS[name]
merged = _section(config, section)
merged[key] = value
config[section] = merged
if new_roots is not None:
gallery = _section(config, "gallery")
monitoring = _section(gallery, "monitoring")
@@ -132,16 +62,6 @@ def _settings_mutation(updates, new_roots):
return mutate
def _set_nested(config, sections, key, value):
"""``config[s1][s2]...[key] = value``, each section a fresh dict."""
if not sections:
config[key] = value
return
merged = _section(config, sections[0])
_set_nested(merged, sections[1:], key, value)
config[sections[0]] = merged
async def _read_json(request):
"""Parse a JSON object body through the package's capped reader."""
from . import _read_json_body
@@ -161,10 +81,6 @@ class SettingsRoutesMixin:
async def save_settings_route(request):
return await self.save_settings(request)
@routes.post("/prompt_manager/settings/validate-source")
async def validate_source_route(request):
return await self.validate_source(request)
async def get_settings(self, request):
"""Get current settings."""
try:
@@ -174,15 +90,11 @@ class SettingsRoutesMixin:
self._public_path(d) for d in GalleryConfig.MONITORING_DIRECTORIES
]
revision = await self._run_in_executor(self._config_revision)
source_status, monitor = await self._run_in_executor(self._source_status)
return web.json_response(
{
"success": True,
**revision,
"sources": source_status,
"monitor": monitor,
"settings": {
"monitoring_enabled": GalleryConfig.MONITORING_ENABLED,
"result_timeout": PromptManagerConfig.RESULT_TIMEOUT,
"webui_display_mode": PromptManagerConfig.WEBUI_DISPLAY_MODE,
"infinite_scroll": PromptManagerConfig.INFINITE_SCROLL,
@@ -203,61 +115,6 @@ class SettingsRoutesMixin:
status=500,
)
def _source_status(self):
"""``(sources, monitor)`` of GET /settings (blocking)."""
from ..config import GalleryConfig
output_dirs, auto_detected = self._output_dirs_with_origin()
return sources.describe_sources(
configured=list(GalleryConfig.MONITORING_DIRECTORIES),
output_dirs=output_dirs,
auto_detected=auto_detected,
monitor=_current_image_monitor(),
enabled=GalleryConfig.MONITORING_ENABLED,
)
async def validate_source(self, request):
"""Check one source path ``{path}``; writes nothing.
200 ``{success, valid, label, path, code, reason}``: a valid source
answers its label and public path, an invalid one its code and reason
only. A body without a string ``path`` is a 400 ``invalid-request``.
"""
data, error_response = await _read_json(request)
if error_response is not None:
too_large = error_response.status == 413
return web.json_response(
{
"success": False,
"code": "too-large" if too_large else "invalid-request",
"error": (
"Request body too large"
if too_large
else "Request body must be a JSON object"
),
},
status=error_response.status,
)
path = data.get("path")
if not isinstance(path, str):
return web.json_response(
{
"success": False,
"code": "invalid-request",
"error": "path must be a string",
},
status=400,
)
try:
result = await self._run_in_executor(sources.validate_source, path)
except Exception:
self.logger.exception("Source validation failed")
return web.json_response(
{"success": False, "code": None, "error": "Failed to check the source"},
status=500,
)
return web.json_response({"success": True, **result})
def _monitored_directories(self):
"""Directories the running image monitor watches, else the configured roots."""
from ..config import GalleryConfig
@@ -294,15 +151,9 @@ class SettingsRoutesMixin:
{"success": False, "error": str(ve)}, status=400
)
# Off the loop: an accepted root is resolved (filesystem access).
new_roots, error = await self._run_in_executor(
self._parse_gallery_roots, data
)
new_roots, error = self._parse_gallery_roots(data)
if error:
return web.json_response({"success": False, **error}, status=400)
roots_change = new_roots is not None and await self._run_in_executor(
self._roots_change, new_roots
)
return web.json_response({"success": False, "error": error}, status=400)
published = {}
@@ -311,30 +162,15 @@ class SettingsRoutesMixin:
# they were written, so the runtime ends on the last one.
published.update(self._publish_settings(updates, new_roots))
async with contextlib.AsyncExitStack() as stack:
hold = None
if roots_change:
try:
hold = await stack.enter_async_context(
self._jobs.hold(
SOURCE_CHANGE_JOB_KIND,
SOURCE_CHANGE_JOB_TITLE,
admission="records",
)
)
except JobBusy as busy:
return busy_response(busy)
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_settings_mutation(updates, new_roots),
expected_revision=expected_revision,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
if hold is not None:
hold.finish("failed")
return self._config_commit_failure(exc)
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_settings_mutation(updates, new_roots),
expected_revision=expected_revision,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
return self._config_commit_failure(exc)
return web.json_response(
{
@@ -360,11 +196,10 @@ class SettingsRoutesMixin:
as ``runtime_applied: false`` with the fields that need a restart;
nothing is retried or rolled back. Never raises.
"""
from ..config import GalleryConfig, PromptManagerConfig
from ..config import PromptManagerConfig
for name, value in updates.items():
owner = GalleryConfig if name in _GALLERY_SETTINGS else PromptManagerConfig
setattr(owner, name, value)
setattr(PromptManagerConfig, name, value)
restart_fields = []
runtime_applied = True
if new_roots is not None:
@@ -377,40 +212,11 @@ class SettingsRoutesMixin:
)
runtime_applied = False
restart_fields.append("gallery_root_paths")
if "MONITORING_ENABLED" in updates:
# The monitor starts with the server: a restart applies the setting
# unless the observer already matches it.
if updates["MONITORING_ENABLED"] != self._observer_alive():
restart_fields.append("monitoring_enabled")
return {
"runtime_applied": runtime_applied,
"restart_required_fields": restart_fields,
}
@staticmethod
def _roots_change(new_roots):
"""Whether a save changes the sources (blocking: it resolves them).
Compared as directories, not spellings: the legacy page resends the
roots resolved, while config.json may hold them as hand-written
(relative, another case), and an unchanged resend is no change.
"""
from ..config import GalleryConfig
if new_roots is None:
return False
return [_root_key(root) for root in new_roots] != [
_root_key(root) for root in GalleryConfig.MONITORING_DIRECTORIES
]
@staticmethod
def _observer_alive():
"""Whether the image monitor's observer is running now. Never raises."""
try:
return sources.monitor_state(_current_image_monitor())[1]
except Exception:
return False
def _config_commit_failure(self, exc):
"""The response for a config commit that wrote nothing."""
not_saved = {"success": False, "persisted": False, "runtime_applied": False}
@@ -477,10 +283,6 @@ class SettingsRoutesMixin:
if not isinstance(data["infinite_scroll"], bool):
raise ValueError("infinite_scroll must be true or false")
updates["INFINITE_SCROLL"] = data["infinite_scroll"]
if "monitoring_enabled" in data:
if not isinstance(data["monitoring_enabled"], bool):
raise ValueError("monitoring_enabled must be true or false")
updates["MONITORING_ENABLED"] = data["monitoring_enabled"]
if "worker_threads" in data:
updates["WORKER_THREADS"] = validate_worker_threads(
data["worker_threads"], PromptManagerConfig.max_worker_threads()
@@ -493,21 +295,15 @@ class SettingsRoutesMixin:
Accepts ``gallery_root_paths`` (list, preferred) or the legacy single
``gallery_root_path`` string. Blank entries are dropped.
Each entry passes the same checks as validate-source
(``sources.source_problem``): the form, then containment as text,
before anything is resolved. Blocking (an accepted root is resolved).
Returns:
(roots, error): ``roots`` is the validated list, or ``None`` when
the payload carries no gallery root key; ``error`` is the coded
body of the 400 when validation failed (``roots`` is then None).
An unusable source is ``invalid-source`` naming its public path
and its position in the list.
the payload carries no gallery root key; ``error`` is a message
when validation failed (and ``roots`` is then ``None``).
"""
if "gallery_root_paths" in data:
raw = data["gallery_root_paths"]
if not isinstance(raw, list):
return None, _request_error("gallery_root_paths must be a list")
return None, "gallery_root_paths must be a list"
elif "gallery_root_path" in data:
raw = [data["gallery_root_path"]]
else:
@@ -516,38 +312,27 @@ class SettingsRoutesMixin:
from ..config import GalleryConfig
roots = []
seen = set()
for index, entry in enumerate(raw):
for entry in raw:
if not isinstance(entry, str):
return None, _request_error("Gallery root paths must be strings")
return None, "Gallery root paths must be strings"
entry = entry.strip()
if not entry:
continue
code, reason = sources.source_problem(entry)
if code is not None:
return None, _invalid_source(index, entry, code, reason)
root = GalleryConfig.resolve_gallery_root(entry)
keys = {("path", _source_identity(root)), ("file", _stat_identity(root))}
keys.discard(("file", None))
if keys & seen:
continue # another spelling of a root already listed
seen |= keys
roots.append(root)
ok, reason = GalleryConfig.validate_gallery_root(entry)
if not ok:
return (
None,
f"Invalid gallery root '{os.path.basename(entry)}': {reason}",
)
roots.append(GalleryConfig.resolve_gallery_root(entry))
return roots, None
def _apply_gallery_roots(self, roots):
"""Install validated gallery roots; returns True when they changed.
Another spelling of the same directories installs the resolved
spelling and is no change (nothing to restart for).
"""
"""Install validated gallery roots; returns True when they changed."""
from ..config import GalleryConfig
if roots == list(GalleryConfig.MONITORING_DIRECTORIES):
return False
if not self._roots_change(roots):
GalleryConfig.MONITORING_DIRECTORIES = roots
return False
GalleryConfig.MONITORING_DIRECTORIES = roots
self._cached_output_dir = None
self.reset_gallery_roots_cache()
-280
View File
@@ -1,280 +0,0 @@
"""Image sources as System › Sources & scans shows them (#249 S4).
- ``source_form_error`` is the lexical check a submitted source path passes
before any filesystem access: no control characters, no ``..`` part, and
only the native path forms (on Windows a drive or a UNC share, never the
``\\\\?\\`` and ``\\\\.\\`` namespaces, a drive-relative path, an alternate
data stream, a device name or a part Windows would silently trim; on POSIX
no Windows drive or backslash form).
- ``source_problem`` is the one refusal every caller answers (validate-source
and the settings save): the form check, then
``GalleryConfig.gallery_root_problem``, which decides containment as text
before it touches the filesystem.
- ``describe_sources`` (blocking: it resolves paths) reports the configured
sources with their validity, the effective sources with their listing
index and watch state, and the image monitor's state. Every path in it is
a public path (``_public_path``), never an absolute one.
A source's watch state is ``watching`` when the monitor's observer is alive,
scheduled that directory and monitoring is on; ``stopping`` when it still
watches it but monitoring was switched off (it stops at the next restart);
``manual`` when monitoring is off or this process has no monitor; else
``restart_pending`` (a source added since the monitor started, monitoring
switched on, or an observer that stopped).
"""
import logging
import ntpath
import os
import re
logger = logging.getLogger("prompt_manager.api")
MAX_SOURCE_PATH = 4096 # characters
FORM_REASONS = {
"empty": "Gallery root must be a non-empty path",
"invalid-form": "Gallery root is not a folder path this server can use",
"traversal": "Gallery root must not contain '..'",
}
_DRIVE = re.compile(r"[A-Za-z]:")
_POSIX_FOREIGN = re.compile(r"^[A-Za-z]:")
# Reserved device names: Windows opens the device for any folder of that name.
_DEVICE_NAME = re.compile(r"^(con|prn|aux|nul|com[0-9¹²³]|lpt[0-9¹²³])(\..*)?$", re.I)
# folder_paths getters and the label of the directory each one reports.
_COMFYUI_LABELS = (
("get_output_directory", "ComfyUI output"),
("get_input_directory", "ComfyUI input"),
("get_temp_directory", "ComfyUI temp"),
("get_user_directory", "ComfyUI user"),
)
WATCH_STATES = ("watching", "stopping", "manual", "restart_pending")
def source_form_error(text, *, windows=None):
"""The form code that refuses ``text`` as a source path, or None.
``windows`` selects the path rules (default: this OS). No filesystem
access happens here.
"""
if windows is None:
windows = os.name == "nt"
if not isinstance(text, str) or len(text) > MAX_SOURCE_PATH:
return "invalid-form"
if any(ord(char) < 32 or ord(char) == 127 for char in text):
return "invalid-form"
text = text.strip()
if not text:
return "empty"
return _windows_form_error(text) if windows else _posix_form_error(text)
def _posix_form_error(text):
if "\\" in text or _POSIX_FOREIGN.match(text):
return "invalid-form" # a Windows path names nothing on this server
return "traversal" if ".." in text.split("/") else None
def _windows_form_error(text):
portable = text.replace("/", "\\")
if portable.startswith(("\\\\?\\", "\\\\.\\")):
return "invalid-form" # verbatim and device namespaces
if portable.startswith("\\\\"):
drive, rest = ntpath.splitdrive(portable)
share = drive[2:].split("\\")
if len(share) != 2 or any(part in ("", ".", "..") for part in share):
return "invalid-form"
else:
drive, rest = ntpath.splitdrive(portable)
if drive and (not _DRIVE.fullmatch(drive) or not rest.startswith("\\")):
return "invalid-form" # "C:folder" is relative to that drive's cwd
if ":" in rest:
return "invalid-form" # an alternate data stream
parts = rest.split("\\")
if ".." in parts:
return "traversal"
for part in parts:
if part in ("", "."):
continue
if part[-1] in ". " or _DEVICE_NAME.match(part):
return "invalid-form"
return None
def source_problem(text):
"""``(code, reason)`` that refuses ``text`` as a source, else ``(None, "")``.
The form check runs first and touches nothing; then containment is
decided as text before any filesystem access (see
``GalleryConfig.gallery_root_problem``). Every caller answers alike.
"""
from ..config import GalleryConfig
code = source_form_error(text)
if code is not None:
return code, FORM_REASONS[code]
return GalleryConfig.gallery_root_problem(text)
def public_spelling(text):
"""How a refused source is named in a response, computed as text only.
Relative to the path anchor it lies under (its public path), else its
last part: the caller sent the path, so naming it reveals nothing, and a
refused path is never resolved to name it.
"""
from ..config import GalleryConfig, _lexical_path, _lexically_inside
text = str(text).strip()
try:
lexical = os.path.normpath(text)
key = _lexical_path(text) # the same length as ``lexical``
if not os.path.isabs(text):
return lexical.replace("\\", "/")
for anchor in GalleryConfig.path_anchors():
if _lexically_inside(key, [anchor]):
rest = lexical[len(anchor.rstrip(os.sep)) :].lstrip(os.sep)
return rest.replace(os.sep, "/") or "."
return os.path.basename(lexical) or lexical
except (TypeError, ValueError):
return ""
def identity(path):
"""The comparison key of a directory: resolved and case-folded (Windows)."""
return os.path.normcase(os.path.realpath(str(path)))
def _comfyui_labels():
"""``{identity: label}`` of the directories ComfyUI reports."""
try:
import folder_paths
except ImportError:
return {}
labels = {}
for getter_name, label in _COMFYUI_LABELS:
getter = getattr(folder_paths, getter_name, None)
if getter is None:
continue
try:
directory = getter()
except Exception:
continue
if isinstance(directory, str) and directory:
labels.setdefault(identity(directory), label)
return labels
def source_label(path, labels=None):
""" "ComfyUI output" for ComfyUI's own directories, else the folder name."""
if labels is None:
labels = _comfyui_labels()
label = labels.get(identity(path))
if label:
return label
return os.path.basename(os.path.normpath(str(path))) or str(path)
def _configured(entry, labels, public_path):
from ..config import GalleryConfig
code, reason = GalleryConfig.gallery_root_problem(entry)
try:
resolved = GalleryConfig.resolve_gallery_root(entry)
except (OSError, TypeError, ValueError):
resolved = str(entry)
return {
"label": source_label(resolved, labels),
"path": public_path(resolved),
"valid": code is None,
"code": code,
"reason": reason or None,
}
def monitor_state(monitor):
"""``(present, observer_alive, scheduled directories)`` of the image monitor."""
if monitor is None:
return False, False, []
status = monitor.get_status()
scheduled = [str(d) for d in status.get("monitored_directories") or []]
return True, bool(status.get("observer_alive")), scheduled
def watch_state(path, *, present, alive, scheduled_keys, enabled):
"""One of ``WATCH_STATES`` (see the module docstring)."""
if alive and identity(path) in scheduled_keys:
return "watching" if enabled else "stopping"
if not present or not enabled:
return "manual"
return "restart_pending"
def describe_sources(*, configured, output_dirs, auto_detected, monitor, enabled):
"""``(sources, monitor)`` for GET /settings (blocking; public paths only).
``configured`` is the configured roots, ``output_dirs`` and
``auto_detected`` what ``_output_dirs_with_origin`` returned, ``monitor``
the running ImageMonitor or None, ``enabled`` the monitoring setting.
"""
from . import _public_path
labels = _comfyui_labels()
present, alive, scheduled = monitor_state(monitor)
scheduled_keys = {identity(path) for path in scheduled}
effective = [
{
"label": source_label(path, labels),
"path": _public_path(path),
"root_index": index,
"watch": watch_state(
path,
present=present,
alive=alive,
scheduled_keys=scheduled_keys,
enabled=enabled,
),
}
for index, path in enumerate(output_dirs)
]
return (
{
"configured": [
_configured(entry, labels, _public_path) for entry in configured
],
"effective": effective,
"auto_detected": bool(auto_detected),
},
{
"present": present,
"observer_alive": alive,
"scheduled": [_public_path(path) for path in scheduled],
},
)
def validate_source(text):
"""``{valid, label, path, code, reason}`` for one submitted path (blocking).
Writes nothing. An invalid path answers its code and reason with no
label or path, so a response never echoes what was typed.
"""
from ..config import GalleryConfig
from . import _public_path
code, reason = source_problem(text)
if code is not None:
return {
"valid": False,
"label": None,
"path": None,
"code": code,
"reason": reason,
}
resolved = GalleryConfig.resolve_gallery_root(text)
return {
"valid": True,
"label": source_label(resolved),
"path": _public_path(resolved),
"code": None,
"reason": None,
}
-221
View File
@@ -1,221 +0,0 @@
"""System routes: the read-only health check, the Overview and usage statistics.
- ``POST /prompt_manager/system/health`` starts the health check
(``database/health.py``) off-loop as a ``read`` job of kind ``health``: one
at a time, beside a scan, not cancellable and with no file outcomes. The
API keeps the last report for this session; ``GET`` returns it or null.
A report describes the database it was taken from: replacing the database
(a restore) forgets it, and a check that started before the swap is not
kept.
- ``GET /prompt_manager/system/overview`` is the System › Overview aggregate.
Every value is real or null: there is no backup status until backups report
one and no health result until a check ran; ``needs_review.pending`` is the
Needs review queue's pending total. The missing-thumbnail count is not part
of it: the page reads ``/images/thumbnails/status``, which walks every
source.
- ``GET /prompt_manager/system/statistics`` lists runs and ratings per
category, with the legacy ``/stats`` totals and the Overview's count of
rated images (images whose linked prompt is rated).
New routes answer errors with ``{success:false, code, error}``.
"""
import asyncio
import os
from collections import namedtuple
from aiohttp import web
from .job_routes import busy_response
from .jobs import HEALTH_JOB_KIND, JobBusy
HEALTH_JOB_TITLE = "Health check"
_HEALTH_STATUS_LINES = {
"passed": "Health check passed",
"issues": "Health check found issues",
"incomplete": "Health check incomplete",
}
# The last health report and the database generation it was taken from.
_StoredReport = namedtuple("_StoredReport", "generation report")
async def _gather_settled(*reads):
"""Await every read, then raise the first failure.
Unlike a plain gather, a failure never answers while the other executor
reads still run against the database.
"""
results = await asyncio.gather(*reads, return_exceptions=True)
for result in results:
if isinstance(result, BaseException):
raise result
return results
def _failure(message, status=500, code=None):
return web.json_response(
{"success": False, "code": code, "error": message}, status=status
)
def health_summary(report):
"""What the Overview shows of a report: when, the outcome, checks with issues."""
if report is None:
return None
return {
"checked_at": report["checked_at"],
"status": report["status"],
"issues": sum(1 for check in report["checks"] if check["status"] == "issues"),
}
class SystemRoutesMixin:
"""Mixin providing the /prompt_manager/system endpoints."""
def _register_system_routes(self, routes):
@routes.post("/prompt_manager/system/health")
async def start_health_check_route(request):
return await self.start_health_check(request)
@routes.get("/prompt_manager/system/health")
async def health_report_route(request):
return await self.get_health_report(request)
@routes.get("/prompt_manager/system/overview")
async def system_overview_route(request):
return await self.get_system_overview(request)
@routes.get("/prompt_manager/system/statistics")
async def system_statistics_route(request):
return await self.get_system_statistics(request)
# ── health ────────────────────────────────────────────────────────
def _health_roots(self):
"""The sources missing files are judged under.
The effective sources, plus every configured source that is not an
existing directory (an unmounted drive): the check then reports
missing files as unavailable instead of counting that source's links.
"""
from ..config import GalleryConfig
roots = [str(path) for path in self._get_all_output_dirs()]
for configured in GalleryConfig.MONITORING_DIRECTORIES:
try:
resolved = GalleryConfig.resolve_gallery_root(configured)
except (OSError, TypeError, ValueError):
continue
if not os.path.isdir(resolved):
roots.append(resolved)
return roots
def _database_replaced(self):
"""Forget what described the previous database (call after swapping ``db``)."""
self._db_generation += 1
self._health_report = None
def _current_health_report(self):
"""The last report, or None when none was kept for the current database."""
stored = self._health_report
generation = self._db_generation
if isinstance(stored, _StoredReport) and stored.generation == generation:
return stored.report
return None
async def start_health_check(self, request):
"""Start the health check job (202), or 409 naming the running check."""
generation = self._db_generation
db = self.db
try:
roots = await self._run_in_executor(self._health_roots)
except Exception:
self.logger.exception("Health check could not resolve the image sources")
return _failure("Failed to start the health check")
async def run(job):
job.update(status_line="Checking the database and image links")
report = await self._run_in_executor(db.run_health_checks, roots)
if generation == self._db_generation:
self._health_report = _StoredReport(generation, report)
else:
self.logger.info("Health report dropped: the database was replaced")
job.update(progress=100, status_line=_HEALTH_STATUS_LINES[report["status"]])
try:
job = self._jobs.start(
HEALTH_JOB_KIND,
HEALTH_JOB_TITLE,
admission="read",
cancellable=False,
scope={},
runner=run,
)
except JobBusy as busy:
return busy_response(busy)
return web.json_response({"success": True, "job": job.summary()}, status=202)
async def get_health_report(self, request):
return web.json_response(
{"success": True, "report": self._current_health_report()}
)
# ── overview ──────────────────────────────────────────────────────
async def get_system_overview(self, request):
from ..config import GalleryConfig
try:
(output_dirs, auto_detected), counts = await _gather_settled(
self._run_in_executor(self._output_dirs_with_origin),
self._run_in_executor(self.db.library_counts),
)
except Exception:
self.logger.exception("System overview failed")
return _failure("Failed to read the overview")
return web.json_response(
{
"success": True,
"library": {
"positive_prompts": counts["positive_prompts"],
"negative_variants": counts["negative_variants"],
},
"images": {
"indexed": counts["images"],
"rated": counts["rated_images"],
},
"health": health_summary(self._current_health_report()),
"backup": None, # no verified backup status exists yet (S8)
"needs_review": {"pending": counts["review_pending"]},
"sources": {
"configured": len(GalleryConfig.MONITORING_DIRECTORIES),
"effective": len(output_dirs),
"auto_detected": auto_detected,
},
},
headers={"Cache-Control": "no-store"},
)
# ── statistics ────────────────────────────────────────────────────
async def get_system_statistics(self, request):
try:
by_category, totals, counts = await _gather_settled(
self._run_in_executor(self.db.usage_by_category),
self._run_in_executor(self.db.get_statistics),
self._run_in_executor(self.db.library_counts),
)
except Exception:
self.logger.exception("System statistics failed")
return _failure("Failed to read statistics")
return web.json_response(
{
"success": True,
"by_category": by_category,
"totals": totals,
"rated_images": counts["rated_images"],
},
headers={"Cache-Control": "no-store"},
)
+25 -121
View File
@@ -31,6 +31,7 @@ routes = server_instance.routes
extension_uri = None # Will be set in __init__.py
import os # noqa: E402
from pathlib import Path # noqa: E402
from typing import Any, Dict, List, Optional, Tuple # noqa: E402
# Import logging system
@@ -50,14 +51,6 @@ config_logger = get_logger("prompt_manager.config")
# may be used as gallery roots in addition to ComfyUI's own directories.
EXTRA_GALLERY_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
# The one answer for every path outside the allowed folders. The filesystem
# root and the home directory answer it too, so no caller learns the server's
# layout (home path, user name) from a refusal (#249 S4).
OUTSIDE_ALLOWED_REASON = (
"Gallery root must be inside a ComfyUI directory (output, input, temp, user) "
f"or one listed in {EXTRA_GALLERY_ROOTS_ENV}"
)
# Environment variable overriding where config.json is read from and saved to.
CONFIG_PATH_ENV = "PROMPT_MANAGER_CONFIG_PATH"
@@ -96,29 +89,6 @@ def write_private_json(path: str, data: Any) -> None:
os.chmod(path, 0o600)
def _lexical_path(path: str) -> str:
"""``path`` normalised for comparison without touching the filesystem.
normpath collapses ``.`` and ``..`` as text and normcase folds case on
Windows; neither opens, stats nor resolves anything (no SMB request for a
UNC path). A symlink is only seen by :func:`_canonical_path` later.
"""
return os.path.normcase(os.path.normpath(path))
def _lexically_inside(path: str, parents: List[str]) -> bool:
"""Whether the normalised ``path`` is one of ``parents`` or below one.
Pure text on normalised spellings (:func:`_lexical_path` or
:func:`_canonical_path`): no filesystem access.
"""
sep = os.path.sep
return any(
path == parent or path.startswith(parent.rstrip(sep) + sep)
for parent in parents
)
def _is_filesystem_root(path: str) -> bool:
"""True for '/' on POSIX and drive roots such as 'C:\\' on Windows."""
return os.path.dirname(path) == path
@@ -298,12 +268,7 @@ class GalleryConfig:
"""
monitoring = new_config.get("monitoring", {})
if "enabled" in monitoring:
if isinstance(monitoring["enabled"], bool):
cls.MONITORING_ENABLED = monitoring["enabled"]
else: # a hand-edited "false" would otherwise switch it on
config_logger.warning(
"Ignoring gallery monitoring enabled: expected true or false"
)
cls.MONITORING_ENABLED = monitoring["enabled"]
if "directories" in monitoring:
directories = cls._validated_directories(monitoring["directories"])
if directories is not None:
@@ -372,56 +337,31 @@ class GalleryConfig:
Filesystem roots are never allowed as parents, even when listed in
the environment, because that would re-open the whole disk.
"""
return [canonical for canonical, _spelling in cls._allowed_parent_pairs()]
@classmethod
def _allowed_parent_pairs(cls) -> List[Tuple[str, str]]:
"""``(canonical, lexical spelling)`` of each allowed parent.
Only trusted directories (ComfyUI's own and the environment's) are
resolved here; the spelling is how ComfyUI or the environment wrote
the directory, so a submitted path can be compared without resolving it.
"""
pairs = []
seen = set()
parents = []
for candidate in _comfyui_directories() + _extra_gallery_roots():
try:
canonical = _canonical_path(candidate)
spelling = _lexical_path(os.path.abspath(candidate))
except (OSError, ValueError):
continue
if _is_filesystem_root(canonical) or not os.path.isdir(canonical):
continue
if canonical not in seen:
seen.add(canonical)
pairs.append((canonical, spelling))
return pairs
if canonical not in parents:
parents.append(canonical)
return parents
@classmethod
def path_anchors(cls) -> List[str]:
"""Canonical directories that relative gallery paths are resolved against.
The ComfyUI base directory comes first, followed by the parent of each
ComfyUI directory outside it (``--output-directory D:\\Renders``) and of
each directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so such a
directory is addressed by its own name (``Renders/sub``) rather than by
an absolute path, and two folders of one name stay apart. The same
anchors drive the public (relative) form of paths in API responses.
directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so an extra
root is addressed by its own name (``gallery/sub``) rather than by an
absolute path. The same anchors drive the public (relative) form of
paths in API responses.
"""
anchors = []
base = _comfyui_base_directory()
candidates = [base] if base else []
try:
base_key = _canonical_path(base) if base else None
except (OSError, ValueError):
base_key = None
for directory in _comfyui_directories():
try:
canonical = _canonical_path(directory)
except (OSError, ValueError):
continue
if base_key is None or not _lexically_inside(canonical, [base_key]):
candidates.append(os.path.dirname(canonical))
candidates.extend(os.path.dirname(root) for root in _extra_gallery_roots())
for candidate in candidates:
try:
@@ -459,69 +399,33 @@ class GalleryConfig:
(True, "") when the directory lies inside one of
:meth:`allowed_gallery_parents`; otherwise (False, reason).
"""
code, reason = cls.gallery_root_problem(path)
return code is None, reason
@classmethod
def gallery_root_problem(cls, path: Any) -> Tuple[Optional[str], str]:
"""``(code, reason)`` why ``path`` may not be a gallery root, else (None, "").
Codes: ``empty``, ``unresolvable``, ``outside-allowed``, ``not-found``
and ``not-directory``. No reason names the path.
Containment is decided twice. First lexically, with no filesystem
access at all: a path (UNC, ``\\\\?\\``, a device, anything) whose
normalised spelling lies outside every allowed parent answers
``outside-allowed`` before it is resolved, so the server never opens
or stats it (no SMB connection, no existence or timing oracle). Only
then is it resolved and checked again, which catches a symlink out.
The filesystem root and the home directory answer ``outside-allowed``
too: one answer for every caller (validate-source, the settings save,
config.json), and the detail goes to the log only.
"""
if not isinstance(path, str) or not path.strip():
return "empty", "Gallery root must be a non-empty path"
pairs = cls._allowed_parent_pairs()
spellings = [spelling for pair in pairs for spelling in pair]
try:
lexical = cls._lexical_candidates(path)
except (TypeError, ValueError):
return "unresolvable", "Gallery root could not be resolved"
if not any(_lexically_inside(candidate, spellings) for candidate in lexical):
return "outside-allowed", OUTSIDE_ALLOWED_REASON
return False, "Gallery root must be a non-empty path"
try:
canonical = cls.resolve_gallery_root(path)
except (OSError, ValueError):
return "unresolvable", "Gallery root could not be resolved"
return False, "Gallery root could not be resolved"
if _is_filesystem_root(canonical):
config_logger.info("Gallery root refused: a filesystem root")
return "outside-allowed", OUTSIDE_ALLOWED_REASON
return False, "Gallery root cannot be a filesystem root"
if canonical == _canonical_path(os.path.expanduser("~")):
config_logger.info("Gallery root refused: the home directory")
return "outside-allowed", OUTSIDE_ALLOWED_REASON
if not _lexically_inside(canonical, [canonical for canonical, _ in pairs]):
return "outside-allowed", OUTSIDE_ALLOWED_REASON
return False, "Gallery root cannot be the home directory"
if not os.path.exists(canonical):
return "not-found", "Gallery root does not exist"
return False, "Gallery root does not exist"
if not os.path.isdir(canonical):
return "not-directory", "Gallery root is not a directory"
return None, ""
return False, "Gallery root is not a directory"
@classmethod
def _lexical_candidates(cls, path: str) -> List[str]:
"""Every spelling ``path`` may resolve to, normalised as text (no IO).
candidate = Path(canonical)
for parent in cls.allowed_gallery_parents():
if candidate.is_relative_to(Path(parent)):
return True, ""
An absolute path is itself; a relative one is joined to each
:meth:`path_anchors` entry, as :meth:`resolve_gallery_root` tries them.
"""
path = path.strip()
if os.path.isabs(path):
return [_lexical_path(path)]
anchors = cls.path_anchors()
return [_lexical_path(os.path.join(anchor, path)) for anchor in anchors]
return (
False,
"Gallery root must be inside a ComfyUI directory "
f"(output, input, temp, user) or one listed in {EXTRA_GALLERY_ROOTS_ENV}",
)
class IntegrationConfig:
-2
View File
@@ -127,8 +127,6 @@ export async function openProduction(page,route,theme,motion='no-preference') {
await expect(page.locator('#primary-nav .nav-count')).toHaveCount(3);
// SAVED VIEWS' Favorites count (D51), the concept's starred images in the concept fixture.
await expect(page.locator('#saved-nav a[data-route="favorites"] .nav-count')).toHaveCount(1);
// SAVED VIEWS' Needs review count (ruling R5): the exact pending count.
await expect(page.locator('#saved-nav a[data-route="review"] .nav-count')).toHaveCount(1);
}
/** Computed outline of the focused element, after keyboard focus so `:focus-visible` applies. */
-212
View File
@@ -1,212 +0,0 @@
// Needs review against the approved concept (#249 PR F, plan Task F3, Mock inventory M4: app.js renderReview
// a749–760, style.css s407–415, s796–814 and their breakpoints, screenshot needs-review-tagging.png). Production runs
// on the review fixture's concept-shaped queue (workspace_review_fixture.py `concept: true`: the concept's first three
// image names and sizes, and the concept's first prompt on the open item), so every remaining difference is a recorded
// deviation applied by mapConceptReview (D3, D29, D119) or a test failure. Paired captures of every state are attached
// as artifacts; no screenshot baseline is added.
import { test, expect } from './fixtures.mjs';
import { openConcept, openProduction, inventoryOf, mutate, saveImageTags, diffInventory, SIZES, THEMES } from './concept-inventory.mjs';
import { reviewInventory, mapConceptReview, taggingInventory, mapConceptTagging, diffTaggingInventory } from './tagging-canonical-comparison.mjs';
const CONCEPT_FILE = 'alpine_00001.png';
const CONCEPT_TAGS = ['cinematic', 'blue_hour', 'architecture'];
const SURFACE = ['h2', 'p.small', 'div.row-actions.spaced-15', 'div.field', 'section.positive.prompt-section',
'section.negative.prompt-section', 'div.dialog-actions'];
// The fake model's server default instructions (tests/browser/server.py TestModel.default_prompt), D84.
const INSTRUCTIONS = 'Describe the image';
const panel = page => page.locator('#workspace-main section.review-panel');
const tagDialog = page => page.getByRole('dialog', { name: 'Tag image', exact: true });
async function seed(request, data = {}) {
await expect(await request.post('/__test__/review/reset', { data: { items: 3, concept: true, ...data } })).toBeOK();
}
/** Production's Needs review page with its first item's sampler paths loaded. */
async function openReview(page, theme) {
await openProduction(page, 'review', theme);
await expect(panel(page).getByRole('combobox', { name: 'Sampler context', exact: true })).toBeEnabled();
await settle(page);
}
async function settle(page) {
await page.mouse.move(0, 0);
await page.evaluate(async () => {
await document.fonts.ready;
await Promise.all([...document.images].map(image => image.decode().catch(() => {})));
});
}
const conceptInventory = page => page.evaluate(reviewInventory, '#main');
const productionInventory = page => page.evaluate(reviewInventory, '#workspace-main');
const node = (tree, prefix) => tree.tag?.startsWith(prefix) ? tree : tree.children?.map(child => node(child, prefix)).find(Boolean) ?? null;
/** Computed values of CSS tokens, read through a probe element in the open page. */
const tokens = (page, declarations) => page.evaluate(text => {
const probe = document.createElement('i');
probe.style.cssText = text;
document.body.append(probe);
const style = getComputedStyle(probe), values = [style.borderTopColor, style.backgroundColor];
probe.remove();
return values;
}, declarations);
async function compareAt(page, request, width, height, theme) {
await seed(request);
await page.setViewportSize({ width, height });
await openConcept(page, 'review', theme);
await settle(page);
const concept = await conceptInventory(page);
await openReview(page, theme);
const actual = await productionInventory(page);
return { concept, actual, reference: mapConceptReview(concept, actual) };
}
test.beforeEach(async ({ page }) => {
page.errors = [];
page.on('pageerror', error => page.errors.push(error.message));
});
test.afterEach(async ({ page, request }) => {
expect(page.errors).toEqual([]);
await request.post('/__test__/jobs/restart');
});
for (const [width, height] of SIZES) for (const theme of THEMES) {
test(`concept and production Metadata review match at ${width}x${height} ${theme}`, async ({ page, request }) => {
const { actual, reference } = await compareAt(page, request, width, height, theme);
expect(diffInventory(reference, actual), `${width}x${height} ${theme}`).toEqual([]);
// Pinned concept values (M4), so a mapping mistake on both sides cannot pass unseen.
const main = actual.main;
expect(main.children.map(child => child.tag)).toEqual(['div.heading', 'div.notice', 'div.gallery-layout']);
const notice = node(main, 'div.notice');
const [accent, soft] = await tokens(page, 'border-top:1px solid var(--ca-accent);background:var(--soft-blue)');
expect([notice.borderTopColor, notice.borderLeftColor, notice.backgroundColor]).toEqual([accent, accent, soft]);
expect([notice.borderTopWidth, notice.borderTopLeftRadius, notice.paddingTop, notice.paddingLeft, notice.marginBottom])
.toEqual(['1px', '6px', '15px', '18px', '22px']);
expect(notice.flexDirection).toBe(width <= 700 ? 'column' : 'row');
expect(notice.children.map(child => child.children.map(item => [item.tag, item.text])))
.toEqual([[['h2', '3 images need a decision'], ['p', 'Multiple sampler paths were found. The existing image links were preserved. Choose the correct context or leave them unresolved.']]]);
const layout = node(main, 'div.gallery-layout');
expect(layout.gridTemplateColumns).toBe(width <= 700 ? `${layout.width}px` : `${layout.width - 22 - 310}px 310px`);
const surface = node(main, 'section.review-panel');
expect(surface.children.map(child => child.tag)).toEqual(SURFACE);
expect(surface.children[0].text).toBe('Resolve the association');
expect(surface.children[1].text).toBe(CONCEPT_FILE);
const actions = surface.children.at(-1);
expect(actions.children.map(button => [button.text, button.primary])).toEqual([['Leave for later', false], ['Apply association', true]]);
expect(node(surface, 'select').options.at(-1)).toEqual(['Leave unresolved', false]);
expect(node(main, 'div.image-grid').children.map(card => node(card, 'h3').text))
.toEqual(['alpine_00001.png', 'dunes_00001.png', 'forest_00001.png']);
expect(actual.noPageScroll).toBe(true);
});
}
test('the gallery layout follows the concept at every breakpoint', async ({ page, request }) => {
await seed(request);
// 310px beside the queue, 275px and a 15px gap at ≤1250px, 280px at ≤950px, one column at ≤700px.
for (const [width, columns, gap] of [[1600, '310px', '22px'], [1200, '275px', '15px'], [900, '280px', '15px'], [390, null, '15px']]) {
await page.setViewportSize({ width, height: 1000 });
await openConcept(page, 'review', 'dark');
const concept = node((await conceptInventory(page)).main, 'div.gallery-layout');
await openReview(page, 'dark');
const actual = node((await productionInventory(page)).main, 'div.gallery-layout');
expect([actual.gridTemplateColumns, actual.columnGap], `${width}px`).toEqual([concept.gridTemplateColumns, concept.columnGap]);
expect([actual.gridTemplateColumns.split(' ').at(-1), actual.columnGap], `${width}px`).toEqual([columns ?? `${actual.width}px`, gap]);
expect(actual.gridTemplateColumns.split(' ').length, `${width}px`).toBe(width <= 700 ? 1 : 2);
}
});
test('SAVED VIEWS rows match the concept with Needs review after Favorites', async ({ page, request }) => {
await seed(request);
await page.setViewportSize({ width: 1600, height: 1100 });
await openConcept(page, 'review', 'dark');
const concept = (await inventoryOf(page, '#main')).chrome.navs.find(nav => nav.id === 'saved-nav');
await openReview(page, 'dark');
const actual = (await inventoryOf(page, '#workspace-main')).chrome.navs.find(nav => nav.id === 'saved-nav');
// Sample data: the concept stars three images; this queue's fixture stars none. Needs review counts the queue (3).
const reference = { ...concept, rows: concept.rows.map(row => row.text === 'Favorites' ? { ...row, count: { ...row.count, text: '0' } } : row) };
expect(diffInventory(reference, actual)).toEqual([]);
expect(actual.rows.map(row => [row.text, row.count?.text, row.current])).toEqual([['Favorites', '0', null], ['Needs review', '3', 'page']]);
// The count's auto start margin follows the row's own text; every other count token is shared.
const { text: favoriteCount, marginInlineStart: favoriteMargin, ...favoriteStyle } = actual.rows[0].count;
const { text: reviewCount, marginInlineStart: reviewMargin, ...reviewStyle } = actual.rows[1].count;
expect(reviewStyle).toEqual(favoriteStyle);
expect(actual.rows[1].icon.d).toBe('M4 3h16v18H4zM8 7h8M8 12h8M8 17h5');
});
test('review Tag dialog matches the needs-review-tagging reference', async ({ page, request }) => {
await seed(request);
await saveImageTags(request, `review/${CONCEPT_FILE}`, CONCEPT_TAGS);
await page.setViewportSize({ width: 1600, height: 1100 });
await openConcept(page, 'review', 'dark');
await page.locator('#main section.surface').getByRole('button', { name: 'Tag image', exact: true }).click();
await expect(tagDialog(page)).toBeVisible();
await settle(page);
const opened = await page.locator('dialog#modal').evaluate(taggingInventory);
await page.keyboard.press('Escape');
await openReview(page, 'dark');
await panel(page).getByRole('button', { name: 'Tag image', exact: true }).click();
await expect(tagDialog(page).locator('img.tag-image-preview')).toHaveAttribute('src', /\/image-annotations\/capture\//);
await expect(tagDialog(page).locator('#manual-image-tags')).toHaveValue(CONCEPT_TAGS.join(', '));
await settle(page);
const actual = await page.locator('dialog#modal').evaluate(taggingInventory);
const numberField = opened.modelPanel.fields.find(field => field.control.id === 'image-general-threshold');
const mapped = mapConceptTagging(opened, { instructions: INSTRUCTIONS, chip: opened.heading.size, numberField });
// D119: an undecided queue item names no prompt, so the heading and the preview's alt are the file name.
const reference = { ...mapped, heading: { ...mapped.heading, h3: { ...mapped.heading.h3, text: CONCEPT_FILE },
preview: { ...mapped.heading.preview, alt: CONCEPT_FILE } } };
expect(actual.scopeNote.text).toBe('Tags belong to this image. Resolve its prompt association separately in Needs review.');
expect(diffTaggingInventory(reference, actual)).toEqual([]);
await page.keyboard.press('Escape');
await expect(tagDialog(page)).toHaveCount(0);
});
test('the comparison notices a dropped notice border and reordered actions', async ({ page, request }) => {
const { actual, reference } = await compareAt(page, request, 1600, 1100, 'dark');
expect(diffInventory(reference, actual)).toEqual([]);
const undoBorder = await mutate(page, { css: '#workspace-main .notice { border: 0 !important; }' });
const borderless = await productionInventory(page);
expect(diffInventory(reference, borderless).map(item => item.path)).toEqual(expect.arrayContaining([
'main.children.1.borderTopWidth', 'main.children.1.borderTopStyle']));
await undoBorder();
await panel(page).locator('.dialog-actions').evaluate(row => row.prepend(row.lastElementChild));
const reordered = await productionInventory(page);
expect(diffInventory(reference, reordered).map(item => item.path)).toEqual(expect.arrayContaining([
'main.children.2.children.1.children.6.children.0.text', 'main.children.2.children.1.children.6.children.0.primary']));
await panel(page).locator('.dialog-actions').evaluate(row => row.prepend(row.lastElementChild));
expect(diffInventory(reference, await productionInventory(page))).toEqual([]);
});
// Paired captures for the PR (no baselines): the populated queue beside the concept in both themes and widths, then
// the states the concept lacks (conflict, limited item, unavailable file, empty queue) as production captures.
test('paired Needs review captures', async ({ page, request }, testInfo) => {
test.setTimeout(120_000);
const shot = async name => {
await settle(page);
const file = testInfo.outputPath(`${name}.png`);
await page.screenshot({ path: file, fullPage: true });
await testInfo.attach(name, { path: file, contentType: 'image/png' });
};
await seed(request);
for (const [width, height] of SIZES) for (const theme of THEMES) {
await page.setViewportSize({ width, height });
await openConcept(page, 'review', theme);
await shot(`concept-review-${width}-${theme}`);
await openReview(page, theme);
await shot(`production-review-${width}-${theme}`);
}
await page.setViewportSize({ width: 1600, height: 1100 });
await seed(request, { items: 0, conflict: true, limited: true, unavailable: true });
await openReview(page, 'dark');
await panel(page).getByRole('combobox', { name: 'Sampler context', exact: true }).selectOption({ index: 0 });
const decided = page.waitForResponse(item => /\/decision$/.test(item.url()));
await panel(page).getByRole('button', { name: 'Apply association', exact: true }).click();
expect((await decided).status()).toBe(409);
await expect(panel(page).locator('.warning-note')).toBeVisible();
await shot('production-review-conflict');
for (const [index, name] of [[1, 'limited'], [2, 'unavailable']]) {
await page.locator('#workspace-main .review-queue article').nth(index).getByRole('button', { name: /^Inspect / }).click();
await expect(panel(page).locator('.warning-note')).toBeVisible();
await shot(`production-review-${name}`);
}
await seed(request, { items: 0 });
await openProduction(page, 'review', 'dark');
await expect(page.locator('#workspace-main .notice h2')).toHaveText('No images need a decision');
await shot('production-review-empty');
});
-12
View File
@@ -19,9 +19,7 @@ from PIL import Image
from PIL.PngImagePlugin import PngInfo
from tests.browser.request_hold import register_request_holds
from tests.browser import slow_reset_watch
from tests.browser.workspace_activity_fixture import register_activity_commands
from tests.browser.workspace_system_fixture import register_system_commands
from tests.browser.workspace_concept_fixture import (
MAX_EXTRA_IMAGES as CONCEPT_MAX_EXTRA_IMAGES,
MAX_SEEDLESS_IMAGES as CONCEPT_MAX_SEEDLESS_IMAGES,
@@ -40,7 +38,6 @@ from tests.browser.workspace_library_editor_fixture import (
)
from tests.browser.workspace_vocabulary_fixture import register_vocabulary_commands
from tests.browser.workspace_tagging_fixture import register_tagging_commands
from tests.browser.workspace_review_fixture import register_review_commands
CANONICAL_ROOT = Path(__file__).resolve().parents[2]
CANONICAL_ASSETS = {
@@ -198,12 +195,9 @@ def create_app(root: Path) -> web.Application:
library_editor_commands = register_library_editor_commands(routes, api, root)
register_vocabulary_commands(routes, api, root)
register_tagging_commands(routes, api, root, model)
register_review_commands(routes, api, root)
reveal_stub = register_reveal_stub(routes, api, root)
register_activity_commands(app, routes, api)
register_system_commands(app, routes, api, root)
request_holds = register_request_holds(app, routes)
slow_reset_watch.install(app, api)
async def canonical_asset(request: web.Request) -> web.Response:
filename, content_type = CANONICAL_ASSETS[request.path]
@@ -228,8 +222,6 @@ def create_app(root: Path) -> web.Application:
for table in (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -595,13 +587,9 @@ def create_app(root: Path) -> web.Application:
return web.json_response("dark")
async def cleanup(application: web.Application) -> None:
# A scan of an ambiguous fixture file captures it for Needs review.
api.close_annotation_captures()
api.db.close_all()
seed()
# Open Needs review change streams end first, as on the real host.
app.on_shutdown.append(api.close_review_events)
app.on_cleanup.append(cleanup)
app.add_routes(routes)
return app
+5 -6
View File
@@ -68,12 +68,12 @@ function remainingFooter(footer,spans) {
const OVERVIEW_GLYPH='M3 11 12 4l9 7M5 10v10h14V10M10 20v-6h4v6';
/**
* The concept's SAVED VIEWS rows (app.js a331–335) as production lists them: Favorites with its count (D51), Needs
* review with the exact pending count (ruling R5), then the saved views. The concept's "3" is sample data: the concept
* fixture records no review items, so production's count reads "0".
* The concept's SAVED VIEWS rows (app.js a331–335) as production lists them: Favorites with its count (D51), then the
* saved views. D50: no "Needs review" row until the tag-review slice (maintainer ruling 2026-10-21).
*/
function savedNav(nav) {
return {...nav,rows:nav.rows.map(row=>row.text==='Needs review'?{...row,count:{...row.count,text:'0'}}:row)};
const rows=nav.rows.filter(row=>row.text!=='Needs review');
return {...nav,rows:rows.map((row,index)=>index===rows.length-1?{...row,gapAfter:null}:row)};
}
/** Recorded shell deviations only (spec §8). Everything else in the concept inventory must equal production. */
@@ -243,9 +243,8 @@ for(const [width,height] of SIZES)test(`a saved view row matches the concept's $
await openProduction(page,'images','dark');
const actual=await save('Save','#workspace-main');
expect(diffInventory(mapShellConcept(reference).chrome.navs,actual.chrome.navs)).toEqual([]);
const referenceRows=reference.chrome.navs.find(nav=>nav.id==='saved-nav').rows;
expect(actual.chrome.navs.find(nav=>nav.id==='saved-nav').rows.map(row=>[row.text,row.count]))
.toEqual([['Favorites',referenceRows[0].count],['Needs review',{...referenceRows[1].count,text:'0'}],['Golden hour',null]]);
.toEqual([['Favorites',reference.chrome.navs.find(nav=>nav.id==='saved-nav').rows[0].count],['Golden hour',null]]);
});
test('the inventory reports a missing shell element by its path instead of failing',async({page})=>{
-237
View File
@@ -1,237 +0,0 @@
"""Report a slow test reset with what the server was doing while it waited.
A reset normally answers within about a second. Once one runs past
SLOW_RESET_SECONDS, this writes to stderr, while the reset is still waiting:
- every thread's stack, so a reset waiting on a lock, a full executor or a
slow disk shows where it waits and who holds it;
- the default executor's threads and queued work;
- how many database connections are open, and the WAL size;
- on Linux, each thread's scheduler state and the kernel function it sleeps
in (``D`` with a journal or writeback wait is a slow disk flush, a nanosleep
is SQLite's busy handler, a futex is a Python lock or an idle worker).
When the reset ends it writes how long it took and, for each executor call it
made, how long the call queued and how long it ran.
Every report line starts with REPORT_PREFIX. The worker fixture
(tools/vendor/start-browser-server.mjs) copies those lines to the job log,
so a slow reset is visible even in a test that still passes.
"""
import asyncio
import contextvars
import faulthandler
import os
import sys
import tempfile
import threading
import time
from typing import Callable, TextIO
from aiohttp import web
SLOW_RESET_SECONDS = 2.0
REPORT_PREFIX = "pm-slow-reset| "
_current: contextvars.ContextVar["ResetWatch | None"] = contextvars.ContextVar(
"pm_reset_watch", default=None
)
def is_reset_path(path: str) -> bool:
return path.startswith("/__test__/") and path.endswith("/reset")
class ResetWatch:
"""Time one reset; dump the server's state if it passes the threshold."""
def __init__(
self,
label: str,
describe: Callable[[], list[str]],
*,
threshold: float = SLOW_RESET_SECONDS,
stream: TextIO | None = None,
) -> None:
self.label = label
self.threshold = threshold
self._describe = describe
self._stream = stream
self._start = time.monotonic()
self._calls: list[str] = []
self._calls_lock = threading.Lock()
self._stalled = threading.Event()
self._timer = threading.Timer(threshold, self._report_stall)
self._timer.daemon = True
self._timer.name = "pm-slow-reset-watch"
def start(self) -> "ResetWatch":
self._timer.start()
return self
def elapsed(self) -> float:
return time.monotonic() - self._start
def record_call(self, name: str, queued: float, ran: float) -> None:
with self._calls_lock:
self._calls.append(
f"executor call {name}: queued {queued:.3f}s, ran {ran:.3f}s"
)
def finish(self, outcome: str) -> None:
self._timer.cancel()
elapsed = self.elapsed()
# A coarse clock (Windows) can fire the timer just short of the
# threshold; a reset that was reported always gets its summary.
if elapsed < self.threshold and not self._stalled.is_set():
return
with self._calls_lock:
calls = list(self._calls)
self._write([f"{self.label} {outcome} after {elapsed:.3f}s", *calls])
def _report_stall(self) -> None:
self._stalled.set()
lines = [f"{self.label} still running after {self.elapsed():.3f}s"]
try:
lines.extend(self._describe())
except Exception as error: # a diagnostic never breaks the reset
lines.append(f"state unavailable: {error!r}")
lines.extend(thread_stacks())
self._write(lines)
def _write(self, lines: list[str]) -> None:
stream = self._stream or sys.stderr
text = "".join(
f"{REPORT_PREFIX}{part}\n" for line in lines for part in line.splitlines()
)
try:
stream.write(text)
stream.flush()
except (OSError, ValueError):
pass
def thread_stacks() -> list[str]:
"""Every thread's stack, from faulthandler.
Formatting sys._current_frames() showed a thread blocked in SQLite as idle
(Python 3.14); faulthandler reports where each thread really is.
"""
# Before Python 3.14 faulthandler prints ids only; name each id here.
names = [
f"thread 0x{thread.ident:016x} is {thread.name}"
for thread in threading.enumerate()
if thread.ident is not None
]
with tempfile.TemporaryFile() as dump:
faulthandler.dump_traceback(file=dump, all_threads=True)
dump.seek(0)
return [*names, *dump.read().decode("utf-8", "replace").splitlines()]
def executor_state(loop: asyncio.AbstractEventLoop) -> list[str]:
executor = getattr(loop, "_default_executor", None)
if executor is None:
return ["default executor: not started"]
threads = getattr(executor, "_threads", ())
queue = getattr(executor, "_work_queue", None)
queued = queue.qsize() if queue is not None else "?"
return [
f"default executor: {len(threads)} of "
f"{getattr(executor, '_max_workers', '?')} threads, {queued} queued"
]
def database_state(model) -> list[str]:
"""How many connections are open, and the WAL size.
Never touches another thread's connection: a connection busy in SQLite
holds its mutex, and reading even ``in_transaction`` from here would block
on it while holding the GIL, freezing every thread until that query ends.
"""
with model._conn_lock:
count = len(model._connections)
wal = model.db_path + "-wal"
size = os.path.getsize(wal) if os.path.exists(wal) else 0
return [f"database: {count} connections, WAL {size} bytes"]
def kernel_wait_state() -> list[str]:
"""Each thread's state and kernel wait channel, where /proc has them."""
lines = []
for thread in threading.enumerate():
task = f"/proc/self/task/{thread.native_id}"
try:
with open(f"{task}/stat", encoding="ascii") as stat:
state = stat.read().rsplit(")", 1)[1].split()[0]
with open(f"{task}/wchan", encoding="ascii") as wchan:
channel = wchan.read().strip() or "0"
except (OSError, IndexError):
continue
# The id matches the "Thread 0x..." headers of the stacks below.
lines.append(
f"kernel: {thread.name} (0x{thread.ident:016x}) state {state}, "
f"waiting in {channel}"
)
return lines
def install(app: web.Application, api) -> None:
"""Watch every reset route, and time the executor calls each one makes."""
run_in_executor = api._run_in_executor
async def timed_run_in_executor(func, *args, **kwargs):
watch = _current.get()
if watch is None:
return await run_in_executor(func, *args, **kwargs)
queued_at = time.monotonic()
started: list[float] = []
def timed(*call_args, **call_kwargs):
started.append(time.monotonic())
return func(*call_args, **call_kwargs)
try:
return await run_in_executor(timed, *args, **kwargs)
finally:
end = time.monotonic()
begin = started[0] if started else end
watch.record_call(
getattr(func, "__qualname__", repr(func)),
begin - queued_at,
end - begin,
)
api._run_in_executor = timed_run_in_executor
@web.middleware
async def watch_resets(request: web.Request, handler):
if not is_reset_path(request.path):
return await handler(request)
loop = asyncio.get_running_loop()
def describe() -> list[str]:
return [
*executor_state(loop),
*database_state(api.db.model),
*kernel_wait_state(),
]
watch = ResetWatch(
f"{request.method} {request.path}", describe, threshold=SLOW_RESET_SECONDS
).start()
token = _current.set(watch)
outcome = "failed"
try:
response = await handler(request)
outcome = f"answered {response.status}"
return response
except web.HTTPException as error:
outcome = f"answered {error.status}"
raise
finally:
_current.reset(token)
watch.finish(outcome)
app.middlewares.append(watch_resets)
-777
View File
@@ -1,777 +0,0 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { SIZES, THEMES, seedConcept, openConcept, openProduction, diffInventory, mutate } from './concept-inventory.mjs';
// #249 System S2 Task 6: System's shell and its Overview section (app.js renderSystem 658–706), the Statistics dialog
// ("Usage statistics", a1600–1604) and the health report (a1439–1456) against the immutable concept, on the
// concept-shaped real fixture. Each concept measurement is mapped only by the recorded deviations (D92–D96, D121,
// D122, D123), each commented where it applies; after that the difference must be empty, and a mutation of a
// row, a label or a token must be noticed.
test.use({ locale: 'en-US', timezoneId: 'UTC' });
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.conceptErrors = [];
page.on('pageerror', error => page.conceptErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/system/reset');
expect(page.conceptErrors).toEqual([]);
});
/** Runs in the page; self-contained. System's heading, section nav, Overview body and Save row under `mainSelector`. */
function measureSystem(mainSelector) {
const main = document.querySelector(mainSelector);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const box = (node, origin = main) => {
const rect = node.getBoundingClientRect(), base = origin.getBoundingClientRect();
return { left: Math.round(rect.left - base.left), width: Math.round(rect.width) };
};
// A surface's top: from `.two-col` beside its neighbour, else from the bottom of the surface above it. This pins the
// concept's `.surface + .surface` 18px offset, which concept.css applies inside `.two-col` too (copied, not ruled).
const top = node => {
const rect = node.getBoundingClientRect(), previous = node.previousElementSibling?.getBoundingClientRect();
if (previous && rect.top >= previous.bottom) return Math.round(rect.top - previous.bottom);
return Math.round(rect.top - node.parentElement.getBoundingClientRect().top);
};
const text = node => node?.textContent.trim() ?? null;
const button = node => ({ text: text(node), className: node.className,
...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopWidth', 'borderTopColor',
'borderTopLeftRadius', 'opacity', 'cursor') });
const heading = main.querySelector(':scope > .heading');
const nav = main.querySelector(':scope > .section-nav');
const layout = main.querySelector(':scope > .settings-layout');
const metrics = layout.querySelector(':scope > .summary-metrics');
const columns = layout.querySelector(':scope > .two-col');
const save = layout.querySelector(':scope > .save-row');
const firstMetric = metrics.querySelector('.metric');
const firstSurface = columns.querySelector('.surface');
const firstOperation = firstSurface.querySelector('.operation');
return {
tree: [...main.children].map(name),
heading: {
title: text(heading.querySelector('h1')), description: text(heading.querySelector('p.muted')),
actions: [...(heading.querySelector('.heading-actions')?.children ?? [])].map(name),
},
nav: {
role: nav.getAttribute('role'), label: nav.getAttribute('aria-label'),
...css(nav, 'display', 'flexWrap', 'columnGap', 'rowGap', 'paddingBottom', 'marginBottom', 'borderBottomWidth', 'borderBottomColor'),
box: box(nav),
buttons: [...nav.children].map(item => ({ pressed: item.getAttribute('aria-pressed'), ...button(item) })),
},
layout: { tree: [...layout.children].map(name), ...css(layout, 'maxWidth'), box: box(layout) },
metrics: {
...css(metrics, 'display', 'gridTemplateColumns', 'columnGap', 'rowGap', 'marginBottom'), box: box(metrics),
items: [...metrics.children].map(item => ({ tree: [...item.children].map(name), value: text(item.querySelector('strong')),
label: text(item.querySelector('span')), box: box(item) })),
metric: css(firstMetric, 'paddingTop', 'paddingLeft', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius', 'backgroundColor'),
strong: css(firstMetric.querySelector('strong'), 'display', 'fontSize', 'fontWeight', 'color'),
span: css(firstMetric.querySelector('span'), 'fontSize', 'color', 'marginTop'),
},
columns: {
...css(columns, 'display', 'gridTemplateColumns', 'columnGap', 'rowGap'), box: box(columns),
surfaces: [...columns.children].map(surface => ({ tree: [...surface.children].map(name), box: box(surface), top: top(surface),
heading: text(surface.querySelector(':scope > h2')),
rows: [...surface.querySelectorAll(':scope > .operation')].map(row => ({ tree: [...row.children].map(name),
title: text(row.querySelector('h3')), description: text(row.querySelector('p')),
action: button(row.querySelector('.row-actions > button')) })) })),
surface: css(firstSurface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius'),
h2: css(firstSurface.querySelector('h2'), 'fontSize', 'fontWeight', 'color', 'marginBottom'),
operation: css(firstOperation, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom',
'borderTopWidth', 'borderTopColor'),
h3: css(firstOperation.querySelector('h3'), 'fontSize', 'fontWeight', 'color'),
p: css(firstOperation.querySelector('p'), 'fontSize', 'color', 'marginTop'),
actions: css(firstOperation.querySelector('.row-actions'), 'display', 'columnGap', 'flexShrink'),
},
save: {
tree: [...save.children].map(name), ...css(save, 'display', 'alignItems', 'justifyContent', 'columnGap', 'marginTop', 'paddingTop'),
status: { text: text(save.querySelector('#settings-status')), ...css(save.querySelector('#settings-status'), 'fontSize', 'color') },
button: button(save.querySelector('button')),
},
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
/** Runs in the page: the open dialog's title, body sequence, tables, operation rows and actions. */
function measureDialog() {
const dialog = document.querySelector('dialog#modal[open]');
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const body = dialog.querySelector('#modal-body');
const firstCell = selector => body.querySelector(selector);
const list = body.querySelector(':scope > .operation-list');
const actions = body.querySelector(':scope > .dialog-actions');
return {
title: dialog.querySelector('#modal-title').textContent,
body: [...body.children].map(name),
tables: [...body.querySelectorAll(':scope > table')].map(table => ({
caption: table.caption?.textContent ?? null,
head: [...table.tHead.rows[0].cells].map(cell => [cell.localName, cell.scope, cell.textContent]),
rows: [...table.tBodies[0].rows].map(row => [...row.cells].map(cell => cell.textContent)),
})),
tableStyle: firstCell(':scope > table') && {
table: css(firstCell(':scope > table'), 'width', 'borderCollapse', 'fontSize', 'marginTop'),
caption: css(firstCell(':scope > table caption'), 'textAlign', 'fontSize', 'fontWeight', 'color', 'paddingBottom'),
th: css(firstCell(':scope > table thead th'), 'textAlign', 'fontSize', 'fontWeight', 'color', 'paddingTop', 'paddingLeft',
'borderBottomWidth', 'borderBottomColor'),
td: css(firstCell(':scope > table tbody td'), 'textAlign', 'fontSize', 'color', 'paddingTop', 'paddingLeft',
'borderBottomWidth', 'borderBottomColor'),
},
operations: list && {
...css(list, 'display', 'rowGap', 'marginTop'),
rows: [...list.children].map(row => ({ tree: [...row.children].map(name), title: row.querySelector('h3').textContent,
chip: row.querySelector('.chip').textContent })),
operation: css(list.querySelector('.operation'), 'display', 'alignItems', 'justifyContent', 'paddingTop', 'borderTopWidth'),
h3: css(list.querySelector('.operation h3'), 'fontSize', 'fontWeight', 'color'),
chip: css(list.querySelector('.operation .chip'), 'fontSize', 'color', 'backgroundColor', 'borderTopLeftRadius', 'paddingLeft'),
},
actions: actions && {
...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
buttons: [...actions.children].map(item => ({ text: item.textContent, className: item.className,
...css(item, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft') })),
},
};
}
/** Runs in the page: a `.notice` (Maintenance's "Check first…") with its copy. */
function measureNotice(mainSelector) {
const notice = document.querySelector(`${mainSelector} .settings-layout > .notice`);
const css = (node, ...keys) => Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]]));
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const run = notice.querySelector(':scope > button');
return {
tree: [...notice.children].map(name), text: [...notice.querySelectorAll(':scope > div > *')].map(node => [node.localName, node.textContent]),
frame: css(notice, 'display', 'alignItems', 'justifyContent', 'columnGap', 'borderTopWidth', 'borderTopColor', 'backgroundColor',
'borderTopLeftRadius', 'paddingTop', 'paddingLeft', 'marginBottom'),
h2: css(notice.querySelector('h2'), 'fontSize', 'fontWeight', 'color'),
p: css(notice.querySelector('p'), 'fontSize', 'color', 'marginTop'),
run: { text: run.textContent, className: run.className, ...css(run, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft') },
};
}
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
const number = value => value.toLocaleString('en-US');
const ratingText = value => value === null ? 'Not rated' : `${number(value)} / 5`;
/** The concept's shell and Overview with the recorded deviations applied, given the server's real overview. */
function mapOverviewConcept(reference, overview) {
const relabel = { 'Database integrity': 'Not checked', 'Positive prompts': number(overview.library.positive_prompts),
'Negative variants': number(overview.library.negative_variants), Images: number(overview.images.indexed) };
const rows = {
// D93: the real pending count; Review metadata opens Needs review (the concept's go-review).
'3 ambiguous image links': row => ({ ...row,
title: `${number(overview.needs_review.pending)} ambiguous image link${overview.needs_review.pending === 1 ? '' : 's'}` }),
// D93: no backup has been recorded (verified backups arrive with S8).
'Backup status': row => ({ ...row, description: 'No retained backup record.' }),
// D93: there is no paging, so the shared settings are grid and density.
'One display configuration': row => ({ ...row, description: 'Grid and density settings are shared across views.' }),
// D93: the real sources, not samples.
'2 sample output sources': row => ({ ...row, title: `${number(overview.sources.effective)} output sources` }),
// D96: the dialog shows runs and ratings by category and library totals, not prompt, tag or date usage.
Statistics: row => ({ ...row, description: 'Runs and ratings by category, with library totals.' }),
};
return {
...reference,
// D92: no "Sample configuration" chip.
heading: { ...reference.heading, actions: [] },
// D122: the concept's literal label is its prototype action name ("system-tab sections").
nav: { ...reference.nav, label: 'System sections' },
metrics: { ...reference.metrics, items: reference.metrics.items.map(item => ({ ...item, value: relabel[item.label] })) },
columns: { ...reference.columns, surfaces: reference.columns.surfaces.map(surface => ({ ...surface,
rows: surface.rows.map(row => (rows[row.title] ?? (value => value))(row)) })) },
save: { ...reference.save,
// D92: truthful status copy, and "Save preferences" is aria-disabled (dimmed) while nothing is unsaved.
status: { ...reference.save.status, text: 'No unsaved changes.' },
button: { ...reference.save.button, opacity: '0.55', cursor: 'not-allowed' } },
};
}
async function conceptOverview(page, theme) {
await openConcept(page, 'system', theme);
await expect(page.locator('#main .summary-metrics')).toBeVisible();
await page.mouse.move(0, 0);
return page.evaluate(measureSystem, '#main');
}
async function productionOverview(page, theme) {
await openProduction(page, 'system/overview', theme);
await expect(page.locator('#workspace-main .summary-metrics')).toBeVisible();
// The thumbnail row is its own read (I2): compare once it answered.
await expect(page.locator('#workspace-main .operation h3').filter({ hasText: 'thumbnails' })).toHaveText('No missing thumbnails');
await page.mouse.move(0, 0);
return page.evaluate(measureSystem, '#workspace-main');
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`system shell and overview match the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const overview = await serverJson(request, '/prompt_manager/system/overview');
// The fixture's thumbnails all exist, so the concept's "No missing thumbnails" is also the real title.
expect((await serverJson(request, '/prompt_manager/images/thumbnails/status')).pending).toBe(0);
const concept = await conceptOverview(page, theme);
expect(concept.heading.actions).toEqual(['span.chip']);
// Paired captures are artifacts only; there are no PNG baselines.
await page.screenshot({ path: testInfo.outputPath(`concept-system-${theme}-${width}.png`), fullPage: true });
const production = await productionOverview(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-system-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapOverviewConcept(concept, overview), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
// The recorded differences, by path: D93 the review row (its real count), the backup row, the
// preferences row and the sources row, and metric 4; D96 the statistics row; D92 the chip, the status and the dimmed
// Save; D122 the nav label. (The fixture's first three metrics are the concept's own counts.)
const UNMAPPED_PATHS = [
'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
'columns.surfaces.1.rows.0.description', 'columns.surfaces.1.rows.1.title', 'columns.surfaces.1.rows.2.description',
'heading.actions.0', 'heading.actions.length', 'metrics.items.3.value', 'nav.label',
'save.button.cursor', 'save.button.opacity', 'save.status.text',
];
test('system comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const overview = await serverJson(request, '/prompt_manager/system/overview');
const concept = await conceptOverview(page, 'dark');
const expected = mapOverviewConcept(concept, overview);
const production = await productionOverview(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs exactly where a deviation is recorded, and nowhere else.
expect(diffInventory(concept, production).map(item => item.path).sort()).toEqual(UNMAPPED_PATHS);
const measure = () => page.evaluate(measureSystem, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed row.
const undoRow = await mutate(page, { remove: ['#workspace-main .two-col > .surface:first-child > .operation:nth-of-type(2)'] });
expect(await paths()).toEqual(expect.arrayContaining(['columns.surfaces.0.rows.length', 'columns.surfaces.0.tree.length']));
await undoRow();
// A changed label.
await page.evaluate(() => { document.querySelector('#workspace-main .metric span').textContent = 'Positive prompt'; });
expect(await paths()).toEqual(['metrics.items.0.label']);
await page.evaluate(() => { document.querySelector('#workspace-main .metric span').textContent = 'Positive prompts'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['metrics.metric.borderTopColor', 'columns.surface.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/** D96: the concept's table, its rows the real ones; then "Library totals" in the same table markup. */
function mapStatisticsConcept(reference, statistics) {
const [runs] = reference.tables;
return {
...reference,
// No sample note (prototype copy); a second table follows the first.
body: ['table', 'table'],
tables: [
{ ...runs, rows: statistics.by_category.map(row => [row.category ?? 'Uncategorized', number(row.runs), ratingText(row.average_rating)]) },
{ caption: 'Library totals', head: [['th', 'col', 'Total'], ['th', 'col', 'Count']],
rows: [['Prompts', number(statistics.totals.total_prompts)], ['Tags', number(statistics.totals.total_tags)],
['Images', number(statistics.totals.total_images)], ['Rated images', number(statistics.rated_images)]] },
],
};
}
// The fixture's image sources and each check's real chip on it: nothing is damaged, Needs review does not exist yet (Q6).
const FIXTURE_SOURCES = ['output', 'input'];
const FIXTURE_CHIPS = Object.freeze({ 'Integrity check': 'Passed', 'Foreign keys & pair references': 'Passed',
'Canonical hashes': 'Passed', 'Pair uniqueness': 'Passed', 'Missing files': 'Passed', 'Ambiguous metadata': 'Not available yet',
Consistency: 'Passed' });
const chipOf = title => {
if (!(title in FIXTURE_CHIPS)) throw new Error(`No expected chip for ${title}`);
return FIXTURE_CHIPS[title];
};
/**
* D95: the concept's health report from its own five rows, labels verbatim, titled "Health report". Only the D95 rows
* are inserted where they are recorded (Pair uniqueness after Canonical hashes, Consistency last); Missing files has a
* row per image source when there are several; each chip is the fixture's expected result.
*/
function mapHealthConcept(reference, sources = FIXTURE_SOURCES) {
const [first] = reference.operations.rows;
const added = title => ({ ...first, title, chip: chipOf(title) });
const rows = reference.operations.rows.flatMap(row => {
const mapped = row.title === 'Missing files' && sources.length > 1
? sources.map(source => ({ ...row, title: `Missing files · ${source}`, chip: chipOf(row.title) }))
: [{ ...row, chip: chipOf(row.title) }];
return row.title === 'Canonical hashes' ? [...mapped, added('Pair uniqueness')] : mapped;
});
return {
...reference,
title: 'Health report',
// No sample note (prototype copy); D95 adds the checked line after the list.
body: ['div.operation-list', 'p.small', 'div.dialog-actions'],
operations: { ...reference.operations, rows: [...rows, added('Consistency')] },
};
}
// Unmapped, the health report differs exactly where D95 is recorded: the title, the body sequence (no sample note, the
// checked line), the rows after Canonical hashes (Pair uniqueness inserted, Missing files per source) and the chips of
// the concept's sample findings.
const UNMAPPED_HEALTH_PATHS = [
'body.0', 'body.1',
...[3, 4].flatMap(row => [`operations.rows.${row}.chip`, `operations.rows.${row}.title`]),
// The concept's five rows end at row 4; rows 5–7 exist only with the D95 rows and the second source.
...[5, 6, 7].flatMap(row => ['chip', 'title', 'tree.0', 'tree.1', 'tree.length'].map(key => `operations.rows.${row}.${key}`)),
'operations.rows.length', 'title',
];
async function conceptDialogs(page, theme, testInfo) {
await openConcept(page, 'system', theme);
await page.locator('#main').getByRole('button', { name: 'Explore', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics');
await page.mouse.move(0, 0);
const statistics = await page.evaluate(measureDialog);
await page.screenshot({ path: testInfo.outputPath(`concept-statistics-${theme}.png`) });
await page.keyboard.press('Escape');
await page.locator('#main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click();
const notice = await page.evaluate(measureNotice, '#main');
await page.locator('#main .notice').getByRole('button', { name: 'Run health checks', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Sample health report');
await page.mouse.move(0, 0);
await page.screenshot({ path: testInfo.outputPath(`concept-health-${theme}.png`) });
return { statistics, notice, health: await page.evaluate(measureDialog) };
}
async function productionDialogs(page, theme, testInfo) {
await openProduction(page, 'system/overview', theme);
await page.locator('#workspace-main').getByRole('button', { name: 'Explore', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics');
await page.mouse.move(0, 0);
const statistics = await page.evaluate(measureDialog);
await page.screenshot({ path: testInfo.outputPath(`production-statistics-${theme}.png`) });
await page.keyboard.press('Escape');
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click();
await expect(page.locator('#workspace-main .notice')).toBeVisible();
const notice = await page.evaluate(measureNotice, '#workspace-main');
await page.locator('#workspace-main .notice').getByRole('button', { name: 'Run health checks', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Health report');
await page.mouse.move(0, 0);
await page.screenshot({ path: testInfo.outputPath(`production-health-${theme}.png`) });
return { statistics, notice, health: await page.evaluate(measureDialog) };
}
for (const theme of THEMES) test(`statistics and health dialogs match the concept ${theme}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const concept = await conceptDialogs(page, theme, testInfo);
expect(concept.statistics.body).toEqual(['p.demo-note', 'table']);
expect(concept.health.body).toEqual(['p.demo-note', 'div.operation-list', 'div.dialog-actions']);
const production = await productionDialogs(page, theme, testInfo);
const statistics = await serverJson(request, '/prompt_manager/system/statistics');
const { report } = await serverJson(request, '/prompt_manager/system/health');
// The fixture the expected chips describe: its two sources, and the concept's checks plus the D95 ones.
const missing = report.checks.find(check => check.id === 'missing_files');
expect(missing.sources.map(source => source.name)).toEqual(FIXTURE_SOURCES);
expect(diffInventory(mapStatisticsConcept(concept.statistics, statistics), production.statistics)).toEqual([]);
expect(diffInventory(mapHealthConcept(concept.health), production.health)).toEqual([]);
// D123: Maintenance's notice is the concept's own, delivered ahead of the rest of the section.
expect(diffInventory(concept.notice, production.notice)).toEqual([]);
// Unmapped, the dialogs differ only where D95 and D96 are recorded.
expect(diffInventory(concept.health, production.health).map(item => item.path).sort()).toEqual(UNMAPPED_HEALTH_PATHS);
expect(diffInventory(concept.statistics, production.statistics).map(item => item.path.split('.')[0])
.filter((part, index, all) => all.indexOf(part) === index).sort()).toEqual(['body', 'tables']);
// Mutation check: a changed chip token in the health report is noticed, then undone to an exact match again.
const undo = await mutate(page, { css: '#modal .operation .chip { background: rgb(255, 0, 0); }' });
expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog)).map(item => item.path))
.toEqual(['operations.chip.backgroundColor']);
await undo();
expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog))).toEqual([]);
// A renamed check (database/health.py's label) is noticed against the concept's own label.
await page.evaluate(() => { document.querySelector('#modal .operation h3').textContent = 'Integrity checks'; });
expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog)).map(item => item.path))
.toEqual(['operations.rows.0.title']);
await page.evaluate(() => { document.querySelector('#modal .operation h3').textContent = 'Integrity check'; });
// And a removed "Library totals" table and a changed caption in the statistics dialog.
await page.keyboard.press('Escape');
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click();
await page.locator('#workspace-main').getByRole('button', { name: 'Explore', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics');
const expectedStatistics = mapStatisticsConcept(concept.statistics, statistics);
const undoTable = await mutate(page, { remove: ['#modal-body > table:nth-of-type(2)'] });
expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path))
.toEqual(expect.arrayContaining(['body.length', 'tables.length']));
await undoTable();
await page.evaluate(() => { document.querySelector('#modal-body caption').textContent = 'Runs per category'; });
expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path)).toEqual(['tables.0.caption']);
});
/** Runs in the page: Appearance's surface, every row's text and control, and the concept's row geometry and tokens. */
function measureAppearance(mainSelector) {
const layout = document.querySelector(`${mainSelector} .settings-layout`);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const surface = layout.querySelector(':scope > section.surface');
const control = row => {
const select = row.querySelector(':scope > select');
if (select) return { kind: 'select', id: select.id, options: [...select.options].map(option => option.textContent),
selected: select.selectedOptions[0]?.textContent ?? null };
const input = row.querySelector(':scope > input');
if (input) return { kind: `input.${input.type}`, id: input.id, value: input.value };
const toggle = row.querySelector(':scope > * input[type="checkbox"]');
if (toggle) return { kind: 'switch', id: toggle.id, wrapper: name(toggle.parentElement), checked: toggle.checked,
...css(toggle.parentElement, 'width', 'height') };
const chip = row.querySelector(':scope > .chip');
return chip ? { kind: name(chip), text: text(chip), ...css(chip, 'color', 'backgroundColor', 'fontSize') } : null;
};
const rows = [...surface.querySelectorAll(':scope > .setting-row')];
const operations = [...surface.querySelectorAll(':scope > .operation')];
const firstRow = rows[0];
return {
layout: [...layout.children].map(name),
surface: { tree: [...surface.children].map(name), heading: text(surface.querySelector(':scope > h2')),
intro: text(surface.querySelector(':scope > p')),
...css(surface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius') },
rows: rows.map(row => ({ tree: [...row.children].map(name), text: [...row.firstElementChild.children].map(node => [node.localName, text(node),
node.localName === 'label' ? node.htmlFor : null]), control: control(row) })),
operations: operations.map(row => ({ tree: [...row.children].map(name), title: text(row.querySelector('h3')),
description: text(row.querySelector('p')), action: text(row.querySelector('.row-actions > button')) })),
row: css(firstRow, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', 'borderTopWidth',
'borderTopColor', 'marginTop'),
label: css(firstRow.querySelector('label'), 'fontSize', 'fontWeight', 'color'),
description: css(firstRow.querySelector('p'), 'fontSize', 'color', 'marginTop', 'maxWidth'),
select: css(firstRow.querySelector('select'), 'fontSize', 'minWidth', 'maxWidth', 'backgroundColor', 'color', 'borderTopColor'),
heading3: css(surface.querySelector('.setting-row h3'), 'fontSize', 'fontWeight', 'color'),
number: css(surface.querySelector('.setting-row input[type="number"]'), 'fontSize', 'minWidth', 'maxWidth', 'backgroundColor'),
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
// D97: the rows Appearance adds for the legacy gallery settings, after the concept's own, as production renders them.
const ADDED_APPEARANCE_ROWS = [
['card-details', 'Show image details on cards', 'Show each card’s title and file details under the preview.', true],
['viewer-details', 'Collapse viewer details by default', 'Open the image viewer with its details panel hidden.', false],
];
/** The concept's Appearance with the recorded deviations applied, given the server's real settings. */
function mapAppearanceConcept(reference, settings) {
const toggleRow = reference.rows.find(row => row.control?.kind === 'switch');
const added = ADDED_APPEARANCE_ROWS.map(([id, label, description, checked]) => ({
tree: toggleRow.tree, text: [['label', label, id], ['p', description, null]],
control: { ...toggleRow.control, id, checked } }));
return {
...reference,
// D97: two added toggles and the "Reset display defaults" operation close the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.setting-row', 'div.setting-row', 'div.operation'] },
rows: [...reference.rows.map(row => row.control?.id === 'timeout'
// The node search timeout is the server's real `result_timeout`, not the concept's sample 30.
? { ...row, control: { ...row.control, value: String(settings.result_timeout) } } : row), ...added],
operations: [{ tree: ['div', 'div.row-actions'], title: 'Reset display defaults',
description: 'Restore this browser’s display defaults, then save. Other settings are kept.', action: 'Reset' }],
};
}
async function conceptAppearance(page, theme) {
await openConcept(page, 'system', theme);
await page.locator('#main .section-nav').getByRole('button', { name: 'Appearance', exact: true }).click();
await expect(page.locator('#main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
await page.mouse.move(0, 0);
return page.evaluate(measureAppearance, '#main');
}
async function productionAppearance(page, theme) {
await openProduction(page, 'system/appearance', theme);
await expect(page.locator('#workspace-main #timeout')).toBeEnabled();
await page.mouse.move(0, 0);
return page.evaluate(measureAppearance, '#workspace-main');
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`appearance matches the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const { settings } = await serverJson(request, '/prompt_manager/settings');
const concept = await conceptAppearance(page, theme);
await page.screenshot({ path: testInfo.outputPath(`concept-appearance-${theme}-${width}.png`), fullPage: true });
const production = await productionAppearance(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-appearance-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapAppearanceConcept(concept, settings), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
test('appearance comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const { settings } = await serverJson(request, '/prompt_manager/settings');
const concept = await conceptAppearance(page, 'dark');
const expected = mapAppearanceConcept(concept, settings);
const production = await productionAppearance(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only where D97 and the real timeout are recorded.
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
.filter((path, index, all) => all.indexOf(path) === index).sort())
.toEqual(['operations.0', 'operations.length', 'rows.11', 'rows.12', 'rows.13', 'rows.length', 'surface.tree'].sort());
const measure = () => page.evaluate(measureAppearance, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed row.
const undoRow = await mutate(page, { remove: ['#workspace-main .setting-row:nth-of-type(3)'] });
expect(await paths()).toEqual(expect.arrayContaining(['rows.length', 'surface.tree.length']));
await undoRow();
// A changed label and a changed option.
await page.evaluate(() => { document.querySelector('#workspace-main label[for="density"]').textContent = 'Thumbnail sizes'; });
expect(await paths()).toEqual(['rows.1.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="density"]').textContent = 'Thumbnail size'; });
await page.evaluate(() => { document.querySelector('#workspace-main #image-quality option').textContent = 'Medium'; });
expect(await paths()).toEqual(expect.arrayContaining(['rows.5.control.options.0']));
await page.evaluate(() => { document.querySelector('#workspace-main #image-quality option').textContent = 'Optimized'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['row.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/**
* Runs in the page: Sources & scans (app.js 679–681): the optional notice, then the "Image sources" surface with its
* panel heading, the source list, the two setting rows and the operations, with the concept's geometry and tokens.
*/
function measureSources(mainSelector) {
const layout = document.querySelector(`${mainSelector} .settings-layout`);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const button = node => ({ text: text(node), className: node.className,
...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') });
const operation = row => ({ tree: [...row.children].map(name), title: text(row.querySelector('h3')),
description: text(row.querySelector('p')), actions: [...row.querySelectorAll('.row-actions > button')].map(button) });
const notice = layout.querySelector(':scope > .notice');
const surface = layout.querySelector(':scope > section.surface');
const heading = surface.querySelector(':scope > .panel-heading');
const list = surface.querySelector(':scope > #source-list');
const control = row => {
const toggle = row.querySelector(':scope > * input[type="checkbox"]');
if (toggle) return { kind: 'switch', id: toggle.id, wrapper: name(toggle.parentElement), checked: toggle.checked,
...css(toggle.parentElement, 'width', 'height') };
const input = row.querySelector(':scope > input');
return { kind: `input.${input.type}`, id: input.id, min: input.min, max: input.max, value: input.value,
...css(input, 'width', 'accentColor') };
};
const rows = [...surface.querySelectorAll(':scope > .setting-row')];
return {
layout: [...layout.children].map(name),
notice: notice && { tree: [...notice.children].map(name),
text: [...notice.querySelectorAll(':scope > div > *')].map(node => [node.localName, text(node)]),
actions: [...notice.querySelectorAll(':scope > .row-actions > button')].map(button),
...css(notice, 'display', 'alignItems', 'columnGap', 'borderTopColor', 'backgroundColor', 'paddingTop', 'marginBottom') },
surface: { tree: [...surface.children].map(name), intro: text(surface.querySelector(':scope > p')),
...css(surface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
heading: { tree: [...heading.children].map(name), title: text(heading.querySelector('h2')),
action: button(heading.querySelector('button')), ...css(heading, 'display', 'alignItems', 'justifyContent', 'marginBottom'),
h2: css(heading.querySelector('h2'), 'fontSize', 'fontWeight', 'color') },
sources: [...list.children].map(operation),
rows: rows.map(row => ({ tree: [...row.children].map(name), text: [...row.firstElementChild.children].map(node => [node.localName, text(node),
node.localName === 'label' ? node.htmlFor : null]), control: control(row) })),
operations: [...surface.querySelectorAll(':scope > .operation')].map(operation),
operation: css(list.querySelector('.operation'), 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop',
'paddingBottom', 'borderTopWidth', 'borderTopColor'),
h3: css(list.querySelector('.operation h3'), 'fontSize', 'fontWeight', 'color'),
p: css(list.querySelector('.operation p'), 'fontSize', 'color', 'marginTop'),
row: css(rows[0], 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', 'borderTopColor'),
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
// D128: "watching until restart" after monitoring was switched off while the observer still runs.
const WATCH_TEXT = { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan', restart_pending: 'restart pending' };
const WATCHED = new Set(['watching', 'stopping']);
// D98: the operation Sources adds after the concept's own, as production renders it.
const REFRESH_FILES = { title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.',
label: 'Refresh' };
/** The concept's Sources with the recorded deviations applied, given the server's real settings and sources. */
function mapSourcesConcept(reference, { settings, sources }) {
const [sample] = reference.sources;
const scan = sample.actions[0];
return {
...reference,
// The real sources replace the two samples (without the "sample path" note). D98: each configured one has Remove,
// a plain button after its scan.
sources: sources.effective.map(source => ({ ...sample, title: source.label,
description: `${source.path} · ${WATCH_TEXT[source.watch]}`,
actions: [{ ...scan, text: WATCHED.has(source.watch) ? 'Scan changes' : 'Scan folder' }, { ...scan, text: 'Remove' }] })),
rows: reference.rows.map(row => {
if (row.control.id === 'monitoring') return { ...row, control: { ...row.control, checked: settings.monitoring_enabled } };
// D113: the range ends at the server's core count (the concept's fixed 16) and shows the real worker count.
return { ...row, control: { ...row.control, max: String(settings.cpu_count), value: String(settings.worker_threads) } };
}),
// D98: "Refresh files" closes the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.operation'] },
operations: [...reference.operations, { ...reference.operations[0], title: REFRESH_FILES.title,
description: REFRESH_FILES.description, actions: [{ ...reference.operations[0].actions[0], text: REFRESH_FILES.label }] }],
};
}
async function conceptSources(page, theme, { notice }) {
await openConcept(page, 'system', theme);
await page.locator('#main .section-nav').getByRole('button', { name: 'Sources & scans', exact: true }).click();
await expect(page.locator('#main .panel-heading > h2')).toHaveText('Image sources');
if (!notice) await page.locator('#main .notice').getByRole('button', { name: 'Later', exact: true }).click();
await expect(page.locator('#main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0);
await page.mouse.move(0, 0);
return page.evaluate(measureSources, '#main');
}
async function productionSources(page, theme, { notice }) {
// about:blank has no storage; the workspace page reads the dismissal.
if (!notice) await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('pm_update_notice_dismissed', '3.2.0'); });
await openProduction(page, 'system/sources', theme);
await expect(page.locator('#workspace-main #source-list .operation')).toHaveCount(2);
await expect(page.locator('#workspace-main #workers')).toBeEnabled();
await expect(page.locator('#workspace-main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0);
await page.mouse.move(0, 0);
return page.evaluate(measureSources, '#workspace-main');
}
/** The fixture's sources as the concept shows its samples: the first one watched, the second not. */
async function watchedSources(request) {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK();
return serverJson(request, '/prompt_manager/settings');
}
for (const theme of THEMES) for (const [width, height] of SIZES) for (const notice of [true, false]) {
const variant = notice ? 'with' : 'without';
test(`sources matches the concept ${variant} the notice ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const server = await watchedSources(request);
const concept = await conceptSources(page, theme, { notice });
await page.screenshot({ path: testInfo.outputPath(`concept-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true });
const production = await productionSources(page, theme, { notice });
await page.screenshot({ path: testInfo.outputPath(`production-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapSourcesConcept(concept, server), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
test('sources comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const server = await watchedSources(request);
const concept = await conceptSources(page, 'dark', { notice: true });
const expected = mapSourcesConcept(concept, server);
const production = await productionSources(page, 'dark', { notice: true });
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only in the real sources, the D98 operation and the D113 range.
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
.filter((path, index, all) => all.indexOf(path) === index).sort())
.toEqual(['operations.2', 'operations.length', 'rows.1', 'sources.0', 'sources.1', 'surface.tree'].sort());
const measure = () => page.evaluate(measureSources, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed notice and a removed source row.
const undoNotice = await mutate(page, { remove: ['#workspace-main .settings-layout > .notice'] });
expect(await paths()).toEqual(expect.arrayContaining(['layout.length', 'notice.tree.length']));
await undoNotice();
const undoRow = await mutate(page, { remove: ['#workspace-main #source-list > .operation:nth-child(2)'] });
expect(await paths()).toEqual(expect.arrayContaining(['sources.length']));
await undoRow();
// A changed label and a changed button.
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for outputs'; });
expect(await paths()).toEqual(['rows.0.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for new outputs'; });
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = 'Add source'; });
expect(await paths()).toEqual(['heading.action.text']);
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = '+ Add source'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['operation.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/**
* Runs in the page: the open "Add image source" dialog (the concept's `simpleInput`, app.js 1138 and 1497): its title,
* the body's tree, the field's label and input, and the actions, with the concept's geometry and tokens.
*/
function measureAddSource() {
const dialog = document.querySelector('dialog#modal[open]');
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const body = dialog.querySelector('#modal-body');
const field = body.querySelector(':scope > .field');
const label = field?.querySelector(':scope > label');
const input = field?.querySelector(':scope > input');
const actions = body.querySelector(':scope > .dialog-actions');
const rect = dialog.getBoundingClientRect();
return {
title: text(dialog.querySelector('#modal-title')),
tree: [...body.children].map(name),
field: field && { tree: [...field.children].map(name), ...css(field, 'display', 'flexDirection', 'rowGap', 'marginBottom') },
label: label && { text: text(label), for: label.htmlFor, ...css(label, 'fontSize', 'fontWeight', 'color') },
input: input && { id: input.id, type: input.type, value: input.value,
...css(input, 'height', 'paddingLeft', 'fontSize', 'color', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
buttons: [...actions.children].map(button => ({ text: text(button), className: button.className,
...css(button, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) },
width: Math.round(rect.width),
};
}
/** D127: the field is "Folder path", starts empty (no sample folder) and its action is "Add". */
function mapAddSourceConcept(reference) {
const [cancel, apply] = reference.actions.buttons;
return { ...reference, label: { ...reference.label, text: 'Folder path' }, input: { ...reference.input, value: '' },
actions: { ...reference.actions, buttons: [cancel, { ...apply, text: 'Add' }] } };
}
async function conceptAddSource(page, theme) {
await conceptSources(page, theme, { notice: false });
await page.locator('#main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source');
await page.mouse.move(0, 0);
return page.evaluate(measureAddSource);
}
async function productionAddSource(page, theme) {
await productionSources(page, theme, { notice: false });
await page.locator('#workspace-main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source');
await page.mouse.move(0, 0);
return page.evaluate(measureAddSource);
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`add source dialog matches the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
await watchedSources(request);
const concept = await conceptAddSource(page, theme);
await page.screenshot({ path: testInfo.outputPath(`concept-add-source-${theme}-${width}.png`) });
const production = await productionAddSource(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-add-source-${theme}-${width}.png`) });
expect(diffInventory(mapAddSourceConcept(concept), production)).toEqual([]);
});
}
test('add source comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
await watchedSources(request);
const concept = await conceptAddSource(page, 'dark');
const expected = mapAddSourceConcept(concept);
const production = await productionAddSource(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only where D127 is recorded.
expect(diffInventory(concept, production).map(item => item.path).sort())
.toEqual(['actions.buttons.1.text', 'input.value', 'label.text']);
const paths = async () => diffInventory(expected, await page.evaluate(measureAddSource)).map(item => item.path);
// A changed label, a swapped action order and a changed style.
await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder'; });
expect(await paths()).toEqual(['label.text']);
await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder path'; });
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text']));
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
const undoStyle = await mutate(page, { css: 'dialog#modal #action-input { border-color: rgb(255, 0, 0); } dialog#modal .dialog-actions { gap: 31px; }' });
expect(await paths()).toEqual(expect.arrayContaining(['input.borderTopColor', 'actions.columnGap']));
await undoStyle();
expect(diffInventory(expected, await page.evaluate(measureAddSource))).toEqual([]);
});
@@ -140,152 +140,3 @@ export function mapConceptTagging(reference,{instructions,limit=10,chip,numberFi
export function withoutChipGeometry(inventory) {
return {...inventory,suggestions:{...inventory.suggestions,rows:inventory.suggestions.rows.map(({paddingTop,paddingLeft,alignItems,...row})=>row)}};
}
/**
* Needs review (plan Task F3, Mock inventory M4: app.js renderReview a749–760, style.css s407–415, s796–814 and their
* breakpoints): the page's main element as a tree, runs in the page. Every shown box below `main` to `depth` levels is
* recorded with its tag and classes, its box relative to its parent, its fonts, edges and layout, a leaf's text, a
* button's label and icon, a select's options; a `display: contents` node is replaced by its children (D20). A flex
* row of actions also records the gaps between its children and its right inset, so a row whose button copy differs
* (D29) still compares by order, spacing and alignment.
*/
export function reviewInventory(selector) {
const main=document.querySelector(selector);
// Half pixels: the concept's 13.5px line heights add up exactly, so a box that loses one line moves by 13.5px.
const px=value=>Math.round(value*2)/2;
const css=(node,...keys)=>{const style=getComputedStyle(node);return Object.fromEntries(keys.map(key=>[key,style[key]]));};
const rect=node=>node.getBoundingClientRect();
const shown=node=>node.getClientRects().length>0&&getComputedStyle(node).visibility!=='hidden';
const relative=(node,origin)=>{const a=rect(node),b=rect(origin);return {left:px(a.left-b.left),top:px(a.top-b.top),width:px(a.width),height:px(a.height)};};
const font=node=>css(node,'fontFamily','fontSize','fontWeight','lineHeight','letterSpacing','color','textTransform','textAlign');
const edges=node=>css(node,'paddingTop','paddingRight','paddingBottom','paddingLeft','marginTop','marginRight','marginBottom','marginLeft',
'borderTopWidth','borderTopStyle','borderTopColor','borderRightWidth','borderRightStyle','borderRightColor','borderBottomWidth',
'borderBottomStyle','borderBottomColor','borderLeftWidth','borderLeftStyle','borderLeftColor','borderTopLeftRadius','backgroundColor',
'boxShadow','outlineStyle');
const layout=node=>css(node,'display','position','flexDirection','flexWrap','justifyContent','alignItems','columnGap','rowGap',
'gridTemplateColumns','overflowX','overflowY','objectFit','minHeight','minWidth');
const tagOf=node=>`${node.localName}${[...node.classList].sort().map(name=>`.${name}`).join('')}`;
const textOf=node=>node.textContent.replace(/\s+/g,' ').trim();
const boxed=node=>[...node.children].flatMap(child=>getComputedStyle(child).display==='contents'?boxed(child):[child]);
const LEAVES=new Set(['button','select','input','img','svg','textarea','progress']);
function nodeOf(node,origin,depth) {
const base={tag:tagOf(node),...relative(node,origin),...font(node),...edges(node),...layout(node)};
const name=node.localName;
if(name==='svg')return {tag:base.tag,width:base.width,height:base.height,d:node.querySelector('path')?.getAttribute('d')??null,
...css(node,'strokeWidth','color','verticalAlign')};
if(name==='button')return {...base,text:textOf(node),label:node.getAttribute('aria-label'),pressed:node.getAttribute('aria-pressed'),
disabled:node.getAttribute('aria-disabled')==='true'||node.disabled,primary:node.classList.contains('ca-primary'),
children:boxed(node).filter(shown).map(child=>nodeOf(child,node,0))};
if(name==='select')return {...base,id:node.id,options:[...node.options].map(option=>[option.textContent,option.disabled])};
if(name==='input')return {...base,id:node.id,type:node.type,label:node.getAttribute('aria-label')};
if(name==='img')return {...base,alt:node.getAttribute('alt')};
const children=boxed(node).filter(shown);
const item={...base,id:node.id||null,role:node.getAttribute('role'),label:node.getAttribute('aria-label'),
text:children.length?null:textOf(node),htmlFor:node.htmlFor||null};
if(/(^|\s)(dialog-actions|row-actions|image-card-actions)(\s|$)/.test(node.className)&&children.length) {
const boxes=children.map(rect);
item.gaps=boxes.slice(1).map((box,index)=>px(box.left-boxes[index].right));
item.rightInset=px(rect(node).right-boxes.at(-1).right);
}
if(depth>0&&!LEAVES.has(name))item.children=children.map(child=>nodeOf(child,node,depth-1));
return item;
}
return {main:nodeOf(main,main,8),noPageScroll:document.documentElement.scrollWidth<=innerWidth+1};
}
/**
* `tree` with `edit(node)` applied to every node `match` picks (a tag prefix or a predicate), depth first, new objects
* only.
*/
function editNodes(tree,match,edit) {
const children=tree.children?.map(child=>editNodes(child,match,edit));
const node=children?{...tree,children}:tree;
const picked=typeof match==='function'?match(node):node.tag?.startsWith(match);
return picked?edit(node):node;
}
const findNode=(tree,prefix)=>tree.tag?.startsWith(prefix)?tree:tree.children?.map(child=>findNode(child,prefix)).find(Boolean)??null;
/**
* The concept's Metadata review inventory (reviewInventory) with the recorded deviations applied, the remaining sample
* data taken from production (`actual`) only where the concept's own copy cannot say it:
* - D29: no "Sample review queue" chip in the notice; "Resolve the association" and "Apply association" without the
* prototype's "sample";
* - D119: queue cards carry no selection checkbox and no star (the overlay row stays, empty); a card's title is its
* file name (an undecided item names no prompt) and its line is the extension only (D11 omits the unknown size and
* seed, which a queue item does not carry); the image's alt is the file name; the prompt blocks' Copy and Send names
* carry no prompt title;
* - D3: the grid keeps the Images grid's `position: relative` (its cards stay in its flow);
* - production's hooks and names: the `review-queue` / `review-panel` classes, the queue's list roles and name, the
* panel heading's id (the region's name) and the prompt sections' region names;
* - sample data: the Sampler context's candidate options (the file's own evidence; the last option, Leave unresolved,
* stays compared) and the widths that follow from copy the deviations above change (a card's text column, the
* panel's action buttons; their gaps and right inset stay compared).
* Where that copy changes the height of a box, the boxes after it move by exactly that much: the stacked notice (390)
* loses the chip and its row gap; a card whose concept line wraps (390: "1536 × 1152 · PNG · Seed 42001") takes
* production's one-line height, its row of the grid shrinks by the most any of its cards lost, and in one column the
* panel below the grid moves up by the grid's loss. The actions keep production's centred place in the card's info.
*/
export function mapConceptReview(reference,actual) {
const named=(node,id)=>({...node,id});
const actualCards=findNode(actual.main,'div.image-grid')?.children??[];
const actualActions=findNode(actual.main,'div.dialog-actions');
let cardIndex=0;
let main={...reference.main,id:actual.main.id};
let noticeLoss=0;
main=editNodes(main,'div.notice',notice=>{
const chip=notice.children.find(child=>child.tag==='span.chip');
if(chip&&notice.flexDirection==='column')noticeLoss=chip.height+parseFloat(notice.rowGap);
return {...notice,height:notice.height-noticeLoss,children:notice.children.filter(child=>child!==chip)};
});
const noticeIndex=main.children.findIndex(child=>child.tag==='div.notice');
main={...main,height:main.height-noticeLoss,children:main.children.map((child,index)=>index>noticeIndex?{...child,top:child.top-noticeLoss}:child)};
const losses=[];
main=editNodes(main,'div.image-grid',grid=>({...grid,tag:'div.image-grid.review-queue',position:'relative',role:'list',label:'Review queue'}));
main=editNodes(main,'article.media-card',card=>{
const twin=actualCards[cardIndex++];
const filename=findNode(card,'button.image-open').label.replace(/^Open /,'');
const art=editNodes(card.children[0],'div.image-overlays',overlays=>({...overlays,height:0,text:'',children:[]}));
const withAlt=editNodes(art,'img',img=>({...img,alt:filename}));
const info=card.children[1];
const [copy,...rest]=info.children;
const twinCopy=twin?.children[1]?.children[0];
const [h3,p]=copy.children;
const twinLine=twinCopy?.children[1];
const loss=twinLine?p.height-twinLine.height:0;
losses.push({top:card.top,loss});
const mappedCopy={...copy,width:twinCopy?.width,height:copy.height-loss,children:[{...h3,text:filename,width:twinCopy?.children[0]?.width},
{...p,text:'PNG',width:twinLine?.width,height:p.height-loss}]};
const [actions,...after]=rest;
const twinActions=twin?.children[1]?.children[1];
const mappedActions=actions&&loss?{...actions,top:twinActions?.top}:actions;
return {...card,role:'listitem',height:card.height-loss,
children:[withAlt,{...info,height:info.height-loss,children:[mappedCopy,...(actions?[mappedActions]:[]),...after]},...card.children.slice(2)]};
});
// Each grid row loses what its most shortened card lost; later rows move up by the rows before them.
const rowLoss=new Map();
for(const {top,loss} of losses)rowLoss.set(top,Math.max(rowLoss.get(top)??0,loss));
const lossBefore=top=>[...rowLoss].filter(([rowTop])=>rowTop<top).reduce((sum,[,loss])=>sum+loss,0);
const gridLoss=[...rowLoss.values()].reduce((sum,loss)=>sum+loss,0);
main=editNodes(main,'div.image-grid',grid=>({...grid,height:grid.height-gridLoss,
children:grid.children.map(card=>({...card,top:card.top-lossBefore(card.top)}))}));
main=editNodes(main,'div.gallery-layout',layout=>{
if(!gridLoss||layout.gridTemplateColumns.trim().split(/\s+/).length>1)return layout;
return {...layout,height:layout.height-gridLoss,children:layout.children.map((child,index)=>index?{...child,top:child.top-gridLoss}:child)};
});
if(gridLoss&&findNode(main,'div.gallery-layout').gridTemplateColumns.trim().split(/\s+/).length===1)main={...main,height:main.height-gridLoss};
main=editNodes(main,'section.surface',surface=>{
const [h2,...rest]=surface.children;
return {...surface,tag:'section.review-panel.surface',children:[named({...h2,text:h2.text.replace('Resolve the sample association','Resolve the association')},'review-panel-title'),...rest]};
});
main=editNodes(main,'select',select=>{
const twin=findNode(actual.main,'select');
return select.id==='review-context'?{...select,options:[...twin.options.slice(0,-1),select.options.at(-1)]}:select;
});
main=editNodes(main,node=>/^section\..*\bprompt-section\b/.test(node.tag),section=>({...section,label:section.tag.includes('positive')?'Positive prompt':'Negative prompt'}));
main=editNodes(main,node=>node.tag?.startsWith('button')&&/^(Copy|Send) (positive|negative) prompt/.test(node.label??''),
button=>({...button,label:button.label.replace(/: .+$/,'')}));
main=editNodes(main,'div.dialog-actions',actions=>({...actions,children:actions.children.map((button,index)=>({...button,
text:button.text.replace('Apply sample association','Apply association'),
left:actualActions?.children[index]?.left,width:actualActions?.children[index]?.width}))}));
return {...reference,main};
}
+24 -76
View File
@@ -354,48 +354,39 @@ test('prompt deletion requires confirmation and can retry after a network failur
}
});
// The duplicate scan reads the output root only, but the fixture's input root holds a different
// shared/same.png. A bare /images/serve/<path> URL tries every gallery root in turn, so once the
// output copy is deleted it answers 200 with the input copy: check each file in its own root.
const serveIn = (root, path) => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}?root=${root}`;
const rootOf = url => new URL(url, 'http://gallery.test').searchParams.get('root');
// Which copy the scan keeps depends on file times: the fixture stamps shared/same.png with the
// wall clock, and the kernel can stamp its bulk copies a fraction of a millisecond earlier.
// listOrder rewrites the finished scan's groups so a test can pick the order it needs.
async function openDuplicateScan(page, request, listOrder = null) {
test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => {
await expect(await request.post('/__test__/reset', { data: { extra_images: 2 } })).toBeOK();
if (listOrder) {
await page.route('**/prompt_manager/scan_duplicates/stream', async route => {
const response = await route.fetch();
const body = (await response.text()).replace(/^data: (.*)$/gm, (line, json) => {
const event = JSON.parse(json);
if (event.type !== 'complete') return line;
for (const group of event.duplicates) group.images = listOrder(group.images);
return `data: ${JSON.stringify(event)}`;
});
await route.fulfill({ response, body });
});
}
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(4);
const originals = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
const output = rootOf(originals.find(url => url.includes('/bulk/')));
const input = rootOf(originals.find(url => url.includes('/shared/same.png') && rootOf(url) !== output));
expect(output).not.toBeNull();
expect(input).not.toBeNull();
await page.locator('#settingsBtn').click();
await page.locator('#scanDuplicatesBtn').click();
await expect(page.locator('#duplicatesCount')).toHaveText('1');
const candidates = page.locator('.duplicate-checkbox');
await expect(candidates).toHaveCount(2);
return { originals, output, input, candidates };
}
async function expectOnlySelectedGone(page, request, { originals, output, input }, selected, retained) {
expect((await request.get(serveIn(output, selected))).status()).toBe(404);
await expect(await request.get(serveIn(output, retained))).toBeOK();
await expect(await request.get(serveIn(input, 'shared/same.png'))).toBeOK();
const selected = await candidates.first().getAttribute('data-image-path');
const retained = await candidates.last().getAttribute('data-image-path');
const serve = path => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}`;
await candidates.first().check();
let deletes = 0;
page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; });
confirmNextDialog(page, false, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
expect(deletes).toBe(0);
await expect(await request.get(serve(selected))).toBeOK();
await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible();
await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled();
await expect(await request.get(serve(selected))).toBeOK();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
expect((await request.get(serve(selected))).status()).toBe(404);
await expect(await request.get(serve(retained))).toBeOK();
await page.reload();
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(3);
const remaining = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
@@ -404,47 +395,4 @@ async function expectOnlySelectedGone(page, request, { originals, output, input
expect(originals).toContain(url);
await expect(await request.get(url)).toBeOK();
}
}
test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => {
const scan = await openDuplicateScan(page, request);
const { output, candidates } = scan;
const selected = await candidates.first().getAttribute('data-image-path');
const retained = await candidates.last().getAttribute('data-image-path');
await candidates.first().check();
let deletes = 0;
page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; });
confirmNextDialog(page, false, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
expect(deletes).toBe(0);
await expect(await request.get(serveIn(output, selected))).toBeOK();
await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible();
await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled();
await expect(await request.get(serveIn(output, selected))).toBeOK();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
test('deleting a duplicate whose path another gallery root also holds removes only the scanned copy', async ({ page, request }) => {
// The order CI saw: a bulk copy stamped first, so shared/same.png is the first Delete candidate.
const ciOrder = ['bulk/0000.png', 'shared/same.png', 'bulk/0001.png'];
const rank = image => ciOrder.indexOf(image.path.replace(/\\/g, '/'));
const scan = await openDuplicateScan(page, request, images => [...images].sort((a, b) => rank(a) - rank(b)));
const selected = await scan.candidates.first().getAttribute('data-image-path');
const retained = await scan.candidates.last().getAttribute('data-image-path');
expect(selected.replace(/\\/g, '/')).toBe('shared/same.png');
await scan.candidates.first().check();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
-49
View File
@@ -284,19 +284,6 @@ test('reload during a scan reattaches to the running job', async ({ page, reques
await expect(pill(page)).toBeHidden();
});
test('an empty Activity says no job has run, and a list that cannot be read says so', async ({ page }) => {
// The empty and failed states were only reached in passing by other tests, so their coverage depended on timing.
const empty = main(page).locator('section.surface > p.empty-state');
await page.goto(`${workspace}#/activity`);
await expect(empty).toHaveText('No jobs have run since ComfyUI started. Scans and other long operations appear here.');
await expect(empty).not.toHaveAttribute('role');
await expect(items(page)).toHaveCount(0);
await page.route(url => url.pathname === '/prompt_manager/jobs', route => route.abort());
await page.reload();
await expect(empty).toHaveText('Activity could not be read. It is tried again shortly.');
await expect(empty).toHaveAttribute('role', 'alert');
});
test('lost server clears the pill and marks jobs ended', async ({ page, request }) => {
// A poll interval after the restart, then failed re-reads backing off for 1 + 2 + 4 s.
test.setTimeout(90_000);
@@ -737,42 +724,6 @@ test('a held legacy job shows no Cancel in the list or its view', async ({ page,
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
});
// Since #289 an item dropped its Cancel job only when it was already in the page, so the first render of jobs the
// store already knew (Activity opened from another page) kept a Cancel job on a job that had ended, and moved a
// running job's progress after its actions. The read Activity starts on opening is held, so that first render is
// what the page shows (the read's own render would otherwise repair it a moment later).
test('jobs that ended before Activity opened show no Cancel job, a running one keeps its progress first', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const finished = await startJob(request, { title: 'Finished scan' });
await finish(request, finished.id);
const cancelled = await startJob(request, { title: 'Cancelled scan' });
await finish(request, cancelled.id, 'cancelled');
await startJob(request, { title: 'Running scan' });
// The page's first list read has all three jobs once the running one shows in the pill.
await page.goto(`${workspace}#/images`);
await expect(pill(page)).toBeVisible();
let hold;
const held = new Promise(resolve => { hold = resolve; });
// Only the first read is held; any later one goes through.
await page.route(url => url.pathname === '/prompt_manager/jobs', route => {
if (!hold) return route.continue();
hold(route);
hold = null;
});
await activityButton(page).click();
const route = await held;
await expect(items(page)).toHaveCount(3);
const item = title => items(page).filter({ has: page.getByRole('link', { name: new RegExp(`${title}$`) }) });
for (const ended of ['Finished scan', 'Cancelled scan']) {
await expect(item(ended).locator('progress, .row-actions')).toHaveCount(0);
await expect(item(ended).getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(0);
}
await expect(item('Running scan').getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(1);
expect(await item('Running scan').evaluate(node => [...node.children].map(child => child.localName + (child.className ? `.${child.className}` : ''))))
.toEqual(['div.panel-heading', 'p', 'progress', 'div.row-actions']);
await route.continue();
});
test('busy start names the running job', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
// The page's timers are Playwright's, so the toast's lifetime below is run out rather than waited for.
-44
View File
@@ -463,50 +463,6 @@ test('collection viewer ignores delayed media callbacks and keeps inspection ret
await page.evaluate(()=>{location.hash='/overview';});await expect(page.locator('.viewer-container,#viewer-dialog')).toHaveCount(0);
});
test('collection viewer leaving an image Viewer.js has not built yet releases it at once',async({page})=>{
await page.goto(workspace);await expect(cards(page)).toHaveCount(2);
let release,held;const waiting=new Promise(resolve=>{held=resolve;});const pending=new Promise(resolve=>{release=resolve;});
await page.route('**/images/serve/shared/same.png?*',async route=>{
const url=new URL(route.request().url());
if(url.searchParams.get('root')==='0'&&url.searchParams.has('_pm_viewer')) {held();await pending;}
await route.continue().catch(()=>{});
});
await cards(page).first().getByRole('button',{name:'Open same.png',exact:true}).click();await waiting;
const preview=page.locator('dialog#viewer-dialog');
await page.evaluate(()=>{window.unbuiltImage=document.querySelector('#viewer-stage > img');});
await preview.getByRole('button',{name:'Next media',exact:true}).click();
await expect(preview.getByRole('button',{name:'Zoom in',exact:true})).toHaveAttribute('aria-disabled','false');
// The left image is off the stage. Viewer.js has not built it, so the viewer is destroyed and the src released at
// once: its late load must never build Viewer.js into an image without a parent (getComputedStyle(null)).
release();
await expect.poll(()=>page.evaluate(()=>[window.unbuiltImage.isConnected,window.unbuiltImage.hasAttribute('src'),!!window.unbuiltImage.viewer])).toEqual([false,false,false]);
await page.evaluate(()=>{location.hash='/overview';});await expect(page.locator('.viewer-container,#viewer-dialog')).toHaveCount(0);
});
test('collection viewer closed while its image finishes loading never builds Viewer.js afterwards',async({page})=>{
await page.goto(workspace);await expect(cards(page)).toHaveCount(2);
let held;const waiting=new Promise(resolve=>{held=resolve;});
// The image is held for the whole test: its late load is delivered below, at the moment Viewer.js cannot cancel.
await page.route('**/images/serve/shared/same.png?*',async route=>{
const url=new URL(route.request().url());
if(url.searchParams.get('root')==='0'&&url.searchParams.has('_pm_viewer')) {held();return;}
await route.continue().catch(()=>{});
});
await cards(page).first().getByRole('button',{name:'Open same.png',exact:true}).click();await waiting;
const preview=page.locator('dialog#viewer-dialog');
// Viewer.js's own cancel removes its image listeners only while `image.complete` is false. An image that has
// finished downloading but whose load event is still queued is already complete: close in that state, then deliver it.
await page.evaluate(()=>{window.finishingImage=document.querySelector('#viewer-stage > img');Object.defineProperty(window.finishingImage,'complete',{get:()=>true});});
await page.keyboard.press('Escape');await expect(preview).toHaveCount(0);
const after=await page.evaluate(async()=>{
const image=window.finishingImage;image.dispatchEvent(new Event('load'));
// Viewer.js builds an inline viewer in a zero-delay timeout after its image loads; this runs after it.
await new Promise(resolve=>setTimeout(resolve));
return {src:image.hasAttribute('src'),built:!!image.parentNode?.querySelector('.viewer-container')};
});
expect(after).toEqual({src:false,built:false});
});
test('collection viewer describes video transforms and releases native playback on navigation',async({page,request})=>{
await expect(await request.post('/__test__/reset',{data:{include_video:true}})).toBeOK();await page.goto(workspace);
await page.locator('article[data-image-key="0:clip.webm"] .image-open').click();
-48
View File
@@ -577,54 +577,6 @@ test('a video replaced while still starting is reset once it settles', async ({
}
});
test('leaving for another route while a video starts waits for it to settle', async ({ context, browserName }) => {
test.setTimeout(180000);
// A WebKit (GStreamer) video detached before its pipeline had started deadlocked the page (TESTING.md D62); a
// route change detaches the whole page, so the shell renders the next route once the video reports metadata or an
// error. The route changes from the video's own `loadstart`, or a few milliseconds after it.
for (let attempt = 0; attempt < 15; attempt++) {
const page = await context.newPage();
try {
if (attempt === 0) await startCoverage(page, browserName);
await page.addInitScript(() => {
const load = HTMLMediaElement.prototype.load;
HTMLMediaElement.prototype.load = function () {
if (!this.hasAttribute('src') && window.__starting?.node === this) window.__starting.reset = { readyState: this.readyState, failed: this.error !== null, connected: this.isConnected };
return load.call(this);
};
});
await openInspect(page);
await page.evaluate(delay => {
const slot = document.querySelector('#local-preview');
new MutationObserver((records, observer) => {
const video = slot.querySelector('video');
if (!video) return;
observer.disconnect();
const leave = () => {
window.__starting = { node: video, readyState: video.readyState };
location.hash = '#/overview';
};
video.addEventListener('loadstart', () => { if (delay) setTimeout(leave, delay); else leave(); }, { once: true });
}).observe(slot, { childList: true });
}, attempt % 5);
await choose(page, localFixture('core-tags.webm', 'video/webm'));
await expect(main(page).getByRole('heading', { level: 1, name: 'Overview', exact: true })).toBeVisible({ timeout: 8000 });
const { readyState, reset } = await page.evaluate(() => ({ readyState: window.__starting.readyState, reset: window.__starting.reset }));
if (attempt % 5 === 0) expect(readyState).toBe(0);
// The video was reset and left the page only once it had settled.
expect(reset.readyState >= 1 || reset.failed).toBe(true);
expect(reset.connected).toBe(true);
// The page still answers.
await page.locator('aside.sidebar nav').getByRole('link', { name: 'Inspect a file', exact: true }).click();
await chooser(page).setInputFiles(parityPng('f02.png'), { timeout: 8000 });
await expect(preview(page).locator('img.local-file-preview')).toBeVisible({ timeout: 8000 });
if (attempt === 0) await saveCoverage(page, browserName);
} finally {
await page.close();
}
}
});
test('closing the viewer on a video that just started never blocks the page', async ({ context, browserName }) => {
test.setTimeout(180000);
// The viewer's video teardown resets its element; on WebKit a reset of a still-starting pipeline blocked the page.
-556
View File
@@ -1,556 +0,0 @@
// Needs review (#249 PR F, plan Tasks F1 and F2, Mock inventory M4): the SAVED VIEWS row and its count, the Metadata
// review page over the real review routes, and Activity's Review metadata. Items are recorded by the real recorder
// (tests/browser/workspace_review_fixture.py) through the monitor's entry point; the conflict item's file was linked
// to another prompt after it was queued. An open page re-reads its queue from the review change stream (F2).
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
const review = '/prompt_manager/workspace#/review';
const main = page => page.locator('#workspace-main');
const savedNav = page => page.getByRole('navigation', { name: 'Saved views', exact: true });
const reviewRow = page => savedNav(page).locator('a[data-route="review"]');
const panel = page => main(page).getByRole('region', { name: 'Resolve the association', exact: true });
const context = page => panel(page).getByRole('combobox', { name: 'Sampler context', exact: true });
const cards = page => main(page).locator('.review-queue article.media-card');
const toast = page => page.locator('#workspace-toast');
const NOTICE = 'Multiple sampler paths were found. The existing image links were preserved. Choose the correct context or leave them unresolved.';
const EMPTY = 'Ambiguous images are added when a metadata scan or a new generation finds more than one sampler path.';
const OPTIONS = ['Sampler 11 · Golden dunes at dusk / Soft negative, blur', 'Sampler 13 · Harbor lights at night / Soft negative, blur',
'Leave unresolved'];
async function seed(request, items = 3, special = {}) {
const response = await request.post('/__test__/review/reset', { data: { items, ...special } });
await expect(response).toBeOK();
return response.json();
}
async function pending(request) {
return (await (await request.get('/prompt_manager/review/items?limit=1')).json()).total;
}
const links = async request => (await (await request.get('/__test__/review/links')).json()).links;
const streams = async request => (await (await request.get('/__test__/review/streams')).json());
const apply = page => panel(page).getByRole('button', { name: 'Apply association', exact: true });
const warning = page => panel(page).locator('.warning-note');
/** The page with its first item's sampler paths loaded. */
async function open(page) {
await page.goto(review);
await expect(context(page)).toBeEnabled();
}
test.beforeEach(async ({ page, browserName }) => {
await startCoverage(page, browserName);
page.reviewErrors = [];
page.on('pageerror', error => page.reviewErrors.push(error.message));
});
test.afterEach(async ({ page, browserName, request }) => {
await saveCoverage(page, browserName);
expect(page.reviewErrors).toEqual([]);
await request.post('/__test__/jobs/restart');
});
test('queue shows persisted items and the nav count', async ({ page, request }) => {
await seed(request, 3);
await open(page);
// SAVED VIEWS: Favorites, then Needs review with the exact pending count (ruling R5).
await expect(savedNav(page).locator('a').nth(1)).toHaveAttribute('data-route', 'review');
await expect(reviewRow(page)).toHaveText('Needs review3');
await expect(reviewRow(page)).toHaveAttribute('aria-current', 'page');
await expect(page).toHaveTitle('Review · Prompt Manager');
await expect(page.locator('#workspace-breadcrumb')).toHaveText('Review');
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Metadata review');
await expect(main(page).locator('.heading p.muted')).toHaveText('Uncertain evidence stays visible until you decide.');
await expect(main(page).locator('.notice h2')).toHaveText('3 images need a decision');
await expect(main(page).locator('.notice p')).toHaveText(NOTICE);
await expect(cards(page)).toHaveCount(3);
await expect(cards(page).first()).toHaveClass(/selected/);
await expect(panel(page).locator('p.small')).toHaveText('review_0000.png');
await expect(context(page).locator('option')).toHaveText(OPTIONS);
await expect(context(page)).toHaveValue(/^[0-9a-f]{64}$/);
await expect(panel(page).locator('.prompt-section.positive .prompt-text')).toHaveText('Golden dunes at dusk');
await expect(panel(page).locator('.prompt-section.negative .prompt-text')).toHaveText('Soft negative, blur');
await context(page).selectOption({ label: OPTIONS[1] });
await expect(panel(page).locator('.prompt-section.positive .prompt-text')).toHaveText('Harbor lights at night');
await context(page).selectOption({ label: 'Leave unresolved' });
await expect(panel(page).locator('.prompt-section.positive .prompt-text')).toHaveText('Multiple possible prompts. Source could not be determined.');
// Info selects another item.
await cards(page).nth(1).getByRole('button', { name: 'Inspect review_0001.png', exact: true }).click();
await expect(panel(page).locator('p.small')).toHaveText('review_0001.png');
await expect(cards(page).nth(1)).toHaveClass(/selected/);
await expect(context(page)).toBeEnabled();
// The card's image opens the shared viewer.
await cards(page).nth(1).getByRole('button', { name: 'Open review_0001.png', exact: true }).click();
await expect(page.locator('#viewer-dialog')).toBeVisible();
await page.keyboard.press('Escape');
await expect(page.locator('#viewer-dialog')).toHaveCount(0);
// Open activity.
await main(page).getByRole('button', { name: 'Open activity', exact: true }).click();
await expect(page).toHaveURL(/#\/activity$/);
});
test('apply candidate links the pair and the item leaves the queue and the count drops', async ({ page, request }) => {
await seed(request, 2);
await open(page);
await expect(reviewRow(page)).toHaveText('Needs review2');
const decided = page.waitForResponse(item => /\/review\/items\/[^/]+\/decision$/.test(item.url()));
await panel(page).getByRole('button', { name: 'Apply association', exact: true }).click();
expect((await decided).status()).toBe(200);
await expect(toast(page)).toHaveText('Association accepted.');
await expect(page).toHaveURL(/#\/images$/);
await expect(reviewRow(page)).toHaveText('Needs review1');
expect(await pending(request)).toBe(1);
// The first sampler path's pair is linked to that file, and only to it.
expect(await links(request)).toEqual([{ filename: 'review_0000.png', prompt: 'Golden dunes at dusk',
pair_id: expect.any(Number), positive: 'Golden dunes at dusk', negative: 'Soft negative, blur' }]);
});
test('conflict leaves links unchanged and explains', async ({ page, request }) => {
// Queued first, then linked to another prompt (before a scan or monitor pass could settle it).
await seed(request, 0, { conflict: true });
const before = await links(request);
expect(before).toEqual([{ filename: 'conflict.png', prompt: 'Misty forest path', pair_id: null, positive: null,
negative: null }]);
await open(page);
await expect(panel(page).locator('p.small')).toHaveText('conflict.png');
await expect(context(page).locator('option')).toHaveText(OPTIONS);
const decided = page.waitForResponse(item => /\/decision$/.test(item.url()));
await apply(page).click();
const answer = await decided;
expect(answer.status()).toBe(409);
expect((await answer.json()).code).toBe('association-conflict');
await expect(warning(page)).toHaveText('This image is already linked to another prompt or negative, so this sampler path cannot be attached safely. The existing links were preserved.');
await expect(page).toHaveURL(/#\/review$/);
await expect(context(page)).toBeEnabled();
await expect(context(page)).toHaveValue(/^[0-9a-f]{64}$/);
expect(await links(request)).toEqual(before);
expect(await pending(request)).toBe(1);
await expect(reviewRow(page)).toHaveText('Needs review1');
// The file can still be left unresolved, which links nothing either.
await context(page).selectOption({ label: 'Leave unresolved' });
await apply(page).click();
await expect(toast(page)).toHaveText('Left unresolved.');
expect(await links(request)).toEqual(before);
expect(await pending(request)).toBe(0);
});
test('a limited item and an unavailable file can only be left unresolved', async ({ page, request }) => {
await seed(request, 0, { limited: true, unavailable: true });
await open(page);
await expect(cards(page)).toHaveCount(2);
await expect(panel(page).locator('p.small')).toHaveText('limited.png');
await expect(context(page).locator('option')).toHaveText(['Leave unresolved']);
await expect(warning(page)).toHaveText('This file has too many sampler paths to list. You can leave it unresolved.');
// The unavailable file: no current root lists it, so its media and Tag are inert.
const unavailable = cards(page).nth(1);
await expect(unavailable.locator('.image-open')).toHaveAttribute('aria-disabled', 'true');
await expect(unavailable.locator('.image-card-actions button').last()).toHaveAttribute('aria-disabled', 'true');
// F3: its empty media button keeps the concept's 1.33 media area, so the card lines up with its neighbours.
await expect(cards(page).first().locator('.image-open img')).toBeVisible();
const sizes = await cards(page).evaluateAll(nodes => nodes.map(node => {
const art = node.querySelector('.image-open').getBoundingClientRect();
return [Math.round(art.width), Math.round(art.height), Math.round(node.getBoundingClientRect().height)];
}));
expect(sizes[1]).toEqual(sizes[0]);
await unavailable.getByRole('button', { name: 'Inspect unavailable.png', exact: true }).click();
await expect(panel(page).locator('p.small')).toHaveText('unavailable.png');
await expect(warning(page)).toHaveText('This file is not under a current image source, so it cannot be opened. You can leave it unresolved.');
await expect(panel(page).getByRole('button', { name: 'Tag image', exact: true })).toHaveAttribute('aria-disabled', 'true');
const observes = [];
page.on('request', item => { if (item.url().endsWith('/image-annotations/observe')) observes.push(item.url()); });
await apply(page).click();
await expect(toast(page)).toHaveText('Left unresolved.');
expect(observes).toEqual([]);
expect(await pending(request)).toBe(1);
});
test('leave unresolved removes the item from pending', async ({ page, request }) => {
await seed(request, 2);
await open(page);
await context(page).selectOption({ label: 'Leave unresolved' });
await panel(page).getByRole('button', { name: 'Apply association', exact: true }).click();
await expect(toast(page)).toHaveText('Left unresolved.');
await expect(page).toHaveURL(/#\/images$/);
expect(await pending(request)).toBe(1);
});
test('leave for later writes nothing and goes to Images', async ({ page, request }) => {
await seed(request, 2);
await open(page);
const writes = [];
page.on('request', item => { if (item.method() !== 'GET') writes.push(item.url()); });
await panel(page).getByRole('button', { name: 'Leave for later', exact: true }).click();
await expect(page).toHaveURL(/#\/images$/);
expect(writes).toEqual([]);
expect(await pending(request)).toBe(2);
});
test('tag image from review saves tags and the item remains pending', async ({ page, request }) => {
await seed(request, 1);
await open(page);
const before = await links(request);
await panel(page).getByRole('button', { name: 'Tag image', exact: true }).click();
const dialog = page.getByRole('dialog', { name: 'Tag image', exact: true });
await expect(dialog).toBeVisible();
await expect(dialog.locator('p.tag-scope-note')).toHaveText('Tags belong to this image. Resolve its prompt association separately in Needs review.');
await dialog.getByRole('textbox', { name: 'Image tags · edit manually', exact: true }).fill('dunes');
await dialog.getByRole('button', { name: 'Save manual tags', exact: true }).click();
await expect(toast(page)).toHaveText('1 tags saved for review_0000.png. Prompt association still awaits review.');
await expect(dialog).toHaveCount(0);
expect(await pending(request)).toBe(1);
await expect(page).toHaveURL(/#\/review$/);
// The item is untouched: still listed, still open, nothing linked.
await expect(cards(page)).toHaveCount(1);
await expect(context(page)).toBeEnabled();
expect(await links(request)).toEqual(before);
const [item] = (await (await request.get('/prompt_manager/review/items')).json()).items;
expect([item.state, item.revision]).toEqual(['pending', 0]);
// The card's Tag opens the same dialog.
await cards(page).first().getByRole('button', { name: 'Tag or auto-tag review_0000.png', exact: true }).click();
await expect(dialog).toBeVisible();
await expect(dialog.locator('p.tag-scope-note')).toBeVisible();
await page.keyboard.press('Escape');
await expect(dialog).toHaveCount(0);
});
test('an empty queue shows the empty notice only', async ({ page, request }) => {
await seed(request, 0);
await page.goto(review);
await expect(main(page).locator('.notice h2')).toHaveText('No images need a decision');
await expect(main(page).locator('.notice p')).toHaveText(EMPTY);
await expect(main(page).locator('.gallery-layout')).toHaveCount(0);
await expect(reviewRow(page)).toHaveText('Needs review0');
});
test('load more appends the next page', async ({ page, request }) => {
await seed(request, 45);
await open(page);
await expect(cards(page)).toHaveCount(40);
const continuation = main(page).locator('.infinite-continuation');
await expect(continuation.locator('p')).toHaveText('40 of 45 images');
await continuation.getByRole('button', { name: 'Load more images', exact: true }).click();
await expect(cards(page)).toHaveCount(45);
await expect(continuation).toHaveCount(0);
});
test('a scan job with review items offers Review metadata in Activity', async ({ page, request }) => {
await seed(request, 1);
const started = await (await request.post('/__test__/jobs/start', { data: { total: 10, review_items: 2 } })).json();
await expect(await request.post(`/__test__/jobs/${started.job.id}/finish`, { data: {} })).toBeOK();
const plain = await (await request.post('/__test__/jobs/start', { data: { total: 10 } })).json();
await expect(await request.post(`/__test__/jobs/${plain.job.id}/finish`, { data: {} })).toBeOK();
await page.goto('/prompt_manager/workspace#/activity');
const item = main(page).locator(`.activity-item[data-job-id="${started.job.id}"]`);
await expect(item.getByRole('button', { name: 'Review metadata', exact: true })).toBeVisible();
await expect(main(page).locator(`.activity-item[data-job-id="${plain.job.id}"] button`)).toHaveCount(0);
await page.goto(`/prompt_manager/workspace#/activity/${started.job.id}`);
const actions = main(page).locator('.heading-actions button');
await expect(actions).toHaveText(['Download report', 'Review metadata']);
await actions.nth(1).click();
await expect(page).toHaveURL(/#\/review$/);
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Metadata review');
});
test('a failed item read says so and Retry reads it again', async ({ page, request }) => {
await seed(request, 1);
await page.route(url => /\/prompt_manager\/review\/items\/[0-9a-f-]{36}$/.test(url.pathname),
route => route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, code: 'review-failed', error: 'x' }) }),
{ times: 1 });
await page.goto(review);
await expect(panel(page).locator('.warning-note')).toHaveText('This item could not be read. Select it again to retry.');
await expect(panel(page).getByText('Could not read prompts from this file.', { exact: true })).toBeVisible();
await expect(panel(page).getByRole('button', { name: 'Apply association', exact: true })).toHaveAttribute('aria-disabled', 'true');
await panel(page).getByRole('button', { name: 'Retry prompt details', exact: true }).click();
await expect(context(page)).toBeEnabled();
await expect(context(page).locator('option')).toHaveText(OPTIONS);
await expect(panel(page).locator('.warning-note')).toBeHidden();
});
test('a failed queue read offers Retry', async ({ page, request }) => {
await seed(request, 1);
await page.route(url => url.pathname.endsWith('/prompt_manager/review/items') && url.searchParams.get('limit') === '40',
route => route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, code: 'review-failed', error: 'x' }) }),
{ times: 1 });
await page.goto(review);
await expect(main(page).getByRole('alert')).toHaveText('The review queue could not be read.');
await main(page).getByRole('button', { name: 'Retry', exact: true }).click();
await expect(context(page)).toBeEnabled();
await expect(main(page).locator('.notice h2')).toHaveText('1 image needs a decision');
});
test('keyboard focus labels and touch targets', async ({ page, request }) => {
await seed(request, 2);
await open(page);
const first = cards(page).first();
for (const [name, role] of [['Open review_0000.png', 'button'], ['Inspect review_0000.png', 'button'],
['Tag or auto-tag review_0000.png', 'button']]) await expect(first.getByRole(role, { name, exact: true })).toBeVisible();
await expect(main(page).getByRole('list', { name: 'Review queue', exact: true })).toBeVisible();
await expect(panel(page)).toBeVisible();
// Keyboard: Info on the second card opens it, and the panel's controls take focus in their order.
const info = cards(page).nth(1).getByRole('button', { name: 'Inspect review_0001.png', exact: true });
await info.focus();
await page.keyboard.press('Enter');
await expect(panel(page).locator('p.small')).toHaveText('review_0001.png');
await expect(context(page)).toBeEnabled();
await context(page).focus();
await expect(context(page)).toBeFocused();
const order = await panel(page).evaluate(node => [...node.querySelectorAll('button, select')]
.filter(control => control.offsetParent !== null).map(control => control.getAttribute('aria-label') || control.textContent.trim()));
expect(order[0]).toBe('Tag image');
expect(order.slice(-2)).toEqual(['Leave for later', 'Apply association']);
// Targets: the panel's controls are 40px on a coarse pointer (the concept's coarse buttons) and 24px otherwise (the
// WCAG 2.2 minimum); the card's buttons keep the concept's compact size, never under 24px (its ≤700px rule).
const coarse = await page.evaluate(() => matchMedia('(pointer: coarse)').matches);
const sized = [
[panel(page).getByRole('button', { name: 'Tag image', exact: true }), coarse ? 40 : 24],
[context(page), coarse ? 40 : 24],
[panel(page).getByRole('button', { name: 'Leave for later', exact: true }), coarse ? 40 : 24],
[apply(page), coarse ? 40 : 24],
[first.getByRole('button', { name: 'Inspect review_0000.png', exact: true }), 24],
[first.getByRole('button', { name: 'Tag or auto-tag review_0000.png', exact: true }), 24],
];
for (const [control, minimum] of sized) {
const box = await control.boundingBox();
const label = await control.evaluate(node => node.getAttribute('aria-label') || node.textContent.trim().slice(0, 40));
expect(box.height, label).toBeGreaterThanOrEqual(minimum);
expect(box.width, label).toBeGreaterThanOrEqual(24);
}
// Enter on Leave for later returns to Images.
await panel(page).getByRole('button', { name: 'Leave for later', exact: true }).focus();
await page.keyboard.press('Enter');
await expect(page).toHaveURL(/#\/images$/);
});
test('an open page shows items the monitor adds and keeps the open choice', async ({ page, request }) => {
await seed(request, 1);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await context(page).selectOption({ label: OPTIONS[1] });
await expect(reviewRow(page)).toHaveText('Needs review1');
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(cards(page)).toHaveCount(2);
await expect(main(page).locator('.notice h2')).toHaveText('2 images need a decision');
await expect(reviewRow(page)).toHaveText('Needs review2');
await expect(cards(page).first()).toHaveClass(/selected/);
await expect(panel(page).locator('p.small')).toHaveText('review_0000.png');
await expect(context(page).locator('option:checked')).toHaveText(OPTIONS[1]);
await expect(warning(page)).toBeHidden();
});
test('an open page shows items a metadata scan adds', async ({ page, request }) => {
await seed(request, 1);
await expect(await request.post('/__test__/review/add', { data: { kind: 'file' } })).toBeOK();
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await expect(cards(page)).toHaveCount(1);
const started = await request.post('/prompt_manager/jobs/scan', { data: { scope: 'new' } });
expect(started.status()).toBe(202);
await expect(cards(page)).toHaveCount(2);
await expect(cards(page).nth(1).getByRole('button', { name: 'Inspect added_0000.png', exact: true })).toBeVisible();
await expect(reviewRow(page)).toHaveText('Needs review2');
});
test('a decision in another tab leaves the open page and opens the next image', async ({ page, request, context: browserContext }) => {
await seed(request, 2);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
const other = await browserContext.newPage();
await open(other);
await context(other).selectOption({ label: 'Leave unresolved' });
await apply(other).click();
await expect(other).toHaveURL(/#\/images$/);
await other.close();
await expect(cards(page)).toHaveCount(1);
await expect(panel(page).locator('p.small')).toHaveText('review_0001.png');
await expect(warning(page)).toHaveText('review_0000.png was decided elsewhere, so the next image is open.');
await expect(main(page).locator('.notice h2')).toHaveText('1 image needs a decision');
await expect(reviewRow(page)).toHaveText('Needs review1');
await expect(context(page)).toBeEnabled();
});
/** The queue reads (`?limit=40`, cursors) and count reads (`?limit=1`) the page sends from now on. */
function reviewReads(page) {
const reads = [];
page.on('request', item => {
const url = new URL(item.url());
if (url.pathname === '/prompt_manager/review/items') reads.push(url.searchParams.get('cursor') ? 'more' : url.search);
});
return reads;
}
test('a visit reads its queue once, and a change reads it once more', async ({ page, request }) => {
// F3 ruling: the stream's ready must not read a queue the first read already holds.
await seed(request, 1);
const reads = reviewReads(page);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(cards(page)).toHaveCount(2);
await expect(reviewRow(page)).toHaveText('Needs review2');
// The page's first read and the change's re-read; the shell's count read at start, and the row follows the page.
expect(reads.filter(search => search === '?limit=40')).toHaveLength(2);
expect(reads.filter(search => search === '?limit=1')).toHaveLength(1);
});
test('a ready whose version cannot be read re-reads the queue once', async ({ page, request }) => {
// The stream's ready names no readable version (an older server, a proxy that rewrote it): the page reads again.
await seed(request, 1);
const reads = reviewReads(page);
// M1: `retry` keeps the browser from reconnecting the ended stream during the test, so exactly one re-read follows.
await page.route(url => url.pathname === '/prompt_manager/review/events', route => route.fulfill({
status: 200, headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-store' },
body: 'retry: 3600000\nevent: ready\ndata: not json\n\n' }));
let reread = 0;
const second = page.waitForResponse(item => item.url().endsWith('/prompt_manager/review/items?limit=40') && ++reread === 2);
await open(page);
await second;
await expect(cards(page)).toHaveCount(1);
expect(reads.filter(search => search === '?limit=40')).toHaveLength(2);
});
const live = page => main(page).locator('.heading .review-live');
const NOT_LIVE = 'Live updates are paused: changes made elsewhere appear once the connection returns.';
test('a stream the browser gives up on is opened again, and the page says it is not live meanwhile', async ({ page, request }) => {
// I3: a 503 (shutdown, or a proxy while the backend restarts) fails an EventSource for good.
await seed(request, 1);
let attempts = 0;
await page.route(url => url.pathname === '/prompt_manager/review/events', route => {
attempts += 1;
if (attempts === 1) return route.fulfill({ status: 503, contentType: 'application/json', body: '{"success":false}' });
return route.fallback();
});
await open(page);
await expect(live(page)).toHaveText(NOT_LIVE);
await expect(live(page)).toBeVisible();
// The page opens the stream again (after 3 s) and is live once it sends ready.
await expect.poll(async () => (await streams(request)).streams, { timeout: 15000 }).toBe(1);
await expect(live(page)).toBeHidden();
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(cards(page)).toHaveCount(2);
expect(attempts).toBe(2);
});
/** Make the page hidden or visible as a tab switch does. */
async function setVisible(page, visible) {
await page.evaluate(shown => {
Object.defineProperty(document, 'hidden', { configurable: true, get: () => !shown });
Object.defineProperty(document, 'visibilityState', { configurable: true, get: () => (shown ? 'visible' : 'hidden') });
document.dispatchEvent(new Event('visibilitychange'));
}, visible);
}
test('a hidden tab closes its stream and a visible one reads only what changed meanwhile', async ({ page, request }) => {
// I4: background tabs must not hold one of the origin's six connections.
await seed(request, 1);
const reads = reviewReads(page);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await setVisible(page, false);
// The server drops a closed stream at its next write: this monitor addition is one.
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect.poll(async () => (await streams(request)).streams).toBe(0);
await expect(cards(page)).toHaveCount(1);
expect(reads.filter(search => search === '?limit=40')).toHaveLength(1);
// Visible again: the reopened stream's ready names a newer version, so the page reads once.
const reread = page.waitForResponse(item => item.url().endsWith('/prompt_manager/review/items?limit=40'));
await setVisible(page, true);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await reread;
await expect(cards(page)).toHaveCount(2);
await expect(live(page)).toBeHidden();
expect(reads.filter(search => search === '?limit=40')).toHaveLength(2);
});
test('a re-read refreshes a card whose media or target changed', async ({ page, request }) => {
// M6 and M10: a root came or went between reads; the card follows, and a known thumbnail is its media.
await seed(request, 2);
let rewrite = false;
await page.route(url => url.pathname === '/prompt_manager/review/items' && url.search === '?limit=40', async route => {
if (!rewrite) return route.fallback();
const response = await route.fetch();
const body = await response.json();
body.items[0] = { ...body.items[0], thumbnail_url: `${body.items[0].url}&thumb=1` };
body.items[1] = { ...body.items[1], url: null, thumbnail_url: null, target: null, unavailable: true };
return route.fulfill({ response, json: body });
});
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
const second = cards(page).nth(1);
await expect(second.getByRole('button', { name: 'Open review_0001.png', exact: true })).not.toHaveAttribute('aria-disabled', 'true');
rewrite = true;
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(cards(page)).toHaveCount(3);
await expect(cards(page).first().locator('img')).toHaveAttribute('src', /&thumb=1$/);
await expect(second.getByRole('button', { name: 'Open review_0001.png', exact: true })).toHaveAttribute('aria-disabled', 'true');
await expect(second.locator('img')).toHaveCount(0);
await expect(second.locator('.image-card-actions button').last()).toHaveAttribute('aria-disabled', 'true');
await expect(cards(page).first()).toHaveClass(/selected/);
});
test('leaving during the observe releases the capture it answers', async ({ page, request }) => {
// M4: the server made the capture; the page left before the answer came.
await seed(request, 1);
await open(page);
let held;
const sent = new Promise(resolve => { held = resolve; });
await page.route(url => url.pathname === '/prompt_manager/image-annotations/observe', async route => {
const response = await route.fetch();
held(() => route.fulfill({ response }));
});
await apply(page).click();
const answer = await sent;
expect((await streams(request)).captures).toBe(1);
await page.evaluate(() => { window.location.hash = '#/tags'; });
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Tags & categories');
await answer();
await expect.poll(async () => (await streams(request)).captures).toBe(0);
});
test('a live re-read keeps the loaded depth and the scroll position', async ({ page, request }) => {
// F3 ruling: a queue change re-reads as many items as Load more had shown, and the page does not jump.
await seed(request, 45);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await main(page).getByRole('button', { name: 'Load more images', exact: true }).click();
await expect(cards(page)).toHaveCount(45);
await cards(page).last().scrollIntoViewIfNeeded();
const scrollTop = () => cards(page).first().evaluate(node => {
for (let el = node.parentElement; el; el = el.parentElement) {
const style = getComputedStyle(el);
if (/(auto|scroll)/.test(style.overflowY) && el.scrollHeight > el.clientHeight) return el.scrollTop;
}
return document.scrollingElement.scrollTop;
});
const before = await scrollTop();
expect(before).toBeGreaterThan(0);
const reads = reviewReads(page);
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(main(page).locator('.notice h2')).toHaveText('46 images need a decision');
await expect(cards(page)).toHaveCount(46);
expect(await scrollTop()).toBe(before);
expect(reads).toEqual(['?limit=40', 'more']);
await expect(main(page).locator('.infinite-continuation')).toHaveCount(0);
});
test('teardown releases captures listeners and requests', async ({ page, request }) => {
await seed(request, 1);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
const inFlight = new Set();
page.on('request', item => { if (item.url().includes('/prompt_manager/review/')) inFlight.add(item); });
page.on('requestfinished', item => inFlight.delete(item));
page.on('requestfailed', item => inFlight.delete(item));
// A decision whose answer never comes: leaving the page must still release its capture.
let held;
const sent = new Promise(resolve => { held = resolve; });
await page.route(url => /\/decision$/.test(url.pathname), route => held(route));
await apply(page).click();
const route = await sent;
expect((await streams(request)).captures).toBe(1);
await page.evaluate(() => { window.location.hash = '#/tags'; });
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Tags & categories');
await expect.poll(async () => (await streams(request)).captures).toBe(0);
await route.abort().catch(() => {});
// The server drops a closed stream at its next write: a monitor addition is one.
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect.poll(async () => (await streams(request)).streams).toBe(0);
await expect.poll(() => inFlight.size).toBe(0);
});
+1 -1
View File
@@ -372,7 +372,7 @@ test('Favorites is the first SAVED VIEWS row, with a real count that follows eve
expect(await favoritesRow(page).evaluate(link => [...link.childNodes].map(node => node.nodeType === Node.TEXT_NODE
? `text:${node.textContent}` : `${node.localName}.${node.getAttribute('class')}`)))
.toEqual(['svg.icon', 'text:Favorites', 'span.nav-count']);
expect(await savedNav(page).locator('a').evaluateAll(links => links.map(link => link.dataset.route))).toEqual(['favorites', 'review']);
expect(await savedNav(page).locator('a').evaluateAll(links => links.map(link => link.dataset.route))).toEqual(['favorites']);
await star(page, '1:shared/same.png');
await expect(toast(page)).toHaveText('Added to favorites.');
@@ -1,401 +0,0 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { seedConcept } from './concept-inventory.mjs';
// System › Sources & scans (#249 System S4 Task 11) on the concept-shaped real fixture, whose two sources are this
// server's `output` and `input` folders: the source list, monitoring and worker rows read GET /settings (S4 Task 9),
// Add/Remove edit the host draft behind the one Save row, and every scan starts the session job through the shared
// "Review metadata scan" (D55), scoped to one source from its row (S4 Task 10). The hooks are workspace_system_fixture.py's.
const workspace = '/prompt_manager/workspace';
const SETTINGS_URL = '/prompt_manager/settings';
const NOTICE_KEY = 'pm_update_notice_dismissed';
const OUTSIDE_REASON = 'Gallery root must be inside a ComfyUI directory (output, input, temp, user) or one listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS';
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.systemErrors = [];
page.on('pageerror', error => page.systemErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/release');
await request.post('/__test__/jobs/restart');
await request.post('/__test__/system/reset');
expect(page.systemErrors).toEqual([]);
});
const toast = page => page.locator('#workspace-toast');
const layoutOf = page => page.locator('#workspace-main .settings-layout');
const sourceList = page => layoutOf(page).locator('#source-list');
const statusOf = page => layoutOf(page).locator('#settings-status');
const saveButton = page => layoutOf(page).getByRole('button', { name: 'Save preferences', exact: true });
const dialog = page => page.locator('dialog#modal[open]');
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
/** Open Sources with the update notice dismissed (or shown, `notice: true`) once the source list has loaded. */
async function openSources(page, { notice = false } = {}) {
if (!notice) await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY);
await page.goto(`${workspace}#/system/sources`);
await expect(sourceList(page).locator('.operation').first()).toBeVisible();
await expect(layoutOf(page).locator('#workers')).toBeEnabled();
return layoutOf(page);
}
/** The source list's rows: `[h3, p, [buttons]]`, then its note line when there is one. */
function rowsOf(page) {
return sourceList(page).evaluate(list => [...list.children].map(row => row.matches('.operation')
? [row.querySelector('h3').textContent, row.querySelector('p').textContent,
[...row.querySelectorAll('.row-actions > button')].map(button => button.textContent)]
: [row.localName, row.className, row.textContent]));
}
const sourceRow = (page, title) => sourceList(page).locator('.operation').filter({ has: page.getByRole('heading', { name: title, exact: true }) });
test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed) without scanning', async ({ page, request }) => {
const writes = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) writes.push(item.url()); });
const layout = await openSources(page, { notice: true });
const notice = layout.locator(':scope > .notice');
await expect(notice.locator('h2')).toHaveText('Bring older images into your prompt history');
await expect(notice.locator('p')).toHaveText('The schema upgrades automatically. An optional metadata scan links existing positives and negatives without regenerating all thumbnails.');
await expect(notice.locator('.row-actions > button')).toHaveText(['Later', 'Review scan']);
await notice.getByRole('button', { name: 'Later', exact: true }).click();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0');
await expect(dialog(page)).toHaveCount(0);
// The dismissal is the legacy admin page's own, so neither page shows the notice again.
await page.reload();
await expect(sourceList(page).locator('.operation').first()).toBeVisible();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
expect(writes).toEqual([]);
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('Review scan opens the shared Review metadata scan with all sources', async ({ page, request }) => {
const layout = await openSources(page, { notice: true });
// The notice's Review scan dismisses it, as the legacy page's does, and reviews every source.
await layout.locator(':scope > .notice').getByRole('button', { name: 'Review scan', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan');
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']);
expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0');
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
// Enrich existing metadata is the same review.
const enrich = layout.locator('.operation').filter({ hasText: 'Enrich existing metadata' });
await expect(enrich.locator('p')).toHaveText('Resolve both prompts, deduplicate recipes, preserve uncertain links.');
await enrich.getByRole('button', { name: 'Review scan', exact: true }).click();
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']);
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('Scan changes opens it with that one source ("1 output folder") and starts a scoped job that opens in Activity', async ({ page, request }) => {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['input'] } })).toBeOK();
const bodies = [];
page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan')) bodies.push(item.postDataJSON()); });
await openSources(page);
const input = sourceRow(page, 'ComfyUI input');
await expect(input.locator('p')).toHaveText('input · watching');
await input.getByRole('button', { name: 'Scan changes', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan');
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['1 output folder', 'New and unresolved files']);
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
// The toast says it started (or, for a scan this quick, that it already finished); Activity shows its progress.
await expect(toast(page)).toHaveText(/^Scan (started\.|finished · )/);
expect(bodies).toEqual([{ scope: 'new', roots: [1], paths: ['input'] }]);
const id = page.url().split('/').pop();
// The job walks only that source (the count is recorded once the walk has listed it).
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.scope)
.toMatchObject({ files: 'new', roots: [1], output_folders: 1 });
await expect(page.locator('#workspace-main h1')).toHaveText('Scan for metadata');
});
test('Add source validates on the server, then marks the host draft dirty; Save persists', async ({ page, request }) => {
// A folder inside ComfyUI's output folder that is not a source yet.
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 1 } })).toBeOK();
const layout = await openSources(page);
expect(await rowsOf(page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']],
['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']],
]);
await expect(layout.locator('.panel-heading > h2')).toHaveText('Image sources');
const add = layout.locator('.panel-heading').getByRole('button', { name: '+ Add source', exact: true });
await add.click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Add image source');
await expect(dialog(page).locator('label[for="action-input"]')).toHaveText('Folder path');
await expect(dialog(page).locator('#action-input')).toHaveValue('');
await expect(dialog(page).locator('.dialog-actions > button')).toHaveText(['Cancel', 'Add']);
await dialog(page).locator('#action-input').fill('output/unthumbnailed');
const validated = page.waitForResponse(response => response.url().endsWith('/prompt_manager/settings/validate-source'));
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
expect(await (await validated).json()).toEqual({ success: true, valid: true, label: 'unthumbnailed', path: 'output/unthumbnailed', code: null, reason: null });
await expect(dialog(page)).toHaveCount(0);
await expect(add).toBeFocused();
// Validation wrote nothing: the draft is dirty and the server still has two sources.
expect(await rowsOf(page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']],
['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']],
['unthumbnailed', 'output/unthumbnailed · not saved yet', ['Remove']],
]);
await expect(statusOf(page)).toHaveText('Unsaved changes.');
expect((await serverJson(request, SETTINGS_URL)).sources.effective).toHaveLength(2);
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const body = (await saved).postDataJSON();
expect(body.gallery_root_paths).toEqual(['output', 'input', 'output/unthumbnailed']);
expect(typeof body.expected_revision).toBe('string');
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
// The list is the server's again: three effective sources, each scannable.
await expect(sourceRow(page, 'unthumbnailed').locator('p')).toHaveText('output/unthumbnailed · manual scan');
await expect(sourceRow(page, 'unthumbnailed').getByRole('button', { name: 'Scan folder', exact: true })).toBeVisible();
expect((await serverJson(request, SETTINGS_URL)).sources.effective.map(source => source.path)).toEqual(['output', 'input', 'output/unthumbnailed']);
});
test('invalid source shows the server reason', async ({ page, request }) => {
const layout = await openSources(page);
await layout.getByRole('button', { name: '+ Add source', exact: true }).click();
const field = dialog(page).locator('#action-input');
for (const [path, reason, submit] of [
['output/not-there', 'Gallery root does not exist', 'click'],
// Enter in the field adds too (M6).
['/etc', OUTSIDE_REASON, 'Enter'],
]) {
await field.fill(path);
if (submit === 'Enter') await field.press('Enter');
else await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText(reason);
await expect(field).toHaveAttribute('aria-invalid', 'true');
await expect(field).toHaveValue(path);
}
// A folder already listed (the server's answer names it) is not added twice.
await field.fill('input');
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('This folder is already a source.');
// A check that cannot finish says so and adds nothing.
await page.route('**/prompt_manager/settings/validate-source', route => route.fulfill({ status: 500,
json: { success: false, code: null, error: 'Failed to check the source' } }));
await field.fill('output/not-checked');
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('The folder could not be checked. Try again.');
await page.unroute('**/prompt_manager/settings/validate-source');
// Typing again clears the reason.
await field.fill('output/unthumbnailed');
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveCount(0);
await expect(field).not.toHaveAttribute('aria-invalid', 'true');
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
await expect(statusOf(page)).toHaveText('No unsaved changes.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2);
});
test('Remove marks the draft; removing the last entry states auto-detect', async ({ page, request }) => {
await openSources(page);
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
expect(await rowsOf(page)).toEqual([['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']]]);
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await sourceRow(page, 'ComfyUI output').getByRole('button', { name: 'Remove', exact: true }).click();
expect(await rowsOf(page)).toEqual([['p', 'small', 'No source is configured. After saving, ComfyUI’s output folder is used.']]);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
// Saved: ComfyUI's output folder is detected and listed without Remove.
await expect.poll(rowsOf.bind(null, page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder']],
['p', 'small', 'No source is configured; ComfyUI’s output folder is used.'],
]);
const { sources } = await serverJson(request, SETTINGS_URL);
expect([sources.configured, sources.auto_detected]).toEqual([[], true]);
});
test('a configured source that does not exist says why and can be removed', async ({ page, request }) => {
await expect(await request.post('/__test__/system/missing-source')).toBeOK();
await openSources(page);
// The fixture's unmounted drive is not inside a ComfyUI folder, so the server says that (it never probes outside).
expect((await rowsOf(page)).at(-1)).toEqual(['unmounted-drive', `unmounted-drive · ${OUTSIDE_REASON}`, ['Remove']]);
await sourceRow(page, 'unmounted-drive').getByRole('button', { name: 'Remove', exact: true }).click();
await expect(sourceList(page).locator('.operation')).toHaveCount(2);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output', 'input']);
});
test('watching only with a live observer; restart pending otherwise', async ({ page, request }) => {
// No image monitor in this process: every source is scanned by hand.
await openSources(page);
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · manual scan', 'input · manual scan']);
// A live observer that scheduled output: output is watched; input was added since it started.
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK();
await page.reload();
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · restart pending']);
await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan folder']);
// An observer that died watches nothing.
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'], alive: false } })).toBeOK();
await page.reload();
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · restart pending', 'input · restart pending']);
});
test('Watch for new outputs saves monitoring_enabled; a restart applies it', async ({ page, request }) => {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'] } })).toBeOK();
const layout = await openSources(page);
const monitoring = layout.getByRole('switch', { name: 'Watch for new outputs', exact: true });
await expect(monitoring).toBeChecked();
await expect(layout.locator('.setting-row').filter({ has: page.locator('#monitoring') }).locator('p'))
.toHaveText('New files enter the same library. Existing scans do not count as new runs.');
await monitoring.uncheck();
await expect(statusOf(page)).toHaveText('Unsaved changes.');
const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const answer = await (await saved).json();
expect(answer.restart_required_fields).toEqual(['monitoring_enabled']);
// D128: saved for the next start, and the status says a restart applies it.
await expect(statusOf(page)).toHaveText('Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// The running observer keeps watching until ComfyUI restarts, and the list says so.
const { settings, monitor } = await serverJson(request, SETTINGS_URL);
expect([settings.monitoring_enabled, monitor.observer_alive]).toEqual([false, true]);
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching until restart', 'input · watching until restart']);
await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan changes']);
// Switched back on before the restart: the observer still runs, so nothing waits for one.
await monitoring.check();
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · watching']);
});
test('worker range max is cpu_count', async ({ page, request }) => {
const { settings } = await serverJson(request, SETTINGS_URL);
const layout = await openSources(page);
const workers = layout.locator('#workers');
await expect(layout.locator('label[for="workers"]')).toHaveText('Worker threads');
await expect(workers).toHaveAttribute('type', 'range');
await expect(workers).toHaveAttribute('min', '1');
await expect(workers).toHaveAttribute('max', String(settings.cpu_count));
await expect(workers).toHaveValue(String(settings.worker_threads));
const next = settings.worker_threads === 1 ? settings.cpu_count : 1;
await workers.fill(String(next));
await expect(statusOf(page)).toHaveText(next === settings.worker_threads ? 'No unsaved changes.' : 'Unsaved changes.');
test.skip(next === settings.worker_threads, 'One core: the range has a single value');
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
expect((await saved).postDataJSON().worker_threads).toBe(next);
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).settings.worker_threads).toBe(next);
});
test('Refresh files calls /gallery/rescan and toasts its counts', async ({ page }) => {
const layout = await openSources(page);
const refresh = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Refresh files', exact: true }) });
await expect(refresh.locator('p')).toHaveText('Read every source again to find files added or removed outside ComfyUI.');
const answered = page.waitForResponse(response => response.url().endsWith('/prompt_manager/gallery/rescan'));
await refresh.getByRole('button', { name: 'Refresh', exact: true }).click();
const { total, roots } = await (await answered).json();
expect(roots).toBe(2);
await expect(toast(page)).toHaveText(`Files refreshed. ${total.toLocaleString('en-US')} files in 2 output folders.`);
// A failed refresh says so.
await page.route('**/prompt_manager/gallery/rescan', route => route.fulfill({ status: 500, json: { success: false, error: 'busy disk' } }));
await refresh.getByRole('button', { name: 'Refresh', exact: true }).click();
await expect(toast(page)).toHaveText('Files could not be refreshed. Try again.');
});
test('Thumbnail generation hands off to the existing gallery until its review dialog', async ({ page }) => {
const layout = await openSources(page);
const thumbnails = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
await expect(thumbnails.locator('p')).toHaveText('Generate missing previews; rebuilding all is a separate explicit option.');
const popup = page.waitForEvent('popup');
await thumbnails.getByRole('button', { name: 'Review options', exact: true }).click();
const gallery = await popup;
expect(new URL(gallery.url()).pathname).toBe('/prompt_manager/gallery');
await gallery.close();
await expect(statusOf(page)).toHaveText('No unsaved changes.');
});
test('a save refused by a running scan shows the busy link toast', async ({ page, request }) => {
const running = await (await request.post('/__test__/jobs/start', { data: { title: 'Scan for metadata' } })).json();
await openSources(page);
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
await saveButton(page).click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
await expect(statusOf(page)).toHaveText('Some changes were not saved. Review the highlighted settings.');
// The draft is kept for the next Save, and the server still has both sources.
await expect(sourceList(page).locator('.operation')).toHaveCount(1);
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2);
// An answer that does not name its job (an older server): the toast names the running one from the job list.
await page.route(`**${SETTINGS_URL}`, route => (route.request().method() === 'POST'
? route.fulfill({ status: 409, json: { success: false, code: 'busy', error: 'Scan for metadata already running' } })
: route.continue()));
await page.locator('#workspace-toast').evaluate(node => { node.textContent = ''; });
await saveButton(page).click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
await page.unroute(`**${SETTINGS_URL}`);
await request.post(`/__test__/jobs/${running.job.id}/finish`, { data: { state: 'finished' } });
});
test('a source list that cannot be read says so and retries', async ({ page }) => {
await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY);
let failing = true;
await page.route(`**${SETTINGS_URL}`, route => (failing && route.request().method() === 'GET'
? route.fulfill({ status: 500, json: { success: false, error: 'Injected failure' } }) : route.continue()));
await page.goto(`${workspace}#/system/sources`);
await expect(sourceList(page).getByRole('alert')).toHaveText('Sources could not be loaded. Your saved data has not changed.');
await expect(sourceList(page).locator('.operation')).toHaveCount(0);
// The server settings rows say so too, and stay disabled.
await expect(layoutOf(page).locator('#workers')).toBeDisabled();
failing = false;
await sourceList(page).getByRole('button', { name: 'Retry', exact: true }).click();
await expect(sourceList(page).locator('.operation')).toHaveCount(2);
});
test('a scan from a list that changed since it was read starts nothing, says why and reads the list again', async ({ page, request }) => {
await openSources(page);
// The legacy page (or another tab) removes ComfyUI output: the page's "input" is index 1, now past the end.
await expect(await request.post(SETTINGS_URL, { data: { gallery_root_paths: ['input'] } })).toBeOK();
const refused = page.waitForResponse(response => response.url().endsWith('/prompt_manager/jobs/scan'));
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click();
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
const answer = await refused;
expect([answer.status(), (await answer.json()).code]).toEqual([409, 'sources-changed']);
await expect(toast(page)).toHaveText('The sources changed since they were loaded. Reload them.');
// No scan started (the other save ran as its own source-change hold).
expect((await serverJson(request, '/prompt_manager/jobs')).jobs.map(job => job.kind)).toEqual(['source_change']);
// Read again: the one source is index 0 now, and its scan starts.
await expect.poll(rowsOf.bind(null, page)).toEqual([['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']]]);
await expect(statusOf(page)).toHaveText('No unsaved changes.');
const started = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan'));
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click();
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
expect((await started).postDataJSON()).toEqual({ scope: 'new', roots: [0], paths: ['input'] });
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
});
test('a save refused for one unusable source names it and marks its row', async ({ page, request }) => {
await expect(await request.post('/__test__/system/missing-source', { data: { inside: true } })).toBeOK();
await openSources(page);
await expect(sourceRow(page, 'unplugged').locator('p')).toHaveText('output/unplugged · manual scan');
// The drive is unplugged after the page read the list; removing another source resends the whole list.
await expect(await request.post('/__test__/system/unplug')).toBeOK();
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const answer = await saved;
expect(answer.status()).toBe(400);
expect(await answer.json()).toMatchObject({ code: 'invalid-source', source: 'output/unplugged', index: 1 });
await expect(statusOf(page)).toHaveText("output/unplugged can't be used: Gallery root does not exist. Remove it or reconnect it, then save.");
await expect(sourceRow(page, 'unplugged')).toHaveAttribute('aria-invalid', 'true');
await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(1);
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(3);
// Removed, the list saves.
await sourceRow(page, 'unplugged').getByRole('button', { name: 'Remove', exact: true }).click();
await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(0);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output']);
});
-623
View File
@@ -1,623 +0,0 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { seedConcept } from './concept-inventory.mjs';
// System › Overview (#249 System S2 Task 6) on the concept-shaped real fixture: the metrics, rows, Statistics dialog and
// Health report read the server, never sample values (D93, D95, D96). The hooks are workspace_system_fixture.py's.
const workspace = '/prompt_manager/workspace';
const OVERVIEW_URL = '/prompt_manager/system/overview';
const THUMBNAILS_URL = '/prompt_manager/images/thumbnails/status';
const number = value => value.toLocaleString('en-US');
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
page.manifest = await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.systemErrors = [];
page.on('pageerror', error => page.systemErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/release');
await request.post('/__test__/system/reset');
expect(page.systemErrors).toEqual([]);
});
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
const layoutOf = page => page.locator('#workspace-main .settings-layout');
async function openOverview(page) {
await page.goto(`${workspace}#/system/overview`);
await expect(layoutOf(page).locator(':scope > .summary-metrics')).toBeVisible();
return layoutOf(page);
}
/** The rows of one Overview surface: `[h3, p, button]`. */
function rowsOf(layout, heading) {
return layout.locator('.two-col > section.surface').filter({ has: layout.page().getByRole('heading', { name: heading, exact: true }) })
.locator('.operation').evaluateAll(rows => rows.map(row => [row.querySelector('h3').textContent,
row.querySelector('p').textContent, row.querySelector('.row-actions button').textContent]));
}
test('overview shows real counts and Not checked', async ({ page, request }) => {
const writes = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) writes.push(item.url()); });
const overview = await serverJson(request, OVERVIEW_URL);
const thumbnails = await serverJson(request, THUMBNAILS_URL);
// The concept fixture: six positives, their saved negatives and 24 linked images, all with thumbnails.
expect([overview.library.positive_prompts, overview.images.indexed, thumbnails.pending]).toEqual([6, 24, 0]);
// I2: the Overview never walks the sources for thumbnails; the row reads the status route itself.
expect(overview).not.toHaveProperty('thumbnails');
// The fixture records no ambiguous file: the Needs review queue is empty.
expect(overview.needs_review).toEqual({ pending: 0 });
const layout = await openOverview(page);
expect(await layout.locator('.summary-metrics > .metric').evaluateAll(items => items.map(item =>
[item.querySelector('strong').textContent, item.querySelector('span').textContent]))).toEqual([
[number(overview.library.positive_prompts), 'Positive prompts'],
[number(overview.library.negative_variants), 'Negative variants'],
[number(overview.images.indexed), 'Images'],
['Not checked', 'Database integrity'],
]);
expect(await rowsOf(layout, 'Library health')).toEqual([
['0 ambiguous image links', 'Review metadata before assigning a positive / negative pair.', 'Review metadata'],
['Backup status', 'No retained backup record.', 'View backups'],
['No missing thumbnails', 'Generate only missing previews when necessary.', 'Review thumbnails'],
]);
expect(await rowsOf(layout, 'Workspace preferences')).toEqual([
['One display configuration', 'Grid and density settings are shared across views.', 'Appearance'],
['2 output sources', 'Monitoring and enrichment share one source list.', 'Manage sources'],
['Statistics', 'Runs and ratings by category, with library totals.', 'Explore'],
]);
expect(writes).toEqual([]);
});
test('Run health checks opens Health report and updates metric 4', async ({ page, request }) => {
await page.goto(`${workspace}#/system/maintenance`);
const layout = layoutOf(page);
await expect(layout.locator(':scope > .notice h2')).toHaveText('Check first. Review changes. Then apply.');
// The rest of Maintenance is still the existing tools' (D94).
await expect(layout.locator(':scope > section.handoff h2')).toHaveText('Your current tools are ready');
const run = layout.getByRole('button', { name: 'Run health checks', exact: true });
await run.click();
const dialog = page.locator('dialog#modal[open]');
await expect(dialog.locator('#modal-title')).toHaveText('Health report');
const { report } = await serverJson(request, '/prompt_manager/system/health');
expect(report.status).toBe('incomplete');
expect(await dialog.locator('.operation-list > .operation').evaluateAll(rows => rows.map(row =>
[row.querySelector('h3').textContent, row.querySelector('span.chip').textContent]))).toEqual([
['Integrity check', 'Passed'], ['Foreign keys & pair references', 'Passed'], ['Canonical hashes', 'Passed'],
['Pair uniqueness', 'Passed'], ['Missing files · output', 'Passed'], ['Missing files · input', 'Passed'],
['Ambiguous metadata', 'Not available yet'], ['Consistency', 'Passed'],
]);
await expect(dialog.locator('#modal-body > p.small')).toHaveText(/^Checked .+\. Some checks could not run, so this report is incomplete\.$/);
await expect(dialog.locator('.dialog-actions button')).toHaveText(['Close', 'Review missing links']);
await dialog.getByRole('button', { name: 'Close', exact: true }).click();
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
await expect(run).toBeFocused();
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click();
await expect(layout.locator('.metric').nth(3).locator('strong')).toHaveText('Incomplete');
// A missing file and an unmounted source: the report counts the readable sources and names the other one.
await expect(await request.post('/__test__/system/missing-file')).toBeOK();
await expect(await request.post('/__test__/system/missing-source')).toBeOK();
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click();
await layout.getByRole('button', { name: 'Run health checks', exact: true }).click();
await expect(dialog.locator('.operation').filter({ hasText: 'Missing files' }).locator('.chip'))
.toHaveText(['1 link needs review', 'Passed', 'Not available']);
await expect(dialog.locator('.operation h3').filter({ hasText: 'Missing files' }))
.toHaveText(['Missing files · output', 'Missing files · input', 'Missing files · unmounted-drive']);
// Review missing links goes to Maintenance, where the cleanup's existing tool is linked.
await dialog.getByRole('button', { name: 'Review missing links', exact: true }).click();
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
await expect(page).toHaveURL(`${workspace}#/system/maintenance`);
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click();
await expect(layout.locator('.metric').nth(3).locator('strong')).toHaveText('1 issue');
});
test('a health check that fails says so and keeps the last result', async ({ page, request }) => {
await expect(await request.post('/__test__/system/fail', { data: { path: '/prompt_manager/system/health' } })).toBeOK();
await page.goto(`${workspace}#/system/maintenance`);
const run = layoutOf(page).getByRole('button', { name: 'Run health checks', exact: true });
await run.click();
await expect(page.locator('#workspace-toast')).toHaveText('The health check could not finish. Nothing was changed.');
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
await expect(run).not.toHaveAttribute('aria-disabled', 'true');
await openOverview(page);
await expect(layoutOf(page).locator('.metric').nth(3).locator('strong')).toHaveText('Not checked');
});
test('thumbnail row reflects the status endpoint', async ({ page, request }) => {
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 3 } })).toBeOK();
const status = await serverJson(request, '/prompt_manager/images/thumbnails/status');
expect(status.pending).toBe(3);
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
await expect(row.locator('h3')).toHaveText('3 missing thumbnails');
// Until the thumbnail review dialog (S5), Review thumbnails opens Sources.
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
await expect(page).toHaveURL(`${workspace}#/system/sources`);
});
test('the thumbnail row paints when its own status read answers', async ({ page, request }) => {
const held = async () => (await serverJson(request, '/__test__/hold')).held[THUMBNAILS_URL] ?? 0;
await expect(await request.post('/__test__/hold', { data: { path: THUMBNAILS_URL } })).toBeOK();
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
// The other values do not wait for the walk of every source.
await expect(row.locator('h3')).toHaveText('Checking thumbnails…');
await expect(layout.locator('.summary-metrics > .metric')).toHaveCount(4);
await expect.poll(held).toBe(1);
await expect(await request.post('/__test__/release')).toBeOK();
await expect(row.locator('h3')).toHaveText('No missing thumbnails');
});
test('a failed thumbnail read is unavailable, never zero', async ({ page }) => {
await page.route(`**${THUMBNAILS_URL}`, route => route.fulfill({ status: 500, contentType: 'application/json',
body: JSON.stringify({ success: false, error: 'Injected failure' }) }));
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
await expect(row.locator('h3')).toHaveText('Thumbnail status unavailable');
await expect(layout.getByRole('alert')).toHaveCount(0);
await expect(layout.locator('.metric').nth(2).locator('strong')).toHaveText('24');
});
test('ambiguous row reads the pending count and Review metadata opens Needs review', async ({ page }) => {
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ hasText: 'ambiguous image links' });
await expect(row.locator('h3')).toHaveText('0 ambiguous image links');
await expect(row.locator('p')).toHaveText('Review metadata before assigning a positive / negative pair.');
const review = row.getByRole('button', { name: 'Review metadata', exact: true });
await expect(review).not.toHaveAttribute('aria-disabled');
await review.click();
// The concept's go-review: the Needs review page.
await expect(page).toHaveURL(`${workspace}#/review`);
await expect(page.getByRole('heading', { name: 'Metadata review', exact: true })).toBeVisible();
});
test('View backups / Appearance / Manage sources switch sections', async ({ page }) => {
for (const [label, section, pressed] of [['View backups', 'backups', 'Backup & export'], ['Appearance', 'appearance', 'Appearance'],
['Manage sources', 'sources', 'Sources & scans']]) {
const layout = await openOverview(page);
await layout.getByRole('button', { name: label, exact: true }).click();
await expect(page).toHaveURL(`${workspace}#/system/${section}`);
await expect(page.locator('#workspace-main .section-nav [aria-pressed="true"]')).toHaveText(pressed);
await expect(layout.locator(':scope > .summary-metrics')).toHaveCount(0);
}
});
test('statistics dialog lists runs by category and library totals', async ({ page, request }) => {
const statistics = await serverJson(request, '/prompt_manager/system/statistics');
expect(statistics.by_category.length).toBeGreaterThan(1);
const layout = await openOverview(page);
const explore = layout.getByRole('button', { name: 'Explore', exact: true });
await explore.click();
const dialog = page.locator('dialog#modal[open]');
await expect(dialog.locator('#modal-title')).toHaveText('Usage statistics');
const tables = dialog.locator('#modal-body > table');
await expect(tables.locator('caption')).toHaveText(['Runs by category', 'Library totals']);
await expect(tables.first().locator('thead th')).toHaveText(['Category', 'Runs', 'Avg. rating']);
const cells = table => table.locator('tbody tr').evaluateAll(rows => rows.map(row => [...row.cells].map(cell => cell.textContent)));
expect(await cells(tables.first())).toEqual(statistics.by_category.map(row => [row.category ?? 'Uncategorized',
number(row.runs), row.average_rating === null ? 'Not rated' : `${number(row.average_rating)} / 5`]));
expect(await cells(tables.nth(1))).toEqual([['Prompts', number(statistics.totals.total_prompts)],
['Tags', number(statistics.totals.total_tags)], ['Images', number(statistics.totals.total_images)],
['Rated images', number(statistics.rated_images)]]);
await page.keyboard.press('Escape');
await expect(explore).toBeFocused();
// A statistics read that fails says so; the Overview stays.
await expect(await request.post('/__test__/system/fail', { data: { path: '/prompt_manager/system/statistics' } })).toBeOK();
await explore.click();
await expect(page.locator('#workspace-toast')).toHaveText('Statistics could not be loaded. Your saved data has not changed.');
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
});
test('load failure shows Retry and no zeros', async ({ page, request }) => {
await expect(await request.post('/__test__/system/fail', { data: { path: OVERVIEW_URL } })).toBeOK();
await page.goto(`${workspace}#/system/overview`);
const layout = layoutOf(page);
await expect(layout.getByRole('alert')).toHaveText('The overview could not be loaded. Your saved data has not changed.');
await expect(layout.locator('.metric')).toHaveCount(0);
await expect(layout).not.toContainText('0');
await layout.getByRole('button', { name: 'Retry', exact: true }).click();
await expect(layout.locator('.summary-metrics > .metric')).toHaveCount(4);
await expect(layout.getByRole('alert')).toHaveCount(0);
});
test('out-of-order overview responses never repaint a newer section', async ({ page, request }) => {
const held = async () => (await serverJson(request, '/__test__/hold')).held[OVERVIEW_URL] ?? 0;
await expect(await request.post('/__test__/hold', { data: { path: OVERVIEW_URL } })).toBeOK();
await page.goto(`${workspace}#/system/overview`);
const layout = layoutOf(page);
await expect(layout.getByRole('status')).toHaveText('Loading the overview…');
await expect.poll(held).toBe(1);
// Leave for Appearance, come back (a second read), then leave again before either answers.
const nav = page.locator('#workspace-main .section-nav');
await nav.getByRole('button', { name: 'Appearance', exact: true }).click();
await nav.getByRole('button', { name: 'Overview', exact: true }).click();
await expect.poll(held).toBe(2);
// Integrations still hands off (S10), so its body is one known panel.
await nav.getByRole('button', { name: 'Integrations', exact: true }).click();
await expect(await request.post('/__test__/release')).toBeOK();
await expect.poll(held).toBe(0);
// Both late answers reached a section that had left: Integrations keeps its own body.
await expect(layout.locator(':scope > section.handoff')).toBeVisible();
await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve))));
await expect(layout.locator('.summary-metrics')).toHaveCount(0);
expect(await layout.evaluate(node => [...node.children].map(child => child.className))).toEqual(['surface handoff', 'save-row']);
// Back on Overview, one fresh read paints one set of metrics.
await nav.getByRole('button', { name: 'Overview', exact: true }).click();
await expect(layout.locator('.summary-metrics')).toHaveCount(1);
await expect(layout.locator('.two-col')).toHaveCount(1);
});
// System › Appearance (#249 System S3 Task 8): the browser display defaults (`gallerySettings`, preferences.mjs) and the
// two host fields behind the one Save row, and the consumers that read them on their next visit.
const SETTINGS_URL = '/prompt_manager/settings';
const INSPECT_PATH = '/prompt_manager/images/inspect/';
const statusOf = page => page.locator('#workspace-main #settings-status');
const saveOf = page => page.locator('#workspace-main .save-row > button');
const storedSettings = page => page.evaluate(() => JSON.parse(localStorage.getItem('gallerySettings') || 'null'));
const imageCards = page => page.locator('#workspace-main article.media-card');
const panel = page => page.getByRole('complementary', { name: 'Selected image details', exact: true });
async function openAppearance(page) {
await page.goto(`${workspace}#/system/appearance`);
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
// The host rows are enabled once the server settings answered.
await expect(page.locator('#timeout')).toBeEnabled();
}
async function saveAppearance(page, text) {
await expect(saveOf(page)).not.toHaveAttribute('aria-disabled', 'true');
await saveOf(page).click();
await expect(statusOf(page)).toHaveText(text);
}
/** Set toggles (by row id) through their Toolkit switches. */
async function toggle(page, values) {
for (const [id, checked] of Object.entries(values)) await page.locator(`#${id}`).setChecked(checked, { force: true });
}
/**
* A request barrier: one request issued and answered after everything the page has asked for so far, so a read the
* last action started has been seen (Playwright reports a page's requests in order).
*/
async function requestBarrier(page) {
expect(await page.evaluate(url => fetch(url, { cache: 'no-store' }).then(response => response.ok), SETTINGS_URL)).toBe(true);
}
async function openImages(page) {
await page.goto(`${workspace}#/images`);
await expect(imageCards(page).first()).toBeVisible();
}
test('defaults apply to a new visit but not to the current view or a saved view', async ({ page, request }) => {
const created = await request.post('/prompt_manager/saved-views', { data: { page: 'images', name: 'Everything',
state: { q: '', category: '', sort: 'newest', source: 'all', folder: '', media: 'all', tags: [], favorites: false } } });
expect(created.status()).toBe(201);
const view = (await created.json()).view;
try {
await openImages(page);
// The current view: the toolbar sets List.
await page.getByRole('group', { name: 'Image layout' }).getByRole('button', { name: 'List', exact: true }).click();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await openAppearance(page);
await expect(page.locator('#default-view')).toHaveValue('grid');
await page.locator('#default-view').selectOption('masonry');
await page.locator('#density').selectOption('compact');
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await saveAppearance(page, 'Saved in this browser.');
const stored = await storedSettings(page);
expect([stored.workspaceLayout, stored.workspaceDensity]).toEqual(['masonry', 'compact']);
// Back in Images (the page session's last visit): the current view keeps its own layout and size.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'comfortable');
// The saved view opens as before.
await page.goto(`${workspace}#/saved/${view.id}`);
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'grid');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'comfortable');
// A new visit starts from the defaults.
await openImages(page);
await page.reload();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'masonry');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'compact');
await expect(page.getByRole('group', { name: 'Image layout' }).getByRole('button', { name: 'Masonry', exact: true }))
.toHaveAttribute('aria-pressed', 'true');
} finally {
await request.delete(`/prompt_manager/saved-views/${view.id}`);
}
});
test('the defaults reach the Images page only; embedded galleries keep their own presentation', async ({ page }) => {
await openAppearance(page);
await page.locator('#default-view').selectOption('list');
await page.locator('#density').selectOption('compact');
await saveAppearance(page, 'Saved in this browser.');
// The prompt history's shared gallery has no toolbar to change them, so it opens as it always did.
await page.goto(`${workspace}#/library/${page.manifest.prompts[0].positive_id}`);
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('#workspace-main .image-grid')).toHaveAttribute('data-layout', 'grid');
await expect(page.locator('#workspace-main .image-grid')).toHaveAttribute('data-density', 'comfortable');
// The Images page starts from them.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'compact');
});
test('metadata off shows an explicit load and loads once on request', async ({ page }) => {
await openAppearance(page);
await toggle(page, { 'metadata-auto': false });
await saveAppearance(page, 'Saved in this browser.');
const reads = [];
page.on('request', item => { if (new URL(item.url()).pathname.startsWith(INSPECT_PATH)) reads.push(item.url()); });
await openImages(page);
const first = imageCards(page).first();
await first.locator('.image-card-actions button').first().click();
await expect(panel(page).getByRole('status')).toHaveText('Metadata loads when you ask for it.');
const load = panel(page).getByRole('button', { name: 'Load metadata', exact: true });
await expect(load).toBeVisible();
expect(reads).toEqual([]);
await load.click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
expect(reads).toHaveLength(1);
// The same file again reads nothing more (the cache is on).
await first.locator('.image-card-actions button').first().click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
await requestBarrier(page);
expect(reads).toHaveLength(1);
// The viewer's details use the same inspector: another file waits there too, and loads on request.
await imageCards(page).nth(1).locator('.image-open').click();
const viewerDetails = page.locator('#viewer-details');
await expect(viewerDetails.getByRole('status')).toHaveText('Metadata loads when you ask for it.');
expect(reads).toHaveLength(1);
await viewerDetails.getByRole('button', { name: 'Load metadata', exact: true }).click();
await expect(viewerDetails.locator('.panel-heading h2')).toBeVisible();
expect(reads).toHaveLength(2);
});
test('cache off rereads the current selection; on reuses the one current record', async ({ page }) => {
const reads = [];
page.on('request', item => { if (new URL(item.url()).pathname.startsWith(INSPECT_PATH)) reads.push(item.url()); });
for (const cache of [true, false]) {
if (!cache) {
await openAppearance(page);
await toggle(page, { 'metadata-cache': false });
await saveAppearance(page, 'Saved in this browser.');
await page.reload();
}
await openImages(page);
const info = imageCards(page).first().locator('.image-card-actions button').first();
await info.click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
await expect.poll(() => reads.length).toBeGreaterThan(0);
await requestBarrier(page);
reads.length = 0;
// Inspect the selected file twice more.
for (const _ of [1, 2]) {
const answered = cache ? null : page.waitForResponse(response => new URL(response.url()).pathname.startsWith(INSPECT_PATH));
await info.click();
if (answered) await answered;
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
}
await requestBarrier(page);
expect(reads, `cache ${cache}`).toHaveLength(cache ? 0 : 2);
}
});
test('paths hidden only changes display', async ({ page }) => {
const rows = async () => page.locator('#viewer-details details').filter({ hasText: 'File & provenance' }).locator('dt')
.evaluateAll(terms => terms.map(term => term.textContent));
const openViewer = async () => {
await openImages(page);
await imageCards(page).first().locator('.image-open').click();
const provenance = page.locator('#viewer-details details').filter({ hasText: 'File & provenance' });
await provenance.locator('summary').click();
await expect(provenance.locator('dl')).toBeVisible();
return provenance;
};
let provenance = await openViewer();
expect(await rows()).toContain('Path');
const before = { show: await provenance.getByRole('button', { name: /^Show .* in folder$/ }).count(),
download: await provenance.getByRole('link', { name: 'Download original', exact: true }).getAttribute('href') };
await page.keyboard.press('Escape');
await openAppearance(page);
await toggle(page, { paths: false });
await saveAppearance(page, 'Saved in this browser.');
provenance = await openViewer();
const shown = await rows();
expect(shown).not.toContain('Path');
expect(shown).toEqual(expect.arrayContaining(['Source', 'Size']));
// Everything else is as it was: the folder action and the original download stay.
expect(await provenance.getByRole('button', { name: /^Show .* in folder$/ }).count()).toBe(before.show);
await expect(provenance.getByRole('link', { name: 'Download original', exact: true })).toHaveAttribute('href', before.download);
});
test('video defaults reach the viewer', async ({ page }) => {
await openAppearance(page);
await toggle(page, { 'video-auto': true, 'video-muted': false, 'video-loop': false });
await saveAppearance(page, 'Saved in this browser.');
expect(await page.evaluate(async () => (await import('/prompt_manager/js/workspace/media-viewer.mjs')).readViewerPreferences()))
.toEqual({ videoAutoplay: true, videoMute: false, videoLoop: false });
const stored = await storedSettings(page);
expect([stored.videoAutoplay, stored.videoMute, stored.videoLoop]).toEqual([true, false, false]);
});
test('host fields round-trip through /settings with revision', async ({ page, request }) => {
const posts = [];
page.on('request', item => { if (item.method() === 'POST' && new URL(item.url()).pathname === SETTINGS_URL) posts.push(item.postDataJSON()); });
const before = await serverJson(request, SETTINGS_URL);
await openAppearance(page);
await expect(page.locator('#open-mode')).toHaveValue(before.settings.webui_display_mode);
await expect(page.locator('#timeout')).toHaveValue(String(before.settings.result_timeout));
// Only whole seconds from 0 to 300: anything else is highlighted and never sent.
for (const bad of ['301', '-1', '2.5']) {
await page.locator('#timeout').fill(bad);
await page.locator('#timeout').press('Tab');
await expect(page.locator('#timeout')).toHaveAttribute('aria-invalid', 'true');
await expect(statusOf(page)).toHaveText('Some changes were not saved. Review the highlighted settings.');
await expect(saveOf(page)).toHaveAttribute('aria-disabled', 'true');
}
await page.locator('#timeout').fill('0');
await page.locator('#timeout').press('Tab');
await expect(page.locator('#timeout')).not.toHaveAttribute('aria-invalid');
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Saved to the server configuration.');
expect(posts).toEqual([{ webui_display_mode: 'popup', result_timeout: 0, expected_revision: before.revision }]);
const after = await serverJson(request, SETTINGS_URL);
expect([after.settings.webui_display_mode, after.settings.result_timeout]).toEqual(['popup', 0]);
expect(after.revision).not.toBe(before.revision);
// No browser key was written for a host field.
expect(await storedSettings(page)).toBeNull();
});
test('server settings that cannot be read say so in their rows', async ({ page }) => {
await page.route(url => url.pathname === SETTINGS_URL, route => (route.request().method() === 'GET'
? route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, error: 'Injected failure' }) })
: route.continue()));
await page.goto(`${workspace}#/system/appearance`);
for (const id of ['open-mode', 'timeout']) {
const row = page.locator('#workspace-main .setting-row').filter({ has: page.locator(`#${id}`) });
await expect(row.locator('p.small')).toHaveText('Server settings could not be read.');
await expect(page.locator(`#${id}`)).toBeDisabled();
}
// The browser rows are this browser's, and work as before.
await expect(page.locator('#default-view')).toBeEnabled();
await expect(page.locator('#workspace-main .setting-row p.small')).toHaveCount(2);
});
test('stale revision shows the conflict status', async ({ page, request }) => {
await openAppearance(page);
// Another page saves after this one read the settings.
const current = await serverJson(request, SETTINGS_URL);
await expect(await request.post(SETTINGS_URL, { data: { result_timeout: 42, expected_revision: current.revision } })).toBeOK();
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Settings changed elsewhere. Reload this section to see them.');
await expect(page.locator('#open-mode')).toHaveAttribute('aria-invalid', 'true');
// The draft is kept and the section now shows the other page's value.
await expect(page.locator('#open-mode')).toHaveValue('popup');
await expect(page.locator('#timeout')).toHaveValue('42');
expect((await serverJson(request, SETTINGS_URL)).settings.webui_display_mode).toBe('newtab');
// With the reloaded revision, Save succeeds.
await saveAppearance(page, 'Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).settings.webui_display_mode).toBe('popup');
});
test('reload keeps saved values; a new tab reads them', async ({ page, context }) => {
await openAppearance(page);
await page.locator('#image-quality').selectOption('high');
await toggle(page, { 'card-details': false });
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Saved in this browser. Saved to the server configuration.');
await page.reload();
await expect(page.locator('#timeout')).toBeEnabled();
await expect(page.locator('#image-quality')).toHaveValue('high');
await expect(page.locator('#card-details')).not.toBeChecked();
await expect(page.locator('#open-mode')).toHaveValue('popup');
const other = await context.newPage();
await openAppearance(other);
await expect(other.locator('#image-quality')).toHaveValue('high');
await expect(other.locator('#card-details')).not.toBeChecked();
// A change saved in one tab reaches the other's clean controls; a draft it overtakes is flagged and kept.
await other.locator('#density').selectOption('large');
await page.locator('#density').selectOption('compact');
await saveAppearance(page, 'Saved in this browser.');
await expect(other.locator('#density')).toHaveValue('large');
await expect(other.locator('#settings-status')).toHaveText('Settings changed elsewhere. Reload this section to see them.');
await expect(other.locator('#density')).toHaveAttribute('aria-invalid', 'true');
await page.locator('#image-quality').selectOption('low');
await saveAppearance(page, 'Saved in this browser.');
await expect(other.locator('#image-quality')).toHaveValue('low');
await other.close();
});
test('a change another tab saved while System was closed is a conflict on return, never saved over silently', async ({ page, context }) => {
await openAppearance(page);
await page.locator('#density').selectOption('compact');
await expect(statusOf(page)).toHaveText('Unsaved changes.');
// Leave System for Images: the draft stays for the page session, but System no longer listens.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
const other = await context.newPage();
await openAppearance(other);
await other.locator('#density').selectOption('large');
await saveAppearance(other, 'Saved in this browser.');
await other.close();
// Back in System: the draft is kept and flagged, and nothing was written over the other tab's value.
await page.evaluate(() => { location.hash = '#/system/appearance'; });
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
await expect(statusOf(page)).toHaveText('Settings changed elsewhere. Reload this section to see them.');
await expect(page.locator('#density')).toHaveAttribute('aria-invalid', 'true');
await expect(page.locator('#density')).toHaveValue('compact');
expect((await storedSettings(page)).workspaceDensity).toBe('large');
// Saving after the conflict was shown is an explicit choice: it wins.
await saveAppearance(page, 'Saved in this browser.');
expect((await storedSettings(page)).workspaceDensity).toBe('compact');
await expect(page.locator('#density')).not.toHaveAttribute('aria-invalid');
});
test('a browser that refuses storage is saved for this tab only, never "Saved"', async ({ page }) => {
await openAppearance(page);
await page.evaluate(() => {
Storage.prototype.setItem = function setItem() { throw new DOMException('Quota exceeded', 'QuotaExceededError'); };
});
await page.locator('#default-view').selectOption('list');
await saveAppearance(page, 'Saved for this tab only; browser storage is unavailable.');
expect(await storedSettings(page)).toBeNull();
// This tab's next Images visit still uses it.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
});
test('added rows reach their consumers', async ({ page }) => {
await openImages(page);
await expect(imageCards(page).first().locator('.media-info h3')).toBeVisible();
await openAppearance(page);
await toggle(page, { 'card-details': false, 'viewer-details': true });
await saveAppearance(page, 'Saved in this browser.');
await page.reload();
await openImages(page);
// Show image details on cards off: no title or file line; Info and Tag stay.
const card = imageCards(page).first();
await expect(card.locator('.media-info h3')).toBeHidden();
await expect(card.locator('.image-card-actions button')).toHaveCount(2);
// Collapse viewer details by default: the viewer opens with its details hidden behind one toggle.
await card.locator('.image-open').click();
const details = page.locator('#viewer-details');
await expect(page.locator('#viewer-dialog')).toBeVisible();
await expect(details).toBeHidden();
const show = page.locator('#viewer-toolbar').getByRole('button', { name: 'Show details', exact: true });
await expect(show).toHaveAttribute('aria-expanded', 'false');
await show.click();
await expect(details).toBeVisible();
await expect(page.locator('#viewer-toolbar').getByRole('button', { name: 'Hide details', exact: true })).toHaveAttribute('aria-expanded', 'true');
});
test('reset restores defaults without touching other keys', async ({ page }) => {
await page.goto(`${workspace}#/system/appearance`);
await page.evaluate(() => localStorage.setItem('gallerySettings', JSON.stringify({ gridColumns: 5, defaultLimit: 200,
thumbnailPromptDismissed: 'kept', workspaceLayout: 'list', videoMute: false, showImageInfo: false })));
await page.reload();
await expect(page.locator('#timeout')).toBeEnabled();
await expect(page.locator('#default-view')).toHaveValue('list');
const reset = page.locator('#workspace-main .operation').filter({ hasText: 'Reset display defaults' }).getByRole('button', { name: 'Reset', exact: true });
await reset.click();
await expect(page.locator('#default-view')).toHaveValue('grid');
await expect(page.locator('#video-muted')).toBeChecked();
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await saveAppearance(page, 'Saved in this browser.');
const stored = await storedSettings(page);
expect(stored).toEqual({ gridColumns: 5, defaultLimit: 200, thumbnailPromptDismissed: 'kept', workspaceLayout: 'grid',
videoMute: true, showImageInfo: true });
});
+6 -105
View File
@@ -18,12 +18,9 @@ const routes = [
['tags', 'Tags & categories', null],
// #249 local Inspect: the native Inspect a file page; Overview's tool card still links the legacy metadata page.
['inspect', 'Inspect a file', null],
// #249 System S2: native System; its undelivered sections hand off per section (D94, asserted below).
['system', 'System', null],
['system', 'System', '/prompt_manager/admin'],
// #249 D51: SAVED VIEWS' Favorites, Images filtered to favorites.
['favorites', 'Favorites', null],
// #249 PR F (ruling R5): SAVED VIEWS' Needs review, the concept's Metadata review page (its own heading).
['review', 'Needs review', null, 'Metadata review'],
// #249 Activity-1: the concept's Activity page in the tools-and-system group.
['activity', 'Activity', null],
];
@@ -79,10 +76,10 @@ test('workspace starts with real persisted statistics without mutating data', as
});
test('all shell destinations deep-link, reload and expose working existing applications', async ({ page, request }) => {
for (const [route, label, legacy, title = label] of routes) {
for (const [route, label, legacy] of routes) {
await page.goto(`${workspace}#/${route}`);
const main = page.locator('#workspace-main');
await expect(main.getByRole('heading', { level: 1, name: title, exact: true })).toBeVisible();
await expect(main.getByRole('heading', { level: 1, name: label, exact: true })).toBeVisible();
await expect(sidebarLink(page, label)).toHaveAttribute('aria-current', 'page');
if (route === 'tags' || route === 'inspect') await expect(main.locator('.handoff')).toHaveCount(0);
if (route === 'inspect') await expect(main.locator('#local-file')).toBeAttached();
@@ -91,7 +88,7 @@ test('all shell destinations deep-link, reload and expose working existing appli
await expect(await request.get(legacy.split('#')[0])).toBeOK();
}
await page.reload();
await expect(main.getByRole('heading', { level: 1, name: title, exact: true })).toBeVisible();
await expect(main.getByRole('heading', { level: 1, name: label, exact: true })).toBeVisible();
}
// D29 and #249 E3: the legacy gallery and Library are reached from Overview's tool cards, not from their pages.
await page.goto(`${workspace}#/overview`);
@@ -311,9 +308,9 @@ test('overview reports loading, offers retry after failure and renders real empt
test('small screens keep navigation and the theme accessible without horizontal page overflow', async ({ page }) => {
await page.setViewportSize({ width: 375, height: 812 });
await page.goto(`${workspace}#/overview`);
for (const [, label, , title = label] of routes) {
for (const [, label] of routes) {
await sidebarLink(page, label).click();
await expect(page.locator('#workspace-main h1')).toHaveText(title);
await expect(page.locator('#workspace-main h1')).toHaveText(label);
}
await expect(page.getByRole('combobox', { name: 'Theme', exact: true })).toBeVisible();
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true);
@@ -412,8 +409,6 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true);
await sidebarLink(page, 'System').click();
// System's Overview is native (S2 Task 6); a section not delivered yet still hands off (D94).
await main.locator('.section-nav').getByRole('button', { name: 'Integrations', exact: true }).click();
const handoff = main.locator('section.handoff');
await expect(handoff).toHaveClass(/\bsurface\b/);
expect(await handoff.evaluate(surfaceStyle)).toMatchObject(SURFACE);
@@ -422,100 +417,6 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true);
});
// #249 System S2 Task 5: the concept's renderSystem shell (app.js 698–706) at `#/system/<section>`, every section a D94
// hand-off until its PR, and the Save row's truthful status (D92).
const SYSTEM_SECTION_LABELS = ['Overview', 'Appearance', 'Sources & scans', 'Integrations', 'Auto-tagging', 'Maintenance',
'Backup & export', 'Diagnostics & logs'];
const SYSTEM_HASHES = ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics'];
const SYSTEM_HANDOFFS = { maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] };
test('System is native: the concept shell, section navigation, sections, hand-offs and the Save row', async ({ page }) => {
const mutations = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) mutations.push(item.url()); });
await page.goto(`${workspace}#/system`);
const main = page.locator('#workspace-main');
await expect(main.locator('h1')).toHaveText('System');
await expect(main.locator('.heading p.muted')).toHaveText('Configuration, maintenance and recovery—with browsing kept in the workspace.');
// D92: no "Sample configuration" chip.
await expect(main.locator('.heading .chip')).toHaveCount(0);
expect(await main.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`)))
.toEqual(['div.heading', 'div.section-nav', 'div.settings-layout']);
const nav = main.getByRole('group', { name: 'System sections', exact: true });
await expect(nav.getByRole('button')).toHaveText(SYSTEM_SECTION_LABELS);
await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true');
await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1);
await expect(page.locator('aside.sidebar a[href="#/system"]')).toHaveAttribute('aria-current', 'page');
const layout = main.locator('.settings-layout');
await expect(layout.locator(':scope > .save-row > span.small#settings-status')).toHaveText('No unsaved changes.');
const save = layout.locator(':scope > .save-row > button.ca-primary');
await expect(save).toHaveText('Save preferences');
await expect(save).toHaveAttribute('aria-disabled', 'true');
// Playwright waits on an aria-disabled control, so the ignored press is forced.
await save.click({ force: true });
await expect(layout.locator('#settings-status')).toHaveText('No unsaved changes.');
for (const [index, label] of SYSTEM_SECTION_LABELS.entries()) {
const hash = SYSTEM_HASHES[index];
const button = nav.getByRole('button', { name: label, exact: true });
await button.click();
await expect(page).toHaveURL(`${workspace}#/system/${hash}`);
await expect(button).toHaveAttribute('aria-pressed', 'true');
await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1);
// The section changes in place: focus stays on the pressed section, not the page title.
await expect(button).toBeFocused();
// S2 Task 6: Overview is native (workspace-system.spec.mjs), and so are Appearance (S3 Task 8) and Sources & scans
// (S4 Task 11, workspace-system-sources.spec.mjs, its update notice shown in a fresh browser); Maintenance shows
// its health-check notice above the hand-off (D123); every other section hands off.
if (hash === 'overview') await expect(layout.locator(':scope > .summary-metrics')).toBeVisible();
if (hash === 'sources') await expect(layout.locator('#source-list .operation')).toHaveCount(2);
expect(await layout.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`)))
.toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'], appearance: ['section.surface', 'div.save-row'],
sources: ['div.notice', 'section.surface', 'div.save-row'],
maintenance: ['div.notice', 'section.surface handoff', 'div.save-row'] }[hash]
?? ['section.surface handoff', 'div.save-row']);
if (['overview', 'appearance', 'sources'].includes(hash)) continue;
const handoff = layout.locator('section.handoff');
await expect(handoff.locator('h2')).toHaveText('Your current tools are ready');
expect(await handoff.locator('a.workspace-link').evaluateAll(links => links.map(link => link.getAttribute('href'))))
.toEqual(SYSTEM_HANDOFFS[hash] ?? ['/prompt_manager/admin']);
await expect(layout.locator('#settings-status')).toHaveText('No unsaved changes.');
await expect(page.locator('aside.sidebar a[href="#/system"]')).toHaveAttribute('aria-current', 'page');
}
// Deep links, reload and history keep the section; an unknown section is Overview under its own hash.
await page.reload();
await expect(nav.getByRole('button', { name: 'Diagnostics & logs', exact: true })).toHaveAttribute('aria-pressed', 'true');
await page.goBack();
await expect(page).toHaveURL(`${workspace}#/system/backups`);
await expect(nav.getByRole('button', { name: 'Backup & export', exact: true })).toHaveAttribute('aria-pressed', 'true');
await page.goto(`${workspace}#/system/general`);
await expect(page).toHaveURL(`${workspace}#/system/overview`);
await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true');
// The sidebar's System returns to the Overview section.
await nav.getByRole('button', { name: 'Integrations', exact: true }).click();
await sidebarLink(page, 'System').click();
await expect(page).toHaveURL(`${workspace}#/system`);
await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true');
// Leaving System and coming back renders it once, with a fresh section.
await sidebarLink(page, 'Images').click();
await expect(main.locator('h1')).toHaveText('Images');
await expect(main.locator('.section-nav')).toHaveCount(0);
await page.goBack();
await expect(main.locator('.section-nav')).toHaveCount(1);
await expect(main.locator('.settings-layout')).toHaveCount(1);
expect(mutations).toEqual([]);
});
test('Overview opens native System from its System card (Q12)', async ({ page }) => {
await page.goto(`${workspace}#/overview`);
const card = page.locator('#workspace-main .tool-card').filter({ has: page.getByRole('heading', { name: 'System', exact: true }) });
await expect(card.locator('p.small')).toHaveText('Configuration, maintenance and recovery—with browsing kept in the workspace.');
const link = card.getByRole('link', { name: 'Open System', exact: true });
await expect(link).toHaveAttribute('href', '#/system/overview');
await link.click();
await expect(page.locator('#workspace-main h1')).toHaveText('System');
await expect(page.locator('#workspace-main .section-nav [aria-pressed="true"]')).toHaveText('Overview');
});
// Task A3: the concept sidebar (index.html h16-41, app.js nav()) rendered from real data (D27, D28).
const NAV_TOTALS = [
['images', 'Images', '/prompt_manager/images/output?mode=workspace&limit=1'],
@@ -150,12 +150,6 @@ def register_activity_commands(app, routes, api):
scope = {
"files": str(data.get("files", "new")),
"output_folders": count(data.get("output_folders"), 2),
# A real scan's count of its files in the Needs review queue.
**(
{"review_items": count(data["review_items"])}
if "review_items" in data
else {}
),
}
groups = SCAN_GROUP_NAMES
try:
@@ -176,8 +176,6 @@ _TABLES = (
"execution_context_texts",
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -28,8 +28,6 @@ EDITOR_METADATA = {
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
-368
View File
@@ -1,368 +0,0 @@
"""Owned, deterministic Needs review items for the workspace page (#249 PR F).
Each file is a PNG whose workflow has two samplers with their own positive
encoders and one shared negative, so the resolver finds it ``ambiguous``. Its
loader node carries ComfyUI's ``Infinity`` stamp, which the legacy pages'
parser (the scan's fallback) cannot read, so nothing links the file and it is
a Needs review case (#284: only ambiguous, unlinked files are queued). It is
recorded through the producers' entry point (``record_review_file``). No path
or SQL comes from a request.
``POST /__test__/review/reset`` also takes the special items F2 needs, queued
after the plain ones in this order:
- ``conflict``: an item whose file was linked to another prompt ("Misty
forest path") after it was queued (the next scan or monitor pass would
remove the item); applying a sampler path then conflicts with that link.
- ``limited``: a graph of 17 samplers, each with its own prompt, which the
recorder stores ``limited`` (more than 16 candidates, so none are stored).
An A1111 block with two negative lines is not queued any more: it offers
no candidate to choose (#284).
- ``unavailable``: a file recorded under the ``temp`` directory while it was a
gallery root; the root is then removed again, so no current root lists it.
``concept: true`` (F3, the concept comparison) names and sizes the plain
files as the concept's first three images (app.js ``images``) and gives
their first sampler the concept's first prompt (``alpine``: its positive and
its first negative, the one the concept's panel shows), so the panel's text
wraps as the concept's does.
``POST /__test__/review/add {kind}`` adds one file while a page is open:
``monitor`` records a new ambiguous, unlinked output from a worker thread
through ``record_review_file`` (the monitor's own entry point; the monitor
never queues an output it linked itself), ``file`` only writes one (a
metadata scan records it). ``GET
/__test__/review/links`` answers each review file's prompt and pair links, and
``GET /__test__/review/streams`` the open change streams and captures.
"""
from __future__ import annotations
import json
import os
import threading
from pathlib import Path
import shutil
import time
from typing import TYPE_CHECKING
from aiohttp import web
from PIL import Image
from PIL.PngImagePlugin import PngInfo
if TYPE_CHECKING:
from py.api import PromptManagerAPI
POSITIVES = ("Golden dunes at dusk", "Harbor lights at night")
NEGATIVE = "Soft negative, blur"
# The prompt the conflict item's file is linked to after it was queued.
FALLBACK = "Misty forest path"
SPECIAL = ("conflict", "limited", "unavailable")
# The concept's first three images (concept app.js `images`) and first prompt.
CONCEPT_IMAGES = (
("alpine_00001.png", (1536, 1152)),
("dunes_00001.png", (1024, 1536)),
("forest_00001.png", (1280, 1280)),
)
CONCEPT_POSITIVE = (
"A remote alpine observatory at blue hour, warm light through glass, dramatic "
"mountain peaks, cinematic composition, atmospheric mist, fine detail"
)
CONCEPT_NEGATIVE = "blurry, low quality, low resolution"
# More candidates than an item stores (16): the ``limited`` item.
LIMITED_SAMPLERS = 17
# More than one queue page (40), so Load more has a second page.
MAX_ITEMS = 45
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
"image_prompt_pairs",
"prompt_pair_tags",
"prompt_pairs",
"generated_images",
"prompt_tags",
"tags",
"prompt_library_state",
"prompts",
)
def _graph(
encoder_class: str = "CLIPTextEncode",
positives: tuple = POSITIVES,
negative: str = NEGATIVE,
) -> dict:
graph = {
# ``Infinity`` (as ComfyUI writes it): the legacy parser cannot read the
# graph, so the scan's fallback links nothing and the file is queued.
"1": {
"class_type": "CheckpointLoaderSimple",
"inputs": {"ckpt_name": "m"},
"is_changed": [float("inf")],
},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": negative}},
}
for index, text in enumerate(positives):
encoder, sampler = str(10 + 2 * index), str(11 + 2 * index)
graph[encoder] = {"class_type": encoder_class, "inputs": {"text": text}}
graph[sampler] = {
"class_type": "KSampler",
"inputs": {
"model": ["1", 0],
"positive": [encoder, 0],
"negative": ["2", 0],
},
}
return graph
def _png(
path: Path,
colour: tuple[int, int, int],
*,
graph: dict | None = None,
parameters: str | None = None,
size: tuple[int, int] = (96, 64),
) -> None:
info = PngInfo()
if parameters is not None:
info.add_text("parameters", parameters)
else:
info.add_text("prompt", json.dumps(graph or _graph()))
path.parent.mkdir(parents=True, exist_ok=True)
with Image.new("RGB", size, colour) as image:
image.save(path, pnginfo=info)
def _record(api: PromptManagerAPI, path: Path, expected: str) -> None:
outcome = api.record_review_file(str(path))
if outcome != expected:
raise RuntimeError(f"{path.name} was {outcome}, not {expected}")
def _stamp(path: Path, age: float) -> None:
stamp = time.time() - age
os.utime(path, (stamp, stamp))
class _Monitor:
"""The monitor's recorder over this server's database (one per server)."""
def __init__(self, api: PromptManagerAPI) -> None:
self.api = api
self.lock = threading.Lock()
def new_output(self, path: Path) -> None:
"""The monitor's Needs review pass for a new output, from a worker thread."""
outcome = []
with self.lock:
worker = threading.Thread(
target=lambda: outcome.append(self.api.record_review_file(str(path)))
)
worker.start()
worker.join()
if outcome != ["recorded"]:
raise RuntimeError(f"{path.name} was {outcome}, not recorded")
def _monitored_output(api: PromptManagerAPI, monitor: _Monitor, path: Path) -> None:
_png(path, (150, 60, 40))
monitor.new_output(path)
def _linked_after_queueing(api: PromptManagerAPI, path: Path) -> None:
from utils.hashing import generate_prompt_hash
_png(path, (150, 60, 40))
_record(api, path, "recorded")
prompt = api.db.get_prompt_by_hash(generate_prompt_hash(FALLBACK))
prompt_id = (
prompt["id"]
if prompt is not None
else api.db.save_prompt(FALLBACK, prompt_hash=generate_prompt_hash(FALLBACK))
)
api.db.link_image_to_prompt(prompt_id, str(path))
def _unavailable(api: PromptManagerAPI, root: Path, path: Path) -> None:
from py.config import GalleryConfig
roots = list(GalleryConfig.MONITORING_DIRECTORIES)
GalleryConfig.MONITORING_DIRECTORIES = [*roots, str(root / "temp")]
try:
_record(api, path, "recorded")
finally:
GalleryConfig.MONITORING_DIRECTORIES = roots
def seed_review(
api: PromptManagerAPI,
root: Path,
items: int,
special: tuple = (),
monitor: _Monitor | None = None,
concept: bool = False,
) -> dict:
"""Replace this server's disposable rows and files with ``items`` queued files."""
with api.db.model.get_connection() as conn:
for table in _TABLES:
conn.execute(f"DELETE FROM {table}")
api.close_annotation_captures()
for name in ("output", "input"):
shutil.rmtree(root / name)
(root / name).mkdir()
shutil.rmtree(root / "temp" / "review", ignore_errors=True)
output = root / "output"
files = []
for number in range(items):
relative = f"review/review_{number:04}.png"
options = {}
if concept:
name, size = CONCEPT_IMAGES[number % len(CONCEPT_IMAGES)]
relative = f"review/{name}" if number < len(CONCEPT_IMAGES) else relative
graph = _graph(
positives=(CONCEPT_POSITIVE, POSITIVES[1]), negative=CONCEPT_NEGATIVE
)
options = {"graph": graph, "size": size}
_png(output / relative, (40 + number * 4, 90, 140), **options)
_stamp(output / relative, 60 + number)
_record(api, output / relative, "recorded")
files.append({"path": relative, "filename": Path(relative).name})
for kind in SPECIAL:
if kind not in special:
continue
relative = f"review/{kind}.png"
path = (root / "temp" if kind == "unavailable" else output) / relative
if kind == "conflict":
_linked_after_queueing(api, path)
elif kind == "limited":
many = tuple(f"{POSITIVES[0]} {n}" for n in range(LIMITED_SAMPLERS))
_png(path, (90, 90, 90), graph=_graph(positives=many))
_record(api, path, "limited")
else:
_png(path, (20, 140, 90))
_unavailable(api, root, path)
files.append({"path": relative, "filename": path.name, "kind": kind})
api.invalidate_gallery_cache()
return {
"files": files,
"positives": list(POSITIVES),
"negative": NEGATIVE,
"fallback": FALLBACK,
}
def add_review_file(
api: PromptManagerAPI, root: Path, kind: str, monitor: _Monitor
) -> dict:
"""One more ambiguous output: through the monitor, or only written."""
output = root / "output"
number = (
len(list((output / "added").glob("*.png")))
if (output / "added").exists()
else 0
)
relative = f"added/added_{number:04}.png"
if kind == "monitor":
_monitored_output(api, monitor, output / relative)
else:
_png(output / relative, (200, 200, 40))
api.invalidate_gallery_cache()
return {"path": relative, "filename": Path(relative).name}
_LINKS = (
"SELECT gi.filename, p.text AS prompt, pp.id AS pair_id, pos.text AS positive, "
"neg.text AS negative FROM generated_images gi "
"JOIN prompts p ON p.id = gi.prompt_id "
"LEFT JOIN image_prompt_pairs ipp ON ipp.image_id = gi.id "
"LEFT JOIN prompt_pairs pp ON pp.id = ipp.pair_id "
"LEFT JOIN prompts pos ON pos.id = pp.positive_prompt_id "
"LEFT JOIN prompts neg ON neg.id = pp.negative_prompt_id "
"ORDER BY gi.filename, gi.id, pp.id"
)
def review_links(api: PromptManagerAPI) -> list:
"""Every file's prompt link and pair links, as plain rows."""
conn = api.db.model.get_connection()
return [dict(row) for row in conn.execute(_LINKS)]
def register_review_commands(
routes: web.RouteTableDef, api: PromptManagerAPI, root: Path
) -> None:
"""Register the review fixture's commands; no path or SQL comes from a request."""
monitors = []
def monitor() -> _Monitor:
if not monitors:
monitors.append(_Monitor(api))
return monitors[0]
@routes.post("/__test__/review/reset")
async def reset(request: web.Request) -> web.Response:
data = await request.json() if request.can_read_body else {}
if (
request.query
or not isinstance(data, dict)
or set(data) - {"items", "concept", *SPECIAL}
):
raise web.HTTPBadRequest()
items = data.get("items", 3)
if isinstance(items, bool) or not isinstance(items, int):
raise web.HTTPBadRequest()
if not 0 <= items <= MAX_ITEMS:
raise web.HTTPBadRequest()
if any(
not isinstance(data.get(kind, False), bool)
for kind in (*SPECIAL, "concept")
):
raise web.HTTPBadRequest()
special = tuple(kind for kind in SPECIAL if data.get(kind))
manifest = await api._run_in_executor(
seed_review,
api,
root,
items,
special,
monitor(),
data.get("concept", False),
)
return web.json_response({"success": True, **manifest})
@routes.post("/__test__/review/add")
async def add(request: web.Request) -> web.Response:
data = await request.json() if request.can_read_body else None
if request.query or not isinstance(data, dict) or set(data) != {"kind"}:
raise web.HTTPBadRequest()
if data["kind"] not in ("monitor", "file"):
raise web.HTTPBadRequest()
added = await api._run_in_executor(
add_review_file, api, root, data["kind"], monitor()
)
return web.json_response({"success": True, **added})
@routes.get("/__test__/review/links")
async def links(request: web.Request) -> web.Response:
if request.query:
raise web.HTTPBadRequest()
return web.json_response(
{"links": await api._run_in_executor(review_links, api)}
)
@routes.get("/__test__/review/streams")
async def streams(request: web.Request) -> web.Response:
"""Open review change streams and held captures (teardown checks)."""
if request.query:
raise web.HTTPBadRequest()
store = api._captures
captures = store.stats()["captures"] if store is not None else 0
return web.json_response(
{"streams": api.review_stream_count(), "captures": captures}
)
-206
View File
@@ -1,206 +0,0 @@
"""System test hooks for the browser tests (#249 System S2 Task 6).
The data is the concept-shaped real fixture (``POST /__test__/concept/reset``,
``workspace_concept_fixture.py``): its parents, saved pairs, images, ratings
and runs are what the Overview, the Statistics dialog and the health check
read. These per-server hooks only change what a System test needs on top:
- ``POST /__test__/system/reset`` restores the configured sources, forgets the
session's health report and ends every injected failure;
- ``POST /__test__/system/fail`` ``{path, times}`` answers the next ``times``
requests for ``path`` (a ``/prompt_manager/system/`` route) with the routes'
500 error envelope;
- ``POST /__test__/system/missing-source`` configures one more source that
does not exist (an unmounted drive); with ``{"inside": true}`` it is an
existing folder inside the output folder (``output/unplugged``), which
``POST /__test__/system/unplug`` then removes (a drive unplugged while the
page shows it);
- ``POST /__test__/system/missing-file`` deletes the newest linked image's
file from disk, leaving its database row (a missing-file link);
- ``POST /__test__/system/unthumbnailed`` ``{count}`` writes ``count`` plain
PNGs with no thumbnail into the output folder;
- ``POST /__test__/system/monitor`` ``{watching, alive}`` stands in a fake
image monitor whose observer is ``alive`` (default true) and schedules the
named folders of this server's root (``["output"]``), so Sources reads
watching, manual scan and restart pending states (S4).
Delays use the shared request holds (``request_hold.py``).
"""
import os
from pathlib import Path
from aiohttp import web
from PIL import Image
FAILABLE_PREFIX = "/prompt_manager/system/"
MAX_UNTHUMBNAILED = 50
def register_system_commands(app, routes, api, root: Path):
# System › Appearance saves the host fields to this server's config.json (S3);
# reset restores both. The file is this server's own, inside its temp root:
# reset rewrites or deletes it, so a server whose configuration points
# anywhere else (a real config.json) refuses to register these hooks.
config_path = root / "config.json"
configured_path = os.environ.get("PROMPT_MANAGER_CONFIG_PATH")
if (
configured_path is None
or Path(configured_path).resolve() != config_path.resolve()
):
raise RuntimeError(
f"PROMPT_MANAGER_CONFIG_PATH must be {config_path}, not {configured_path!r}"
)
from py.config import GalleryConfig, PromptManagerConfig
from utils import image_monitor
configured = list(GalleryConfig.MONITORING_DIRECTORIES)
config_bytes = config_path.read_bytes() if config_path.exists() else None
host_values = (
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
GalleryConfig.MONITORING_ENABLED,
)
monitor = getattr(image_monitor, "_monitor_instance", None)
failures: dict[str, int] = {}
@web.middleware
async def inject_failures(request, handler):
left = failures.get(request.path, 0)
if left <= 0:
return await handler(request)
failures[request.path] = left - 1
return web.json_response(
{"success": False, "code": None, "error": "Injected failure"}, status=500
)
app.middlewares.append(inject_failures)
unplugged = root / "output" / "unplugged"
def reset():
failures.clear()
if unplugged.is_dir():
unplugged.rmdir()
GalleryConfig.MONITORING_DIRECTORIES = list(configured)
api._health_report = None
(
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
GalleryConfig.MONITORING_ENABLED,
) = host_values
image_monitor._monitor_instance = monitor
if config_bytes is None:
config_path.unlink(missing_ok=True)
else:
# Replace in one step, as the server writes it (Windows preflight).
staged = config_path.with_name(config_path.name + ".reset")
staged.write_bytes(config_bytes)
os.replace(staged, config_path)
@routes.post("/__test__/system/reset")
async def system_reset(request):
reset()
return web.json_response({"success": True})
@routes.post("/__test__/system/fail")
async def system_fail(request):
data = await request.json()
path, times = data.get("path"), data.get("times", 1)
if (
not isinstance(path, str)
or not path.startswith(FAILABLE_PREFIX)
or isinstance(times, bool)
or not isinstance(times, int)
or not 1 <= times <= 10
):
raise web.HTTPBadRequest()
failures[path] = times
return web.json_response({"success": True})
@routes.post("/__test__/system/missing-source")
async def missing_source(request):
data = await request.json() if request.can_read_body else {}
if data.get("inside") is True:
unplugged.mkdir(exist_ok=True)
absent = str(unplugged)
else:
absent = str(root / "unmounted-drive")
if os.path.exists(absent):
raise web.HTTPConflict()
GalleryConfig.MONITORING_DIRECTORIES = [*configured, absent]
return web.json_response(
{"success": True, "configured": len(GalleryConfig.MONITORING_DIRECTORIES)}
)
@routes.post("/__test__/system/unplug")
async def unplug(request):
if not unplugged.is_dir():
raise web.HTTPConflict()
unplugged.rmdir()
return web.json_response({"success": True})
@routes.post("/__test__/system/missing-file")
async def missing_file(request):
with api.db.model.get_connection() as conn:
row = conn.execute(
"SELECT file_path FROM generated_images ORDER BY id DESC LIMIT 1"
).fetchone()
if row is None:
raise web.HTTPNotFound()
Path(row[0]).unlink()
api.invalidate_gallery_cache()
return web.json_response({"success": True, "filename": Path(row[0]).name})
@routes.post("/__test__/system/unthumbnailed")
async def unthumbnailed(request):
data = await request.json()
count = data.get("count")
if (
isinstance(count, bool)
or not isinstance(count, int)
or not 1 <= count <= MAX_UNTHUMBNAILED
):
raise web.HTTPBadRequest()
folder = root / "output" / "unthumbnailed"
folder.mkdir(exist_ok=True)
for number in range(count):
with Image.new("RGB", (64, 64), "#2b3442") as image:
image.save(folder / f"plain_{number + 1:03}.png")
api.invalidate_gallery_cache()
return web.json_response({"success": True})
@routes.post("/__test__/system/monitor")
async def fake_monitor(request):
data = await request.json()
watching, alive = data.get("watching", []), data.get("alive", True)
if (
not isinstance(watching, list)
or not all(name in ("output", "input") for name in watching)
or not isinstance(alive, bool)
):
raise web.HTTPBadRequest()
image_monitor._monitor_instance = _FakeMonitor(
[str(root / name) for name in watching], alive
)
return web.json_response({"success": True})
return reset
class _FakeMonitor:
"""The image monitor's status surface (``get_status``) with no observer thread."""
def __init__(self, directories, alive):
self.monitored_directories = directories
self._alive = alive
def get_status(self):
return {
"running": True,
"monitored_directories": list(self.monitored_directories),
"handler_active": True,
"observer_alive": self._alive,
}
@@ -44,8 +44,6 @@ MAX_EXTRA_IMAGES = 160
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -114,8 +114,6 @@ def seed_workspace_visual(api: PromptManagerAPI, root: Path) -> None:
for table in (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -81,8 +81,6 @@ MAX_EXTRA_CATEGORIES = 200
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
+1 -10
View File
@@ -1,20 +1,11 @@
import { expect } from '../../tools/vendor/node_modules/@playwright/test/index.mjs';
import runtime from '../../tools/vendor/comfyui-runtime.json' with { type: 'json' };
import { hostBaseDirectory } from '../../tools/vendor/comfyui-host.mjs';
export function installHostHooks(test, expectedTypes = ['PromptManager', 'PromptManagerText', 'PromptSearchList']) {
test.beforeEach(async ({ page, context, request }, testInfo) => {
const response = await request.get('/system_stats');
expect(response.ok()).toBe(true);
const stats = await response.json();
// comfyui.config.mjs's launcher reports the host it owns. The execution suite has no
// web server: it starts its own host per worker behind a port refusal.
if (testInfo.config.webServer) {
const ownedBase = process.env.PM_COMFY_HOST_BASE;
expect(ownedBase, 'the launcher reported the host it started').toBeTruthy();
expect(hostBaseDirectory(stats), 'the host answering the port is the one this run started').toBe(ownedBase);
}
const info = stats.system;
const info = (await response.json()).system;
expect(info.comfyui_version).toBe(runtime.serverVersion);
expect(info.pytorch_version).toBe(`${runtime.torchVersion}+cpu`);
expect(info.required_frontend_version).toBe(runtime.frontendVersion);
+4 -4
View File
@@ -594,12 +594,12 @@
"anchor": null,
"expect": {
"metadata_status": "available",
"state": "resolved",
"reason": "",
"state": "ambiguous",
"reason": "Multiple sampler branches",
"reason_code": null,
"selection": null,
"positive": "a red fox in snow, detailed fur",
"negative": "blurry, lowres",
"positive": null,
"negative": null,
"candidate_count": 0,
"facts": {
"seed": {
File diff suppressed because one or more lines are too long
-17
View File
@@ -355,20 +355,3 @@ test("keepalive is forwarded only when asked, so a request can outlive its page"
for (const call of fetch.calls) call.resolve(jsonResponse(200, { success: true }));
await Promise.all(writes);
});
test("ApiError keeps a validated running job on busy and a validated source on invalid-source", () => {
const body = { running: { id: "a".repeat(32), title: "Scan for metadata" }, source: "output/gone" };
const busy = new ApiError(409, "busy", "busy", null, body);
assert.deepEqual([busy.running, busy.source], [{ id: "a".repeat(32), title: "Scan for metadata" }, null]);
assert.notEqual(busy.running, body.running, "server objects must be copied");
const invalid = new ApiError(400, "refused", "invalid-source", null, body);
assert.deepEqual([invalid.running, invalid.source], [null, "output/gone"]);
for (const running of [null, "x", { id: 1, title: "t" }, { id: "a b", title: "t" }, { id: "a", title: "" },
{ id: "a", title: "x".repeat(201) }, { id: "a".repeat(65), title: "t" }]) {
assert.equal(new ApiError(409, "busy", "busy", null, { running }).running, null);
}
for (const source of [null, 7, "", "x".repeat(4097)]) {
assert.equal(new ApiError(400, "refused", "invalid-source", null, { source }).source, null);
}
assert.deepEqual([new ApiError(409, "old caller").running, new ApiError(409, "old caller").source], [null, null]);
});
-47
View File
@@ -61,50 +61,3 @@ test("infinite scrolling is off by default and must be a boolean", () => {
assert.equal(normalizeGallerySettings({ infiniteScroll: true }).infiniteScroll, true);
assert.equal(normalizeGallerySettings({ infiniteScroll: "yes" }).infiniteScroll, false);
});
test("the workspace keys have their own defaults and validation", () => {
const { WORKSPACE_LAYOUTS, WORKSPACE_DENSITIES } = require("../../web/js/gallery-settings.js");
assert.equal(DEFAULT_SETTINGS.workspaceLayout, "grid");
assert.equal(DEFAULT_SETTINGS.workspaceDensity, "comfortable");
assert.deepEqual(WORKSPACE_LAYOUTS, ["grid", "list", "masonry"]);
assert.deepEqual(WORKSPACE_DENSITIES, ["comfortable", "large", "compact"]);
assert.equal(normalizeGallerySettings({ workspaceLayout: "masonry" }).workspaceLayout, "masonry");
assert.equal(normalizeGallerySettings({ workspaceLayout: "mosaic" }).workspaceLayout, "grid");
assert.equal(normalizeGallerySettings({ workspaceDensity: "large" }).workspaceDensity, "large");
assert.equal(normalizeGallerySettings({ workspaceDensity: 3 }).workspaceDensity, "comfortable");
// The legacy gallery's own view mode is separate: it has no masonry.
assert.equal(normalizeGallerySettings({ defaultViewMode: "masonry" }).defaultViewMode, "grid");
});
test("a legacy gallery Save keeps the workspace keys and every other stored key", () => {
const fs = require("node:fs");
const path = require("node:path");
const vm = require("node:vm");
const settings = require("../../web/js/gallery-settings.js");
const storage = new Map([["gallerySettings", JSON.stringify({ workspaceLayout: "masonry", workspaceDensity: "compact", someOtherTool: 1 })]]);
const controls = {
lazyLoadingToggle: { checked: true }, imageQualitySelect: { value: "high" }, defaultViewModeSelect: { value: "list" },
defaultLimitSelect: { value: "200" }, gridColumnsSelect: { value: "6" }, showImageInfoToggle: { checked: false },
autoLoadMetadataToggle: { checked: true }, cacheMetadataToggle: { checked: true }, showFilePathsToggle: { checked: false },
sidebarCollapsedByDefaultToggle: { checked: true }, infiniteScrollToggle: { checked: false }, debugModeToggle: { checked: false },
apiTimeoutInput: { value: "30" }, checkThumbnailsAtStartup: { checked: true }, videoAutoplayToggle: { checked: true },
videoMuteToggle: { checked: false }, videoLoopToggle: { checked: true },
};
const context = vm.createContext({
document: { addEventListener() {}, getElementById: id => controls[id] },
window: {}, console: { debug() {}, warn() {}, error() {} }, GallerySettings: settings,
localStorage: { getItem: key => storage.get(key) ?? null, setItem: (key, value) => storage.set(key, value) },
});
const file = path.join(__dirname, "../../web/js/gallery.js");
const Manager = vm.runInContext(`${fs.readFileSync(file, "utf8")}\nGalleryManager;`, context, { filename: file });
const manager = Object.create(Manager.prototype);
Object.assign(manager, { showNotification() {}, hideSettings() {}, applySettings() {} });
manager.saveSettings();
const saved = JSON.parse(storage.get("gallerySettings"));
assert.equal(saved.workspaceLayout, "masonry");
assert.equal(saved.workspaceDensity, "compact");
assert.equal(saved.someOtherTool, 1);
assert.equal(saved.defaultViewMode, "list");
assert.equal(saved.showImageInfo, false);
assert.equal(saved.videoMute, false);
});
-26
View File
@@ -82,32 +82,6 @@ test('a zeroed conditioning negative shows as an explicit empty negative without
}finally{scope.dispose();f.restore();}
});
test('a prompt that left out run-time trigger words says so under its text (D125)',()=>{
const f=dom(),scope=createScope(),container=new f.Element('div'),sent=[];
const note='Text added at run time, such as trigger words, is not included.';
try{
module.mountPromptBlocks(container,{parentScope:scope,sender:{open:roles=>sent.push(roles)},prompts:{positive:{status:'resolved',text:'own text',source:'runtime-parts-omitted'},negative:{status:'resolved',text:'blur',source:'graph-inferred'}}});
const notes=f.walk(container,node=>node.textContent===note);
assert.equal(notes.length,1,'one note, under the positive only');
assert.equal(notes[0].parent.attributes['aria-label'],'Positive prompt');
container.click(f.walk(container,node=>node.dataset.sendRole==='pair')[0]);
assert.deepEqual(sent,[{positive:'own text',negative:'blur'}],'the note is never sent');
}finally{scope.dispose();f.restore();}
});
test('a prompt a later stage added conditioning to says so, and both lines show together (D126)',()=>{
const f=dom(),scope=createScope(),container=new f.Element('div');
const added='Conditioning added at a later stage is not included.', runtime='Text added at run time, such as trigger words, is not included.';
try{
const view=module.mountPromptBlocks(container,{parentScope:scope,prompts:{positive:{status:'resolved',text:'base',source:'added-conditioning'}}});
assert.deepEqual(f.walk(container,node=>node.className==='small muted').map(node=>node.textContent),[added]);
view.update({positive:{status:'resolved',text:'base',source:'runtime-parts-omitted-and-added-conditioning'}});
assert.deepEqual(f.walk(container,node=>node.className==='small muted').map(node=>node.textContent),[runtime,added]);
view.update({positive:{status:'resolved',text:'base',source:'graph-inferred'}});
assert.equal(f.walk(container,node=>node.className==='small muted').length,0);
}finally{scope.dispose();f.restore();}
});
const unresolved=message=>({status:'unresolved',text:null,source:null,...(message===undefined?{}:{message})});
const statusLines=(f,container)=>f.walk(container,node=>node.tagName==='p'&&node.className==='prompt-text muted').map(node=>node.textContent);
test('unsupported node status names the class',()=>{
-18
View File
@@ -182,24 +182,6 @@ test('start returns busy with the running job', async () => {
world.dispose();
});
test('track follows a job another route started, with no read of its own', async () => {
const { createActivityStore } = await source;
const { world, ...deps } = harness();
const store = createActivityStore(deps);
const seen = [];
store.subscribe(() => seen.push(store.find(ID_A)?.state));
const job = store.track(summary({ kind: 'health', title: 'Health check' }));
assert.equal(job.id, ID_A);
assert.equal(world.gets, 0, 'the started job is known without a list read');
assert.deepEqual(world.open().map(stream => stream.url), [`/prompt_manager/jobs/${ID_A}/events`]);
assert.deepEqual(world.toasts, [], 'the route that started it says so itself');
send(world.open()[0], summary({ kind: 'health', title: 'Health check', state: 'finished', progress: 100 }));
assert.deepEqual(seen, ['running', 'finished']);
assert.equal(world.open().length, 0, 'an ended job is no longer followed');
assert.throws(() => store.track({ id: 'nope' }), /Invalid job/);
world.dispose();
});
test('a list that is not a job list counts as a failed read', async () => {
const { createActivityStore } = await source;
const { world, api, ...deps } = harness();
-16
View File
@@ -478,19 +478,3 @@ test('a refresh deferred behind a page load is dropped when the stream is dispos
await new Promise(resolve=>setImmediate(resolve));
assert.equal(requests.length,2,'no refresh runs after dispose');
});
test('defaults apply to a new visit but not to the current view or a saved view', async () => {
const { initialImagesView } = await source;
assert.equal(typeof initialImagesView, 'function', 'initialImagesView is exported');
const preferences = { workspaceLayout: 'masonry', workspaceDensity: 'compact' };
// A new visit starts from System › Appearance.
assert.deepEqual(initialImagesView({ preferences }), { layout: 'masonry', density: 'compact' });
// The current view (the last visit) keeps its own toolbar choices.
assert.deepEqual(initialImagesView({ prior: { layout: 'list', density: 'large' }, preferences }), { layout: 'list', density: 'large' });
assert.deepEqual(initialImagesView({ prior: {}, preferences }), { layout: 'grid', density: 'comfortable' });
// A saved view opens as it always did.
assert.deepEqual(initialImagesView({ saved: { q: '' }, preferences }), { layout: 'grid', density: 'comfortable' });
// Without readable preferences nothing changes.
assert.deepEqual(initialImagesView({}), { layout: 'grid', density: 'comfortable' });
assert.deepEqual(initialImagesView({ preferences: {} }), { layout: 'grid', density: 'comfortable' });
});
@@ -825,73 +825,3 @@ test('refresh re-reads the selected file and keeps the shown record until the ne
assert.equal(controller.state.record.identity.path, 'folder/b.png');
controller.dispose(); assert.equal(await controller.refresh(), undefined);
});
test('metadata off shows an explicit load and loads once on request', async () => {
const { api, requests } = pendingApi(), updates = [];
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, autoLoad: false });
controller.subscribe(state => updates.push(state));
// Selecting reads nothing: the state says the metadata waits for a request.
assert.equal(controller.select(image()), undefined);
assert.equal(requests.length, 0);
assert.equal(controller.state.deferred, true);
assert.equal(controller.state.loading, false);
assert.equal(controller.state.record, null);
// Selecting the same file again still reads nothing.
controller.select({ ...image() });
assert.equal(requests.length, 0);
const load = controller.load();
assert.equal(requests.length, 1);
assert.equal(controller.state.loading, true);
assert.equal(controller.state.deferred, false);
// A second press while it reads, and the same file selected again once it loaded, read nothing more.
controller.load();
assert.equal(requests.length, 1);
requests[0].resolve(envelope()); await load;
assert.equal(controller.state.record.generation.seed.value, '9007199254740993');
controller.load(); controller.select(image());
assert.equal(requests.length, 1);
// Another file waits for its own request again.
controller.select({ ...image(), path: 'folder/b.png', filename: 'b.png', url: image().url.replace('a%20%23%25', 'b') });
assert.equal(controller.state.deferred, true);
assert.equal(requests.length, 1);
// The default controller loads on selection, as before.
const eager = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts });
eager.select(image());
assert.equal(requests.length, 2);
assert.equal(eager.state.deferred, false);
eager.dispose(); controller.dispose();
assert.equal(controller.load(), undefined);
});
test('cache off rereads the current selection; on reuses the one current record', async () => {
for (const cache of [true, false]) {
const { api, requests } = pendingApi();
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, cache });
const first = controller.select(image());
// A read in flight is shared either way.
controller.select(image());
assert.equal(requests.length, 1);
requests[0].resolve(envelope()); await first;
const again = controller.select(image());
assert.equal(requests.length, cache ? 1 : 2, `cache ${cache}`);
if (!cache) { requests[1].resolve(envelope()); await again; assert.equal(controller.state.record.identity.path, 'folder/a #%.png'); }
controller.dispose();
}
});
test('cache off never rereads a selection whose source changed, and refresh never loads a deferred one', async () => {
const { api, requests } = pendingApi();
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, cache: false });
const first = controller.select(image());
requests[0].reject(new ApiError(409, refreshMessage, 'source-changed')); await first;
assert.equal(controller.state.refreshRequired, true);
controller.select(image());
assert.equal(requests.length, 1);
controller.dispose();
const deferred = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, autoLoad: false });
deferred.select(image());
assert.equal(deferred.refresh(), undefined);
assert.equal(requests.length, 1);
assert.equal(deferred.state.deferred, true);
deferred.dispose();
});
-11
View File
@@ -136,17 +136,6 @@ test('an unusable semantics table is refused', () => {
for (const doc of [null, {schema: 2}, {schema: 1, samplers: []}]) assert.throws(() => resolver.compileSemantics(doc), resolver.SemanticsError);
});
test('the lineage section is compiled, and a bad one refuses the table', () => {
const doc = JSON.parse(fs.readFileSync(new URL('../../web/js/inspection-semantics.json', import.meta.url), 'utf8'));
assert.deepEqual(semantics.lineage.get('Image Filter\u00001'), { strategy: 'continue', input: 'latents' });
// Lineage describes no prompt: its classes are not table classes for the stage walk.
assert.equal(semantics.allClasses.has('VAEEncode'), false);
const row = { class: 'VAEEncode', slot: 0, strategy: 'continue', input: 'pixels' };
for (const lineage of [undefined, {}, [null], [{ ...row, slot: -1 }], [{ ...row, strategy: 'passthrough' }], [{ ...row, input: '' }], [row, row]]) {
assert.throws(() => resolver.compileSemantics({ ...doc, lineage }), resolver.SemanticsError, JSON.stringify(lineage));
}
});
// The committed server answers (tests/fixtures/local-inspect/resolver-vectors.json, written by
// `python3 -m tests.local_resolver_cases --write` and checked current by the Python suite).
test('committed server answers equal the local evidence', () => {
+5 -38
View File
@@ -124,51 +124,18 @@ test('real inspection navigation preserves slideshow across two advances and def
} finally {view.dispose();controller.dispose();parentScope.dispose();globalThis.setTimeout=oldSetTimeout;globalThis.clearTimeout=oldClearTimeout;f.restore();}
});
test('Viewer.js is built only on a loaded image and an unbuilt viewer is released at once', async () => {
const {reportedLocationAdapter:adapter} = await import('../../web/js/workspace/history-media.mjs');
const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c'),occurrenceMedia('2',0,'d')];
const instances = [];
// On a loaded image Viewer.js schedules its inline build; until it runs `ready` is false and destroy() cancels it.
f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.ready = false; this.viewed = false; instances.push(this); }
destroy() { this.destroyed++; } resize() {} };
const globals = {ResizeObserver:globalThis.ResizeObserver, requestAnimationFrame:globalThis.requestAnimationFrame, cancelAnimationFrame:globalThis.cancelAnimationFrame};
Object.assign(globalThis, {ResizeObserver:class { observe() {} disconnect() {} }, requestAnimationFrame:() => 1, cancelAnimationFrame() {}});
const view = module.createMediaViewer({container:f.container,parentScope,inspectionController:controller,adapter,sender:{open(){}}});
try {
const stageImage = () => f.walk(f.container,n => n.tagName === 'img')[0];
view.open({items,selectedKey:'d'.repeat(64)});
// Viewer.js's own cancel misses an image whose load event is still queued, so none exists before the load.
const unloaded = stageImage(), dropped = [];unloaded.removeAttribute = key => dropped.push(key);
assert.equal(instances.length, 0, 'no Viewer.js before the image loads');
f.walk(f.container,n => n.attributes['aria-label'] === 'Previous media')[0].fire('click');
assert.deepEqual(dropped, ['src'], 'an image left before it loads is released at once');
unloaded.fire('load');
assert.equal(instances.length, 0, 'its late load builds no Viewer.js');
stageImage().fire('load');
const [loading] = instances, image = loading.node, removed = [];
image.removeAttribute = key => removed.push(key);
// Next drops the image whose build is pending; a deferred destroy would let that build run there.
f.walk(f.container,n => n.attributes['aria-label'] === 'Next media')[0].fire('click');
stageImage().fire('load');assert.equal(instances.length, 2, 'the next image builds its own Viewer.js once loaded');
assert.equal(loading.destroyed, 1, 'an unbuilt viewer is destroyed at once');
assert.deepEqual(removed, ['src'], 'and its image is released at once');
loading.options.viewed.call(loading);
assert.equal(loading.destroyed, 1, 'never destroyed twice');
} finally { view.dispose(); parentScope.dispose(); f.restore(); Object.assign(globalThis, globals); }
});
test('an inline Viewer still loading its image is destroyed only after it has viewed it', async () => {
const {reportedLocationAdapter:adapter} = await import('../../web/js/workspace/history-media.mjs');
const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c')];
const instances = [];
// A built (`ready`) inline Viewer.js's destroy() leaves an in-flight view running; its callback then needs a live instance.
f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.ready = true; this.viewed = false; instances.push(this); }
// Viewer.js's inline destroy() leaves an in-flight view running; its callback then needs a live instance.
f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.viewed = false; instances.push(this); }
destroy() { this.destroyed++; } resize() {} };
const globals = {ResizeObserver:globalThis.ResizeObserver, requestAnimationFrame:globalThis.requestAnimationFrame, cancelAnimationFrame:globalThis.cancelAnimationFrame};
Object.assign(globalThis, {ResizeObserver:class { observe() {} disconnect() {} }, requestAnimationFrame:() => 1, cancelAnimationFrame() {}});
const view = module.createMediaViewer({container:f.container,parentScope,inspectionController:controller,adapter,sender:{open(){}}});
try {
view.open({items,selectedKey:'c'.repeat(64)});f.walk(f.container,n => n.tagName === 'img')[0].fire('load');
view.open({items,selectedKey:'c'.repeat(64)});
const [loading] = instances, image = loading.node, removed = [];
image.removeAttribute = key => removed.push(key);
f.walk(f.container,n => n.attributes['aria-label'] === 'Close image viewer')[0].fire('click');
@@ -181,7 +148,7 @@ test('an inline Viewer still loading its image is destroyed only after it has vi
loading.options.viewed.call(loading);
assert.equal(loading.destroyed, 1, 'never destroyed twice');
view.open({items,selectedKey:'c'.repeat(64)});f.walk(f.container,n => n.tagName === 'img')[0].fire('load');
view.open({items,selectedKey:'c'.repeat(64)});
const shown = instances[1]; shown.viewed = true; shown.options.viewed.call(shown);
f.walk(f.container,n => n.attributes['aria-label'] === 'Close image viewer')[0].fire('click');
assert.equal(shown.destroyed, 1, 'a viewed image is destroyed at once');
@@ -269,7 +236,7 @@ test('viewer uses adapter.mediaUrl for image and video', async () => {
const img = f.walk(f.container,n => n.tagName === 'img')[0];
assert.equal(img.src, image.url, 'the blob URL is used exactly');
assert.doesNotMatch(img.src, /\?/);
img.fire('load');assert.equal(instances[0].node, img, 'Viewer.js shows the blob image');
assert.equal(instances[0].node, img, 'Viewer.js shows the blob image');
const download = f.walk(f.container,n => n.attributes['aria-label'] === 'Download original')[0];
assert.equal(download.href, image.url);assert.equal(download.download, 'a.png');
f.walk(f.container,n => n.attributes['aria-label'] === 'Next media')[0].fire('click');
-193
View File
@@ -1,193 +0,0 @@
// Browser display defaults (#249 System S3 Task 7): the workspace reads and writes the legacy gallery's own
// `gallerySettings` object (Q3, "One display configuration"), writes only the keys it changed, never consumes the
// legacy paging keys, and says when this browser refused to keep a change.
import test from 'node:test';
import assert from 'node:assert/strict';
import gallerySettings from '../../web/js/gallery-settings.js';
const module = await import('../../web/js/workspace/preferences.mjs').catch(error => {
if (error.code !== 'ERR_MODULE_NOT_FOUND') throw error;
return {};
});
globalThis.GallerySettings = gallerySettings;
const KEY = 'gallerySettings';
/** A localStorage stand-in holding `text` under the settings key; `setItem` can be made to throw. */
function storage(text = null, { quota = false } = {}) {
const writes = [];
const store = {
text,
writes,
getItem(key) { assert.equal(key, KEY); return store.text; },
setItem(key, value) {
assert.equal(key, KEY);
writes.push(value);
if (quota) { const error = new Error('Quota exceeded'); error.name = 'QuotaExceededError'; throw error; }
store.text = value;
},
removeItem() { assert.fail('Preferences never remove the shared settings'); },
};
return store;
}
const denied = () => { const error = new Error('Denied'); error.name = 'SecurityError'; throw error; };
test('normalizes new workspace keys and keeps legacy keys', () => {
assert.equal(typeof module.readPreferences, 'function', 'preferences.mjs exports readPreferences');
module.resetTabPreferences();
const legacy = { defaultViewMode: 'list', gridColumns: 5, defaultLimit: 200, thumbnailPromptDismissed: 'sig', extra: 1 };
const store = storage(JSON.stringify({ ...legacy, workspaceLayout: 'masonry', workspaceDensity: 'compact', showFilePaths: false }));
const { values, persisted } = module.readPreferences(() => store);
assert.equal(persisted, true);
assert.equal(values.workspaceLayout, 'masonry');
assert.equal(values.workspaceDensity, 'compact');
assert.equal(values.showFilePaths, false);
// Invalid workspace values fall back to the defaults, as every legacy key does.
store.text = JSON.stringify({ workspaceLayout: 'mosaic', workspaceDensity: 7, videoMute: 'no' });
const fallback = module.readPreferences(() => store).values;
assert.equal(fallback.workspaceLayout, 'grid');
assert.equal(fallback.workspaceDensity, 'comfortable');
assert.equal(fallback.videoMute, true);
// The legacy normalizer knows the new keys and still keeps unknown ones.
const normalized = gallerySettings.normalizeGallerySettings({ ...legacy, workspaceLayout: 'list' });
assert.equal(normalized.workspaceLayout, 'list');
assert.equal(normalized.workspaceDensity, 'comfortable');
assert.equal(normalized.extra, 1);
assert.equal(normalized.thumbnailPromptDismissed, 'sig');
assert.deepEqual(gallerySettings.WORKSPACE_LAYOUTS, ['grid', 'list', 'masonry']);
assert.deepEqual(gallerySettings.WORKSPACE_DENSITIES, ['comfortable', 'large', 'compact']);
});
test('write touches only dirty keys', () => {
module.resetTabPreferences();
const stored = { defaultViewMode: 'list', gridColumns: 5, thumbnailsGenerated: true, extra: { kept: true }, videoMute: false };
const store = storage(JSON.stringify(stored));
const result = module.writePreferences({ workspaceLayout: 'masonry', showImageInfo: false }, () => store);
assert.deepEqual(result, { persisted: true, conflicts: [] });
assert.equal(store.writes.length, 1);
// Every other key, legacy or unknown, is written back exactly as it was stored: no defaults are filled in.
assert.deepEqual(JSON.parse(store.text), { ...stored, workspaceLayout: 'masonry', showImageInfo: false });
// It re-reads the stored object, so a key another page wrote after this one read is kept.
store.text = JSON.stringify({ ...JSON.parse(store.text), gridColumns: 9 });
module.writePreferences({ videoLoop: false }, () => store);
assert.equal(JSON.parse(store.text).gridColumns, 9);
assert.equal(JSON.parse(store.text).workspaceLayout, 'masonry');
// An empty or malformed stored value starts from an empty object, never the defaults.
for (const text of [null, '{broken', '[]', '42']) {
const fresh = storage(text);
assert.equal(module.writePreferences({ cacheMetadata: false }, () => fresh).persisted, true);
assert.deepEqual(JSON.parse(fresh.text), { cacheMetadata: false });
}
// Only workspace keys with valid values are accepted; nothing is written for a bad change.
for (const bad of [{ gridColumns: 4 }, { defaultLimit: 50 }, { workspaceLayout: 'mosaic' }, { videoMute: 'yes' }, { imageQuality: 'ultra' }]) {
const untouched = storage(JSON.stringify(stored));
assert.throws(() => module.writePreferences(bad, () => untouched), TypeError);
assert.equal(untouched.writes.length, 0);
}
// A key changed elsewhere since this tab read it (its baseline) is reported, and the explicit save still wins.
const elsewhere = storage(JSON.stringify({ videoLoop: false }));
const outcome = module.writePreferences({ videoLoop: true, videoMute: false }, () => elsewhere, { baseline: { videoLoop: true, videoMute: true } });
assert.deepEqual(outcome, { persisted: true, conflicts: ['videoLoop'] });
assert.equal(JSON.parse(elsewhere.text).videoLoop, true);
});
test('denied storage reports tab-only', () => {
module.resetTabPreferences();
const read = module.readPreferences(denied);
assert.equal(read.persisted, false);
assert.equal(read.values.workspaceLayout, 'grid');
const result = module.writePreferences({ workspaceLayout: 'list', videoAutoplay: true }, denied);
assert.deepEqual(result, { persisted: false, conflicts: [] });
// The change is kept for this tab: every reader in it (a new Images visit, the viewer) sees it.
const tab = module.readPreferences(denied);
assert.equal(tab.persisted, false);
assert.equal(tab.values.workspaceLayout, 'list');
assert.equal(tab.values.videoAutoplay, true);
// A getItem that throws is denied storage too.
const throwing = { getItem: denied, setItem: denied };
assert.equal(module.readPreferences(() => throwing).values.workspaceLayout, 'list');
assert.equal(module.writePreferences({ workspaceDensity: 'large' }, () => throwing).persisted, false);
// Once storage accepts a write, the tab copy of that key gives way to the stored value.
const working = storage(JSON.stringify({ workspaceLayout: 'masonry' }));
assert.equal(module.writePreferences({ workspaceLayout: 'masonry' }, () => working).persisted, true);
assert.equal(module.readPreferences(() => working).values.workspaceLayout, 'masonry');
module.resetTabPreferences();
assert.equal(module.readPreferences(denied).values.videoAutoplay, false);
});
test('quota error keeps previous bytes', () => {
module.resetTabPreferences();
const before = JSON.stringify({ gridColumns: 6, workspaceDensity: 'large' });
const store = storage(before, { quota: true });
const result = module.writePreferences({ workspaceDensity: 'compact' }, () => store);
assert.deepEqual(result, { persisted: false, conflicts: [] });
assert.equal(store.writes.length, 1);
assert.equal(store.text, before);
// This tab keeps the change it could not store.
assert.equal(module.readPreferences(() => store).values.workspaceDensity, 'compact');
module.resetTabPreferences();
assert.equal(module.readPreferences(() => store).values.workspaceDensity, 'large');
});
/** A window stand-in that records its storage listeners. */
function fakeWindow() {
const listeners = new Set();
return {
listeners,
addEventListener(type, listener) { assert.equal(type, 'storage'); listeners.add(listener); },
removeEventListener(type, listener) { assert.equal(type, 'storage'); listeners.delete(listener); },
fire(event) { for (const listener of [...listeners]) listener(event); },
};
}
test('storage event updates clean values and flags dirty conflicts', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ workspaceLayout: 'list', videoMute: false }));
const win = fakeWindow();
const seen = [];
const unsubscribe = module.subscribePreferences(win, next => seen.push(next), () => store);
assert.equal(win.listeners.size, 1);
// Another key, or another storage area's event, is not ours.
win.fire({ key: 'pm_update_notice_dismissed', storageArea: store });
win.fire({ key: KEY, storageArea: { other: true } });
assert.equal(seen.length, 0);
win.fire({ key: KEY, storageArea: store });
win.fire({ key: null, storageArea: store }); // localStorage.clear() elsewhere
assert.equal(seen.length, 2);
assert.equal(seen[0].values.workspaceLayout, 'list');
unsubscribe();
assert.equal(win.listeners.size, 0);
// The reconciliation of the System drafts: clean values follow the other page; a key this tab has a draft for and
// the other page changed is a conflict, and the draft stays.
const baseline = { workspaceLayout: 'grid', videoMute: true, videoLoop: true };
const next = { workspaceLayout: 'list', videoMute: true, videoLoop: false };
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { videoMute: false, videoLoop: false }, next }), { conflicts: [] });
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { workspaceLayout: 'masonry' }, next }), { conflicts: ['workspaceLayout'] });
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { workspaceLayout: 'list' }, next }), { conflicts: [] });
});
test('legacy paging keys are never consumed by the workspace', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ defaultLimit: 500, gridColumns: 12, infiniteScroll: true, lazyLoading: false,
defaultViewMode: 'list', debugMode: true, apiTimeout: 9, thumbnailsGenerated: true, checkThumbnailsAtStartup: false }));
const { values } = module.readPreferences(() => store);
assert.deepEqual(Object.keys(values).sort(), [...module.WORKSPACE_PREFERENCE_KEYS].sort());
for (const key of ['defaultLimit', 'gridColumns', 'infiniteScroll', 'lazyLoading', 'defaultViewMode', 'debugMode', 'apiTimeout',
'thumbnailsGenerated', 'checkThumbnailsAtStartup', 'thumbnailPromptDismissed']) assert.equal(Object.hasOwn(values, key), false, key);
// The display defaults are the gallery's own defaults for these keys.
const defaults = module.preferenceDefaults();
for (const key of module.WORKSPACE_PREFERENCE_KEYS) assert.equal(defaults[key], gallerySettings.DEFAULT_SETTINGS[key], key);
});
test('a later save flushes the changes kept for this tab only', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ gridColumns: 4 }), { quota: true });
assert.equal(module.writePreferences({ workspaceLayout: 'list' }, () => store).persisted, false);
// Storage accepts writes again: saving another key stores the tab-only change with it, so "Saved" is true for both.
const working = storage(store.text);
assert.deepEqual(module.writePreferences({ videoLoop: false }, () => working), { persisted: true, conflicts: [] });
assert.deepEqual(JSON.parse(working.text), { gridColumns: 4, workspaceLayout: 'list', videoLoop: false });
// Nothing is overlaid any more: another page's later change reads through.
working.text = JSON.stringify({ ...JSON.parse(working.text), workspaceLayout: 'masonry' });
assert.equal(module.readPreferences(() => working).values.workspaceLayout, 'masonry');
module.resetTabPreferences();
});
-788
View File
@@ -1,788 +0,0 @@
// Run with: node --test tests/js/workspace-review-state.test.js
const test = require('node:test');
const assert = require('node:assert/strict');
const { createApiClient } = require('../../web/js/api-client.js');
// Only an absent module is tolerated, so RED is an assertion failure, not a crash.
const source = import('../../web/js/workspace/review-state.mjs').catch(error => {
if (error.code !== 'ERR_MODULE_NOT_FOUND') throw error;
return {};
});
async function load() {
const module = await source;
assert.equal(typeof module.createReviewController, 'function', 'review-state.mjs is not implemented');
return module;
}
const EPOCH = '11111111-1111-4111-8111-111111111111';
const SCOPE = 'c'.repeat(64);
const SHA = 'd'.repeat(64);
const PRINT = 'e'.repeat(64);
const HASH_A = 'a'.repeat(64);
const HASH_B = 'b'.repeat(64);
const HASH_C = 'f'.repeat(64);
const ITEMS = '/prompt_manager/review/items';
const OBSERVE = '/prompt_manager/image-annotations/observe';
const CAPTURE = '/prompt_manager/image-annotations/capture/';
const flush = () => new Promise(resolve => setImmediate(resolve));
async function settle(times = 8) { for (let i = 0; i < times; i++) await flush(); }
function response(status, body) {
return { ok: status >= 200 && status < 300, status, statusText: `Status ${status}`, json: async () => body };
}
const fail = (status, code, error = `failure ${code}`, extra = {}) => response(status, { success: false, code, error, ...extra });
function item(id, overrides = {}) {
const path = `dunes/${id}.png`;
return {
id, state: 'pending', revision: 0, created_at: `2026-10-22T10:00:0${id.length}+00:00`, filename: `${id}.png`,
target: { root_index: 0, root_scope: SCOPE, path }, unavailable: false, candidate_count: 3, limited: false,
url: `/prompt_manager/images/serve/${path}?root=0&root_scope=${SCOPE}`, is_video: false, ...overrides,
};
}
const candidate = (hash, sampler, label, positive, negative, state = 'resolved') => ({
hash, sampler_id: sampler, state, reason: '', label,
positive: positive === null ? null : { text: positive }, negative: negative === null ? null : { text: negative },
});
function detailOf(entry, overrides = {}) {
return {
success: true, library_epoch: EPOCH, ...entry, total: 2, sha256: SHA, dependency_fingerprint: PRINT,
candidates: [
candidate(HASH_C, '9', 'No positive / Soft negative', null, 'blur', 'unresolved'),
candidate(HASH_A, '12', 'Dunes / Soft negative', 'golden dunes', 'blur'),
candidate(HASH_B, '28', 'Dunes / Harsh negative', 'golden dunes', 'noise'),
],
...overrides,
};
}
const page = (items, overrides = {}) => ({ success: true, library_epoch: EPOCH, total: items.length, items, next_cursor: null, ...overrides });
const observed = (token = 'capture-1') => ({
success: true, library_epoch: EPOCH, target: item('one').target, filename: 'one.png', title: 'one.png', tags: [], tag_revision: 0,
capture: { token, sha256: SHA, preview_url: CAPTURE + token, size: 10, width: 8, height: 8, media_type: 'image',
expires_at: '2026-10-22T12:00:00+00:00', generate: { supported: true, reason: null } },
});
/** Real createApiClient over a fetch whose answers come from routes['METHOD path'](call). */
function harness(routes = {}) {
const calls = [];
const fetch = (url, init = {}) => {
const call = { url, method: init.method, body: init.body === undefined ? undefined : JSON.parse(init.body) };
calls.push(call);
const path = url.startsWith(CAPTURE) ? CAPTURE : url.split('?')[0];
const route = routes[`${call.method} ${path}`];
return new Promise((resolve, reject) => {
init.signal?.addEventListener('abort', () => reject(new DOMException('Aborted', 'AbortError')));
Promise.resolve().then(() => (route ? route(call) : response(200, { success: true }))).then(resolve, reject);
});
};
return {
api: createApiClient(fetch), calls,
of: (method, prefix) => calls.filter(call => call.method === method && call.url.startsWith(prefix)),
};
}
const QUEUE = [item('one'), item('two')];
function routes(extra = {}) {
return {
[`GET ${ITEMS}`]: () => response(200, page(QUEUE)),
[`GET ${ITEMS}/one`]: () => response(200, detailOf(QUEUE[0])),
[`GET ${ITEMS}/two`]: () => response(200, detailOf(QUEUE[1], { candidates: [], limited: true, candidate_count: 0 })),
[`POST ${OBSERVE}`]: () => response(200, observed()),
[`POST ${ITEMS}/one/decision`]: call => response(200, { success: true, library_epoch: EPOCH, total: 1,
item: { ...QUEUE[0], state: call.body.choice === 'unresolved' ? 'unresolved' : 'selected', revision: 1 } }),
...extra,
};
}
async function loaded(extra) {
const { createReviewController } = await load();
const h = harness(routes(extra));
const c = createReviewController({ api: h.api });
await c.load();
await settle();
return { c, h };
}
test('notice titles for 0, 1 and 3', async () => {
const { noticeTitle } = await load();
assert.equal(noticeTitle(0), 'No images need a decision');
assert.equal(noticeTitle(1), '1 image needs a decision');
assert.equal(noticeTitle(3), '3 images need a decision');
assert.equal(noticeTitle(1234), '1,234 images need a decision');
});
test('loads pending items and the exact count', async () => {
const { c, h } = await loaded();
assert.equal(c.state.status, 'ready');
assert.equal(c.state.total, 2);
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
const [list] = h.of('GET', ITEMS);
assert.equal(list.url, `${ITEMS}?limit=40`);
// The first item is selected, as the concept selects its first queue image.
assert.equal(c.state.selectedId, 'one');
assert.ok(Object.isFrozen(c.state));
});
test('selecting a card loads detail and option labels, first workable preselected, unusable disabled', async () => {
const { c } = await loaded();
assert.equal(c.state.detailStatus, 'ready');
assert.deepEqual(c.state.options.map(option => [option.label, option.disabled]), [
['Sampler 9 · No positive / Soft negative', true],
['Sampler 12 · Dunes / Soft negative', false],
['Sampler 28 · Dunes / Harsh negative', false],
['Leave unresolved', false],
]);
assert.equal(c.state.options.at(-1).value, 'unresolved');
assert.equal(c.state.choice, HASH_A);
assert.deepEqual(c.state.prompts, { positive: { status: 'resolved', text: 'golden dunes' }, negative: { status: 'resolved', text: 'blur' } });
c.choose(HASH_C);
assert.equal(c.state.choice, HASH_A, 'a disabled option cannot be chosen');
c.choose(HASH_B);
assert.equal(c.state.prompts.negative.text, 'noise');
c.choose('unresolved');
assert.equal(c.state.prompts.positive.status, 'ambiguous');
await c.select('two');
await settle();
assert.equal(c.state.selectedId, 'two');
assert.deepEqual(c.state.options.map(option => option.label), ['Leave unresolved']);
assert.equal(c.state.choice, 'unresolved');
assert.match(c.state.warning, /too many sampler paths/i);
});
test('apply sends the selected candidate hash or unresolved with a fresh capture', async () => {
const { c, h } = await loaded();
c.choose(HASH_B);
const done = await c.apply();
assert.equal(done, true);
const [observe] = h.of('POST', OBSERVE);
assert.deepEqual(observe.body, { target: QUEUE[0].target });
const [decision] = h.of('POST', `${ITEMS}/one/decision`);
assert.deepEqual(decision.body, { expected_epoch: EPOCH, expected_revision: 0, capture_token: 'capture-1', sha256: SHA,
dependency_fingerprint: PRINT, choice: HASH_B });
assert.equal(c.state.phase, 'done');
assert.equal(c.state.total, 1);
assert.equal(c.state.toast, 'Association accepted.');
// The server releases a used capture itself.
assert.equal(h.of('DELETE', CAPTURE).length, 0);
const again = await loaded();
again.c.choose('unresolved');
assert.equal(await again.c.apply(), true);
const [unresolved] = again.h.of('POST', `${ITEMS}/one/decision`);
assert.equal(unresolved.body.choice, 'unresolved');
assert.equal(unresolved.body.capture_token, 'capture-1');
assert.equal(again.c.state.toast, 'Left unresolved.');
});
test('an unavailable file can only be left unresolved, without a capture', async () => {
const gone = item('one', { target: null, unavailable: true, url: null });
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page([gone])),
[`GET ${ITEMS}/one`]: () => response(200, detailOf(gone)),
});
assert.ok(c.state.options.slice(0, -1).every(option => option.disabled));
assert.equal(c.state.choice, 'unresolved');
assert.match(c.state.warning, /not under a current image source/i);
assert.equal(await c.apply(), true);
assert.equal(h.of('POST', OBSERVE).length, 0);
const [decision] = h.of('POST', `${ITEMS}/one/decision`);
assert.equal(decision.body.capture_token, null);
assert.equal(decision.body.sha256, SHA);
});
test('association conflict keeps the selection and shows the inline warning', async () => {
const message = 'This image is already linked to another prompt or negative, so this sampler path cannot be attached safely.';
const { c, h } = await loaded({ [`POST ${ITEMS}/one/decision`]: () => fail(409, 'association-conflict', message) });
assert.equal(await c.apply(), false);
assert.equal(c.state.selectedId, 'one');
assert.equal(c.state.choice, HASH_A);
assert.equal(c.state.phase, 'idle');
assert.equal(c.state.warning, message);
// The refused decision's capture is released.
assert.equal(h.of('DELETE', CAPTURE).length, 1);
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'no reload');
});
test('storage-changed requires reload', async () => {
let lists = 0;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => { lists += 1; return response(200, page(QUEUE)); },
[`POST ${ITEMS}/one/decision`]: () => fail(409, 'storage-changed', 'Prompt storage was restored; reload the review queue.'),
});
assert.equal(await c.apply(), false);
await settle();
assert.equal(lists, 2, 'the queue is read again under the new epoch');
assert.equal(c.state.warning, 'Prompt storage was restored; reload the review queue.');
assert.equal(c.state.canRetry, false);
assert.equal(h.of('DELETE', CAPTURE).length, 1);
});
test('lost response keeps the selection and retry resends once', async () => {
let attempts = 0;
const { c, h } = await loaded({
[`POST ${ITEMS}/one/decision`]: call => {
attempts += 1;
if (attempts === 1) return Promise.reject(new TypeError('Failed to fetch'));
return response(200, { success: true, library_epoch: EPOCH, total: 1, item: { ...QUEUE[0], state: 'selected', revision: 1 } });
},
});
assert.equal(await c.apply(), false);
assert.equal(c.state.selectedId, 'one');
assert.equal(c.state.canRetry, true);
assert.match(c.state.warning, /may not have been saved/i);
assert.equal(h.of('DELETE', CAPTURE).length, 0, 'the capture is kept for the retry');
const [first] = h.of('POST', `${ITEMS}/one/decision`);
assert.equal(await c.retry(), true);
const sent = h.of('POST', `${ITEMS}/one/decision`);
assert.equal(sent.length, 2);
assert.deepEqual(sent[1].body, first.body, 'the identical decision is resent');
assert.equal(h.of('POST', OBSERVE).length, 1, 'no second capture');
assert.equal(await c.retry(), false, 'nothing left to resend');
assert.equal(h.of('POST', `${ITEMS}/one/decision`).length, 2);
});
test('a retry answered by review-conflict reloads the queue', async () => {
let attempts = 0, lists = 0;
const { c } = await loaded({
[`GET ${ITEMS}`]: () => { lists += 1; return response(200, page(lists === 1 ? QUEUE : [QUEUE[1]])); },
[`POST ${ITEMS}/one/decision`]: () => {
attempts += 1;
if (attempts === 1) return Promise.reject(new TypeError('Failed to fetch'));
return fail(409, 'review-conflict', 'This item was decided or changed since it was opened. Reload the queue.');
},
});
await c.apply();
assert.equal(await c.retry(), false);
await settle();
assert.equal(lists, 2);
assert.equal(c.state.selectedId, 'two');
});
test('a review-conflict on a still pending item reads its detail again', async () => {
let details = 0, decisions = 0;
const { c, h } = await loaded({
[`GET ${ITEMS}/one`]: () => { details += 1; return response(200, detailOf({ ...QUEUE[0], revision: details - 1 })); },
[`POST ${ITEMS}/one/decision`]: () => {
decisions += 1;
return decisions === 1 ? fail(409, 'review-conflict', 'This item was decided or changed since it was opened. Reload the queue.')
: response(200, { success: true, library_epoch: EPOCH, total: 1, item: { ...QUEUE[0], state: 'selected', revision: 2 } });
},
});
assert.equal(await c.apply(), false);
await settle();
assert.equal(details, 2, 'the item is read again');
assert.equal(c.state.detail.revision, 1);
assert.equal(c.state.warning, 'This item was decided or changed since it was opened. Reload the queue.');
assert.equal(await c.apply(), true);
assert.equal(h.of('POST', `${ITEMS}/one/decision`).at(-1).body.expected_revision, 1);
});
test('leave for later writes nothing', async () => {
const { c, h } = await loaded();
c.leave();
assert.equal(c.state.phase, 'done');
assert.equal(c.state.toast, null);
assert.equal(h.of('POST', '/').length, 0);
assert.equal(h.of('DELETE', '/').length, 0);
});
test('load more pages with the cursor and keeps the selection', async () => {
const more = [item('three')];
const { c, h } = await loaded({
[`GET ${ITEMS}`]: call => response(200, call.url.includes('cursor=')
? page(more, { total: 3 }) : page(QUEUE, { total: 3, next_cursor: 'abc_-1' })),
});
assert.equal(c.state.nextCursor, 'abc_-1');
await c.loadMore();
assert.equal(h.of('GET', ITEMS).at(-1).url, `${ITEMS}?limit=40&cursor=abc_-1`);
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two', 'three']);
assert.equal(c.state.nextCursor, null);
assert.equal(c.state.selectedId, 'one');
});
test('a failed queue read is an error state and load retries', async () => {
let lists = 0;
const { c } = await loaded({ [`GET ${ITEMS}`]: () => { lists += 1; return lists === 1 ? fail(500, 'review-failed') : response(200, page(QUEUE)); } });
assert.equal(c.state.status, 'error');
await c.load();
await settle();
assert.equal(c.state.status, 'ready');
});
test('tag save toast carries the review suffix and the item stays pending', async () => {
const { c, h } = await loaded({
['GET /prompt_manager/autotag/models']: () => response(200, { success: true, models: {}, default_prompt: 'x',
wd14_general_threshold: 0.35, wd14_character_threshold: 0.85, suggestion_limit: 10 }),
['PUT /prompt_manager/image-annotations/tags']: call => response(200, { success: true, library_epoch: EPOCH, sha256: SHA,
tags: call.body.tags, tag_revision: 1, replayed: false }),
});
const { createImageTaggingController } = await import('../../web/js/workspace/image-tagging-state.mjs');
const before = h.calls.filter(call => call.url.startsWith(ITEMS)).length;
// The Tag dialog opened from Needs review is the shared controller with the review source.
const tagging = createImageTaggingController({ api: h.api, target: QUEUE[0].target, source: 'review' });
assert.equal(tagging.state.showScopeNote, true);
await tagging.open();
await settle();
tagging.setManual('dunes');
await tagging.save();
await settle();
assert.equal(tagging.state.toast, '1 tags saved for one.png. Prompt association still awaits review.');
tagging.dispose();
assert.equal(h.calls.filter(call => call.url.startsWith(ITEMS)).length, before, 'a tag save neither decides nor re-reads the queue');
assert.deepEqual(c.state.items.map(entry => [entry.id, entry.state]), [['one', 'pending'], ['two', 'pending']]);
assert.equal(c.state.phase, 'idle');
});
test('dispose releases a kept capture and stops reads', async () => {
const { c, h } = await loaded({ [`POST ${ITEMS}/one/decision`]: () => Promise.reject(new TypeError('Failed to fetch')) });
await c.apply();
c.dispose();
await settle();
const released = h.of('DELETE', CAPTURE);
assert.equal(released.length, 1);
assert.equal(released[0].url, CAPTURE + 'capture-1');
assert.equal(await c.retry(), false);
});
// F2: an open page re-reads its queue when the server says it changed (the review change stream).
function refreshOf(c) {
assert.equal(typeof c.refresh, 'function', 'refresh is not implemented');
return c.refresh();
}
test('refresh re-reads the queue quietly and keeps the selection and choice', async () => {
let queue = QUEUE;
const { c, h } = await loaded({ [`GET ${ITEMS}`]: () => response(200, page(queue)) });
c.choose(HASH_B);
const statuses = [];
c.subscribe(state => statuses.push(state.status));
queue = [...QUEUE, item('three')];
await refreshOf(c);
await settle();
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two', 'three']);
assert.equal(c.state.total, 3);
assert.equal(c.state.selectedId, 'one');
assert.equal(c.state.choice, HASH_B);
assert.equal(c.state.warning, '');
assert.ok(statuses.every(status => status === 'ready'), `no loading state: ${statuses}`);
assert.equal(h.of('GET', `${ITEMS}/one`).length, 1, 'the open item is not read again');
});
test('refreshes asked for during a refresh collapse into one more read', async () => {
let release;
const held = new Promise(resolve => { release = resolve; });
let lists = 0;
const { c, h } = await loaded({ [`GET ${ITEMS}`]: async () => {
lists += 1;
if (lists === 2) await held;
return response(200, page(QUEUE));
} });
const first = refreshOf(c);
await settle();
refreshOf(c);
refreshOf(c);
release();
await first;
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 3, 'the load, the refresh and one collapsed re-read');
assert.equal(c.state.status, 'ready');
});
test('a refresh after the open item was decided elsewhere opens the next one and says so', async () => {
let queue = QUEUE;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(queue)),
[`GET ${ITEMS}/three`]: () => response(200, detailOf(item('three'))),
});
queue = [item('three')];
await refreshOf(c);
await settle();
assert.equal(c.state.selectedId, 'three');
assert.equal(c.state.detailStatus, 'ready');
assert.equal(c.state.warning, 'one.png was decided elsewhere, so the next image is open.');
assert.equal(h.of('GET', `${ITEMS}/three`).length, 1);
});
test('a refresh to an empty queue clears the selection', async () => {
let queue = QUEUE;
const { c } = await loaded({ [`GET ${ITEMS}`]: () => response(200, page(queue)) });
queue = [];
await refreshOf(c);
await settle();
assert.equal(c.state.total, 0);
assert.deepEqual(c.state.items, []);
assert.equal(c.state.selectedId, null);
assert.equal(c.state.status, 'ready');
});
test('a failed refresh keeps the queue and says so', async () => {
let lists = 0;
const { c } = await loaded({ [`GET ${ITEMS}`]: () => { lists += 1; return lists === 1 ? response(200, page(QUEUE)) : fail(500, 'review-failed'); } });
await refreshOf(c);
await settle();
assert.equal(c.state.status, 'ready');
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
assert.equal(c.state.warning, 'The review queue could not be read again. Reload the page to see the latest images.');
});
test('dispose during a decision releases its capture', async () => {
const { c, h } = await loaded({ [`POST ${ITEMS}/one/decision`]: () => new Promise(() => {}) });
const applying = c.apply();
await settle();
assert.equal(c.state.phase, 'applying');
c.dispose();
assert.equal(await applying, false);
await settle();
assert.deepEqual(h.of('DELETE', CAPTURE).map(call => call.url), [CAPTURE + 'capture-1']);
await refreshOf(c);
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'a disposed controller reads nothing');
});
test('a change heard during the first read re-reads once after it, and a failed first read stays an error', async () => {
const { createReviewController } = await load();
let release;
let lists = 0;
const h = harness(routes({ [`GET ${ITEMS}`]: async () => {
lists += 1;
if (lists === 1) await new Promise(resolve => { release = resolve; });
return response(200, page(QUEUE));
} }));
const c = createReviewController({ api: h.api });
const first = c.load();
await settle();
refreshOf(c);
assert.equal(lists, 1, 'the first read is not interrupted');
release();
await first;
await settle();
assert.equal(lists, 2, 'one re-read after the first read');
assert.equal(c.state.status, 'ready');
const failing = harness(routes({ [`GET ${ITEMS}`]: () => fail(500, 'review-failed') }));
const broken = createReviewController({ api: failing.api });
const reading = broken.load();
refreshOf(broken);
await reading;
await settle();
assert.equal(broken.state.status, 'error');
assert.equal(failing.of('GET', `${ITEMS}?`).length, 1, 'Retry is the way back, not a re-read');
});
// F3 rulings: a live re-read keeps the loaded depth, and the stream's `ready` reads nothing when the first read is current.
/** A keyset pager over `source()`: pages of 40 with an index cursor, each answering `version()`. */
function pager(source, version = () => 0, { onPage } = {}) {
return call => {
const query = new URL(call.url, 'http://x').searchParams;
const cursor = query.get('cursor');
const all = source();
const from = cursor ? Number(cursor.slice(1)) : 0;
const answer = onPage?.(cursor);
if (answer) return answer;
const items = all.slice(from, from + 40);
const next = from + 40 < all.length ? `c${from + 40}` : null;
return response(200, page(items, { total: all.length, next_cursor: next, review_revision: version() }));
};
}
const many = (count, prefix = 'i') => Array.from({ length: count }, (_, index) => item(`${prefix}${String(index).padStart(3, '0')}`));
function syncOf(c, version) {
assert.equal(typeof c.sync, 'function', 'sync is not implemented');
return c.sync(version);
}
const version = revision => ({ library_epoch: EPOCH, review_revision: revision });
test('a refresh keeps the loaded depth: it re-reads as many items as were loaded', async () => {
let queue = many(100);
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => queue) });
await c.loadMore();
assert.equal(c.state.items.length, 80);
const before = h.of('GET', `${ITEMS}?`).length;
queue = [...queue, item('late')];
await refreshOf(c);
await settle();
assert.equal(c.state.items.length, 80, 'the Load more page is kept');
assert.deepEqual(c.state.items.map(entry => entry.id), queue.slice(0, 80).map(entry => entry.id));
assert.equal(c.state.total, 101);
assert.equal(c.state.nextCursor, 'c80', 'Load more continues after the kept depth');
assert.deepEqual(h.of('GET', `${ITEMS}?`).slice(before).map(call => call.url),
[`${ITEMS}?limit=40`, `${ITEMS}?limit=40&cursor=c40`]);
assert.equal(c.state.selectedId, 'i000');
});
test('a refresh after decisions elsewhere fills the loaded depth from what is left', async () => {
let queue = many(85);
const { c } = await loaded({ [`GET ${ITEMS}`]: pager(() => queue) });
await c.loadMore();
await c.loadMore();
assert.equal(c.state.items.length, 85);
queue = queue.filter((_, index) => index % 10 !== 5);
await refreshOf(c);
await settle();
assert.equal(c.state.items.length, queue.length);
assert.equal(c.state.nextCursor, null);
});
test('a deep refresh whose next page meets a restore reads again from the first page', async () => {
let restored = false;
let queue = many(60);
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => queue, () => 0, {
onPage: cursor => {
if (cursor && restored) { restored = false; return fail(409, 'storage-changed', 'The library was restored.'); }
return null;
},
}) });
await c.loadMore();
const before = h.of('GET', `${ITEMS}?`).length;
restored = true;
queue = many(60, 'r');
await refreshOf(c);
await settle();
assert.deepEqual(h.of('GET', `${ITEMS}?`).slice(before).map(call => call.url),
[`${ITEMS}?limit=40`, `${ITEMS}?limit=40&cursor=c40`, `${ITEMS}?limit=40`, `${ITEMS}?limit=40&cursor=c40`]);
assert.equal(c.state.items.length, 60);
assert.equal(c.state.items[0].id, 'r000');
assert.notEqual(c.state.warning, 'The review queue could not be read again. Reload the page to see the latest images.',
'a restore met mid-read is not a failed re-read');
assert.equal(c.state.status, 'ready');
});
test('ready naming the version the first read answered reads nothing more', async () => {
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => QUEUE, () => 4) });
await syncOf(c, version(4));
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'one list read per visit');
assert.equal(h.of('GET', `${ITEMS}/one`).length, 1);
});
test('ready naming a newer version, another epoch or no version re-reads once', async () => {
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => QUEUE, () => 4) });
await syncOf(c, version(5));
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 2);
await syncOf(c, { library_epoch: '22222222-2222-4222-8222-222222222222', review_revision: 4 });
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 3);
await syncOf(c, {});
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 4, 'an unknown version is read, as before versions existed');
});
test('ready heard during the first read re-reads only when that read is older', async () => {
const { createReviewController } = await load();
for (const [readRevision, readyRevision, reads] of [[4, 4, 1], [5, 4, 1], [4, 5, 2]]) {
let release;
let lists = 0;
const answer = pager(() => QUEUE, () => (lists === 1 ? readRevision : readyRevision));
const h = harness(routes({ [`GET ${ITEMS}`]: async call => {
lists += 1;
if (lists === 1) await new Promise(resolve => { release = resolve; });
return answer(call);
} }));
const c = createReviewController({ api: h.api });
const first = c.load();
await settle();
syncOf(c, version(readyRevision));
release();
await first;
await settle();
assert.equal(lists, reads, `first read r${readRevision}, ready r${readyRevision}`);
c.dispose();
}
});
test('a change heard during the first read still re-reads after a current ready', async () => {
const { createReviewController } = await load();
let release;
let lists = 0;
const h = harness(routes({ [`GET ${ITEMS}`]: async call => {
lists += 1;
if (lists === 1) await new Promise(resolve => { release = resolve; });
return pager(() => QUEUE, () => 4)(call);
} }));
const c = createReviewController({ api: h.api });
const first = c.load();
await settle();
syncOf(c, version(4));
refreshOf(c);
release();
await first;
await settle();
assert.equal(lists, 2);
});
// Final review round (I1, I2, M3–M5, M7).
test('I1: a change heard during a refused Apply is re-read once the phase is idle again', async () => {
let queue = [item('one')], releaseDecision;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(queue, { review_revision: queue.length })),
[`POST ${ITEMS}/one/decision`]: () => new Promise(resolve => {
releaseDecision = () => resolve(fail(409, 'association-conflict', 'conflict'));
}),
});
const applying = c.apply();
await settle();
assert.equal(c.state.phase, 'applying');
queue = [item('one'), item('two')]; // the monitor records an item while the decision is in flight
c.refresh();
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'no read while the decision is in flight');
releaseDecision();
await applying;
await settle();
assert.equal(c.state.phase, 'idle');
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
assert.equal(c.state.total, 2);
assert.equal(c.state.warning, 'conflict', 'the refusal stays the warning');
assert.equal(h.of('GET', `${ITEMS}?`).length, 2, 'exactly one owed re-read');
});
test('I1: a change heard during an Apply whose observe is refused is re-read too', async () => {
let queue = [item('one')], releaseObserve;
const { c } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(queue)),
[`POST ${OBSERVE}`]: () => new Promise(resolve => {
releaseObserve = () => resolve(fail(409, 'capture-capacity', 'busy'));
}),
});
const applying = c.apply();
await settle();
queue = [item('one'), item('two')];
c.refresh();
releaseObserve();
await applying;
await settle();
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
});
const forty = Array.from({ length: 41 }, (_, i) => item(`i${String(i).padStart(2, '0')}`));
function deepRoutes(state) {
return {
[`GET ${ITEMS}`]: call => {
const cursor = new URL(call.url, 'http://x').searchParams.get('cursor');
if (cursor) {
if (state.moreGate) return state.moreGate.then(() => state.more ?? response(200, page([forty[40]], { total: 41 })));
return state.more ?? response(200, page([forty[40]], { total: 41 }));
}
const body = response(200, page(state.first, { total: state.first.length + 1, review_revision: state.revision ?? 1, next_cursor: 'cur' }));
return state.gate ? state.gate.then(() => state.fail ?? body) : body;
},
[`GET ${ITEMS}/i00`]: () => response(200, detailOf(forty[0])),
};
}
test('I2: Load more during a re-read waits for it and continues from the re-read queue', async () => {
const state = { first: forty.slice(0, 40) };
const { c } = await loaded(deepRoutes(state));
// Another tab decides i05; the re-read is slow and the user clicks Load more meanwhile.
let open;
state.first = forty.slice(0, 40).filter(entry => entry.id !== 'i05');
state.gate = new Promise(resolve => { open = resolve; });
const refreshing = c.refresh();
await settle();
const more = c.loadMore();
await settle();
assert.equal(c.state.more, 'loading', 'the click shows as loading while the re-read runs');
open();
await Promise.all([refreshing, more]);
await settle();
const ids = c.state.items.map(entry => entry.id);
assert.ok(!ids.includes('i05'), 'the re-read was applied');
assert.equal(ids.at(-1), 'i40', 'Load more appended the next page after the re-read');
assert.equal(c.state.more, 'idle');
});
test('I2: a re-read started during Load more takes over the click and reads one page deeper', async () => {
const state = { first: forty.slice(0, 40) };
const { c, h } = await loaded(deepRoutes(state));
let openMore;
state.moreGate = new Promise(resolve => { openMore = resolve; });
const more = c.loadMore();
await settle();
state.moreGate = null;
state.first = forty.slice(0, 40).filter(entry => entry.id !== 'i05');
await c.refresh();
openMore();
await more;
await settle();
const ids = c.state.items.map(entry => entry.id);
assert.ok(!ids.includes('i05'));
assert.equal(ids.at(-1), 'i40', 'the aborted Load more page is part of the re-read');
assert.equal(c.state.more, 'idle');
assert.equal(h.of('GET', `${ITEMS}?`).length, 4, 'load, the aborted more, and the re-read of two pages');
});
test('M7: a failed re-read that took over Load more clears its loading state', async () => {
const state = { first: forty.slice(0, 40) };
const { c } = await loaded(deepRoutes(state));
state.moreGate = new Promise(() => {});
c.loadMore();
await settle();
assert.equal(c.state.more, 'loading');
state.moreGate = null;
state.gate = Promise.resolve();
state.fail = fail(500, 'review-failed');
await c.refresh();
await settle();
assert.equal(c.state.more, 'idle', 'Load more can be clicked again');
assert.equal(c.state.warning, COPY_REFRESH_FAILED);
});
const COPY_REFRESH_FAILED = 'The review queue could not be read again. Reload the page to see the latest images.';
test('M3: Apply fences on the epoch of the detail it decides', async () => {
const OTHER = '22222222-2222-4222-8222-222222222222';
const { c, h } = await loaded({ [`POST ${OBSERVE}`]: () => response(200, { ...observed(), library_epoch: OTHER }) });
assert.equal(await c.apply(), false);
await settle();
assert.equal(h.of('POST', `${ITEMS}/one/decision`).length, 0, 'a restored storage sends no decision');
assert.deepEqual(h.of('DELETE', CAPTURE).map(call => call.url), [CAPTURE + 'capture-1']);
assert.equal(h.of('GET', `${ITEMS}?`).length, 2, 'the queue is read again');
assert.equal(h.of('GET', `${ITEMS}/one`).length, 2, 'the open item is read again under the new storage');
assert.equal(c.state.phase, 'idle');
});
test('M3: the decision carries the detail epoch', async () => {
const { c, h } = await loaded();
assert.equal(await c.apply(), true);
assert.equal(h.of('POST', `${ITEMS}/one/decision`)[0].body.expected_epoch, EPOCH);
});
test('M3: a quiet re-read under another epoch reads the open item again', async () => {
const OTHER = '22222222-2222-4222-8222-222222222222';
let epoch = EPOCH;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(QUEUE, { library_epoch: epoch })),
[`GET ${ITEMS}/one`]: () => response(200, detailOf(QUEUE[0], { library_epoch: epoch })),
});
epoch = OTHER;
await c.refresh();
await settle();
assert.equal(h.of('GET', `${ITEMS}/one`).length, 2);
assert.equal(c.state.detail.library_epoch, OTHER);
});
test('M4: leaving during the observe releases the capture it then answers', async () => {
let answer;
const { c, h } = await loaded({ [`POST ${OBSERVE}`]: () => new Promise(resolve => { answer = resolve; }) });
const applying = c.apply();
await settle();
c.dispose();
answer(response(200, observed('late')));
assert.equal(await applying, false);
await settle();
assert.deepEqual(h.of('DELETE', CAPTURE).map(call => call.url), [CAPTURE + 'late']);
assert.equal(h.of('POST', `${ITEMS}/one/decision`).length, 0);
});
test('M5: an unanswered decision that landed finishes as recorded, not as decided elsewhere', async () => {
let queue = QUEUE;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(queue)),
[`POST ${ITEMS}/one/decision`]: () => Promise.reject(new TypeError('Failed to fetch')),
});
await c.apply();
assert.equal(c.state.canRetry, true);
queue = [QUEUE[1]]; // the server did record it: its change arrives
await c.refresh();
await settle();
assert.equal(c.state.phase, 'done');
assert.equal(c.state.toast, 'Association accepted.');
assert.doesNotMatch(c.state.warning, /decided elsewhere/);
assert.equal(h.of('DELETE', CAPTURE).length, 0, 'the server used the capture');
});
+11 -76
View File
@@ -4,7 +4,7 @@ const source = import('../../web/js/workspace/state.mjs');
test('workspace routes resolve only supported deep links', async () => {
const { resolveRoute } = await source;
for (const route of ['overview', 'images', 'library', 'tags', 'inspect', 'system', 'activity', 'review']) {
for (const route of ['overview', 'images', 'library', 'tags', 'inspect', 'system', 'activity']) {
assert.equal(resolveRoute(`#/${route}`).id, route);
}
for (const hash of ['', '#', '#/unknown', '#/images/other', '#/<script>', '#/activity/abc', `#/activity/${'A'.repeat(32)}`]) {
@@ -23,50 +23,8 @@ test('tags is a native route titled Tags; only unbuilt routes hand off to legacy
assert.equal(routeTitle(route), route.label);
}
assert.equal(routeTitle(resolveRoute('#/library/7')), 'Prompt library');
// System is native from S2 (#249 System): its undelivered sections hand off per section (D94), not the route.
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity', 'review', 'system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
});
test('system resolves its sections; #/overview stays the production hub', async () => {
const { resolveRoute, routeTitle, isHandoff, routes } = await source;
// `#/system` is the Overview section and keeps the sidebar's hash.
const system = resolveRoute('#/system');
assert.deepEqual([system.id, system.section, system.hash, routeTitle(system), isHandoff(system)],
['system', 'overview', '#/system', 'System', false]);
for (const section of ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics']) {
const route = resolveRoute(`#/system/${section}`);
assert.deepEqual([route.id, route.section, route.hash], ['system', section, `#/system/${section}`]);
}
// An unknown child is the Overview section with a replaced hash.
for (const hash of ['#/system/general', '#/system/', '#/system/__proto__', '#/system/backups/1', '#/system/<b>']) {
const route = resolveRoute(hash);
assert.deepEqual([route.id, route.section, route.hash], ['system', 'overview', '#/system/overview'], hash);
}
// D28: the production hub is untouched.
const hub = resolveRoute('#/overview');
assert.deepEqual([hub.id, hub.section], ['overview', undefined]);
// Q12: Overview's System card opens the native System Overview, with no legacy href or action.
const card = routes.find(route => route.id === 'system');
assert.equal(card.href, '#/system/overview');
assert.equal(card.native, true);
assert.equal(card.action, 'Open System');
});
test('the router renders again when only the System section changes', async () => {
const { mountRouter } = await source;
const win = new EventTarget();
win.location = { hash: '#/system' };
const replacements = [];
win.history = { state: null, replaceState(state, unused, hash) { replacements.push(hash); win.location.hash = hash; } };
const seen = [];
const dispose = mountRouter(win, route => seen.push([route.id, route.section]));
for (const hash of ['#/system/appearance', '#/system/appearance', '#/system/bogus', '#/system/overview', '#/images']) {
win.location.hash = hash;
win.dispatchEvent(new Event('hashchange'));
}
assert.deepEqual(seen, [['system', 'overview'], ['system', 'appearance'], ['system', 'overview'], ['images', undefined]]);
assert.deepEqual(replacements, ['#/system/overview']);
dispose();
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
for (const id of ['system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), true, id);
});
test('inspect is the native Inspect a file page with the concept description', async () => {
@@ -157,10 +115,9 @@ test('positive library deep links preserve the selected identity without duplica
test('nav counts reject negative, non-integer and missing totals', async () => {
const { summarizeNavCounts } = await source;
const counts = summarizeNavCounts({images: {total: 24}, favorites: {total: 3}, library: {total: 0}, tags: {total: 7},
review: {total: 3}});
const counts = summarizeNavCounts({images: {total: 24}, favorites: {total: 3}, library: {total: 0}, tags: {total: 7}});
assert.ok(counts instanceof Map);
assert.deepEqual([...counts], [['images', 24], ['favorites', 3], ['library', 0], ['tags', 7], ['review', 3]]);
assert.deepEqual([...counts], [['images', 24], ['favorites', 3], ['library', 0], ['tags', 7]]);
assert.equal(counts.has('inspect'), false, 'Inspect a file has no count');
for (const invalid of [undefined, null, {}, {total: -1}, {total: 1.5}, {total: '3'}, {total: NaN},
{total: Infinity}, {total: Number.MAX_SAFE_INTEGER + 1}, {total: null}, [], 'x']) {
@@ -170,8 +127,7 @@ test('nav counts reject negative, non-integer and missing totals', async () => {
assert.equal(result.get('library'), 2);
assert.equal(result.get('tags'), null);
}
assert.deepEqual([...summarizeNavCounts()], [['images', null], ['favorites', null], ['library', null], ['tags', null],
['review', null]]);
assert.deepEqual([...summarizeNavCounts()], [['images', null], ['favorites', null], ['library', null], ['tags', null]]);
});
test('nav counts settle independently so a slow images read never delays the others', async () => {
@@ -187,14 +143,14 @@ test('nav counts settle independently so a slow images read never delays the oth
const settled = [];
const done = loadNavCounts(api, new AbortController().signal, (id, count) => settled.push([id, count]));
await new Promise(resolve => setTimeout(resolve, 0));
assert.deepEqual(settled.sort(), [['favorites', 2], ['library', 3], ['review', null], ['tags', null]]);
assert.deepEqual(settled.sort(), [['favorites', 2], ['library', 3], ['tags', null]]);
releaseImages({success: true, total: 40});
const counts = await done;
assert.deepEqual(settled.at(-1), ['images', 40]);
assert.deepEqual([...counts], [['images', 40], ['favorites', 2], ['library', 3], ['tags', null], ['review', null]]);
assert.deepEqual([...counts], [['images', 40], ['favorites', 2], ['library', 3], ['tags', null]]);
});
test('nav counts issue five bounded requests and drop only the failed count', async () => {
test('nav counts issue four bounded requests and drop only the failed count', async () => {
const { loadNavCounts } = await source;
const calls = [];
const signal = new AbortController().signal;
@@ -202,15 +158,13 @@ test('nav counts issue five bounded requests and drop only the failed count', as
calls.push([url, options.signal]);
if (url.startsWith('/prompt_manager/library')) throw new Error('Unavailable');
if (url.startsWith('/prompt_manager/vocabulary/tags')) return {success: true, total: 5};
if (url.startsWith('/prompt_manager/review/items')) return {success: true, total: 2};
return {success: true, total: 12};
}};
const counts = await loadNavCounts(api, signal);
assert.deepEqual([...counts], [['images', 12], ['favorites', 12], ['library', null], ['tags', 5], ['review', 2]]);
assert.deepEqual([...counts], [['images', 12], ['favorites', 12], ['library', null], ['tags', 5]]);
const urls = calls.map(([url]) => new URL(url, 'http://host'));
assert.deepEqual(urls.map(url => url.pathname),
['/prompt_manager/images/output', '/prompt_manager/images/output', '/prompt_manager/library', '/prompt_manager/vocabulary/tags',
'/prompt_manager/review/items']);
['/prompt_manager/images/output', '/prompt_manager/images/output', '/prompt_manager/library', '/prompt_manager/vocabulary/tags']);
for (const url of urls) assert.equal(url.searchParams.get('limit'), '1');
assert.equal(urls[0].searchParams.get('mode'), 'workspace');
assert.equal(urls[0].searchParams.has('favorites'), false);
@@ -292,22 +246,3 @@ test('saved-view links resolve to their id and re-render only when the id change
assert.deepEqual(seen,[['saved',3],['saved',4],['images',undefined]]);
dispose();
});
test('Needs review is a native SAVED VIEWS route titled Review, with its own count', async () => {
const { resolveRoute, isHandoff, routeTitle, routes, loadNavCounts, REVIEW_WRITE_COUNTS, NAV_COUNT_ROUTES } = await source;
const route = resolveRoute('#/review');
assert.equal(route.id, 'review');
assert.equal(route.label, 'Needs review');
assert.equal(routeTitle(route), 'Review');
assert.equal(isHandoff(route), false);
// No href: Overview lists no legacy card for it.
assert.equal(routes.find(item => item.id === 'review').href, undefined);
assert.ok(NAV_COUNT_ROUTES.includes('review'));
assert.deepEqual([...REVIEW_WRITE_COUNTS], ['review']);
assert.ok(Object.isFrozen(REVIEW_WRITE_COUNTS));
const calls = [];
const api = {get: async url => { calls.push(url); return {success: true, total: 4}; }};
const counts = await loadNavCounts(api, new AbortController().signal, () => {}, REVIEW_WRITE_COUNTS);
assert.deepEqual(calls, ['/prompt_manager/review/items?limit=1']);
assert.deepEqual([...counts], [['review', 4]]);
});
@@ -1,188 +0,0 @@
// System › Appearance (#249 System S3 Task 8): the concept's "Browsing & display" rows (app.js 676–678) on the drafts
// of the one Save row, their stored values (Q3 mappings), the host timeout's 0–300 whole seconds and the D97 rows.
const test = require('node:test');
const assert = require('node:assert/strict');
const gallerySettings = require('../../web/js/gallery-settings.js');
const appearanceModule = import('../../web/js/workspace/system-appearance.mjs');
const stateModule = import('../../web/js/workspace/system-state.mjs');
const coreModule = import('../../web/js/shared/toolkit/core.js');
globalThis.GallerySettings = gallerySettings;
/** A minimal DOM whose elements count their live listeners (a listener's signal aborts it, as in a browser). */
function fakeDOM() {
const previous = global.document;
const elements = [];
class Element {
constructor(tag) {
this.tagName = tag.toUpperCase();
this.children = [];
this.attributes = {};
this.listeners = [];
this.dataset = {};
this.className = '';
this.id = '';
this.value = '';
this._text = '';
elements.push(this);
}
get textContent() { return this._text + this.children.map(child => child.textContent).join(''); }
set textContent(text) { this.children = []; this._text = String(text); }
append(...nodes) {
for (const node of nodes) {
const child = typeof node === 'string' ? Object.assign(new Element('#text'), { _text: node }) : node;
child.parentNode = this;
this.children.push(child);
}
}
remove() { if (this.parentNode) this.parentNode.children = this.parentNode.children.filter(child => child !== this); this.parentNode = null; }
replaceChildren(...nodes) { this.children = []; this._text = ''; this.append(...nodes); }
setAttribute(name, value) { this.attributes[name] = String(value); }
getAttribute(name) { return Object.hasOwn(this.attributes, name) ? this.attributes[name] : null; }
removeAttribute(name) { delete this.attributes[name]; }
addEventListener(type, callback, options = {}) {
if (options.signal?.aborted) return;
const entry = { type, callback };
this.listeners.push(entry);
options.signal?.addEventListener('abort', () => { this.listeners = this.listeners.filter(item => item !== entry); });
}
removeEventListener(type, callback) { this.listeners = this.listeners.filter(item => item.type !== type || item.callback !== callback); }
dispatch(type) { for (const { callback } of this.listeners.filter(item => item.type === type)) callback({ type, target: this }); }
click() { this.dispatch('click'); }
}
global.document = {
createElement: tag => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const walk = (node, found = []) => { found.push(node); node.children.forEach(child => walk(child, found)); return found; };
return {
listenerCount: () => elements.reduce((sum, element) => sum + element.listeners.length, 0),
find: (root, predicate) => walk(root).find(predicate),
tags: node => node.children.map(child => child.tagName.toLowerCase()),
restore() { global.document = previous; },
};
}
const BROWSER = { workspaceLayout: 'grid', workspaceDensity: 'comfortable', autoLoadMetadata: true, cacheMetadata: true,
imageQuality: 'medium', videoAutoplay: false, videoMute: true, videoLoop: true, showFilePaths: true, showImageInfo: true,
sidebarCollapsedByDefault: false };
async function mounted({ host = { webui_display_mode: 'newtab', result_timeout: 5 }, browser = BROWSER, hostFails = false } = {}) {
const { mountAppearance } = await appearanceModule;
const { createSystemDrafts } = await stateModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
let hostValues = null;
const drafts = createSystemDrafts({
readBrowser: () => ({ ...browser }), readRuntime: () => ({}),
readHost: async () => { if (hostFails) throw new Error('offline'); return { values: hostValues, revision: 'sha256:one' }; },
writeBrowser: async () => ({ persisted: true }), writeHost: async () => ({ success: true, persisted: true }), writeRuntime: async () => ({}),
});
await drafts.load('browser');
const scope = createScope();
const [surface] = mountAppearance(null, { scope, drafts });
const byId = id => dom.find(surface, node => node.dataset?.setting === id);
return { dom, drafts, scope, surface, byId, loadHost: async () => { hostValues = host; await drafts.load('host'); },
fixHost: () => { hostFails = false; } };
}
test('the surface is the concept section with the concept rows in order, then the D97 rows', async () => {
const { APPEARANCE_ROWS } = await appearanceModule;
const { dom, surface, scope } = await mounted();
try {
assert.equal(surface.className, 'surface');
assert.equal(surface.children[0].textContent, 'Browsing & display');
assert.equal(surface.children[1].textContent, 'One set of defaults; toolbar changes apply to the current view.');
const rows = surface.children.slice(2);
const titles = rows.map(row => row.children[0].children[0].textContent);
assert.deepEqual(titles, ['Default layout', 'Thumbnail size', 'Continuous browsing', 'Load metadata on selection',
'Cache metadata in this browser', 'Image loading quality', 'Autoplay videos', 'Mute videos by default', 'Loop videos',
'Show file paths', 'Open from ComfyUI', 'Node search auto-hide (seconds)', 'Show image details on cards',
'Collapse viewer details by default', 'Reset display defaults']);
const continuous = rows[2];
assert.deepEqual(dom.tags(continuous.children[0]), ['h3', 'p']);
assert.equal(continuous.children[1].className, 'chip blue');
assert.equal(continuous.children[1].textContent, 'Infinite scroll');
assert.equal(rows.at(-1).className, 'operation');
assert.equal(APPEARANCE_ROWS.filter(Array.isArray).length, 13);
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
});
test('added rows reach their consumers: each control edits its stored key with the Q3 values', async () => {
const { dom, drafts, scope, byId, loadHost } = await mounted();
try {
// Host rows stay disabled until the server settings answered.
assert.equal(byId('open-mode').disabled, true);
assert.equal(byId('timeout').disabled, true);
await loadHost();
assert.equal(byId('open-mode').disabled, false);
assert.equal(byId('timeout').value, '5');
assert.deepEqual(byId('image-quality').children.map(option => [option.value, option.textContent]),
[['medium', 'Optimized'], ['high', 'Original'], ['low', 'Fast']]);
assert.deepEqual(byId('open-mode').children.map(option => [option.value, option.textContent]),
[['newtab', 'New tab'], ['popup', 'Popup window']]);
const edits = [['default-view', 'masonry'], ['density', 'large'], ['image-quality', 'low'], ['open-mode', 'popup']];
for (const [id, value] of edits) { byId(id).value = value; byId(id).dispatch('change'); }
for (const id of ['metadata-auto', 'metadata-cache', 'video-muted', 'video-loop', 'paths', 'card-details']) {
byId(id).checked = false; byId(id).dispatch('change');
}
for (const id of ['video-auto', 'viewer-details']) { byId(id).checked = true; byId(id).dispatch('change'); }
assert.deepEqual(drafts.changes('browser'), { workspaceLayout: 'masonry', workspaceDensity: 'large', imageQuality: 'low',
autoLoadMetadata: false, cacheMetadata: false, videoMute: false, videoLoop: false, showFilePaths: false, showImageInfo: false,
videoAutoplay: true, sidebarCollapsedByDefault: true });
assert.deepEqual(drafts.changes('host'), { webui_display_mode: 'popup' });
} finally { scope.dispose(); dom.restore(); }
});
test('the auto-hide timeout takes whole seconds from 0 to 300 only', async () => {
const { resultTimeout } = await appearanceModule;
assert.deepEqual(['0', '30', ' 300 ', '301', '-1', '2.5', '', 'abc', '1e2'].map(resultTimeout), [0, 30, 300, null, null, null, null, null, null]);
const { dom, drafts, scope, byId, loadHost } = await mounted();
try {
await loadHost();
const timeout = byId('timeout');
timeout.value = '45'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), { result_timeout: 45 });
timeout.value = '301'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), {});
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
timeout.value = '0'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), { result_timeout: 0 });
assert.deepEqual([...drafts.failed('host')], []);
} finally { scope.dispose(); dom.restore(); }
});
test('reset restores defaults without touching other keys', async () => {
const { DISPLAY_KEYS } = await appearanceModule;
const changed = { ...BROWSER, workspaceLayout: 'list', videoMute: false, showImageInfo: false, sidebarCollapsedByDefault: true };
const { dom, drafts, scope, surface, loadHost } = await mounted({ browser: changed });
try {
await loadHost();
drafts.set('host', 'result_timeout', 9);
const reset = dom.find(surface.children.at(-1), node => node.tagName === 'BUTTON');
assert.equal(reset.textContent, 'Reset');
reset.click();
assert.deepEqual(drafts.changes('browser'), { workspaceLayout: 'grid', videoMute: true, showImageInfo: true, sidebarCollapsedByDefault: false });
// The host fields and every key outside the display defaults are left as they are.
assert.deepEqual(drafts.changes('host'), { result_timeout: 9 });
assert.deepEqual([...DISPLAY_KEYS].sort(), Object.keys(BROWSER).sort());
} finally { scope.dispose(); dom.restore(); }
});
test('host rows that could not be read stay disabled and say so; a later read clears it', async () => {
const { HOST_UNAVAILABLE } = await appearanceModule;
const { dom, drafts, scope, byId, surface, fixHost } = await mounted({ hostFails: true });
const notes = () => surface.children.slice(2).flatMap(row => row.children[0].children.slice(2).map(note => [row.children[0].children[0].textContent, note.className, note.textContent]));
try {
assert.deepEqual(notes(), []);
assert.equal(await drafts.reloadHost(), false);
assert.deepEqual(notes(), [['Open from ComfyUI', 'small', HOST_UNAVAILABLE], ['Node search auto-hide (seconds)', 'small', HOST_UNAVAILABLE]]);
assert.equal(byId('timeout').disabled, true);
// Repaints keep one line per row.
drafts.set('browser', 'videoMute', false);
assert.equal(notes().length, 2);
fixHost();
assert.equal(await drafts.reloadHost(), true);
assert.deepEqual(notes(), []);
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
});
-214
View File
@@ -1,214 +0,0 @@
// System controls (#249 System S2 Task 5): the concept's `setting()` (app.js 624–634, its toggles hydrated into Toolkit
// switches by `hydrateSwitches` 708–717), `operation()` (645–654) and `sectionNav()` (306–308) as DOM builders whose
// listeners live on the caller's scope.
const test = require('node:test');
const assert = require('node:assert/strict');
const controlsModule = import('../../web/js/workspace/system-controls.mjs');
const coreModule = import('../../web/js/shared/toolkit/core.js');
/** A minimal DOM whose elements count their live listeners (a listener's signal aborts it, as in a browser). */
function fakeDOM() {
const previous = global.document;
const elements = [];
class Element {
constructor(tag) {
this.tagName = tag.toUpperCase();
this.children = [];
this.attributes = {};
this.listeners = [];
this.dataset = {};
this.className = '';
this.id = '';
this.value = '';
this._text = '';
elements.push(this);
}
get textContent() { return this._text + this.children.map(child => child.textContent).join(''); }
set textContent(text) { this.children = []; this._text = String(text); }
append(...nodes) {
for (const node of nodes) {
const child = typeof node === 'string' ? Object.assign(new Element('#text'), { _text: node }) : node;
child.parentNode = this;
this.children.push(child);
}
}
replaceChildren(...nodes) { this.children = []; this._text = ''; this.append(...nodes); }
setAttribute(name, value) { this.attributes[name] = String(value); }
getAttribute(name) { return Object.hasOwn(this.attributes, name) ? this.attributes[name] : null; }
removeAttribute(name) { delete this.attributes[name]; }
addEventListener(type, callback, options = {}) {
if (options.signal?.aborted) return;
const entry = { type, callback };
this.listeners.push(entry);
options.signal?.addEventListener('abort', () => { this.listeners = this.listeners.filter(item => item !== entry); });
}
removeEventListener(type, callback) { this.listeners = this.listeners.filter(item => item.type !== type || item.callback !== callback); }
dispatch(type) { for (const { callback } of this.listeners.filter(item => item.type === type)) callback({ type, target: this }); }
click() { this.dispatch('click'); }
}
global.document = {
createElement: tag => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const walk = (node, found = []) => { found.push(node); node.children.forEach(child => walk(child, found)); return found; };
return {
listenerCount: () => elements.reduce((sum, element) => sum + element.listeners.length, 0),
find: (root, predicate) => walk(root).find(predicate),
tags: node => node.children.map(child => child.tagName.toLowerCase()),
restore() { global.document = previous; },
};
}
test('settingRow is the concept row: div > label[for] + p, then the control', async () => {
const { settingRow } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const changes = [];
const row = settingRow({ id: 'timeout', label: 'Node search auto-hide (seconds)',
description: 'Node search behavior only. Use 0 to keep results visible.',
control: { type: 'number', value: 30, min: 0, max: 300 }, scope, onChange: value => changes.push(value) });
assert.equal(row.className, 'setting-row');
assert.deepEqual(dom.tags(row), ['div', 'input']);
const [text, input] = row.children;
assert.deepEqual(dom.tags(text), ['label', 'p']);
assert.equal(text.children[0].textContent, 'Node search auto-hide (seconds)');
assert.equal(text.children[0].htmlFor, 'timeout');
assert.equal(text.children[1].textContent, 'Node search behavior only. Use 0 to keep results visible.');
assert.deepEqual([input.id, input.type, input.value, input.min, input.max], ['timeout', 'number', '30', '0', '300']);
assert.equal(input.dataset.setting, 'timeout');
input.value = '45';
input.dispatch('change');
assert.deepEqual(changes, [45]);
const select = settingRow({ id: 'open-mode', label: 'Open from ComfyUI', description: 'Keep the node pack’s popup / new-tab preference.',
control: { type: 'select', options: [['newtab', 'New tab'], ['popup', 'Popup window']], value: 'popup' }, scope,
onChange: value => changes.push(value) }).children[1];
assert.equal(select.tagName, 'SELECT');
assert.deepEqual(select.children.map(option => [option.value, option.textContent]), [['newtab', 'New tab'], ['popup', 'Popup window']]);
assert.equal(select.value, 'popup');
select.value = 'newtab';
select.dispatch('change');
assert.deepEqual(changes, [45, 'newtab']);
// The concept's range is 1–16 unless the row says otherwise (D113: the CPU count).
const range = settingRow({ id: 'workers', label: 'Worker threads', description: 'Balance background scans.',
control: { type: 'range', value: 4 } }).children[1];
assert.deepEqual([range.type, range.min, range.max, range.value], ['range', '1', '16', '4']);
const password = settingRow({ id: 'civitai-key', label: 'CivitAI API key', description: 'Credentials stay on the server.',
control: { type: 'password', value: '' } }).children[1];
assert.equal(password.type, 'password');
// A ready-made control is placed as given and takes the row id.
const custom = document.createElement('textarea');
const customRow = settingRow({ id: 'custom', label: 'Custom', description: 'Any control.', control: custom });
assert.equal(customRow.children[1], custom);
assert.equal(custom.id, 'custom');
assert.throws(() => settingRow({ id: '', label: 'x', description: 'y', control: { type: 'number' } }), TypeError);
assert.throws(() => settingRow({ id: 'x', label: 'x', description: 'y', control: { type: 'colour' } }), TypeError);
} finally {
scope.dispose();
dom.restore();
}
});
test('a toggle row holds the Toolkit switch, labelled by the row label', async () => {
const { settingRow } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const changes = [];
const row = settingRow({ id: 'video-auto', label: 'Autoplay videos', description: 'Videos stay within the same gallery.',
control: { type: 'toggle', checked: false }, scope, onChange: value => changes.push(value) });
const field = row.children[1];
// createSwitch('', '', id): `label.ca-switch` > input[type=checkbox][role=switch] + an empty label text.
assert.equal(field.tagName, 'LABEL');
assert.equal(field.className, 'ca-switch');
const input = field.children[0];
assert.deepEqual([input.type, input.role, input.id, input.checked], ['checkbox', 'switch', 'video-auto', false]);
assert.equal(row.children[0].children[0].htmlFor, input.id);
assert.equal(input.dataset.setting, 'video-auto');
input.checked = true;
input.dispatch('change');
assert.deepEqual(changes, [true]);
// The concept's toggles are on unless configured off (`val !== false`).
const on = settingRow({ id: 'video-loop', label: 'Loop videos', description: 'Repeat a clip.', control: { type: 'toggle' } });
assert.equal(on.children[1].children[0].checked, true);
} finally {
scope.dispose();
dom.restore();
}
});
test('operationRow is the concept operation with a danger class only when flagged', async () => {
const { operationRow } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const clicks = [];
const row = operationRow({ title: 'Missing-file links',
description: 'Remove stale database links only. Files on disk are never deleted by this operation.',
label: 'Review plan', danger: true, scope, onClick: button => clicks.push(button) });
assert.equal(row.className, 'operation');
assert.deepEqual(dom.tags(row), ['div', 'div']);
assert.deepEqual(dom.tags(row.children[0]), ['h3', 'p']);
assert.equal(row.children[0].children[0].textContent, 'Missing-file links');
assert.equal(row.children[1].className, 'row-actions');
const button = row.children[1].children[0];
assert.deepEqual([button.tagName, button.type, button.textContent, button.className], ['BUTTON', 'button', 'Review plan', 'danger']);
button.click();
assert.deepEqual(clicks, [button]);
const plain = operationRow({ title: 'Statistics', description: 'Usage by prompt, category, tag and date.', label: 'Explore',
scope, onClick() {} });
assert.equal(plain.children[1].children[0].className, '');
assert.throws(() => operationRow({ title: 'x', description: 'y', label: 'z', onClick() {} }), TypeError);
} finally {
scope.dispose();
dom.restore();
}
});
test('sectionNav marks the active section and reports the chosen one', async () => {
const { sectionNav } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const chosen = [];
const nav = sectionNav({ label: 'System sections', sections: [['overview', 'Overview'], ['appearance', 'Appearance']],
active: 'appearance', scope, onSelect: id => chosen.push(id) });
assert.equal(nav.className, 'section-nav');
assert.equal(nav.getAttribute('role'), 'group');
assert.equal(nav.getAttribute('aria-label'), 'System sections');
assert.deepEqual(nav.children.map(button => [button.textContent, button.getAttribute('aria-pressed'), button.dataset.value]),
[['Overview', 'false', 'overview'], ['Appearance', 'true', 'appearance']]);
nav.children[0].click();
assert.deepEqual(chosen, ['overview']);
} finally {
scope.dispose();
dom.restore();
}
});
test('disposing the scope leaves no listener behind', async () => {
const { settingRow, operationRow, sectionNav } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
settingRow({ id: 'a', label: 'A', description: 'a', control: { type: 'toggle' }, scope, onChange() {} });
settingRow({ id: 'b', label: 'B', description: 'b', control: { type: 'select', options: ['Grid', 'List'] }, scope, onChange() {} });
operationRow({ title: 'C', description: 'c', label: 'Go', scope, onClick() {} });
sectionNav({ label: 'System sections', sections: [['overview', 'Overview']], active: 'overview', scope, onSelect() {} });
assert.equal(dom.listenerCount() > 0, true);
scope.dispose();
assert.equal(dom.listenerCount(), 0);
} finally {
scope.dispose();
dom.restore();
}
});
-255
View File
@@ -1,255 +0,0 @@
// System › Overview (#249 System S2 Task 6): the concept's Overview copy on real values (D93), the Health report rows
// (D95), the Statistics tables (D96) and the health check run that waits for its read job.
const test = require('node:test');
const assert = require('node:assert/strict');
const overviewModule = import('../../web/js/workspace/system-overview.mjs');
const OVERVIEW = Object.freeze({
library: { positive_prompts: 1234, negative_variants: 7 },
images: { indexed: 24, rated: 20 },
health: null,
backup: null,
needs_review: { pending: null },
sources: { configured: 2, effective: 2, auto_detected: false },
});
test('metrics are the server counts and Not checked before any health check', async () => {
const { metricValues } = await overviewModule;
assert.deepEqual(metricValues(OVERVIEW), [
[(1234).toLocaleString(), 'Positive prompts'], ['7', 'Negative variants'], ['24', 'Images'],
['Not checked', 'Database integrity'],
]);
});
test('metric 4 is the last health result', async () => {
const { integrityText } = await overviewModule;
assert.equal(integrityText(null), 'Not checked');
assert.equal(integrityText({ status: 'passed', issues: 0 }), 'Passed');
assert.equal(integrityText({ status: 'issues', issues: 1 }), '1 issue');
assert.equal(integrityText({ status: 'issues', issues: 3 }), '3 issues');
assert.equal(integrityText({ status: 'incomplete', issues: 0 }), 'Incomplete');
// Issues found beside a check that could not run (Needs review, Q6) are still named.
assert.equal(integrityText({ status: 'incomplete', issues: 2 }), '2 issues');
});
test('library health rows never invent a review count, a backup or thumbnails', async () => {
const { ambiguousTitle, backupText } = await overviewModule;
assert.equal(ambiguousTitle({ pending: null }), 'Ambiguous image links');
assert.equal(ambiguousTitle(undefined), 'Ambiguous image links');
assert.equal(ambiguousTitle({ pending: 1 }), '1 ambiguous image link');
assert.equal(ambiguousTitle({ pending: 3 }), '3 ambiguous image links');
assert.equal(backupText(null), 'No retained backup record.');
assert.equal(backupText({ verified_at: 'not a time' }), 'No retained backup record.');
assert.match(backupText({ verified_at: '2026-10-22T09:30:00Z' }), /^Last verified backup .+\.$/);
});
test('the thumbnail row is its own read of the thumbnail status', async () => {
const { thumbnailTitle, THUMBNAILS_LOADING } = await overviewModule;
// Until `/images/thumbnails/status` answers the row says it is checking; a failed read is unavailable, never 0.
assert.equal(THUMBNAILS_LOADING, 'Checking thumbnails…');
assert.equal(thumbnailTitle(undefined), THUMBNAILS_LOADING);
assert.equal(thumbnailTitle({ pending: 0 }), 'No missing thumbnails');
assert.equal(thumbnailTitle({ pending: 1 }), '1 missing thumbnail');
assert.equal(thumbnailTitle({ pending: 12 }), '12 missing thumbnails');
assert.equal(thumbnailTitle(null), 'Thumbnail status unavailable');
assert.equal(thumbnailTitle({ pending: 'x' }), 'Thumbnail status unavailable');
// With no image source there is nothing to count.
assert.equal(thumbnailTitle({ pending: 0 }, { effective: 0 }), 'Thumbnail status unavailable');
assert.equal(thumbnailTitle({ pending: 2 }, { effective: 1 }), '2 missing thumbnails');
});
test('an integrity count SQLite stopped at reads as a lower bound', async () => {
const { checkChip } = await overviewModule;
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 100, capped: true }), '100+ problems found');
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 37, capped: false }), '37 problems found');
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 1 }), '1 problem found');
});
test('the sources row tells configured, available and detected sources apart', async () => {
const { sourcesCopy } = await overviewModule;
const shared = 'Monitoring and enrichment share one source list.';
assert.deepEqual(sourcesCopy({ configured: 2, effective: 2, auto_detected: false }), ['2 output sources', shared]);
assert.deepEqual(sourcesCopy({ configured: 2, effective: 1, auto_detected: false }),
['1 output source', `1 of 2 configured sources available. ${shared}`]);
assert.deepEqual(sourcesCopy({ configured: 0, effective: 1, auto_detected: true }),
['1 output source', `None configured is available; ComfyUI's output folder is used. ${shared}`]);
});
const REPORT = Object.freeze({
checked_at: '2026-10-23T08:00:00Z',
status: 'issues',
checks: [
{ id: 'integrity', label: 'Integrity check', status: 'passed', count: 0 },
{ id: 'foreign_keys', label: 'Foreign keys & pair references', status: 'issues', count: 2 },
{ id: 'canonical_hashes', label: 'Canonical hashes', status: 'issues', count: 1 },
{ id: 'pair_uniqueness', label: 'Pair uniqueness', status: 'failed', count: null },
{ id: 'missing_files', label: 'Missing files', status: 'issues', count: 2,
sources: [{ name: 'output', status: 'checked', count: 2 }] },
{ id: 'ambiguous_metadata', label: 'Ambiguous metadata', status: 'unavailable', count: null },
{ id: 'consistency', label: 'Consistency', status: 'issues', count: null },
],
});
test('health rows give each check a chip from its real status, in the server order', async () => {
const { healthRows } = await overviewModule;
assert.deepEqual(healthRows(REPORT), [
['Integrity check', 'Passed'],
['Foreign keys & pair references', '2 broken references'],
['Canonical hashes', '1 hash does not match its text'],
['Pair uniqueness', 'Could not check'],
['Missing files', '2 links need review'],
['Ambiguous metadata', 'Not available yet'],
['Consistency', 'Issues found'],
]);
});
test('missing files is reported per image source when there are several', async () => {
const { healthRows } = await overviewModule;
const report = { ...REPORT, checks: [{ id: 'missing_files', label: 'Missing files', status: 'issues', count: 1, partial: true,
sources: [{ name: 'output', status: 'checked', count: 1 }, { name: 'input', status: 'checked', count: 0 },
{ name: 'archive', status: 'unavailable', count: null }] }] };
assert.deepEqual(healthRows(report), [
['Missing files · output', '1 link needs review'],
['Missing files · input', 'Passed'],
['Missing files · archive', 'Not available'],
]);
const unavailable = { ...REPORT, checks: [{ id: 'missing_files', label: 'Missing files', status: 'unavailable', count: null, sources: [] }] };
assert.deepEqual(healthRows(unavailable), [['Missing files', 'Not available']]);
});
test('the report summary says when it was checked and whether it is complete', async () => {
const { healthSummary } = await overviewModule;
const when = new Date(REPORT.checked_at).toLocaleString();
assert.equal(healthSummary({ ...REPORT, status: 'passed' }), `Checked ${when}. Every check passed.`);
assert.equal(healthSummary({ ...REPORT, status: 'incomplete' }), `Checked ${when}. Some checks could not run, so this report is incomplete.`);
assert.equal(healthSummary({ ...REPORT, checked_at: 'x' }), 'Nothing was changed; review the checks with issues.');
});
test('statistics rows are the real runs and ratings, then the library totals', async () => {
const { categoryRows, totalsRows } = await overviewModule;
const statistics = {
by_category: [{ category: 'Landscapes', runs: 1200, average_rating: 4.5 },
{ category: null, runs: 3, average_rating: null }],
totals: { total_prompts: 12, total_tags: 15, total_images: 24 },
rated_images: 20,
};
assert.deepEqual(categoryRows(statistics), [['Landscapes', (1200).toLocaleString(), '4.5 / 5'], ['Uncategorized', '3', 'Not rated']]);
assert.deepEqual(totalsRows(statistics), [['Prompts', '12'], ['Tags', '15'], ['Images', '24'], ['Rated images', '20']]);
});
/** A fake api answering each `get`/`post` from a queue per URL, recording the calls. */
function fakeApi(answers) {
const calls = [];
const answer = (method, url) => {
calls.push(`${method} ${url}`);
const queue = answers[`${method} ${url}`];
if (!queue?.length) throw new Error(`unexpected ${method} ${url}`);
const next = queue.shift();
return next instanceof Error ? Promise.reject(next) : Promise.resolve(next);
};
return { calls, get: url => answer('GET', url), post: url => answer('POST', url) };
}
const JOB = '0123456789abcdef0123456789abcdef';
const job = (state, id = JOB, kind = 'health') => ({ id, kind, state });
/**
* A fake Activity store holding `jobs`: `track(job)` lists a started job, `load()` applies the next of `reads` (the
* server's job lists) and `push(job)` is a stream update; each notifies as the real store does. It never polls.
*/
function fakeActivity({ jobs = [], reads = [] } = {}) {
const listeners = new Set();
const emit = () => { for (const listener of [...listeners]) listener(); };
const store = {
loads: 0,
find: id => jobs.find(item => item.id === id),
running: () => jobs.filter(item => item.state === 'running'),
subscribe(listener) { listeners.add(listener); return () => listeners.delete(listener); },
listeners: () => listeners.size,
async load() {
store.loads += 1;
if (reads.length) jobs = reads.shift();
emit();
},
track(item) {
if (!item?.id) throw new TypeError('Invalid job');
store.push(item);
return item;
},
push(item) {
jobs = [...jobs.filter(other => other.id !== item.id), item];
emit();
},
};
return store;
}
test('a health check tracks its job in the Activity store, then reads the report', async () => {
const { runHealthCheck } = await overviewModule;
const api = fakeApi({
'POST /prompt_manager/system/health': [{ job: job('running') }],
'GET /prompt_manager/system/health': [{ report: REPORT }],
});
const activity = fakeActivity();
const checking = runHealthCheck(api, { activity });
await new Promise(resolve => setImmediate(resolve));
// The store follows the started job's event stream; nothing reads or polls the job here.
assert.equal(activity.loads, 0);
assert.equal(activity.find(JOB).state, 'running');
assert.deepEqual(api.calls, ['POST /prompt_manager/system/health']);
activity.push(job('finished'));
assert.equal(await checking, REPORT);
assert.deepEqual(api.calls, ['POST /prompt_manager/system/health', 'GET /prompt_manager/system/health']);
assert.equal(activity.listeners(), 0);
});
test('a check already running elsewhere is waited for, not refused', async () => {
const { runHealthCheck } = await overviewModule;
const busy = Object.assign(new Error('busy'), { status: 409, code: 'busy' });
const api = fakeApi({
'POST /prompt_manager/system/health': [busy],
'GET /prompt_manager/system/health': [{ report: REPORT }],
});
const activity = fakeActivity({ reads: [[job('running', 'f'.repeat(32), 'scan'), job('running')]] });
const checking = runHealthCheck(api, { activity });
await new Promise(resolve => setImmediate(resolve));
assert.equal(activity.loads, 1);
activity.push(job('finished'));
assert.equal(await checking, REPORT);
// The other check ended between the refusal and the read: its report is the latest.
const ended = fakeApi({ 'POST /prompt_manager/system/health': [busy], 'GET /prompt_manager/system/health': [{ report: REPORT }] });
assert.equal(await runHealthCheck(ended, { activity: fakeActivity({ reads: [[]] }) }), REPORT);
});
test('a failed, lost or reportless check rejects; other errors pass through', async () => {
const { runHealthCheck, waitForJob, HEALTH_FAILED_TEXT } = await overviewModule;
const failed = fakeApi({ 'POST /prompt_manager/system/health': [{ job: job('failed') }] });
await assert.rejects(runHealthCheck(failed, { activity: fakeActivity() }), { message: HEALTH_FAILED_TEXT });
// The store marks a job lost when the server no longer lists it (ComfyUI restarted).
const restarted = fakeActivity({ jobs: [job('running')] });
const waiting = waitForJob(restarted, JOB);
restarted.push(job('lost'));
assert.equal(await waiting, 'lost');
const empty = fakeApi({ 'POST /prompt_manager/system/health': [{ job: job('finished') }],
'GET /prompt_manager/system/health': [{ report: null }] });
await assert.rejects(runHealthCheck(empty, { activity: fakeActivity() }), { message: HEALTH_FAILED_TEXT });
const broken = fakeApi({ 'POST /prompt_manager/system/health': [Object.assign(new Error('server'), { status: 500 })] });
await assert.rejects(runHealthCheck(broken, { activity: fakeActivity() }), { message: 'server' });
const invalid = fakeApi({ 'POST /prompt_manager/system/health': [{ job: null }] });
await assert.rejects(runHealthCheck(invalid, { activity: fakeActivity() }), { message: 'Invalid job' });
});
test('waiting stops when its signal aborts', async () => {
const { waitForJob } = await overviewModule;
const controller = new AbortController();
const activity = fakeActivity({ jobs: [job('running')] });
const waiting = waitForJob(activity, JOB, { signal: controller.signal });
await new Promise(resolve => setImmediate(resolve));
controller.abort(new Error('left'));
await assert.rejects(waiting, { message: 'left' });
assert.equal(activity.listeners(), 0);
const gone = new AbortController();
gone.abort(new Error('already'));
await assert.rejects(waitForJob(fakeActivity(), JOB, { signal: gone.signal }), { message: 'already' });
});
-145
View File
@@ -1,145 +0,0 @@
// System › Sources & scans (#249 System S4 Task 11): the concept's "Image sources" list (app.js 679–681) built from the
// server's configured, effective and watched sources (GET /settings), the host draft of `gallery_root_paths`, the
// 3.2.0 update notice's dismissal and the Refresh files toast (D98).
const test = require('node:test');
const assert = require('node:assert/strict');
const sourcesModule = import('../../web/js/workspace/system-sources.mjs');
const STATUS = Object.freeze({
configured: [
{ label: 'ComfyUI output', path: 'output', valid: true, code: null, reason: null },
{ label: 'ComfyUI input', path: 'input', valid: true, code: null, reason: null },
],
effective: [
{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'watching' },
{ label: 'ComfyUI input', path: 'input', root_index: 1, watch: 'manual' },
],
auto_detected: false,
});
test('each effective source is the concept row: label, "<path> · watching|manual scan", Scan changes or Scan folder', async () => {
const { sourceRows } = await sourcesModule;
const saved = ['output', 'input'];
const { rows, note } = sourceRows({ sources: STATUS, saved, current: saved });
assert.equal(note, null);
assert.deepEqual(rows, [
{ key: 'output', title: 'ComfyUI output', description: 'output · watching', scan: { label: 'Scan changes', root: 0, path: 'output' },
remove: 'output' },
{ key: 'input', title: 'ComfyUI input', description: 'input · manual scan', scan: { label: 'Scan folder', root: 1, path: 'input' },
remove: 'input' },
]);
});
test('watching only with a live observer; restart pending otherwise (D98)', async () => {
const { sourceRows, WATCH_COPY } = await sourcesModule;
assert.deepEqual(WATCH_COPY, { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan',
restart_pending: 'restart pending' });
const sources = { ...STATUS, effective: STATUS.effective.map(source => ({ ...source, watch: 'restart_pending' })) };
const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] });
assert.deepEqual(rows.map(row => [row.description, row.scan.label]),
[['output · restart pending', 'Scan folder'], ['input · restart pending', 'Scan folder']]);
});
test('a configured source that cannot be used shows the server reason and only Remove', async () => {
const { sourceRows } = await sourcesModule;
const sources = { ...STATUS, configured: [...STATUS.configured,
{ label: 'unmounted-drive', path: 'unmounted-drive', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] };
const saved = ['output', 'input', 'unmounted-drive'];
const { rows } = sourceRows({ sources, saved, current: saved });
assert.deepEqual(rows.at(-1), { key: 'unmounted-drive', title: 'unmounted-drive',
description: 'unmounted-drive · Gallery root does not exist', scan: null, remove: 'unmounted-drive' });
});
test('an added source is listed as not saved yet with its validated label; a removed one leaves the list', async () => {
const { sourceRows } = await sourcesModule;
const labels = new Map([['output/archive', 'archive']]);
const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['input', 'output/archive'], labels });
assert.equal(note, null);
assert.deepEqual(rows.map(row => [row.title, row.description, row.scan?.label ?? null, row.remove]), [
['ComfyUI input', 'input · manual scan', 'Scan folder', 'input'],
['archive', 'output/archive · not saved yet', null, 'output/archive'],
]);
// Without a recorded label (a reload of System), the folder name stands in.
assert.equal(sourceRows({ sources: STATUS, saved: [], current: ['output/new-collection'] }).rows[0].title, 'new-collection');
});
test('removing the last entry states auto-detect (D98)', async () => {
const { sourceRows, NOTES } = await sourcesModule;
const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: [] });
assert.deepEqual(rows, []);
assert.equal(note, NOTES.removedAll);
assert.equal(NOTES.removedAll, 'No source is configured. After saving, ComfyUI’s output folder is used.');
});
test('auto-detected sources are listed without Remove, with the line that says how they were chosen', async () => {
const { sourceRows, NOTES } = await sourcesModule;
const auto = { configured: [], effective: [{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'manual' }], auto_detected: true };
assert.deepEqual(sourceRows({ sources: auto, saved: [], current: [] }), { rows: [
{ key: 'auto:0', title: 'ComfyUI output', description: 'output · manual scan', scan: { label: 'Scan folder', root: 0, path: 'output' },
remove: null },
], note: NOTES.autoDetected });
// Every configured source unusable: ComfyUI's output folder is used instead.
const fallback = { ...auto, configured: [{ label: 'gone', path: 'gone', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] };
const result = sourceRows({ sources: fallback, saved: ['gone'], current: ['gone'] });
assert.deepEqual(result.rows.map(row => [row.title, row.remove]), [['gone', 'gone'], ['ComfyUI output', null]]);
assert.equal(result.note, NOTES.fallback);
assert.equal(sourceRows({ sources: { configured: [], effective: [], auto_detected: false }, saved: [], current: [] }).note, NOTES.none);
});
test('a path listed twice in the draft is one row', async () => {
const { sourceRows } = await sourcesModule;
assert.equal(sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['output', 'output'] }).rows.length, 1);
});
test('roots equal to the saved list are the saved list itself, so the draft is clean again', async () => {
const { draftRoots } = await sourcesModule;
const saved = ['output', 'input'];
assert.equal(draftRoots(['output', 'input'], saved), saved);
assert.deepEqual(draftRoots(['input'], saved), ['input']);
assert.notEqual(draftRoots(['input', 'output'], saved), saved);
assert.deepEqual(draftRoots([], undefined), []);
});
test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed); storage that throws keeps it for this view', async () => {
const { noticeDismissed, dismissNotice, NOTICE_KEY, NOTICE_VERSION } = await sourcesModule;
assert.deepEqual([NOTICE_KEY, NOTICE_VERSION], ['pm_update_notice_dismissed', '3.2.0']);
const values = new Map();
const storage = { getItem: key => values.get(key) ?? null, setItem: (key, value) => values.set(key, String(value)) };
assert.equal(noticeDismissed(() => storage), false);
values.set(NOTICE_KEY, '3.1.0');
assert.equal(noticeDismissed(() => storage), false);
assert.equal(dismissNotice(() => storage), true);
assert.equal(values.get(NOTICE_KEY), '3.2.0');
assert.equal(noticeDismissed(() => storage), true);
const denied = () => { throw new Error('SecurityError'); };
assert.equal(noticeDismissed(denied), false);
assert.equal(dismissNotice(denied), false);
assert.equal(dismissNotice(() => ({ setItem() { throw new Error('QuotaExceededError'); } })), false);
});
test('Refresh files toasts the rescan counts, or says it finished when they are missing', async () => {
const { refreshedText } = await sourcesModule;
assert.equal(refreshedText({ total: 1234, roots: 2 }), 'Files refreshed. 1,234 files in 2 output folders.');
assert.equal(refreshedText({ total: 1, roots: 1 }), 'Files refreshed. 1 file in 1 output folder.');
assert.equal(refreshedText({ total: -1, roots: 1 }), 'Files refreshed.');
assert.equal(refreshedText(null), 'Files refreshed.');
});
test('monitoring switched off: a source still watched says so until the restart (D128)', async () => {
const { sourceRows } = await sourcesModule;
const sources = { ...STATUS, effective: [{ ...STATUS.effective[0], watch: 'stopping' }, STATUS.effective[1]] };
const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] });
assert.deepEqual(rows.map(row => [row.description, row.scan.label]),
[['output · watching until restart', 'Scan changes'], ['input · manual scan', 'Scan folder']]);
});
test('a valid configured source missing from the effective list is another spelling of a listed folder (D128)', async () => {
const { sourceRows, DUPLICATE_SOURCE } = await sourcesModule;
assert.equal(DUPLICATE_SOURCE, 'same folder as another source');
const sources = { ...STATUS, configured: [...STATUS.configured,
{ label: 'output', path: 'output/.', valid: true, code: null, reason: null }] };
const saved = ['output', 'input', 'output/.'];
assert.deepEqual(sourceRows({ sources, saved, current: saved }).rows.at(-1), { key: 'output/.', title: 'output',
description: 'output/. · same folder as another source', scan: null, remove: 'output/.' });
});
-366
View File
@@ -1,366 +0,0 @@
// System draft store (#249 System S2 Task 5): one Save row for browser, host and runtime fields (concept app.js
// renderSystem `.save-row` 705, `save-settings` 1320). It saves browser, then host with the GET revision, then runtime;
// a domain that fails keeps its fields dirty, and the status copy (D92) says what really happened.
const test = require('node:test');
const assert = require('node:assert/strict');
const source = import('../../web/js/workspace/system-state.mjs');
/** Adapters that record every call; `host` answers like POST /prompt_manager/settings. */
function adapters({ browser = {}, host = {}, runtime = {}, revision = 'sha256:one' } = {}) {
const calls = [];
const outcome = { browser: null, host: null, runtime: null };
return {
calls,
outcome,
readBrowser: () => { calls.push(['readBrowser']); return { ...browser }; },
readHost: async () => { calls.push(['readHost']); return { values: { ...host }, revision }; },
readRuntime: async () => { calls.push(['readRuntime']); return { ...runtime }; },
writeBrowser: async changes => {
calls.push(['writeBrowser', changes]);
if (outcome.browser) return outcome.browser(changes);
return { persisted: true };
},
writeHost: async (changes, options) => {
calls.push(['writeHost', changes, options]);
if (outcome.host) return outcome.host(changes, options);
return { success: true, persisted: true, revision: 'sha256:two', runtime_applied: true };
},
writeRuntime: async changes => {
calls.push(['writeRuntime', changes]);
if (outcome.runtime) return outcome.runtime(changes);
return { success: true };
},
};
}
async function loaded(domains) {
const { createSystemDrafts } = await source;
const drafts = createSystemDrafts(domains);
await Promise.all(['browser', 'host', 'runtime'].map(domain => drafts.load(domain)));
domains.calls.length = 0;
return drafts;
}
test('sections follow the concept order and the production hashes', async () => {
const { SYSTEM_SECTIONS, isSystemSection } = await source;
assert.deepEqual(SYSTEM_SECTIONS.map(([id, label]) => [id, label]), [
['overview', 'Overview'], ['appearance', 'Appearance'], ['sources', 'Sources & scans'],
['integrations', 'Integrations'], ['tagging', 'Auto-tagging'], ['maintenance', 'Maintenance'],
['backups', 'Backup & export'], ['diagnostics', 'Diagnostics & logs'],
]);
assert.equal(isSystemSection('backups'), true);
for (const value of ['general', '', 'constructor', '__proto__', undefined]) assert.equal(isSystemSection(value), false, value);
});
test('drafts read baselines and report dirty only for changed values', async () => {
const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 } });
const drafts = await loaded(domains);
assert.equal(drafts.get('browser', 'layout'), 'grid');
assert.equal(drafts.get('host', 'result_timeout'), 30);
assert.equal(drafts.dirty(), false);
drafts.set('host', 'result_timeout', 45);
assert.equal(drafts.get('host', 'result_timeout'), 45);
assert.equal(drafts.dirty(), true);
assert.equal(drafts.dirty('host'), true);
assert.equal(drafts.dirty('browser'), false);
// Setting the saved value back is clean again.
drafts.set('host', 'result_timeout', 30);
assert.equal(drafts.dirty(), false);
assert.throws(() => drafts.set('elsewhere', 'x', 1), TypeError);
assert.throws(() => drafts.get('elsewhere', 'x'), TypeError);
});
test('save commits browser then host then runtime', async () => {
const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 }, runtime: { level: 'INFO' } });
const drafts = await loaded(domains);
drafts.set('runtime', 'level', 'DEBUG');
drafts.set('host', 'result_timeout', 45);
drafts.set('browser', 'layout', 'list');
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'saved', host: 'saved', runtime: 'applied' });
assert.deepEqual(domains.calls, [
['writeBrowser', { layout: 'list' }],
['writeHost', { result_timeout: 45 }, { expected_revision: 'sha256:one' }],
['writeRuntime', { level: 'DEBUG' }],
]);
assert.equal(drafts.dirty(), false);
// The next host save sends the revision the last one returned.
drafts.set('host', 'result_timeout', 60);
await drafts.save();
assert.deepEqual(domains.calls.at(-1), ['writeHost', { result_timeout: 60 }, { expected_revision: 'sha256:two' }]);
});
test('host failure keeps host fields dirty and browser saved', async () => {
const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 } });
const drafts = await loaded(domains);
domains.outcome.host = () => { throw Object.assign(new Error('Settings could not be written to config.json.'), { code: 'config-write-failed' }); };
drafts.set('browser', 'layout', 'list');
drafts.set('host', 'result_timeout', 45);
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'saved', host: 'failed', runtime: 'clean' });
assert.equal(drafts.dirty('browser'), false);
assert.equal(drafts.dirty('host'), true);
assert.equal(drafts.get('host', 'result_timeout'), 45);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
assert.deepEqual([...drafts.failed('browser')], []);
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
// A save answered `persisted: false` without an error is not saved either.
domains.outcome.host = () => ({ success: false, persisted: false });
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'failed', runtime: 'clean' });
// Editing the failed field clears its highlight and reads as unsaved again.
drafts.set('host', 'result_timeout', 50);
assert.deepEqual([...drafts.failed('host')], []);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('stale revision reports conflict and keeps the draft', async () => {
const domains = adapters({ host: { result_timeout: 30 } });
const drafts = await loaded(domains);
domains.outcome.host = () => { throw Object.assign(new Error('Settings changed since they were loaded.'), { code: 'settings-changed' }); };
drafts.set('host', 'result_timeout', 45);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'conflict', runtime: 'clean' });
assert.equal(drafts.dirty('host'), true);
assert.equal(drafts.get('host', 'result_timeout'), 45);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
// The conflict read the host values again (their new revision), keeping the draft.
assert.deepEqual(domains.calls.slice(-2).map(([name]) => name), ['writeHost', 'readHost']);
// Reloading the section reads the new revision and keeps the draft (Q10).
domains.outcome.host = null;
await drafts.load('host');
assert.equal(drafts.get('host', 'result_timeout'), 45);
await drafts.save();
assert.deepEqual(domains.calls.at(-1)[2], { expected_revision: 'sha256:one' });
});
test('host save without a read revision never posts', async () => {
const { createSystemDrafts } = await source;
// Never read: an edit made before the host values (and their revision) arrived must not merge blindly.
const unread = adapters({ host: { result_timeout: 30 } });
const drafts = createSystemDrafts(unread);
drafts.set('host', 'result_timeout', 45);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'failed', runtime: 'clean' });
assert.deepEqual(unread.calls, []);
assert.equal(drafts.dirty('host'), true);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
// A read of an unreadable configuration has no revision either.
const unreadable = adapters({ host: { result_timeout: 30 }, revision: null });
const blind = createSystemDrafts(unreadable);
await blind.load('host');
blind.set('host', 'result_timeout', 45);
assert.equal((await blind.save()).host, 'failed');
assert.equal(unreadable.calls.filter(([name]) => name === 'writeHost').length, 0);
// Once read, the same draft saves with the revision.
await drafts.load('host');
assert.equal((await drafts.save()).host, 'saved');
assert.deepEqual(unread.calls.at(-1), ['writeHost', { result_timeout: 45 }, { expected_revision: 'sha256:one' }]);
});
test('reloadHost reads the host values and says whether it could', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ host: { result_timeout: 30 } });
const drafts = createSystemDrafts(domains);
assert.equal(await drafts.reloadHost(), true);
assert.equal(drafts.get('host', 'result_timeout'), 30);
const broken = createSystemDrafts({ ...domains, readHost: async () => { throw new Error('offline'); } });
assert.equal(await broken.reloadHost(), false);
broken.set('host', 'result_timeout', 45);
assert.equal((await broken.save()).host, 'failed');
});
test('no domain called when clean', async () => {
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = await loaded(domains);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls, []);
drafts.set('browser', 'layout', 'list');
drafts.discard();
assert.equal(drafts.dirty(), false);
assert.equal(drafts.get('browser', 'layout'), 'grid');
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls, []);
});
test('status text per outcome', async () => {
const { statusText, SETTINGS_STATUS } = await source;
assert.deepEqual(SETTINGS_STATUS, {
clean: 'No unsaved changes.',
dirty: 'Unsaved changes.',
browser: 'Saved in this browser.',
host: 'Saved to the server configuration.',
runtime: 'Applied until the server restarts.',
partial: 'Some changes were not saved. Review the highlighted settings.',
tabOnly: 'Saved for this tab only; browser storage is unavailable.',
conflict: 'Settings changed elsewhere. Reload this section to see them.',
restart: 'Restart ComfyUI to apply:',
});
const clean = { browser: 'clean', host: 'clean', runtime: 'clean' };
assert.equal(statusText(clean), 'No unsaved changes.');
assert.equal(statusText({ ...clean, browser: 'saved' }), 'Saved in this browser.');
assert.equal(statusText({ ...clean, host: 'saved' }), 'Saved to the server configuration.');
assert.equal(statusText({ ...clean, runtime: 'applied' }), 'Applied until the server restarts.');
assert.equal(statusText({ ...clean, browser: 'tab-only' }), 'Saved for this tab only; browser storage is unavailable.');
assert.equal(statusText({ browser: 'saved', host: 'saved', runtime: 'applied' }),
'Saved in this browser. Saved to the server configuration. Applied until the server restarts.');
assert.equal(statusText({ ...clean, browser: 'saved', runtime: 'failed' }),
'Some changes were not saved. Review the highlighted settings.');
assert.equal(statusText({ browser: 'failed', host: 'conflict', runtime: 'clean' }),
'Settings changed elsewhere. Reload this section to see them.');
});
test('the store says unsaved, then the save outcome, and notifies subscribers', async () => {
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = await loaded(domains);
const seen = [];
const unsubscribe = drafts.subscribe(() => seen.push(drafts.statusText()));
assert.equal(drafts.statusText(), 'No unsaved changes.');
drafts.set('browser', 'layout', 'list');
domains.outcome.browser = () => ({ persisted: false });
await drafts.save();
assert.deepEqual(seen, ['Unsaved changes.', 'Saved for this tab only; browser storage is unavailable.']);
unsubscribe();
drafts.set('browser', 'layout', 'masonry');
assert.equal(seen.length, 2);
// The tab-only value is this tab's baseline now.
drafts.set('browser', 'layout', 'list');
assert.equal(drafts.dirty(), false);
});
test('an edit made while a save runs stays dirty', async () => {
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = await loaded(domains);
let release;
domains.outcome.browser = () => new Promise(resolve => { release = () => resolve({ persisted: true }); });
drafts.set('browser', 'layout', 'list');
const saving = drafts.save();
// A second Save while one runs joins it instead of writing twice.
assert.equal(drafts.save(), saving);
assert.equal(drafts.saving(), true);
await new Promise(resolve => setImmediate(resolve));
drafts.set('browser', 'layout', 'masonry');
release();
assert.deepEqual(await saving, { browser: 'saved', host: 'clean', runtime: 'clean' });
assert.equal(drafts.saving(), false);
assert.equal(drafts.dirty('browser'), true);
assert.equal(drafts.get('browser', 'layout'), 'masonry');
assert.equal(domains.calls.filter(([name]) => name === 'writeBrowser').length, 1);
});
test('a failed read keeps the store usable and drafts survive a reload (Q10)', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = createSystemDrafts(domains);
await drafts.load('browser');
drafts.set('browser', 'layout', 'list');
await drafts.load('browser');
assert.equal(drafts.get('browser', 'layout'), 'list');
assert.equal(drafts.dirty(), true);
const failing = createSystemDrafts({ ...domains, readHost: async () => { throw new Error('offline'); } });
await assert.rejects(failing.load('host'), /offline/);
assert.equal(failing.dirty(), false);
assert.throws(() => createSystemDrafts({ ...domains, writeHost: null }), TypeError);
});
test('a rejected value drops its draft and is highlighted; a conflict elsewhere keeps the drafts', async () => {
const domains = adapters({ browser: { videoMute: true, videoLoop: true }, host: { result_timeout: 5 } });
const drafts = await loaded(domains);
assert.deepEqual(drafts.saved('browser'), { videoMute: true, videoLoop: true });
drafts.set('host', 'result_timeout', 12);
assert.equal(drafts.dirty('host'), true);
drafts.reject('host', 'result_timeout');
assert.equal(drafts.dirty('host'), false);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
assert.equal(drafts.get('host', 'result_timeout'), 5);
// A valid value clears the highlight.
drafts.set('host', 'result_timeout', 7);
assert.deepEqual([...drafts.failed('host')], []);
drafts.discard();
drafts.set('browser', 'videoMute', false);
drafts.conflict('browser', []);
assert.equal(drafts.statusText(), 'Unsaved changes.');
drafts.conflict('browser', ['videoMute']);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
assert.deepEqual([...drafts.failed('browser')], ['videoMute']);
assert.equal(drafts.get('browser', 'videoMute'), false);
assert.throws(() => drafts.reject('nowhere', 'x'), TypeError);
});
test('a rejected value says why in the status until the next edit', async () => {
const drafts = await loaded(adapters({ host: { result_timeout: 5 } }));
drafts.reject('host', 'result_timeout');
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
drafts.set('host', 'result_timeout', 9);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('a browser write refused as changed elsewhere keeps the draft and reads the browser values again', async () => {
const domains = adapters({ browser: { workspaceDensity: 'comfortable' } });
const drafts = await loaded(domains);
drafts.set('browser', 'workspaceDensity', 'compact');
domains.outcome.browser = () => { throw Object.assign(new Error('changed'), { code: 'settings-changed' }); };
assert.deepEqual(await drafts.save(), { browser: 'conflict', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls.map(([name]) => name), ['writeBrowser', 'readBrowser']);
assert.equal(drafts.get('browser', 'workspaceDensity'), 'compact');
assert.deepEqual([...drafts.failed('browser')], ['workspaceDensity']);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
});
test('host read state says whether the server settings could be read', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ host: { result_timeout: 5 } });
let fail = true;
const drafts = createSystemDrafts({ ...domains, readHost: async () => { if (fail) throw new Error('offline'); return { values: {}, revision: 'r' }; } });
assert.equal(drafts.hostState(), 'pending');
assert.equal(await drafts.reloadHost(), false);
assert.equal(drafts.hostState(), 'failed');
fail = false;
assert.equal(await drafts.reloadHost(), true);
assert.equal(drafts.hostState(), 'loaded');
});
test('a host save names the settings that apply after a restart (D128)', async () => {
const { statusText, RESTART_LABELS } = await source;
assert.deepEqual(RESTART_LABELS, { monitoring_enabled: 'Watch for new outputs' });
const saved = { browser: 'clean', host: 'saved', runtime: 'clean' };
assert.equal(statusText({ ...saved, restart: ['monitoring_enabled'] }),
'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// A field without a row label (the sources say "restart pending" per row) adds nothing.
assert.equal(statusText({ ...saved, restart: ['gallery_root_paths'] }), 'Saved to the server configuration.');
assert.equal(statusText({ ...saved, restart: [] }), 'Saved to the server configuration.');
const domains = adapters({ host: { monitoring_enabled: true } });
const drafts = await loaded(domains);
domains.outcome.host = () => ({ success: true, persisted: true, revision: 'sha256:two',
restart_required_fields: ['monitoring_enabled', 7] });
drafts.set('host', 'monitoring_enabled', false);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'saved', runtime: 'clean', restart: ['monitoring_enabled'] });
assert.equal(drafts.statusText(), 'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// The next edit is unsaved again.
drafts.set('host', 'monitoring_enabled', true);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('a host save refused for one source says the server reason and names it until the next edit', async () => {
const domains = adapters({ host: { gallery_root_paths: ['output', 'output/gone'] } });
const drafts = await loaded(domains);
const message = "output/gone can't be used: Gallery root does not exist. Remove it or reconnect it, then save.";
domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source', source: 'output/gone' }); };
drafts.set('host', 'gallery_root_paths', ['output/gone']);
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'clean', host: 'failed', runtime: 'clean',
invalid: { source: 'output/gone', message } });
assert.equal(drafts.statusText(), message);
assert.equal(drafts.invalidSource(), 'output/gone');
assert.deepEqual([...drafts.failed('host')], ['gallery_root_paths']);
// Any other failure keeps the generic copy and names no source.
domains.outcome.host = () => { throw Object.assign(new Error('busy'), { code: 'busy' }); };
await drafts.save();
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
assert.equal(drafts.invalidSource(), null);
domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source' }); };
await drafts.save();
assert.equal(drafts.invalidSource(), null);
drafts.set('host', 'gallery_root_paths', ['output']);
assert.equal(drafts.invalidSource(), null);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
-143
View File
@@ -76,7 +76,6 @@ function fakeDOM() {
get activeElement() { return active ?? body; },
createElement: tag => new Element(tag),
createElementNS: (namespace, tag) => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const all = (root, predicate, found = []) => {
if (predicate(root)) found.push(root);
@@ -791,145 +790,3 @@ test('the Filters button shows the active refinement count', async () => {
dom.restore();
}
});
// #249 System S2 Task 5: the scan review's dialog and backup box (D55) generalised for every record-changing review.
const scanModule = import('../../web/js/workspace/activity-scan.mjs');
const bodyShape = dom => byId(dom, 'modal-body').children.map(node => [node.tagName.toLowerCase(), node.className].filter(Boolean).join('.'));
test('backupNotice is the concept .notice with its copy and one Back up now action', async () => {
const { backupNotice, SCAN_BACKUP_COPY } = await dialogsModule;
const dom = fakeDOM();
try {
let calls = 0;
const notice = backupNotice({ copy: SCAN_BACKUP_COPY, onBackup: () => { calls++; } });
assert.equal(notice.className, 'notice');
assert.deepEqual(tags(notice), ['div', 'div']);
assert.deepEqual(tags(notice.children[0]), ['h3', 'p']);
assert.equal(notice.children[0].children[0].textContent, 'Back up the database first (recommended)');
assert.equal(notice.children[0].children[1].textContent,
'Downloads a copy of the prompts database before the scan changes any records.');
assert.equal(notice.children[1].className, 'row-actions');
const action = button(notice, 'Back up now');
action.click();
assert.equal(calls, 1);
assert.throws(() => backupNotice({ copy: { heading: '', text: 'x' }, onBackup() {} }), TypeError);
assert.throws(() => backupNotice({ copy: SCAN_BACKUP_COPY }), TypeError);
} finally {
dom.restore();
}
});
test('openReviewDialog lays out heading, grid, options, sentence, backup box and actions', async () => {
const { openReviewDialog, closeWorkspaceDialog } = await dialogsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const trigger = new dom.Element('button');
dom.body.append(trigger);
const option = new dom.Element('div');
option.className = 'setting-row';
const started = [];
const review = openReviewDialog({ scope, title: 'Review maintenance plan', heading: 'Missing-file links',
grid: [['Scope', '2 stale image links'], ['Media on disk', 'No source files removed']], options: [option],
sentence: 'Cancelling leaves all data untouched.', backup: { copy: { heading: 'Back up first', text: 'A copy.' }, onBackup() {} },
cancelLabel: 'Cancel', primary: { label: 'Apply reviewed operation' }, onStart: () => started.push('start'), trigger });
assert.equal(byId(dom, 'modal-title').textContent, 'Review maintenance plan');
assert.deepEqual(bodyShape(dom), ['h3', 'dl.meta-grid', 'div.setting-row', 'p.small', 'div.notice', 'div.dialog-actions']);
const grid = byId(dom, 'modal-body').children[1];
assert.deepEqual(grid.children.map(entry => [tags(entry), entry.children[0].textContent, entry.children[1].textContent]), [
[['dt', 'dd'], 'Scope', '2 stale image links'], [['dt', 'dd'], 'Media on disk', 'No source files removed']]);
review.setValue('Scope', '3 stale image links');
assert.equal(grid.children[0].children[1].textContent, '3 stale image links');
const actions = byId(dom, 'modal-body').children.at(-1);
assert.deepEqual(actions.children.map(node => [node.textContent, node.className]),
[['Cancel', ''], ['Apply reviewed operation', 'ca-primary']]);
actions.children[1].click();
assert.deepEqual(started, ['start']);
assert.equal(modal(dom).open, false);
assert.equal(document.activeElement, trigger);
// A danger primary (the concept's Restore) has no sentence, heading, options or backup box when none is given.
openReviewDialog({ scope, title: 'Restore preview', grid: [['Integrity', 'Verified']], backup: null,
cancelLabel: 'Cancel', primary: { label: 'Restore', danger: true }, onStart() {} });
assert.deepEqual(bodyShape(dom), ['dl.meta-grid', 'div.dialog-actions']);
assert.equal(button(byId(dom, 'modal-body'), 'Restore').className, 'danger');
button(byId(dom, 'modal-body'), 'Cancel').click();
assert.equal(modal(dom).open, false);
assert.throws(() => openReviewDialog({ scope, title: 'x', grid: [['a']], cancelLabel: 'Cancel', primary: { label: 'Go' },
onStart() {} }), TypeError);
assert.throws(() => openReviewDialog({ scope, title: 'x', grid: [], cancelLabel: 'Cancel', primary: {}, onStart() {} }), TypeError);
} finally {
scope.dispose();
closeWorkspaceDialog();
dom.restore();
}
});
test('review dialog backup box keeps the dialog open and toasts the outcome', async () => {
const { openReviewDialog, closeWorkspaceDialog, SCAN_BACKUP_COPY } = await dialogsModule;
const { downloadDatabaseBackup, BACKUP_DONE, BACKUP_FAILED } = await scanModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
const created = URL.createObjectURL;
const revoked = URL.revokeObjectURL;
URL.createObjectURL = () => 'blob:backup';
URL.revokeObjectURL = () => {};
try {
const toasts = [];
let answer;
const fetchUrl = () => new Promise(resolve => { answer = resolve; });
openReviewDialog({ scope, title: 'Review metadata scan', grid: [['Sources', '2 output folders']],
backup: { copy: SCAN_BACKUP_COPY, onBackup: () => downloadDatabaseBackup({ fetchUrl, notify: text => toasts.push(text) }) },
cancelLabel: 'Cancel', primary: { label: 'Start scan' }, onStart() { throw new Error('not started'); } });
const action = button(byId(dom, 'modal-body'), 'Back up now');
action.click();
assert.equal(action.textContent, 'Backing up…');
assert.equal(action.getAttribute('aria-disabled'), 'true');
// A second press while the copy downloads is ignored.
action.click();
answer({ ok: true, blob: async () => new Blob(['db']), headers: { get: () => 'attachment; filename="prompts_backup_1.db"' } });
await tick();
assert.deepEqual(toasts, [BACKUP_DONE]);
assert.equal(modal(dom).open, true);
assert.equal(action.textContent, 'Back up now');
assert.equal(action.getAttribute('aria-disabled'), null);
action.click();
answer({ ok: false, status: 500 });
await tick();
assert.deepEqual(toasts, [BACKUP_DONE, BACKUP_FAILED]);
assert.equal(modal(dom).open, true);
} finally {
URL.createObjectURL = created;
URL.revokeObjectURL = revoked;
scope.dispose();
closeWorkspaceDialog();
dom.restore();
}
});
test('the scan review is a thin openReviewDialog caller with its scope switch', async () => {
const { openScanReviewDialog, closeWorkspaceDialog } = await dialogsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const starts = [];
openScanReviewDialog({ scope, sources: 2, onBackup() {}, onStart: value => starts.push(value) });
assert.equal(byId(dom, 'modal-title').textContent, 'Review metadata scan');
assert.deepEqual(bodyShape(dom), ['dl.meta-grid', 'div.setting-row', 'p.small', 'div.notice', 'div.dialog-actions']);
const grid = byId(dom, 'modal-body').children[0];
assert.deepEqual(grid.children.map(entry => entry.textContent), ['Sources2 output folders', 'ScopeNew and unresolved files']);
const allFiles = byId(dom, 'scan-all-files');
allFiles.checked = true;
allFiles.dispatchEvent(new Event('change'));
assert.equal(grid.children[1].children[1].textContent, 'All files');
button(byId(dom, 'modal-body'), 'Start scan').click();
assert.deepEqual(starts, [{ scope: 'all' }]);
} finally {
scope.dispose();
closeWorkspaceDialog();
dom.restore();
}
});
-2
View File
@@ -29,7 +29,6 @@ VECTORS_PATH = (
# Committed record sets stay small; the large bound cases are built at test time.
MAX_VECTOR_CHARS = 20_000
COMMITTED_FUZZ = (0, 60)
COMMITTED_PASSTHROUGH = (0, 40)
SPACES = ["", " ", "\t", "\n", "\x1c", "\x85", " ", "", "
 "]
WORDS = ["fox", "snow", "café", "猫", "\U0001f600", "a,", "detailed fur", ""]
@@ -1381,7 +1380,6 @@ def committed_vectors():
}
cases.update(fuzz_cases(*COMMITTED_FUZZ))
cases.update(selection.selection_fuzz_cases(*COMMITTED_FUZZ))
cases.update(selection.passthrough_fuzz_cases(*COMMITTED_PASSTHROUGH))
vectors = [
{
"name": name,
+1 -304
View File
@@ -12,10 +12,8 @@ import json
import random
try:
from tests import passthrough_graphs as shapes
from tests.local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import passthrough_graphs as shapes
from local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
@@ -439,213 +437,10 @@ def _parameters():
return cases
def _passthrough():
"""Chains joined through lineage nodes, and the floor for added conditioning."""
cases = {
"hires-fix": shapes.hires_fix(),
"hires-fix-differs": shapes.hires_fix(second_positive=shapes.OTHER),
"hires-fix-unlisted": shapes.hires_fix(filter_class="MysteryImageEffect"),
"hires-fix-generated": shapes.generated_text_chain(),
"flux-three-stage": shapes.flux_three_stage(),
"flux-concat": shapes.flux_three_stage(composite="ConditioningConcat"),
"flux-average": shapes.flux_three_stage(composite="ConditioningAverage"),
"edit-batch": shapes.edit_batch(3),
}
graph = shapes.flux_three_stage(combined=(("37", 0), ("31", 0)))
graph["37"] = shapes.encode(shapes.OTHER)
cases["flux-without-base"] = graph
graph = shapes.flux_three_stage()
graph["38"] = shapes.encode("watermark")
graph["60"]["inputs"]["negative"] = ["38", 0]
cases["flux-other-negative"] = graph
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["10", 0]
graph["10"]["inputs"]["latent_image"] = ["20", 0]
cases["hires-fix-cycle"] = graph
graph = shapes.hires_fix()
graph["17"]["inputs"]["pixels"] = ["18", 0]
graph["18"] = {"class_type": "Reroute", "inputs": {"value": ["16", 0]}}
cases["hires-fix-reroute"] = graph
graph = shapes.hires_fix()
graph["24"] = {"class_type": "SaveImage", "inputs": {"images": ["11", 0]}}
cases["hires-fix-both-saved"] = graph
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["99", 0]
cases["hires-fix-missing-latent"] = graph
graph = shapes.flux_three_stage()
graph["38"] = {"class_type": "MysteryNegative", "inputs": {"seed": 1}}
graph["60"]["inputs"]["negative"] = ["38", 0]
cases["flux-negative-unsupported"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "Reroute", "inputs": {"input": ["35", 0]}}
graph["60"]["inputs"]["positive"] = ["39", 0]
cases["flux-reroute-input"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "CLIPTextEncodeSDXL",
"inputs": {"text_g": shapes.POSITIVE, "text_l": shapes.STYLE},
}
graph["60"]["inputs"]["positive"] = ["39", 0]
cases["flux-encoder-parts"] = graph
cases.update(_passthrough_edges())
cases.update(_runtime_parts())
return cases
def _runtime_parts():
"""Option A: a PromptManager part fed by a run-time node is left out."""
cases = {"trigger-prepend": shapes.hires_fix_with_trigger_words()}
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"].update(prepend_text="masterpiece", append_text=["8", 0])
graph["20"]["inputs"]["latent_image"] = ["4", 0]
cases["trigger-append-single"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["8"] = {"class_type": "PrimitiveString", "inputs": {"value": "trig9word"}}
cases["traced-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["8"]["class_type"] = "MysteryWords"
cases["unknown-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": "kept", "string_b": ["8", 0], "delimiter": ""},
}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
cases["indirect-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"].update(prepend_text="", text=["8", 0])
cases["runtime-main-text"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["class_type"] = "PromptManagerPair"
graph["2"]["inputs"]["negative_text"] = shapes.NEGATIVE
graph["10"]["inputs"]["negative"] = ["2", 1]
graph["20"]["inputs"]["negative"] = ["2", 1]
cases["trigger-pair"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["30"] = {"class_type": "PromptSearchList", "inputs": {"search": "x"}}
graph["2"]["inputs"]["append_text"] = ["30", 0]
cases["search-list-append"] = graph
for shape in ("text-encode", "chained", "concat"):
cases[f"through-{shape}"] = shapes.trigger_words_through(shape)
cases["branches-with-and-without"] = (
shapes.branches_with_and_without_trigger_words()
)
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = shapes.prompt_manager(shapes.POSITIVE)
graph["20"]["inputs"]["positive"] = ["9", 0]
cases["stage-with-and-without"] = graph
graph = shapes.hires_fix()
graph["9"] = {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 2,
"delimiter": " ",
"return_list": ["98", 0],
"string_1": "static words",
"string_2": "more",
},
}
graph["98"] = {"class_type": "PrimitiveBoolean", "inputs": {"value": False}}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
cases["join-multi-flag"] = graph
graph = shapes.flux_three_stage()
graph["31"] = shapes.encode(shapes.OTHER)
cases["flux-other-full-prompt"] = graph
graph = shapes.flux_three_stage()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["2", 0], "text": "x"},
}
graph["8"] = shapes.trigger_toggle()
graph["30"]["inputs"]["prepend_text"] = ["8", 0]
cases["flux-trigger"] = graph
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {"guider": ["63", 0], "latent_image": ["66", 0]},
}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"positive": ["35", 0], "negative": ["33", 0]},
}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["51", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
cases["flux-custom-advanced-chained"] = graph
return {f"runtime-{name}": graph for name, graph in cases.items()}
def _passthrough_edges():
"""Lineage and floor edges: branching refiners, other slots, cycles, guiders."""
cases = {}
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["17", 0], denoise=0.4)
graph["31"] = {"class_type": "SaveImage", "inputs": {"images": ["30", 0]}}
cases["hires-fix-two-refiners"] = graph
graph = shapes.hires_fix()
graph["10"]["class_type"] = "SamplerCustom"
graph["11"]["inputs"]["samples"] = ["10", 1]
cases["hires-fix-denoised-slot"] = graph
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["4", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["30", 0]}}
graph["32"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["31", 0]}}
graph["30"]["inputs"]["latent_image"] = ["32", 0]
cases["hires-fix-self-latent"] = graph
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["32", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["32", 0]}}
graph["32"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["31", 0]}}
graph["33"] = {"class_type": "SaveImage", "inputs": {"images": ["30", 0]}}
cases["hires-fix-lineage-cycle"] = graph
graph = shapes.flux_three_stage()
graph["39"] = shapes.encode(shapes.OTHER)
graph["40"]["inputs"]["positive"] = ["39", 0]
cases["flux-base-stages-differ"] = graph
graph = shapes.flux_three_stage()
del graph["60"]["inputs"]["negative"]
cases["flux-no-last-negative"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "MysteryPrompt", "inputs": {"seed": 1}}
graph["40"]["inputs"]["positive"] = ["39", 0]
graph["50"]["inputs"]["positive"] = ["39", 0]
cases["flux-unreadable-base"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "MysteryUpscaler",
"inputs": {"image": ["51", 0], "model": ["1", 0], "positive": ["34", 0]},
}
graph["52"]["inputs"]["images"] = ["39", 0]
cases["flux-unsupported-ancestry"] = graph
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["62", 0],
"guider": ["63", 0],
"sampler": ["64", 0],
"sigmas": ["65", 0],
"latent_image": ["66", 0],
},
}
graph["62"] = {"class_type": "RandomNoise", "inputs": {"noise_seed": 1}}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"model": ["1", 0], "positive": ["35", 0], "negative": ["33", 0]},
}
graph["64"] = {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "euler"}}
graph["65"] = {"class_type": "BasicScheduler", "inputs": {"steps": 4}}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["52", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
cases["flux-custom-advanced-last"] = graph
return {f"edge-{name}": graph for name, graph in cases.items()}
def selection_cases():
"""Named record sets for every Task 9 path, each a ``read_inspection_file`` dict."""
cases = {}
for group in (_samplers(), _previews(), _detailers(), _passthrough()):
for group in (_samplers(), _previews(), _detailers()):
for name, graph in group.items():
cases[f"select-{name}"] = {"prompt": json.dumps(graph, ensure_ascii=False)}
cases.update(_parameters())
@@ -777,101 +572,3 @@ def selection_fuzz_cases(seed, count):
records.update(builder.records)
cases[f"select-fuzz-{seed}-{index}"] = records
return cases
_LINEAGE_HOPS = (
("VAEDecode", "samples"),
("VAEDecodeTiled", "samples"),
("Image Filter", "images"),
("ImageUpscaleWithModel", "image"),
("ImageScale", "image"),
("ImageScaleBy", "image"),
("VAEEncode", "pixels"),
("LatentUpscaleBy", "samples"),
("Reroute", "value"),
("MysteryImageEffect", "image"),
)
def passthrough_fuzz_cases(seed, count):
"""Seeded sampler chains joined by random lineage (and unlisted) hops.
Each later stage reads the earlier one's output through zero to four hops,
with a shared, a different, an added-to (combine, concat, average) or a
run-time positive, and a shared or different negative, under a saver or a
preview. The server is the oracle; nothing states the expected answer.
"""
rng = random.Random(seed)
cases = {}
for index in range(count):
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": shapes.prompt_manager(rng.choice([shapes.POSITIVE, "fox"])),
"3": shapes.encode(shapes.NEGATIVE),
"4": {"class_type": "EmptyLatentBatch", "inputs": {"batch_size": 2}},
"5": shapes.encode(shapes.STYLE),
"6": shapes.encode(shapes.OTHER),
}
previous, next_id = ["4", 0], 10
for stage in range(rng.randint(2, 3)):
positive = ["2", 0]
choice = rng.randrange(8)
if stage and choice == 1:
positive = ["6", 0]
elif stage and choice in (2, 3, 4):
name = (
"ConditioningCombine",
"ConditioningConcat",
"ConditioningAverage",
)[choice - 2]
names = shapes._COMPOSITE_INPUTS[name]
parts = [["2", 0], rng.choice([["5", 0], ["6", 0]])]
rng.shuffle(parts)
graph[str(next_id)] = {
"class_type": name,
"inputs": dict(zip(names, parts)),
}
positive, next_id = [str(next_id), 0], next_id + 1
elif stage and choice == 5:
graph[str(next_id)] = {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["2", 0]},
}
positive, next_id = [str(next_id), 0], next_id + 1
negative = ["6", 0] if stage and rng.random() < 0.15 else ["3", 0]
latent = previous
if stage:
image = previous
for _ in range(rng.randrange(5)):
name, key = rng.choice(_LINEAGE_HOPS)
graph[str(next_id)] = {"class_type": name, "inputs": {key: image}}
image, next_id = [str(next_id), 0], next_id + 1
latent = image
sampler = shapes.ksampler(positive, negative, latent)
if stage and rng.random() < 0.2:
sampler = {
"class_type": "UltimateSDUpscale",
"inputs": {
"image": latent,
"model": ["1", 0],
"positive": positive,
"negative": negative,
},
}
graph[str(next_id)] = sampler
previous, next_id = [str(next_id), 0], next_id + 1
if rng.random() < 0.3:
saver = rng.choice(["SaveImage", "PreviewImage"])
graph[str(next_id)] = {
"class_type": saver,
"inputs": {"images": previous},
}
next_id += 1
graph[str(next_id)] = {
"class_type": "SaveImage",
"inputs": {"images": previous},
}
cases[f"select-passthrough-fuzz-{seed}-{index}"] = {
"prompt": json.dumps(graph, ensure_ascii=False)
}
return cases
-332
View File
@@ -1,332 +0,0 @@
"""Synthetic multi-stage graphs for the pass-through lineage rule.
Each builder reproduces one graph shape with neutral texts and generic names:
a hires-fix or refiner chain (sampler, decode, image filter, model upscale,
scale, encode, sampler), a three-stage Flux run whose last stage combines the
base prompt with a style trigger, an image-edit batch of samplers with distinct
prompts, and a chain whose prompt is generated at run time. They are shared by
the resolver tests and the browser-port parity cases; no user file is involved.
"""
import copy
POSITIVE = "a lighthouse on a cliff, golden hour"
NEGATIVE = "blurry, lowres"
OTHER = "a red bicycle in the rain"
STYLE = "film grain, analog photo"
def encode(text):
return {"class_type": "CLIPTextEncode", "inputs": {"text": text, "clip": ["1", 1]}}
def prompt_manager(text, prepend=""):
return {
"class_type": "PromptManager",
"inputs": {
"text": text,
"category": "",
"tags": "",
"search_text": "",
"prepend_text": prepend,
"append_text": "",
"clip": ["1", 1],
},
}
def ksampler(positive, negative, latent, denoise=1.0):
return {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"steps": 20,
"cfg": 7.0,
"sampler_name": "euler",
"scheduler": "normal",
"denoise": denoise,
"model": ["1", 0],
"positive": positive,
"negative": negative,
"latent_image": latent,
},
}
def hires_fix(second_positive=None, filter_class="Image Filter"):
"""Two KSamplers joined in image space, like a hires-fix or refiner run.
Sampler 10 runs on a custom batch latent; its output goes through a tiled
decode, an interactive image filter, a model upscale and a scale (sized by a
resolution node reading the filtered image), then is encoded again for
sampler 20 (denoise 0.25), which is decoded and saved.
"""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": prompt_manager(POSITIVE),
"3": encode(NEGATIVE),
"4": {
"class_type": "EmptyLatentBatch",
"inputs": {"width": 1024, "height": 512, "batch_size": 4},
},
"10": ksampler(["2", 0], ["3", 0], ["4", 0]),
"11": {
"class_type": "VAEDecodeTiled",
"inputs": {"tile_size": 512, "samples": ["10", 0], "vae": ["1", 2]},
},
"12": {
"class_type": filter_class,
"inputs": {"timeout": 600, "ontimeout": "send none", "images": ["11", 0]},
},
"13": {"class_type": "UpscaleModelLoader", "inputs": {"model_name": "x4"}},
"14": {
"class_type": "ImageUpscaleWithModel",
"inputs": {"upscale_model": ["13", 0], "image": ["12", 0]},
},
"15": {
"class_type": "ResolutionCalculator",
"inputs": {"scale_factor": 1.5, "image": ["12", 0]},
},
"16": {
"class_type": "ImageScale",
"inputs": {
"upscale_method": "nearest-exact",
"width": ["15", 0],
"height": ["15", 1],
"crop": "disabled",
"image": ["14", 0],
},
},
"17": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["16", 0], "vae": ["1", 2]},
},
"20": ksampler(["2", 0], ["3", 0], ["17", 0], denoise=0.25),
"21": {
"class_type": "VAEDecodeTiled",
"inputs": {"tile_size": 512, "samples": ["20", 0], "vae": ["1", 2]},
},
"22": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "hires", "images": ["21", 0]},
},
"23": {"class_type": "PlaySound|pysssss", "inputs": {"any": ["21", 0]}},
}
if second_positive is not None:
graph["5"] = encode(second_positive)
graph["20"]["inputs"]["positive"] = ["5", 0]
return graph
_COMPOSITE_INPUTS = {
"ConditioningCombine": ("conditioning_1", "conditioning_2"),
"ConditioningConcat": ("conditioning_to", "conditioning_from"),
"ConditioningAverage": ("conditioning_to", "conditioning_from"),
}
def flux_three_stage(combined=(("30", 0), ("31", 0)), composite="ConditioningCombine"):
"""FluxGuidance base, a refiner through an image filter, then a tiled upscale.
Samplers 40 and 50 share FluxGuidance(PromptManager 30) and a zeroed empty
negative. Sampler 50 refines 40's filtered image (denoise 0.25). The final
UltimateSDUpscale 60 reads 50's image, scaled down, and a positive that is
``composite`` of ``combined`` (by default the base PromptManager and a
style-trigger encoder 31).
"""
names = _COMPOSITE_INPUTS[composite]
graph = {
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "flux"}},
"2": {"class_type": "DualCLIPLoader", "inputs": {"type": "flux"}},
"3": {"class_type": "VAELoader", "inputs": {"vae_name": "ae"}},
"30": prompt_manager(POSITIVE),
"31": encode(STYLE),
"32": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "", "clip": ["2", 0]},
},
"33": {
"class_type": "ConditioningZeroOut",
"inputs": {"conditioning": ["32", 0]},
},
"34": {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["30", 0]},
},
"35": {
"class_type": composite,
"inputs": dict(zip(names, map(list, combined))),
},
"36": {
"class_type": "EmptyLatentBatch",
"inputs": {"width": 1080, "height": 1920, "batch_size": 4},
},
"40": ksampler(["34", 0], ["33", 0], ["36", 0]),
"41": {
"class_type": "VAEDecode",
"inputs": {"samples": ["40", 0], "vae": ["3", 0]},
},
"42": {
"class_type": "Image Filter",
"inputs": {"timeout": 600, "images": ["41", 0]},
},
"43": {"class_type": "PreviewImage", "inputs": {"images": ["42", 0]}},
"44": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["42", 0], "vae": ["3", 0]},
},
"50": ksampler(["34", 0], ["33", 0], ["44", 0], denoise=0.25),
"51": {
"class_type": "VAEDecode",
"inputs": {"samples": ["50", 0], "vae": ["3", 0]},
},
"52": {
"class_type": "ImageScaleDownBy",
"inputs": {"scale_by": 0.75, "images": ["51", 0]},
},
"53": {"class_type": "PreviewImage", "inputs": {"images": ["51", 0]}},
"54": {
"class_type": "Image Comparer (rgthree)",
"inputs": {"image_a": ["51", 0], "image_b": ["42", 0]},
},
"60": {
"class_type": "UltimateSDUpscale",
"inputs": {
"upscale_by": 2.0,
"seed": 7,
"steps": 4,
"denoise": 0.35,
"image": ["52", 0],
"model": ["1", 0],
"positive": ["35", 0],
"negative": ["33", 0],
"vae": ["3", 0],
},
},
"61": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "flux", "images": ["60", 0]},
},
}
return graph
def edit_batch(count=3):
"""Image-edit samplers with distinct prompts, all saved with one prefix."""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": {"class_type": "LoadImage", "inputs": {"image": "ref.png"}},
"3": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["2", 0], "vae": ["1", 2]},
},
"4": {"class_type": "PrimitiveString", "inputs": {"value": "edit"}},
}
for index in range(count):
base = 100 + index * 10
graph[str(base)] = {
"class_type": "TextEncodeQwenImageEdit",
"inputs": {
"prompt": f"edit number {index}: change the pose",
"clip": ["1", 1],
},
}
graph[str(base + 1)] = ksampler([str(base), 0], [str(base), 0], ["3", 0])
graph[str(base + 2)] = {
"class_type": "VAEDecode",
"inputs": {"samples": [str(base + 1), 0], "vae": ["1", 2]},
}
graph[str(base + 3)] = {
"class_type": "SaveImage",
"inputs": {"filename_prefix": ["4", 0], "images": [str(base + 2), 0]},
}
return graph
def trigger_toggle(words="trig9word"):
"""LoraManager's TriggerWord Toggle: its output is decided at run time."""
return {
"class_type": "TriggerWord Toggle (LoraManager)",
"inputs": {
"group_mode": True,
"default_active": True,
"toggle_trigger_words": {"__value__": [{"text": words, "active": True}]},
"orinalMessage": words,
"trigger_words": ["7", 2],
},
}
def hires_fix_with_trigger_words():
"""``hires_fix`` whose PromptManager prepends LoRA trigger words (node 8)."""
graph = hires_fix()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["1", 1], "text": "<lora:x:1>"},
}
graph["8"] = trigger_toggle()
graph["2"]["inputs"]["prepend_text"] = ["8", 0]
return graph
def prompt_manager_text(text, prepend=""):
"""PromptManagerText: a STRING PromptManager (prepend/main/append)."""
return {
"class_type": "PromptManagerText",
"inputs": {
"text": text,
"category": "",
"tags": "",
"search_text": "",
"prepend_text": prepend,
"append_text": "",
},
}
def trigger_words_through(shape):
"""``hires_fix_with_trigger_words`` with nodes between the toggle and sampler.
``text-encode``: PromptManagerText (prepend <- toggle) -> CLIPTextEncode.
``chained``: a PromptManager whose ``text`` reads that PromptManagerText.
``concat``: PromptManagerText -> StringConcatenate(..., "extra") -> encoder.
"""
graph = hires_fix_with_trigger_words()
graph["9"] = prompt_manager_text(POSITIVE, ["8", 0])
if shape == "chained":
graph["2"]["inputs"].update(prepend_text="", text=["9", 0])
elif shape == "text-encode":
graph["2"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["9", 0]}}
else:
graph["19"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": ["9", 0], "string_b": "extra", "delimiter": ", "},
}
graph["2"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["19", 0]}}
return graph
def branches_with_and_without_trigger_words():
"""Two saved branches: one PromptManager prepends trigger words, one does not."""
graph = hires_fix_with_trigger_words()
graph["9"] = prompt_manager(POSITIVE)
graph["20"]["inputs"].update(positive=["9", 0], latent_image=["4", 0])
return graph
def generated_text_chain():
"""``hires_fix`` whose positive text comes from a run-time text generator."""
graph = hires_fix()
graph["2"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": ["6", 0], "clip": ["1", 1]},
}
graph["6"] = {
"class_type": "TextGenerate",
"inputs": {"prompt": "write a prompt", "model": ["1", 0]},
}
return graph
def copied(graph):
return copy.deepcopy(graph)
+2 -11
View File
@@ -30,11 +30,6 @@ FILES = "media_annotation_files"
BYTES = "media_annotation_bytes"
IMAGE_TAGS = "media_image_tags"
TABLES = (METADATA, FILES, BYTES, IMAGE_TAGS)
REVIEW_TABLES = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
)
# The exact DDL the plan pins; the module must produce precisely these objects.
DDL = {
METADATA: (
@@ -207,14 +202,10 @@ class AnnotationSchemaTests(unittest.TestCase):
)
def drop_group(self, path):
"""Remove the group from a closed database file (an old backup).
The review queue group cites annotation bytes and predates nothing, so
an old backup lacks it as well.
"""
"""Remove the group from a closed database file (an old backup)."""
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
for table in REVIEW_TABLES + tuple(reversed(TABLES)):
for table in reversed(TABLES):
raw.execute(f"DROP TABLE {table}")
def test_fresh_install_and_reopen_is_idempotent(self):
+3 -45
View File
@@ -52,7 +52,6 @@ class AdminAPITestCase(AioHTTPTestCase):
self.outside_dir = Path(self.tmpdir) / "outside"
self.outside_dir.mkdir()
self.addAsyncCleanup(self._drain_and_remove)
self.config_path = os.path.join(self.tmpdir, "config.json")
self._orig_env = {
key: os.environ.pop(key, None) for key in (CONFIG_PATH_ENV, EXTRA_ROOTS_ENV)
@@ -90,26 +89,6 @@ class AdminAPITestCase(AioHTTPTestCase):
sys.modules.pop("folder_paths", None)
else:
sys.modules["folder_paths"] = self._orig_folder_paths
async def _drain_and_remove(self):
"""The last cleanup: no job outlives the temporary database.
Registered first, so it runs after every cleanup a test added (which
may release its own gated jobs). A job still running then, such as a
scan a failed assertion never awaited, is cancelled and awaited
before the database and the tree are removed.
"""
api = getattr(self, "api", None)
jobs = api._jobs.list() if api is not None else []
tasks = [
job.task for job in jobs if job.task is not None and not job.task.done()
]
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
if api is not None:
# A scan of an ambiguous file captures it for Needs review.
api.close_annotation_captures()
PromptModel.close_all_instances() # restore may have swapped api.db
assert_closed(self, self.tmpdir)
shutil.rmtree(self.tmpdir, ignore_errors=True)
@@ -889,23 +868,6 @@ class TestDeleteDuplicatesEndpoint(AdminAPITestCase):
self.assertFalse(target.exists())
self.assertTrue(thumb_dir.exists())
async def test_deleted_files_leave_the_cached_gallery_listing(self):
# The listing is cached for 30 s; a delete must not leave the gallery
# listing (and counting) a file that is gone.
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
for name in ("keep.png", "gone.png"):
(self.output_dir / name).write_bytes(b"same bytes")
async def listed():
resp = await self.client.get("/prompt_manager/images/output")
data = await resp.json()
return data["total"], sorted(i["filename"] for i in data["images"])
self.assertEqual(await listed(), (2, ["gone.png", "keep.png"]))
resp = await self._post({"image_paths": ["gone.png"]})
self.assertEqual((await resp.json())["deleted_count"], 1)
self.assertEqual(await listed(), (1, ["keep.png"]))
def test_unexpected_error_per_file_is_reported_without_path(self):
self.api._delete_one_duplicate = lambda *a: (_ for _ in ()).throw(
PermissionError(13, "Permission denied", os.path.join(self.tmpdir, "x.png"))
@@ -2242,9 +2204,7 @@ class TestActivityJobs(AdminAPITestCase):
for name in names:
(self.output_dir / name).write_bytes(b"not an image")
def fake_ingest(
media_file, metadata, detailed=False, ambiguous=None, untraced=None
):
def fake_ingest(media_file, metadata, detailed=False):
key = Path(media_file).relative_to(self.output_dir).as_posix()
outcome = names[key]
if isinstance(outcome, Exception):
@@ -2373,10 +2333,8 @@ class TestActivityJobs(AdminAPITestCase):
first_done, release = threading.Event(), threading.Event()
original = self.api._ingest_scan_batch_sync
def gated(batch_results, roots=(), outcomes=None, review=None):
counts = original(
batch_results, roots=roots, outcomes=outcomes, review=review
)
def gated(batch_results, roots=(), outcomes=None):
counts = original(batch_results, roots=roots, outcomes=outcomes)
first_done.set()
release.wait(5)
return counts
+1 -9
View File
@@ -1045,15 +1045,7 @@ class TestFailures(AnnotationAPITestCase):
await self.api.cleanup_annotation_captures(None)
self.assertFalse(os.path.exists(directory))
self.assertNotIn(identity, observers._restore_observers)
# The shutdown cleanup is final: a later observation (or a monitor
# recording still in flight) creates no new store or directory.
status, body = await self.observe({"target": self.target("sub/a.png")})
self.assertEqual((status, body["code"]), (500, "capture-error"), body)
self.assertIsNone(self.api._captures)
self.assertNotIn(identity, observers._restore_observers)
# An explicit, non-final close still lets a later observation start
# a fresh store (the tests' own store swaps rely on it).
self.api._captures_closed = False
# A later observation starts a fresh store.
await self.observe_ok()
self.assertNotEqual(self.api._captures.directory, directory)
-13
View File
@@ -471,19 +471,6 @@ class TestUiRoutes(CoreRoutesTestCase):
self.assertIn(("POST", "/prompt_manager/jobs/scan"), table)
self.assertIn(("POST", "/prompt_manager/jobs/image-tags"), table)
async def test_review_routes_are_registered(self):
table = {
(route.method, route.resource.canonical)
for route in self.app.router.routes()
if route.resource is not None
}
for entry in (
("GET", "/prompt_manager/review/items"),
("GET", "/prompt_manager/review/items/{item_id}"),
("POST", "/prompt_manager/review/items/{item_id}/decision"),
):
self.assertIn(entry, table)
class TestStaticRoutes(CoreRoutesTestCase):
File diff suppressed because it is too large Load Diff
-364
View File
@@ -1,364 +0,0 @@
"""Needs review change notices (#249 PR F2).
``GET /prompt_manager/review/events`` is a server-sent event stream: one
``ready`` event when it opens, then one ``change`` event after every
committed queue change (an item recorded by the scan or the monitor, a
spelling refresh, a decision from any tab) and after a restore. An open
Needs review page re-reads its queue on ``change``; nothing polls. A refused
decision and a duplicate recording change nothing and send nothing.
"""
import asyncio
import json
import sqlite3
import threading
try:
from tests.test_api_review import ReviewRoutesTestCase
from tests.test_review import unlinkable_graph, write_png
except ImportError: # discovered with tests/ as the top-level directory
from test_api_review import ReviewRoutesTestCase
from test_review import unlinkable_graph, write_png
try: # guarded so a missing module fails assertions, not test loading
from database import review_changes
except ImportError:
review_changes = None
try:
from py.api import review_events
except ImportError:
review_events = None
EVENTS_URL = "/prompt_manager/review/events"
# A safety bound for one read; every wait below ends on an event first.
READ_SECONDS = 10
class ReviewEventsTestCase(ReviewRoutesTestCase):
def _require_events(self):
self.assertIsNotNone(review_changes, "database.review_changes is missing")
self.assertIsNotNone(review_events, "py.api.review_events is missing")
async def _stream(self):
resp = await self.client.get(EVENTS_URL)
self.assertEqual(resp.status, 200)
self.assertEqual(resp.headers["Content-Type"], "text/event-stream")
self.assertEqual(resp.headers["Cache-Control"], "no-store")
self.addAsyncCleanup(self._close, resp)
self.assertEqual(await self._event(resp), "ready")
return resp
@staticmethod
async def _close(resp):
resp.close()
async def _event(self, resp):
"""The next event's name (keepalive comments are skipped)."""
return (await self._event_with_data(resp))[0]
async def _event_with_data(self, resp):
"""The next event's name and its parsed ``data`` line."""
name, data = None, None
while True:
line = await asyncio.wait_for(resp.content.readline(), READ_SECONDS)
self.assertTrue(line, "the stream ended")
text = line.decode("utf-8").rstrip("\n")
if text.startswith("event: "):
name = text[len("event: ") :]
elif text.startswith("data: "):
data = json.loads(text[len("data: ") :])
elif text == "" and name is not None:
return name, data
async def _ready_version(self):
resp = await self.client.get(EVENTS_URL)
self.assertEqual(resp.status, 200)
self.addAsyncCleanup(self._close, resp)
name, data = await self._event_with_data(resp)
self.assertEqual(name, "ready")
return data
def _queueable(self, name="two.png", color=(255, 0, 0)):
"""A file the producers queue: ambiguous, and nothing links it."""
path = self.output_dir / name
path.parent.mkdir(parents=True, exist_ok=True)
write_png(path, prompt=unlinkable_graph("cat", "dog"), color=color)
return path
async def _record_in_thread(self, path):
"""Record as the monitor does: from a worker thread, not the loop."""
self._roots()
outcomes = []
thread = threading.Thread(
target=lambda: outcomes.append(self.api.record_review_file(str(path)))
)
thread.start()
await asyncio.get_running_loop().run_in_executor(None, thread.join)
return outcomes[0]
class TestReviewEvents(ReviewEventsTestCase):
async def test_monitor_recording_sends_one_change(self):
self._require_events()
stream = await self._stream()
self.assertEqual(await self._record_in_thread(self._queueable()), "recorded")
self.assertEqual(await self._event(stream), "change")
async def test_duplicate_recording_and_refused_decision_send_nothing(self):
# A spy on the notice itself: a wrong notice would otherwise pass as
# the next real change's event.
self._require_events()
from unittest.mock import patch
from database import review as review_db
from database import review_decisions
calls = []
original = review_changes.notify_review_change
def spy(db_path):
calls.append(db_path)
original(db_path)
path = self._queueable("two.png")
self._record(path)
with (
patch.object(review_db, "notify_review_change", spy),
patch.object(review_decisions, "notify_review_change", spy),
):
# The same bytes again: nothing changes.
self.assertEqual(await self._record_in_thread(path), "duplicate")
self.assertEqual(calls, [], "a duplicate recording sent a notice")
item, detail, observed = await self._open()
# A stale revision: refused with zero writes.
body = {
**self._body(detail, observed, "unresolved"),
"expected_revision": 9,
}
status, data = await self._decide(item["id"], body)
self.assertEqual(status, 409, data)
self.assertEqual(calls, [], "a refused decision sent a notice")
self._record(self._ambiguous("other.png", color=(0, 0, 9)))
self.assertEqual(len(calls), 1, "a real recording sends one notice")
self.assertEqual(len(self._items()), 2)
async def test_a_burst_of_changes_coalesces_into_one_more_event(self):
# M9: a busy scan records file after file; an open page hears the
# first at once and the rest as one trailing ``change``.
self._require_events()
from unittest.mock import patch
path = self.api.db.model.db_path
with (
patch.object(review_events, "COALESCE_SECONDS", 0.3),
patch.object(review_events, "HEARTBEAT_SECONDS", 0.6),
):
stream = await self._stream()
review_changes.notify_review_change(path)
self.assertEqual(await self._event(stream), "change")
for _ in range(5):
review_changes.notify_review_change(path)
await asyncio.sleep(0.02)
# Every change event up to the next keepalive: exactly one.
changes = 0
while True:
line = await asyncio.wait_for(stream.content.readline(), READ_SECONDS)
self.assertTrue(line, "the stream ended")
if line == b": keepalive\n":
break
changes += line == b"event: change\n"
self.assertEqual(changes, 1)
async def test_decision_from_any_tab_sends_a_change(self):
self._require_events()
self._record(self._ambiguous())
stream = await self._stream()
item, detail, observed = await self._open()
status, data = await self._decide(
item["id"], self._body(detail, observed, "unresolved")
)
self.assertEqual(status, 200, data)
self.assertEqual(await self._event(stream), "change")
async def test_restore_sends_a_change(self):
self._require_events()
stream = await self._stream()
from database.execution_history import notify_history_restore
notify_history_restore(self.api.db.model.db_path)
self.assertEqual(await self._event(stream), "change")
async def test_scan_recording_sends_a_change(self):
self._require_events()
self._roots()
self._queueable("sub/two.png")
stream = await self._stream()
job = await self._scan("all")
self.assertEqual(job["scope"]["review_items"], 1)
self.assertEqual(await self._event(stream), "change")
async def test_a_linked_file_leaving_the_queue_sends_a_change(self):
"""The queue heals (#284 maintainer test): a pending item whose file is
linked now leaves the queue, and the open page hears about it."""
self._require_events()
self._roots()
path = self._ambiguous("two.png")
self._record(path) # as the pre-fix scan queued fallback-linked files
stream = await self._stream()
job = await self._scan("new")
self.assertEqual(job["scope"]["review_settled"], 1)
self.assertEqual(self._items(), [])
self.assertEqual(await self._event(stream), "change")
async def test_ready_carries_the_version_a_list_read_answers(self):
# F3 (no double read per visit): the page skips the re-read on
# ``ready`` when its first read already answered this version.
self._require_events()
self._record(self._ambiguous())
page = await self._list()
self.assertIsInstance(page.get("review_revision"), int, page)
version = await self._ready_version()
self.assertEqual(
version,
{
"library_epoch": page["library_epoch"],
"review_revision": page["review_revision"],
},
)
async def test_a_queue_change_moves_the_version_ready_carries(self):
self._require_events()
before = await self._list()
self._record(self._ambiguous())
after = await self._list()
self.assertIn("review_revision", after)
self.assertGreater(after["review_revision"], before["review_revision"])
version = await self._ready_version()
self.assertEqual(version["review_revision"], after["review_revision"])
self.assertNotEqual(version["review_revision"], before["review_revision"])
async def test_ready_names_no_version_when_it_cannot_be_read(self):
# The page then re-reads, as it did before versions existed.
self._require_events()
def broken():
raise sqlite3.DatabaseError("locked")
self.api.db.review_queue_version = broken
self.assertEqual(await self._ready_version(), {})
async def test_a_stream_counts_as_open_once_it_has_sent_ready(self):
# The browser fixture waits on this count before it changes the
# queue, so a change after it is a ``change`` event and never part of
# the version ``ready`` named.
self._require_events()
reading, release = asyncio.Event(), asyncio.Event()
original = self.api._queue_version
async def held():
reading.set()
await release.wait()
return await original()
self.api._queue_version = held
opening = asyncio.ensure_future(self._ready_version())
await asyncio.wait_for(reading.wait(), READ_SECONDS)
self.assertEqual(self.api.review_stream_count(), 0)
release.set()
await asyncio.wait_for(opening, READ_SECONDS)
self.assertEqual(self.api.review_stream_count(), 1)
async def test_change_events_carry_no_data(self):
self._require_events()
stream = await self._stream()
self._record(self._ambiguous())
self.assertEqual(await self._event_with_data(stream), ("change", {}))
async def test_closing_the_stream_unsubscribes(self):
self._require_events()
path = self.api.db.model.db_path
stream = await self._stream()
self.assertEqual(review_changes.observer_count(path), 1)
self.assertEqual(self.api.review_stream_count(), 1)
stream.close()
# The server notices the closed socket at its next write: a change.
self._record(self._ambiguous())
for _ in range(100):
if not self.api.review_stream_count():
break
await asyncio.sleep(0.02)
self.assertEqual(self.api.review_stream_count(), 0)
self.assertEqual(review_changes.observer_count(path), 0)
async def test_shutdown_ends_every_stream(self):
self._require_events()
stream = await self._stream()
await self.api.close_review_events(None)
line = await asyncio.wait_for(stream.content.read(), READ_SECONDS)
self.assertEqual(line, b"")
self.assertEqual(self.api.review_stream_count(), 0)
# A stream opened after shutdown ends at once.
resp = await self.client.get(EVENTS_URL)
self.assertEqual(resp.status, 503)
resp.close()
async def test_keepalive_comment_is_skipped_by_clients(self):
self._require_events()
with self.patch_heartbeat(0.05):
stream = await self._stream()
line = await asyncio.wait_for(stream.content.readline(), READ_SECONDS)
self.assertEqual(line, b": keepalive\n")
def patch_heartbeat(self, seconds):
from unittest.mock import patch
return patch.object(review_events, "HEARTBEAT_SECONDS", seconds)
class TestReviewChangeRegistry(ReviewEventsTestCase):
def test_a_failing_observer_never_fails_a_write(self):
self._require_events()
path = self.api.db.model.db_path
seen = []
def broken():
raise RuntimeError("observer failed")
unsubscribe_broken = review_changes.subscribe_review_changes(path, broken)
unsubscribe = review_changes.subscribe_review_changes(
path, lambda: seen.append(1)
)
try:
with self.assertLogs(
"prompt_manager.database.review_changes", level="WARNING"
):
self._record(self._ambiguous())
self.assertEqual(seen, [1])
finally:
unsubscribe_broken()
unsubscribe()
self.assertEqual(review_changes.observer_count(path), 0)
def test_notice_comes_after_the_commit(self):
self._require_events()
path = self.api.db.model.db_path
seen = []
def committed():
# A separate connection sees only committed rows.
other = sqlite3.connect(path)
try:
seen.append(
other.execute("SELECT COUNT(*) FROM media_review_items").fetchone()[
0
]
)
finally:
other.close()
unsubscribe = review_changes.subscribe_review_changes(path, committed)
try:
self._record(self._ambiguous())
finally:
unsubscribe()
self.assertEqual(seen, [1])
-234
View File
@@ -1,234 +0,0 @@
"""A slow page-workflow reset reports where the server waited.
CI run 39904288262 had one ``/__test__/reset`` wait 16 s with nothing in the
server's stderr. tests/browser/slow_reset_watch.py reports any reset past
SLOW_RESET_SECONDS with every thread's stack, taken while it still waits.
These tests hold a real SQLite write lock behind a real test server's reset.
"""
import glob
import io
import os
import sqlite3
import subprocess
import sys
import tempfile
import threading
import time
import unittest
import urllib.request
from pathlib import Path
from tests.browser import slow_reset_watch
from tests.browser.slow_reset_watch import REPORT_PREFIX, ResetWatch, is_reset_path
REPO_ROOT = Path(__file__).resolve().parent.parent
SERVER_TIMEOUT_S = 60
HOLD_SECONDS = slow_reset_watch.SLOW_RESET_SECONDS + 4
# The reset may take a little longer than the hold, never the 30 s busy timeout.
RESET_MARGIN_SECONDS = 10
# While the reset waits, the server still answers other requests this fast.
# Reading the waiting connection from the report's thread froze it (the GIL is
# held while SQLite's connection mutex is awaited) until the lock was released.
RESPONSIVE_SECONDS = 2.0
PROBE_INTERVAL_SECONDS = 0.1
def blocked_here(gate: threading.Event) -> None:
gate.wait()
class TestResetWatch(unittest.TestCase):
def test_reset_paths_are_the_test_reset_routes(self):
self.assertTrue(is_reset_path("/__test__/reset"))
self.assertTrue(is_reset_path("/__test__/tagging/reset"))
self.assertFalse(is_reset_path("/prompt_manager/reset"))
self.assertFalse(is_reset_path("/__test__/reset/extra"))
def test_a_stalled_reset_reports_state_and_stacks_while_it_waits(self):
stream = io.StringIO()
reported = threading.Event()
class Stream(io.StringIO):
def flush(self):
reported.set()
stream = Stream()
gate = threading.Event()
worker = threading.Thread(target=blocked_here, args=(gate,), name="held-up")
worker.start()
self.addCleanup(worker.join)
self.addCleanup(gate.set)
watch = ResetWatch(
"POST /__test__/reset",
lambda: ["database: 1 connections"],
threshold=0.05,
stream=stream,
).start()
self.assertTrue(reported.wait(SERVER_TIMEOUT_S))
report = stream.getvalue().splitlines()
self.assertTrue(all(line.startswith(REPORT_PREFIX) for line in report))
text = "\n".join(report)
self.assertIn("POST /__test__/reset still running after", text)
self.assertIn("database: 1 connections", text)
self.assertIn(f"thread 0x{worker.ident:016x} is held-up", text)
self.assertIn("in blocked_here", text)
watch.record_call("seed", 0.25, 1.5)
watch.finish("answered 200")
self.assertIn("POST /__test__/reset answered 200 after", stream.getvalue())
self.assertIn(
"executor call seed: queued 0.250s, ran 1.500s", stream.getvalue()
)
def test_a_fast_reset_writes_nothing_and_stops_its_timer(self):
stream = io.StringIO()
watch = ResetWatch("reset", lambda: [], threshold=60, stream=stream).start()
watch.finish("answered 200")
self.assertTrue(watch._timer.finished.is_set())
self.assertEqual(stream.getvalue(), "")
def test_a_failing_state_probe_still_reports_the_stacks(self):
def broken():
raise RuntimeError("no state")
reported = threading.Event()
class Stream(io.StringIO):
def flush(self):
reported.set()
stream = Stream()
ResetWatch("reset", broken, threshold=0.01, stream=stream).start()
self.assertTrue(reported.wait(SERVER_TIMEOUT_S))
self.assertIn("state unavailable: RuntimeError('no state')", stream.getvalue())
self.assertIn("most recent call first", stream.getvalue())
class TestSlowResetOnTheServer(unittest.TestCase):
"""A reset behind another process's write lock, on tests/browser/server.py."""
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
env = {**os.environ, "TMPDIR": self.tmp.name}
env.update(TEMP=self.tmp.name, TMP=self.tmp.name)
self.stderr = tempfile.TemporaryFile()
self.addCleanup(self.stderr.close)
process = subprocess.Popen(
[sys.executable, "-m", "tests.browser.server", "--port", "0"],
cwd=REPO_ROOT,
env=env,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=self.stderr,
)
def stop():
if process.poll() is None:
process.kill()
process.wait(timeout=SERVER_TIMEOUT_S)
for stream in (process.stdin, process.stdout):
stream.close()
self.addCleanup(stop)
line = process.stdout.readline().decode()
self.assertTrue(line.startswith("PM_TEST_SERVER_PORT="), line)
self.base = f"http://127.0.0.1:{int(line.split('=', 1)[1])}"
deadline = time.monotonic() + SERVER_TIMEOUT_S
while True:
try:
with urllib.request.urlopen( # nosec B310
self.base + "/prompt_manager/test", timeout=1
):
break
except OSError:
self.assertIsNone(process.poll(), "the server exited early")
self.assertLess(time.monotonic(), deadline, "never ready")
time.sleep(0.1)
[self.db_path] = glob.glob(
os.path.join(self.tmp.name, "pm-browser-*", "browser.db")
)
def reset(self) -> tuple[int, float]:
request = urllib.request.Request(
self.base + "/__test__/reset",
data=b"{}",
headers={"Content-Type": "application/json"},
method="POST",
)
start = time.monotonic()
with urllib.request.urlopen( # nosec B310
request, timeout=SERVER_TIMEOUT_S
) as response:
return response.status, time.monotonic() - start
def server_stderr(self) -> str:
self.stderr.seek(0)
return self.stderr.read().decode("utf-8", "replace")
def test_a_reset_held_behind_a_write_lock_names_the_wait_and_ends_with_it(self):
self.assertEqual(self.reset()[0], 200, self.server_stderr())
self.assertNotIn(REPORT_PREFIX, self.server_stderr())
held = threading.Event()
def hold_write_lock():
holder = sqlite3.connect(self.db_path, timeout=SERVER_TIMEOUT_S)
try:
holder.execute("BEGIN IMMEDIATE")
held.set()
time.sleep(HOLD_SECONDS)
holder.rollback()
finally:
holder.close()
lock = threading.Thread(target=hold_write_lock)
lock.start()
self.addCleanup(lock.join)
self.assertTrue(held.wait(SERVER_TIMEOUT_S))
outcome = []
resetting = threading.Thread(target=lambda: outcome.append(self.reset()))
resetting.start()
self.addCleanup(resetting.join)
slowest = 0.0
while resetting.is_alive():
start = time.monotonic()
with urllib.request.urlopen( # nosec B310
self.base + "/prompt_manager/test", timeout=SERVER_TIMEOUT_S
):
slowest = max(slowest, time.monotonic() - start)
time.sleep(PROBE_INTERVAL_SECONDS)
resetting.join()
lock.join()
self.assertEqual(len(outcome), 1, self.server_stderr())
[(status, elapsed)] = outcome
self.assertLess(slowest, RESPONSIVE_SECONDS)
self.assertEqual(status, 200)
self.assertGreaterEqual(elapsed, HOLD_SECONDS - 0.5)
self.assertLess(elapsed, HOLD_SECONDS + RESET_MARGIN_SECONDS)
report = [
line[len(REPORT_PREFIX) :]
for line in self.server_stderr().splitlines()
if line.startswith(REPORT_PREFIX)
]
text = "\n".join(report)
self.assertIn("POST /__test__/reset still running after", text)
self.assertIn("default executor:", text)
if sys.platform.startswith("linux"):
# SQLite's busy handler sleeps between its retries.
self.assertRegex(
text, r"kernel: asyncio_\d+ \(0x[0-9a-f]{16}\) state \w, waiting in "
)
# The plain reset's seed() waits on its first DELETE.
self.assertRegex(text, r"line \d+ in seed\n")
self.assertIn("POST /__test__/reset answered 200 after", text)
self.assertRegex(
text,
r"executor call create_app.<locals>.seed: queued \d+\.\d+s, ran \d+\.\d+s",
)
if __name__ == "__main__":
unittest.main()
-57
View File
@@ -68,55 +68,6 @@ class CardSeed(unittest.TestCase):
self.assertIn(expected, (111, 222))
self.assertEqual(info_seed(info), str(expected))
def test_a_chain_the_inspector_names_one_stage_of_shows_no_differing_seed(self):
# The inspector shows the named final stage's seed (D19, pass-through
# chains); the parser shows another stage's. Never two seeds: none.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["21"] = {
"class_type": "FaceDetailer",
"inputs": {
"seed": 333,
"image": ["20", 0],
"model": ["4", 0],
"positive": ["6", 0],
"negative": ["7", 0],
},
}
info = graph(seeded(111), extra=extra)
self.assertEqual(extract_from_records(info["prompt"], None)["seed"], 111)
self.assertIsNone(info_seed(info))
# The same chain on one seed keeps it; so does a refiner over a decode.
extra["21"]["inputs"]["seed"] = 111
self.assertEqual(info_seed(graph(seeded(111), extra=extra)), "111")
refiner = seeded(222, latent_image=["22", 0])
extra["22"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["20", 0]}}
del extra["21"]
self.assertIsNone(info_seed(graph(seeded(111), refiner, extra=extra)))
refiner["inputs"]["seed"] = 111
self.assertEqual(info_seed(graph(seeded(111), refiner, extra=extra)), "111")
def test_a_seed_that_agrees_with_the_named_stage_is_kept(self):
# Review I3: a saved sampler (seed 7) beside a preview-only sampler
# (seed 5): the parser and the inspector both show 7, as on main.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["21"] = {"class_type": "SaveImage", "inputs": {"images": ["20", 0]}}
extra["30"] = {"class_type": "VAEDecode", "inputs": {"samples": ["11", 0]}}
extra["31"] = {"class_type": "PreviewImage", "inputs": {"images": ["30", 0]}}
info = graph(seeded(7), seeded(5), extra=extra)
self.assertEqual(info_seed(info), "7")
def test_a_named_ambiguous_chain_keeps_the_parser_seed(self):
# Review M2: multistage-differs names a sampler, but its facts merge,
# and the inspector fills the parser's seed: the card keeps it.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["8"] = encoder("another scene")
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["22"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["20", 0]}}
refiner = ksampler(("8", 0), ("7", 0), seed=222, latent_image=["22", 0])
self.assertEqual(info_seed(graph(seeded(111), refiner, extra=extra)), "111")
def test_card_seed_linked_input_follows_shared_parser(self):
primitive = {"class_type": "PrimitiveNode", "inputs": {"value": 555}}
info = graph(seeded(["20", 0]), extra={"20": primitive})
@@ -256,7 +207,6 @@ class CardSeed(unittest.TestCase):
self.assertIsNone(parsed_positive(unparsable))
def test_every_shared_case_reads_the_projected_graphs(self):
withheld = []
for name, case in CASES.items():
with self.subTest(case=name):
info = {k: v for k, v in case.items() if v is not None}
@@ -275,18 +225,11 @@ class CardSeed(unittest.TestCase):
except GenerationParamsError:
self.assertIsNone(params)
continue
if name == "shared-prompt-pipeline":
# The inspector names the detailer (seed 333); the parser
# reads the base sampler (111): the card shows none.
self.assertEqual(expected["seed"], 111)
expected = {**expected, "seed": None}
withheld.append(name)
self.assertEqual(params, expected)
# Prompt text is the same as from a plain parse of the records.
if len(values) == len(info):
raw = extract_from_records(case["prompt"], case["workflow"])
self.assertEqual(params["positivePrompt"], raw["positivePrompt"])
self.assertEqual(withheld, ["shared-prompt-pipeline"])
if __name__ == "__main__":
+5 -84
View File
@@ -24,9 +24,6 @@ from database.operations import PromptDatabase
from tests.open_handles import assert_closed
from utils.hashing import generate_prompt_hash
# A bound on thread hand-offs that only a hung test reaches (never a pacing timer).
THREAD_WAIT_SECONDS = 120
def _count_prompts(path):
"""Row count read through a brand-new connection (closed before return)."""
@@ -353,13 +350,7 @@ class TestRestoreFromFile(BackupTestCase):
def worker():
self.db.search_prompts(text="old") # opens this thread's connection
ready.set()
# Wait for the restore itself, not for a timer: a stalled Windows
# runner once took longer than 5 s to restore, and the worker then
# read the old database before the restore had run.
if not proceed.wait(THREAD_WAIT_SECONDS):
results["error"] = "the restore did not finish"
self.db.close()
return
proceed.wait(5)
try:
results["texts"] = [p["text"] for p in self.db.search_prompts()]
except Exception as exc: # noqa: BLE001 - reported through results
@@ -369,15 +360,12 @@ class TestRestoreFromFile(BackupTestCase):
thread = threading.Thread(target=worker)
thread.start()
self.assertTrue(ready.wait(THREAD_WAIT_SECONDS))
ready.wait(5)
try:
self.db.model.restore_from_file(src)
finally:
proceed.set()
thread.join(THREAD_WAIT_SECONDS)
self.db.model.restore_from_file(src)
proceed.set()
thread.join(5)
self.assertFalse(thread.is_alive())
self.assertNotIn("error", results)
self.assertEqual(results["texts"], ["restored"])
self.assertEqual([p["text"] for p in self.db.search_prompts()], ["restored"])
@@ -466,9 +454,6 @@ class TestRestoreFromFile(BackupTestCase):
def test_restore_old_backup_without_annotations(self):
tables = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -505,70 +490,6 @@ class TestRestoreFromFile(BackupTestCase):
0,
)
def test_restore_annotations_era_backup_without_review_queue(self):
tables = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
)
src = self._make_source_db(["predates the review queue"])
with closing(sqlite3.connect(src)) as conn:
for table in tables:
conn.execute(f"DROP TABLE {table}")
conn.commit()
self.assertTrue(PromptModel.verify_database_file(src)[0])
ok, reason = PromptModel.verify_database_file(src, require_library_state=True)
self.assertFalse(ok)
self.assertEqual(reason, "Missing review queue schema")
self.db.model.restore_from_file(src)
self.assertEqual(_texts(self.path), ["predates the review queue"])
with closing(sqlite3.connect(self.path)) as conn:
self.assertEqual(
conn.execute("SELECT * FROM media_review_metadata").fetchall(),
[(1, 1, 0)],
)
self.assertEqual(
conn.execute("SELECT COUNT(*) FROM media_review_items").fetchone()[0],
0,
)
ok, reason = PromptModel.verify_database_file(
self.path, require_library_state=True
)
self.assertTrue(ok, reason)
with closing(sqlite3.connect(src)) as conn: # the source is untouched
self.assertEqual(
conn.execute(
"SELECT COUNT(*) FROM sqlite_master WHERE name LIKE 'media_review%'"
).fetchone()[0],
0,
)
def test_restore_rejects_review_queue_without_annotations(self):
src = self._make_source_db(["review without annotations"])
with closing(sqlite3.connect(src)) as conn:
conn.execute("PRAGMA foreign_keys=OFF")
for table in (
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
"media_annotation_metadata",
):
conn.execute(f"DROP TABLE {table}")
conn.commit()
before = Path(src).read_bytes()
live_before = _texts(self.path)
ok, reason = PromptModel.verify_database_file(src)
self.assertFalse(ok)
self.assertIn("image annotation", reason)
with self.assertRaises(ValueError):
self.db.model.restore_from_file(src)
self.assertEqual(Path(src).read_bytes(), before)
self.assertEqual(_texts(self.path), live_before)
def test_restore_preserves_favorites_and_closes_handles(self):
self.db.set_image_favorite("output", "live.png")
with tempfile.TemporaryDirectory() as directory:
-614
View File
@@ -1,614 +0,0 @@
"""The read-only health check (System › Overview and Maintenance).
``run_health_checks`` reads one snapshot on its own ``mode=ro`` connection:
SQLite's integrity and foreign-key checks, stored prompt hashes against
``utils.hashing``, duplicate pair rows, the legacy consistency queries, image
links whose file is gone (only under readable sources) and the Needs-review
count when one is supplied. It never writes, and it never turns a check it
could not run into a count.
"""
import hashlib
import ntpath
import os
import sqlite3
import sys
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from database import health # noqa: E402
from database.operations import PromptDatabase # noqa: E402
from utils.hashing import generate_prompt_hash # noqa: E402
try:
from tests.open_handles import assert_closed # noqa: E402
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed # noqa: E402
CHECK_IDS = [
"integrity",
"foreign_keys",
"canonical_hashes",
"pair_uniqueness",
"missing_files",
"ambiguous_metadata",
"consistency",
]
def _digest(path):
if not os.path.exists(path):
return None
with open(path, "rb") as handle:
return hashlib.sha256(handle.read()).hexdigest()
class HealthTestCase(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.root = Path(os.path.realpath(self.temp.name))
self.output = self.root / "output"
self.output.mkdir()
self.db_path = str(self.root / "prompts.db")
self.db = PromptDatabase(self.db_path)
def tearDown(self):
self.db.close_all()
assert_closed(self, self.root)
self.temp.cleanup()
def save(self, text, **metadata):
return self.db.save_prompt(
text, prompt_hash=generate_prompt_hash(text), **metadata
)
def image(self, prompt_id, name, *, exists=True, folder=None):
folder = Path(folder or self.output)
path = folder / name
if exists:
path.write_bytes(b"png")
return self.db.link_image_to_prompt(prompt_id, str(path))
def run_checks(self, roots=None, **kwargs):
roots = [str(self.output)] if roots is None else roots
return self.db.run_health_checks(roots, **kwargs)
@staticmethod
def check(report, check_id):
return next(c for c in report["checks"] if c["id"] == check_id)
def raw(self):
"""A plain connection for planting damage the app itself never writes."""
conn = sqlite3.connect(self.db_path)
conn.execute("PRAGMA foreign_keys=OFF")
return conn
class TestHealthReport(HealthTestCase):
def test_clean_library_passes_except_unavailable_review(self):
prompt = self.save("A clean prompt")
self.db.save_prompt_pair("Cat", "Blur")
self.image(prompt, "a.png")
report = self.run_checks(review_count=0)
self.assertTrue(report["success"])
self.assertEqual([c["id"] for c in report["checks"]], CHECK_IDS)
self.assertEqual(
[c["label"] for c in report["checks"]],
[
"Integrity check",
"Foreign keys & pair references",
"Canonical hashes",
"Pair uniqueness",
"Missing files",
"Ambiguous metadata",
"Consistency",
],
)
for check in report["checks"]:
self.assertEqual(check["status"], "passed", check)
self.assertEqual(check["count"], 0, check)
self.assertEqual(report["status"], "passed")
self.assertRegex(report["checked_at"], r"^\d{4}-\d\d-\d\dT.*\+00:00$")
def test_health_check_never_writes(self):
prompt = self.save("Untouched")
self.image(prompt, "kept.png")
self.image(prompt, "gone.png", exists=False)
with self.raw() as conn:
conn.execute("UPDATE prompts SET hash='stale' WHERE id=?", (prompt,))
conn.close()
# The app's own connection stays open, as in the running server.
self.db.get_statistics()
files = [self.db_path, self.db_path + "-wal"]
before = [_digest(path) for path in files]
shm_size = os.path.getsize(self.db_path + "-shm")
media = sorted(os.listdir(self.output))
opened = []
real_connect = sqlite3.connect
class Recording(sqlite3.Connection):
changes_at_close = None
def close(self):
Recording.changes_at_close = self.total_changes
super().close()
def connect(*args, **kwargs):
conn = real_connect(*args, factory=Recording, **kwargs)
opened.append((args, kwargs, conn))
return conn
with patch.object(health.sqlite3, "connect", side_effect=connect):
report = self.run_checks()
self.assertEqual(report["status"], "incomplete")
self.assertEqual(len(opened), 1)
args, kwargs, conn = opened[0]
self.assertTrue(str(args[0]).endswith("?mode=ro"), args)
self.assertTrue(kwargs.get("uri"))
self.assertEqual(Recording.changes_at_close, 0)
with self.assertRaises(sqlite3.ProgrammingError): # closed in finally
conn.execute("SELECT 1")
self.assertEqual([_digest(path) for path in files], before)
# The -shm holds WAL read marks every reader updates; it never grows.
self.assertEqual(os.path.getsize(self.db_path + "-shm"), shm_size)
self.assertEqual(sorted(os.listdir(self.output)), media)
with self.db.model.get_connection() as live:
stored = live.execute(
"SELECT hash FROM prompts WHERE id=?", (prompt,)
).fetchone()[0]
self.assertEqual(stored, "stale")
def test_connection_closed_when_a_check_raises(self):
self.save("Any")
with (
patch.object(
health,
"_canonical_hashes",
side_effect=sqlite3.OperationalError("boom"),
),
self.assertLogs("prompt_manager.database.health", "ERROR"),
):
report = self.run_checks(review_count=0)
canonical = self.check(report, "canonical_hashes")
self.assertEqual(canonical["status"], "failed")
self.assertIsNone(canonical["count"])
self.assertEqual(canonical["detail"], ["The check could not finish."])
self.assertEqual(self.check(report, "integrity")["status"], "passed")
self.assertEqual(report["status"], "incomplete")
self.db.close_all()
assert_closed(self, self.root)
def test_unopenable_database_fails_every_database_check(self):
missing = PromptDatabase.__new__(PromptDatabase)
missing.model = type("M", (), {"db_path": str(self.root / "nope.db")})()
with self.assertLogs("prompt_manager.database.health", "ERROR"):
report = missing.run_health_checks([str(self.output)], review_count=0)
statuses = {c["id"]: c["status"] for c in report["checks"]}
self.assertEqual(statuses["integrity"], "failed")
self.assertEqual(statuses["missing_files"], "failed")
self.assertEqual(statuses["ambiguous_metadata"], "passed")
self.assertEqual(report["status"], "incomplete")
self.assertFalse(os.path.exists(self.root / "nope.db"))
def test_integrity_and_foreign_key_issues_reported(self):
prompt = self.save("Linked")
with self.raw() as conn:
conn.execute("CREATE TABLE probe(a)")
conn.execute("CREATE INDEX probe_a ON probe(a)")
conn.executemany("INSERT INTO probe VALUES(?)", [(i,) for i in range(16)])
conn.execute(
"INSERT INTO generated_images(prompt_id,image_path,filename,file_path)"
" VALUES(?,?,?,?)",
(9999, str(self.output / "orphan.png"), "orphan.png", "orphan"),
)
conn.execute(
"INSERT INTO prompt_tags(prompt_id,tag_id) VALUES(?,?)", (prompt, 777)
)
conn.close()
with self.raw() as conn:
conn.execute("PRAGMA writable_schema=ON")
conn.execute(
"UPDATE sqlite_master SET sql='CREATE INDEX probe_a ON probe(a DESC)'"
" WHERE name='probe_a'"
)
conn.close()
self.db.close_all() # the next read sees the edited schema
(self.output / "orphan.png").write_bytes(b"png")
with self.raw() as conn:
expected = [r[0] for r in conn.execute("PRAGMA integrity_check(100)")]
conn.close()
self.assertGreater(len(expected), 10)
report = self.run_checks(review_count=0)
integrity = self.check(report, "integrity")
self.assertEqual(integrity["status"], "issues")
self.assertEqual(integrity["count"], len(expected))
self.assertEqual(integrity["detail"], expected[:10])
self.assertIn("missing from index probe_a", integrity["detail"][0])
foreign = self.check(report, "foreign_keys")
self.assertEqual(foreign["status"], "issues")
self.assertEqual(foreign["count"], 2)
self.assertEqual(
sorted(foreign["detail"]),
[
"generated_images row 1 references a missing prompts row",
"prompt_tags row 1 references a missing tags row",
],
)
consistency = self.check(report, "consistency")
self.assertEqual(consistency["status"], "issues")
self.assertEqual(consistency["count"], 1)
self.assertEqual(
consistency["detail"], ["Image 1 references non-existent prompt 9999"]
)
self.assertEqual(report["status"], "issues")
def test_canonical_hash_mismatch_counts_rows(self):
ids = [self.save(f"Prompt {n}") for n in range(14)]
with self.raw() as conn:
for n, prompt_id in enumerate(ids[:12]):
conn.execute(
"UPDATE prompts SET hash=? WHERE id=?", (f"bad{n}", prompt_id)
)
conn.execute("UPDATE prompts SET hash=NULL WHERE id=?", (ids[12],))
conn.close()
# Normalised spellings hash alike: this one is not a mismatch.
self.save(" MIXED Case ")
report = self.run_checks(review_count=0)
canonical = self.check(report, "canonical_hashes")
self.assertEqual(canonical["status"], "issues")
self.assertEqual(canonical["count"], 13)
self.assertEqual(len(canonical["detail"]), 10)
self.assertEqual(
canonical["detail"][0],
f"Prompt {ids[0]}: the stored hash does not match its text",
)
def test_duplicate_pair_rows_reported(self):
pair = self.db.save_prompt_pair("Cat", "")
positive = self.db.get_prompt_pair(pair)["positive_prompt_id"]
self.db.save_prompt_pair("Dog", "Blur")
with self.raw() as conn:
# Only the partial index keeps positive-only pairs unique; a restored
# or hand-edited database may have lost it.
conn.execute("DROP INDEX idx_prompt_pairs_positive_only")
conn.executemany(
"INSERT INTO prompt_pairs(positive_prompt_id,negative_prompt_id)"
" VALUES(?,NULL)",
[(positive,), (positive,)],
)
conn.close()
report = self.run_checks(review_count=0)
pairs = self.check(report, "pair_uniqueness")
self.assertEqual(pairs["status"], "issues")
self.assertEqual(pairs["count"], 2)
self.assertEqual(
pairs["detail"],
[f"Positive prompt {positive} with no negative: 3 pair rows"],
)
def test_integrity_count_at_the_limit_is_flagged_capped(self):
class Rows:
def __init__(self, rows):
self.rows = rows
def execute(self, _sql):
return iter([(row,) for row in self.rows])
capped = health._integrity(Rows(["bad"] * health.INTEGRITY_LIMIT), [])
self.assertEqual(capped["status"], "issues")
self.assertEqual(capped["count"], health.INTEGRITY_LIMIT)
self.assertTrue(capped["capped"])
exact = health._integrity(Rows(["bad"] * 3), [])
self.assertEqual(exact["count"], 3)
self.assertFalse(exact["capped"])
self.assertFalse(health._integrity(Rows(["ok"]), [])["capped"])
def test_an_unexpected_error_fails_only_its_own_check(self):
self.save("Any")
with (
patch.object(health, "_pair_uniqueness", side_effect=KeyError("boom")),
self.assertLogs("prompt_manager.database.health", "ERROR"),
):
report = self.run_checks(review_count=0)
statuses = {c["id"]: c["status"] for c in report["checks"]}
self.assertEqual(statuses.pop("pair_uniqueness"), "failed")
self.assertEqual(set(statuses.values()), {"passed"})
class TestDatabaseUri(unittest.TestCase):
"""The ``mode=ro`` URI of the database file, for every path form SQLite opens."""
def test_posix_path_is_a_local_file_uri(self):
self.assertEqual(
health.readonly_uri("/data/my db#1?.db", windows=False),
"file:///data/my%20db%231%3F.db?mode=ro",
)
def test_windows_drive_path(self):
self.assertEqual(
health.readonly_uri(r"C:\Users\Me\prompts.db", windows=True),
"file:///C:/Users/Me/prompts.db?mode=ro",
)
def test_windows_unc_path_has_an_empty_authority(self):
# pathlib spells \\server\share as file://server/share, which SQLite
# rejects (a non-local authority); an empty one keeps the UNC path.
uri = health.readonly_uri(r"\\server\share\Comfy UI\prompts.db", windows=True)
self.assertEqual(uri, "file:////server/share/Comfy%20UI/prompts.db?mode=ro")
self.assertTrue(ntpath.isabs(r"\\server\share\Comfy UI\prompts.db"))
def test_uri_opens_a_path_with_uri_characters(self):
with tempfile.TemporaryDirectory() as temp:
# "?" is not a valid Windows file name character; "#", "%" and spaces are.
name = "a #b%20 c.db" if os.name == "nt" else "a #b?c%20.db"
path = os.path.join(os.path.realpath(temp), name)
conn = sqlite3.connect(path)
try:
conn.execute("CREATE TABLE t(a)")
conn.commit()
finally:
conn.close()
ro = sqlite3.connect(health.readonly_uri(path), uri=True)
try:
self.assertEqual(ro.execute("SELECT COUNT(*) FROM t").fetchone(), (0,))
with self.assertRaises(sqlite3.OperationalError):
ro.execute("INSERT INTO t VALUES(1)")
finally:
ro.close()
@unittest.skipUnless(os.name == "nt", "a native UNC path exists on Windows only")
def test_native_unc_path_opens(self):
with tempfile.TemporaryDirectory() as temp:
path = os.path.join(os.path.realpath(temp), "prompts.db")
conn = sqlite3.connect(path)
try:
conn.execute("CREATE TABLE t(a)")
conn.commit()
finally:
conn.close()
drive, rest = ntpath.splitdrive(path)
unc = "\\\\localhost\\" + drive[0] + "$" + rest
if not os.path.exists(unc):
self.skipTest("the administrative share is not reachable here")
ro = sqlite3.connect(health.readonly_uri(unc), uri=True)
try:
self.assertEqual(ro.execute("SELECT COUNT(*) FROM t").fetchone(), (0,))
finally:
ro.close()
class TestMissingFiles(HealthTestCase):
def test_missing_files_counted_under_readable_sources_only(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
for n in range(12):
self.image(prompt, f"gone-{n}.png", exists=False)
elsewhere = self.root / "elsewhere"
elsewhere.mkdir()
self.image(prompt, "outside.png", exists=False, folder=elsewhere)
report = self.run_checks(review_count=0)
missing = self.check(report, "missing_files")
self.assertEqual(missing["status"], "issues")
self.assertEqual(missing["count"], 12)
self.assertEqual(len(missing["detail"]), 10)
self.assertRegex(missing["detail"][0], r"^Image \d+: gone-0\.png is missing$")
self.assertNotIn(str(self.root), " ".join(missing["detail"]))
with self.db.model.get_connection() as conn:
self.assertEqual(
conn.execute("SELECT COUNT(*) FROM generated_images").fetchone()[0],
14,
)
def test_links_outside_every_source_are_named_not_counted(self):
prompt = self.save("Images")
elsewhere = self.root / "elsewhere"
elsewhere.mkdir()
self.image(prompt, "outside.png", exists=False, folder=elsewhere)
missing = self.check(self.run_checks(review_count=0), "missing_files")
self.assertEqual(missing["status"], "passed")
self.assertEqual(missing["count"], 0)
self.assertEqual(
missing["detail"],
["1 link outside the image sources was not checked."],
)
def test_missing_file_vs_unavailable_root(self):
# Per source (S2 ruling): the readable source is counted, the absent
# one reports "unavailable" with no count, and the report is incomplete.
prompt = self.save("Images")
absent = self.root / "unmounted"
absent.mkdir()
self.image(prompt, "on-nas.png", folder=absent)
self.image(prompt, "gone.png", exists=False)
os.unlink(absent / "on-nas.png")
absent.rmdir()
report = self.run_checks([str(self.output), str(absent)], review_count=0)
missing = self.check(report, "missing_files")
self.assertEqual(missing["status"], "issues")
self.assertEqual(missing["count"], 1)
self.assertTrue(missing["partial"])
self.assertEqual(
missing["sources"],
[
{"name": "output", "status": "checked", "count": 1},
{"name": "unmounted", "status": "unavailable", "count": None},
],
)
self.assertEqual(
missing["detail"][0],
"1 image source is unavailable, so its files were not counted.",
)
self.assertRegex(missing["detail"][1], r"^Image \d+: gone\.png is missing$")
# The link under the unavailable source is neither counted nor "outside".
self.assertEqual(len(missing["detail"]), 2)
self.assertNotIn(str(self.root), " ".join(missing["detail"]))
self.assertEqual(report["status"], "incomplete")
def test_partial_sources_with_no_missing_file_are_not_passed(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
absent = self.root / "unmounted"
missing = self.check(
self.run_checks([str(self.output), str(absent)], review_count=0),
"missing_files",
)
self.assertEqual(missing["status"], "unavailable")
self.assertEqual(missing["count"], 0)
self.assertTrue(missing["partial"])
self.assertEqual(
[s["status"] for s in missing["sources"]], ["checked", "unavailable"]
)
def test_every_source_unavailable_counts_nothing(self):
prompt = self.save("Images")
absent = self.root / "unmounted"
self.image(prompt, "gone.png", exists=False, folder=self.output)
report = self.run_checks([str(absent)], review_count=0)
missing = self.check(report, "missing_files")
self.assertEqual(missing["status"], "unavailable")
self.assertIsNone(missing["count"])
self.assertFalse(missing["partial"])
self.assertEqual(
missing["sources"],
[{"name": "unmounted", "status": "unavailable", "count": None}],
)
self.assertEqual(
missing["detail"],
["1 image source is unavailable, so missing files were not counted."],
)
def test_each_source_counts_its_own_missing_files(self):
prompt = self.save("Images")
archive = self.root / "archive"
archive.mkdir()
self.image(prompt, "a.png", exists=False)
self.image(prompt, "b.png", exists=False, folder=archive)
self.image(prompt, "c.png", exists=False, folder=archive)
missing = self.check(
self.run_checks([str(self.output), str(archive)], review_count=0),
"missing_files",
)
self.assertEqual(missing["count"], 3)
self.assertFalse(missing["partial"])
self.assertEqual(
[(s["name"], s["count"]) for s in missing["sources"]],
[("output", 1), ("archive", 2)],
)
def test_no_sources_is_unavailable(self):
self.save("Images")
missing = self.check(self.run_checks([], review_count=0), "missing_files")
self.assertEqual(missing["status"], "unavailable")
self.assertIsNone(missing["count"])
self.assertEqual(missing["detail"], ["No image source is available."])
self.assertEqual(missing["sources"], [])
self.assertFalse(missing["partial"])
def test_a_bad_stored_path_fails_only_missing_files(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
with self.raw() as conn:
conn.execute(
"INSERT INTO generated_images(prompt_id,image_path,filename)"
" VALUES(?,?,?)",
(prompt, str(self.root / "else\x00where.png"), "bad.png"),
)
conn.close()
real_realpath = os.path.realpath
def realpath(path, *args, **kwargs):
# POSIX (and Windows before Python 3.11) raise for the NUL byte;
# newer ntpath returns it unchanged. Raise everywhere, as POSIX does.
if "\x00" in str(path):
raise ValueError("embedded null byte")
return real_realpath(path, *args, **kwargs)
with (
patch.object(health.os.path, "realpath", side_effect=realpath),
self.assertLogs("prompt_manager.database.health", "ERROR"),
):
report = self.run_checks(review_count=0)
statuses = {c["id"]: c["status"] for c in report["checks"]}
self.assertEqual(statuses.pop("missing_files"), "failed")
self.assertEqual(set(statuses.values()), {"passed"})
def test_files_are_checked_after_the_read_snapshot_ends(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
self.image(prompt, "gone.png", exists=False)
opened = []
real_connect = sqlite3.connect
def connect(*args, **kwargs):
conn = real_connect(*args, **kwargs)
opened.append(conn)
return conn
real_exists = os.path.exists
in_snapshot = []
def exists(path):
for conn in opened:
try:
in_snapshot.append(conn.in_transaction)
except sqlite3.ProgrammingError: # closed: no snapshot
in_snapshot.append(False)
return real_exists(path)
with (
patch.object(health.sqlite3, "connect", side_effect=connect),
patch.object(health.os.path, "exists", side_effect=exists),
):
missing = self.check(self.run_checks(review_count=0), "missing_files")
self.assertEqual((missing["status"], missing["count"]), ("issues", 1))
self.assertEqual(len(opened), 1)
self.assertTrue(in_snapshot)
self.assertNotIn(True, in_snapshot)
class TestAmbiguousMetadata(HealthTestCase):
def test_ambiguous_metadata_unavailable_without_review_queue(self):
self.save("Any")
report = self.run_checks()
ambiguous = self.check(report, "ambiguous_metadata")
self.assertEqual(ambiguous["status"], "unavailable")
self.assertIsNone(ambiguous["count"])
self.assertEqual(ambiguous["detail"], ["Needs review is not available yet."])
self.assertEqual(report["status"], "incomplete")
def test_review_count_reports_images_needing_review(self):
report = self.run_checks(review_count=3)
ambiguous = self.check(report, "ambiguous_metadata")
self.assertEqual(ambiguous["status"], "issues")
self.assertEqual(ambiguous["count"], 3)
self.assertEqual(ambiguous["detail"], [])
self.assertEqual(report["status"], "issues")
def test_invalid_review_count_is_rejected(self):
for value in (-1, True, "3", 1.5):
with self.assertRaises(ValueError):
self.run_checks(review_count=value)
class TestOverallStatus(unittest.TestCase):
def test_overall_status_precedence(self):
def checks(*statuses):
return [{"status": status} for status in statuses]
self.assertEqual(health.overall_status(checks("passed", "passed")), "passed")
self.assertEqual(health.overall_status(checks("passed", "issues")), "issues")
self.assertEqual(
health.overall_status(checks("issues", "unavailable")), "incomplete"
)
self.assertEqual(health.overall_status(checks("failed")), "incomplete")
# A check that found issues under only some sources is incomplete too.
partial = [{"status": "issues", "partial": True}, {"status": "passed"}]
self.assertEqual(health.overall_status(partial), "incomplete")
if __name__ == "__main__":
unittest.main()
+3 -18
View File
@@ -23,11 +23,7 @@ sys.modules.setdefault("server", _mock_server)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from py.config import ( # noqa: E402
OUTSIDE_ALLOWED_REASON,
GalleryConfig,
PromptManagerConfig,
)
from py.config import GalleryConfig, PromptManagerConfig # noqa: E402
EXTRA_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
@@ -162,10 +158,9 @@ class GalleryRootTestCase(unittest.TestCase):
class TestValidateGalleryRootRejects(GalleryRootTestCase):
def test_filesystem_root_rejected(self):
"""Refused as outside the allowed folders: no reason names the layout."""
ok, reason = GalleryConfig.validate_gallery_root(os.path.abspath(os.sep))
self.assertFalse(ok)
self.assertEqual(reason, OUTSIDE_ALLOWED_REASON)
self.assertIn("root", reason.lower())
@unittest.skipUnless(os.name == "nt", "drive roots only exist on Windows")
def test_windows_drive_root_rejected(self):
@@ -175,17 +170,7 @@ class TestValidateGalleryRootRejects(GalleryRootTestCase):
def test_home_directory_rejected(self):
ok, reason = GalleryConfig.validate_gallery_root(os.path.expanduser("~"))
self.assertFalse(ok)
self.assertEqual(reason, OUTSIDE_ALLOWED_REASON)
def test_home_directory_inside_an_allowed_parent_is_still_refused(self):
"""Even when the environment opens the home's parent, home answers
like any outside path."""
home = os.path.expanduser("~")
os.environ[EXTRA_ROOTS_ENV] = os.path.dirname(home)
self.assertEqual(
GalleryConfig.gallery_root_problem(home),
("outside-allowed", OUTSIDE_ALLOWED_REASON),
)
self.assertIn("home", reason.lower())
def test_directory_outside_comfyui_rejected(self):
outside = Path(self.tmpdir) / "elsewhere"
-45
View File
@@ -5,10 +5,7 @@ returns for the same input (tests/test_generation_params_js_parity.py checks
that directly when Node is available).
"""
import json
import os
import shutil
import subprocess # nosec B404 - runs a fixed Node one-liner
import sys
import unittest
@@ -20,7 +17,6 @@ from utils.generation_params import ( # noqa: E402
extract_generation_params,
extract_from_records,
parse_js_json,
parse_page_json,
)
try:
@@ -223,47 +219,6 @@ class ParsesLikeJavaScript(unittest.TestCase):
run("seed-beyond-double-precision")["seed"], 4974052037430000001.0
)
def test_page_parse_reads_nan_numbers_as_null(self):
# gallery.js and admin.js replace NaN with null before JSON.parse.
self.assertEqual(
parse_page_json('{"a": NaN, "b": [NaN], "t": "NaN"}'),
{"a": None, "b": [None], "t": "NaN"},
)
self.assertEqual(
parse_page_json('{"seed": 9007199254740993}'), {"seed": 9007199254740992.0}
)
for text in ('{"a": Infinity}', "[-Infinity]", "not json", "", None):
with self.subTest(text=text):
self.assertIsNone(parse_page_json(text))
@unittest.skipUnless(shutil.which("node"), "Node is not installed")
def test_page_parse_equals_the_legacy_pages_for_nan_numbers(self):
# gallery.js/admin.js: two NaN regexes, then JSON.parse (Infinity throws).
texts = [
'{"1": {"is_changed": [NaN]}, "2": {"is_changed": NaN, "v": 1.5}}',
'{"a":NaN,"b":[NaN, 2]}',
'{"a": Infinity}',
'{"seed": 9007199254740993}',
]
script = (
"const texts = JSON.parse(require('fs').readFileSync(0, 'utf8'));"
"console.log(JSON.stringify(texts.map((t) => { try { return {v: JSON.parse("
"t.replace(/:\\s*NaN\\b/g, ': null').replace(/\\bNaN\\b/g, 'null'))}; }"
" catch (e) { return null; } })));"
)
completed = subprocess.run( # nosec B603 B607 - fixed argv, no shell
["node", "-e", script],
input=json.dumps(texts),
capture_output=True,
text=True,
check=True,
timeout=60,
)
expected = json.loads(completed.stdout)
for text, page in zip(texts, expected):
with self.subTest(text=text):
self.assertEqual(parse_page_json(text), page and page["v"])
def test_duplicate_keys_keep_the_last_value(self):
self.assertEqual(parse_js_json('{"a": 1, "a": 2}'), {"a": 2})
+3 -8
View File
@@ -800,15 +800,10 @@ class TestUpstreamVerifiedNodeSemantics(unittest.TestCase):
prompt["31"] = dict(prompt["3"], inputs=dict(prompt["3"]["inputs"]))
prompt["31"]["inputs"]["latent_image"] = ["3", 0]
prompt["8"]["inputs"]["samples"] = ["31", 0]
# 31 continues 3's latent: the stages are compared, as for KSampler.
for anchor in (None, "9"):
result = _inspect({"prompt": prompt}, output_node_id=anchor)
self.assertEqual(
(result["state"], result["sampler_id"]), ("resolved", "31")
)
prompt["31"]["inputs"]["positive"] = ["7", 0]
# Unanchored, two listed samplers are two branches, as for KSampler.
self.assertEqual(_inspect({"prompt": prompt})["state"], "ambiguous")
result = _inspect({"prompt": prompt}, output_node_id="9")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_seed_variance_enhancer_passes_its_conditioning(self):
prompt = _f02(
+3 -105
View File
@@ -55,9 +55,6 @@ def _minimal():
"runtime": [],
"savers": {},
"non_file_outputs": [],
"lineage": [
{"class": "VAEEncode", "slot": 0, "strategy": "continue", "input": "pixels"}
],
"stage_inputs": {"shaped": ["positive"], "with_model": ["samples"]},
"facts": {
"configuration_outputs": {},
@@ -202,26 +199,16 @@ class SpecEntriesTest(unittest.TestCase):
},
)
# Combine and concat keep every part in effect (additive); an average
# blends its parts, so a part may be weighted away.
self.assertEqual(
table.composites["ConditioningCombine"],
{
"strategy": "combine",
"inputs": ("conditioning_1", "conditioning_2"),
"additive": True,
},
{"strategy": "combine", "inputs": ("conditioning_1", "conditioning_2")},
)
for name, additive in (
("ConditioningConcat", {"additive": True}),
("ConditioningAverage", {}),
):
for name in ("ConditioningConcat", "ConditioningAverage"):
self.assertEqual(
table.composites[name],
{
"strategy": "combine",
"inputs": ("conditioning_to", "conditioning_from"),
**additive,
},
)
@@ -270,12 +257,7 @@ class SpecEntriesTest(unittest.TestCase):
)
# D18: decided at run time.
# LoraManager's TriggerWord Toggle outputs the LoRA's trigger words as
# filtered at run time (option A leaves them out of a PromptManager).
self.assertEqual(
table.runtime,
frozenset({"PromptSearchList", "TriggerWord Toggle (LoraManager)"}),
)
self.assertEqual(table.runtime, frozenset({"PromptSearchList"}))
comfy = "{name}_{counter:05}_.{ext}"
for name, image in (
@@ -482,89 +464,6 @@ class SpecEntriesTest(unittest.TestCase):
self.assertIsInstance(table.composites["ConditioningCombine"]["inputs"], tuple)
class LineageTableTest(_TableCase):
"""``lineage``: outputs whose image or latent continues one input's."""
def test_repository_lineage_lists_the_pass_through_nodes(self):
table = load_semantics()
self.assertEqual(
{key: dict(entry) for key, entry in table.lineage.items()},
{
("VAEDecode", 0): {"strategy": "continue", "input": "samples"},
("VAEDecodeTiled", 0): {"strategy": "continue", "input": "samples"},
("VAEEncode", 0): {"strategy": "continue", "input": "pixels"},
("VAEEncodeTiled", 0): {"strategy": "continue", "input": "pixels"},
("ImageUpscaleWithModel", 0): {
"strategy": "continue",
"input": "image",
},
("ImageScale", 0): {"strategy": "continue", "input": "image"},
("ImageScaleBy", 0): {"strategy": "continue", "input": "image"},
("ImageScaleToTotalPixels", 0): {
"strategy": "continue",
"input": "image",
},
("LatentUpscale", 0): {"strategy": "continue", "input": "samples"},
("LatentUpscaleBy", 0): {"strategy": "continue", "input": "samples"},
("Image Filter", 0): {"strategy": "continue", "input": "images"},
("Image Filter", 1): {"strategy": "continue", "input": "latents"},
},
)
# Lineage describes no prompt: its classes are not prompt categories.
self.assertFalse({name for name, _ in table.lineage} & table.all_classes)
def test_lineage_rows_are_validated(self):
def row(**changes):
entry = {
"class": "VAEEncode",
"slot": 0,
"strategy": "continue",
"input": "pixels",
}
entry.update(changes)
return entry
cases = (
({"lineage": {}}, "must be a list"),
({"lineage": ["VAEEncode"]}, "must be an object"),
({"lineage": [row(strategy="passthrough")]}, "unknown strategy"),
({"lineage": [row(slot=-1)]}, "slot"),
({"lineage": [row(slot=True)]}, "slot"),
({"lineage": [row(input="")]}, "input name"),
({"lineage": [row(note="x")]}, "field"),
({"lineage": [row(), row()]}, "duplicate"),
({"lineage": [row(**{"class": ""})]}, "class"),
)
for index, (change, fragment) in enumerate(cases):
with self.subTest(case=index):
doc = _minimal()
doc.update(change)
self.assertRejected(doc, fragment)
doc = _minimal()
doc["lineage"].append(row(slot=1, input="latents"))
self.assertEqual(len(self.load_doc(doc).lineage), 2)
def test_a_lineage_class_may_also_carry_conditioning(self):
# InpaintModelConditioning-like: conditioning slots, a latent slot.
doc = _minimal()
doc["lineage"].append(
{"class": "FluxGuidance", "slot": 1, "strategy": "continue", "input": "x"}
)
self.assertIn(("FluxGuidance", 1), self.load_doc(doc).lineage)
def test_composite_additive_must_be_a_boolean(self):
doc = _minimal()
doc["composites"]["ConditioningCombine"] = {
"strategy": "combine",
"inputs": ["conditioning_1", "conditioning_2"],
"additive": "yes",
}
self.assertRejected(doc, "additive must be a boolean")
doc["composites"]["ConditioningCombine"]["additive"] = True
table = self.load_doc(doc)
self.assertIs(table.composites["ConditioningCombine"]["additive"], True)
class ValidationTest(_TableCase):
def test_minimal_table_loads(self):
table = self.load_doc(_minimal())
@@ -602,7 +501,6 @@ class ValidationTest(_TableCase):
{"literal", "strip-newlines", "join-multi", "concat"}
),
"savers": frozenset({"comfy-counter", "vhs-counter"}),
"lineage": frozenset({"continue"}),
},
)
with self.assertRaises(TypeError):
-98
View File
@@ -1706,104 +1706,6 @@ class TestLibrary(unittest.TestCase):
self.db.list_library_images(positive, pair_id=foreign, archive="all")
class LibraryCounts(unittest.TestCase):
"""System › Overview counts and usage statistics follow Library eligibility."""
setUp = TestLibrary.setUp
tearDown = TestLibrary.tearDown
save = TestLibrary.save
state = TestLibrary.state
def test_counts_use_library_eligibility(self):
self.db.save_prompt_pair("Cat", "Blur")
archived = self.db.save_prompt_pair("Cat", "Noise")
self.db.archive_prompt_pair(archived)
titled_pair = self.db.save_prompt_pair("Dog", "Titled negative")
titled = self.db.get_prompt_pair(titled_pair)["negative_prompt_id"]
self.state(titled, "A titled parent") # zero pairs of its own: counted
legacy = self.save("Legacy")
self.db.link_image_to_prompt(legacy, str(Path(self.temp.name) / "a.png"))
self.db.link_image_to_prompt(legacy, str(Path(self.temp.name) / "b.png"))
counts = self.db.library_counts()
# Cat, Dog, the titled negative and Legacy; "Blur" and "Noise" are
# negative-only.
self.assertEqual(counts["positive_prompts"], 4)
# Cat+Blur and Dog+Titled negative; the archived pair is excluded.
self.assertEqual(counts["negative_variants"], 2)
self.assertEqual(counts["images"], 2)
self.assertEqual(counts["rated_images"], 0)
listed = self.db.list_library()
self.assertEqual(counts["positive_prompts"], listed["total"])
self.assertEqual(counts["negative_variants"], listed["variant_total"])
def test_counts_of_an_empty_library_are_zero(self):
self.assertEqual(
self.db.library_counts(),
{
"positive_prompts": 0,
"negative_variants": 0,
"images": 0,
"rated_images": 0,
"review_pending": 0,
},
)
def test_rated_images_follow_their_linked_prompt_rating(self):
# An image carries no rating of its own: it shows its linked prompt's
# (the Images "Highest rated" sort), so "rated" counts those images.
folder = Path(self.temp.name)
rated = self.save("Rated", rating=4)
unrated = self.save("Unrated")
for name in ("a.png", "b.png", "c.png"):
self.db.link_image_to_prompt(rated, str(folder / name))
self.db.link_image_to_prompt(unrated, str(folder / "d.png"))
counts = self.db.library_counts()
self.assertEqual(counts["images"], 4)
self.assertEqual(counts["rated_images"], 3)
self.db.update_prompt_rating(unrated, 2)
self.assertEqual(self.db.library_counts()["rated_images"], 4)
def test_statistics_sum_real_run_counts(self):
a = self.save("Alpha", category="Landscapes", rating=5)
b = self.save("Beta", category="Landscapes", rating=4)
self.save("Gamma", category="Landscapes") # unrated: not in the average
c = self.save("Delta", category="Portraits", rating=3)
blank = self.save("Epsilon", category=" ")
uncategorised = self.save("Zeta")
self.db.record_prompt_use(a, times=3)
self.db.record_prompt_use(b, times=2)
self.db.record_prompt_use(c)
self.db.record_prompt_use(blank)
self.db.record_prompt_use(uncategorised, times=4)
# A negative-only prompt's runs are the pair's runs: never counted twice.
pair = self.db.save_prompt_pair("Alpha", "Only negative")
negative = self.db.get_prompt_pair(pair)["negative_prompt_id"]
with self.db.model.get_connection() as connection:
connection.execute(
"UPDATE prompts SET category='Landscapes',run_count=50 WHERE id=?",
(negative,),
)
rows = self.db.usage_by_category()
self.assertEqual(
rows,
[
{"category": "Landscapes", "runs": 5, "average_rating": 4.5},
{"category": None, "runs": 5, "average_rating": None},
{"category": "Portraits", "runs": 1, "average_rating": 3.0},
],
)
def test_counts_and_usage_write_nothing(self):
self.save("Alpha", category="A")
with self.db.model.get_connection() as connection:
before = connection.total_changes
self.db.library_counts()
self.db.usage_by_category()
with self.db.model.get_connection() as connection:
self.assertEqual(connection.total_changes, before)
self.assertFalse(connection.in_transaction)
class DisplayTitleUnits(unittest.TestCase):
"""The first-line title is cut like the Library's JavaScript title()."""
+1 -28
View File
@@ -52,7 +52,7 @@ from utils.prompt_inspector import (
)
from utils.prompt_inspector import inspect_prompt_evidence, unreadable_evidence
from utils.prompt_pair_resolver import ADDED_CONDITIONING_REASON, INSPECTION_PROFILE
from utils.prompt_pair_resolver import INSPECTION_PROFILE
try:
from tests import inspection_parity as parity
@@ -1293,33 +1293,6 @@ class LocalResolverParityTests(unittest.TestCase):
self.assertEqual(
answers["select-preview-dropped"]["resolution"]["sampler_id"], "3"
)
# Pass-through lineage: the chain shapes reach every new outcome.
self.assertEqual(
{
name: answers[f"select-{name}"]["resolution"]["reason"]
for name in ("hires-fix", "flux-three-stage", "hires-fix-differs")
},
{
"hires-fix": "",
"flux-three-stage": ADDED_CONDITIONING_REASON,
"hires-fix-differs": "Sampler stages have different prompts",
},
)
def test_local_roles_match_fuzzed_pass_through_chains(self):
cases = {}
for seed in range(4):
cases.update(selection_cases.passthrough_fuzz_cases(seed, 120))
answers = self._compare(cases)
reasons = [answer["resolution"]["reason"] for answer in answers.values()]
for reason in (
"",
"Sampler stages have different prompts",
"A later stage adds conditioning to the base prompt",
"Combined conditioning has different texts",
"Multiple sampler branches",
):
self.assertGreater(reasons.count(reason), 5, reason)
def test_local_roles_match_fuzzed_selection_sets(self):
cases = {}
-88
View File
@@ -471,94 +471,6 @@ class CaptureStoreTestCase(unittest.TestCase):
self.assertEqual(results["capture"].size, 60)
self.capture(workers, "b.png")
def _hold_worker(self, store, relative, **options):
"""Start a capture that holds a worker until ``proceed`` is set."""
entered, proceed = threading.Event(), threading.Event()
original = CaptureStore._read
results = {}
def blocking(store_, stream, size):
if threading.current_thread() is thread and not entered.is_set():
entered.set()
self.assertTrue(proceed.wait(WAIT))
return original(store_, stream, size)
def worker():
try:
results["capture"] = self.capture(store, relative, **options)
except Exception as error: # pragma: no cover - reported below
results["error"] = error
patcher = patch.object(CaptureStore, "_read", blocking)
patcher.start()
self.addCleanup(patcher.stop)
thread = threading.Thread(target=worker)
thread.start()
self.assertTrue(entered.wait(WAIT))
def finish():
proceed.set()
thread.join(WAIT)
self.assertFalse(thread.is_alive())
self.assertNotIn("error", results)
return results["capture"]
self.addCleanup(proceed.set)
return finish
def test_background_capture_never_takes_the_last_worker(self):
for name in ("a", "b", "c"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_workers=2)
finish = self._hold_worker(store, "a.png", background=True)
# The second background capture waits, then fails: one worker is free
# but it is the interactive request's.
started = time.monotonic()
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.05):
with self.assertRaises(CaptureCapacity):
self.capture(store, "b.png", background=True)
self.assertGreaterEqual(time.monotonic() - started, 0.04)
# An interactive capture (Tag dialog, Apply) still gets the last one.
interactive = self.capture(store, "c.png")
self.assertEqual(interactive.size, 60)
finish()
def test_background_capture_waits_for_a_worker_to_free(self):
for name in ("a", "b"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_workers=2)
finish = self._hold_worker(store, "a.png", background=True)
waited = {}
def background():
waited["capture"] = self.capture(store, "b.png", background=True)
thread = threading.Thread(target=background)
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", WAIT):
thread.start()
finish()
thread.join(WAIT)
self.assertFalse(thread.is_alive())
self.assertEqual(waited["capture"].size, 60)
def test_single_worker_store_admits_no_background_capture(self):
self.write("a.png", bytes(60))
store = self.make_store(max_workers=1)
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
with self.assertRaises(CaptureCapacity):
self.capture(store, "a.png", background=True)
self.assertEqual(self.capture(store, "a.png").size, 60)
def test_background_capture_leaves_the_last_capture_slot(self):
for name in ("a", "b", "c"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_captures=2)
self.capture(store, "a.png")
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
with self.assertRaises(CaptureCapacity):
self.capture(store, "b.png", background=True)
self.assertEqual(self.capture(store, "c.png").size, 60)
# -- lifetime ------------------------------------------------------------
def test_idle_and_hard_expiry_and_in_use_retention(self):
-360
View File
@@ -1,360 +0,0 @@
"""ComfyUI's ``NaN``/``Infinity`` metadata ingests, and is stored as strict JSON.
ComfyUI serializes the prompt with Python's ``json.dumps``, so a node whose
``IS_CHANGED`` returns ``float("nan")`` (the "always re-run" idiom) writes
``"is_changed": [NaN]`` into every saved PNG; widgets can hold ``Infinity``.
The legacy pages replaced ``NaN`` with ``null`` before ``JSON.parse``, so such
files must ingest. Whatever is stored or answered as JSON stays strict (the
non-finite numbers become ``null``) and the prompt text and hash never change.
"""
import json
import logging
import os
import sqlite3
import sys
import tempfile
import time
import types
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
from PIL import Image, PngImagePlugin
from database.operations import PromptDatabase
from tests.history_waits import drain_writer
from tests.test_api_admin import AdminAPITestCase
from tests.test_api_images import ImageAPITestCase
from utils.hashing import generate_prompt_hash
from utils.image_monitor import ImageGenerationHandler
try:
from tests.open_handles import assert_closed
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed
# The words "NaN" and "Infinity" inside the prompt are text, never numbers.
POSITIVE = " A cat named NaN beside an Infinity pool, ratio: NaN, [NaN] "
NEGATIVE = "blurry"
NAN, INF = float("nan"), float("inf")
def graph(positive=POSITIVE, negative=NEGATIVE):
"""An API prompt graph with non-finite numbers where ComfyUI writes them."""
g = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": positive}},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": negative}},
"3": {
"class_type": "KSampler",
"inputs": {"positive": ["1", 0], "negative": ["2", 0], "model": ["4", 0]},
},
"4": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"5": {
"class_type": "Image Filter",
"inputs": {"strength": INF, "floor": -INF},
"_meta": {"title": "Image Filter"},
"is_changed": [NAN],
},
}
return g
WORKFLOW = {
"nodes": [
{"id": 5, "type": "Image Filter", "widgets_values": [NAN, INF, -INF]},
],
"links": [],
}
def comfy_text(value):
"""The record text ComfyUI writes: Python's default, non-strict JSON."""
text = json.dumps(value)
assert "NaN" in text and "Infinity" in text
return text
def write_png(path, prompt_graph=None, workflow=WORKFLOW):
info = PngImagePlugin.PngInfo()
info.add_text("prompt", comfy_text(prompt_graph or graph()))
if workflow is not None:
info.add_text("workflow", comfy_text(workflow))
path.parent.mkdir(parents=True, exist_ok=True)
with Image.new("RGB", (4, 4)) as image:
image.save(path, pnginfo=info)
return path
def _reject(name):
raise ValueError(f"non-strict JSON constant {name}")
def strict_loads(text):
return json.loads(text, parse_constant=_reject)
class StrictRows:
"""Assertions over the rows a write left behind."""
def image_rows(self, db):
with db.model.get_connection() as conn:
return [
dict(row)
for row in conn.execute(
"SELECT prompt_id, workflow_data, prompt_metadata, parameters "
"FROM generated_images ORDER BY id"
)
]
def assert_strict_snapshot(self, row, *, workflow=True):
prompt = strict_loads(row["prompt_metadata"])
self.assertEqual(prompt["5"]["is_changed"], [None])
self.assertEqual(prompt["5"]["inputs"], {"strength": None, "floor": None})
self.assertEqual(prompt["1"]["inputs"]["text"], POSITIVE)
stored_workflow = strict_loads(row["workflow_data"])
if workflow:
self.assertEqual(
stored_workflow["nodes"][0]["widgets_values"], [None, None, None]
)
if row["parameters"] is not None:
strict_loads(row["parameters"])
def assert_prompt_text(self, db, prompt_id):
with db.model.get_connection() as conn:
row = conn.execute(
"SELECT text, hash FROM prompts WHERE id=?", (prompt_id,)
).fetchone()
self.assertEqual(row["text"], POSITIVE.strip())
self.assertEqual(row["hash"], generate_prompt_hash(POSITIVE))
def dumps_strict(value, **kwargs):
from utils.json_strict import dumps_strict as dumps
return dumps(value, **kwargs)
class TestDumpsStrict(unittest.TestCase):
def test_finite_values_serialize_byte_identically(self):
value = {"b": [1, 2.5, "NaN", None, True], "a": {"x": "Infinity"}}
for kwargs in ({}, {"sort_keys": True, "separators": (",", ":")}):
with self.subTest(kwargs=kwargs):
self.assertEqual(
dumps_strict(value, **kwargs),
json.dumps(value, allow_nan=False, **kwargs),
)
def test_non_finite_numbers_become_null_and_strings_stay(self):
value = {"n": NAN, "i": [INF, -INF], "s": "NaN Infinity", "t": (NAN, 1)}
encoded = dumps_strict(value, sort_keys=True)
self.assertEqual(
strict_loads(encoded),
{"n": None, "i": [None, None], "s": "NaN Infinity", "t": [None, 1]},
)
def test_other_errors_still_raise(self):
cycle = []
cycle.append(cycle)
for value in (cycle, [NAN, cycle]):
with self.assertRaisesRegex(ValueError, "Circular"):
dumps_strict(value)
with self.assertRaises(TypeError):
dumps_strict({"x": object()})
self.assertEqual(
dumps_strict({"x": object()}, default=lambda _: NAN), '{"x": null}'
)
class TestScanIngestsNonFiniteMetadata(StrictRows, AdminAPITestCase):
def ingest(self, name, prompt_graph=None):
path = write_png(self.output_dir / name, prompt_graph)
extracted = self.api._extract_comfyui_metadata(str(path))
return self.api._ingest_scan_batch_sync([(path, extracted)])
def test_resolved_pair_with_nan_is_added_with_strict_snapshots(self):
counts = self.ingest("nan.png")
self.assertEqual(counts["failed"], 0)
self.assertEqual(counts["added"], 1)
rows = self.image_rows(self.api.db)
self.assertEqual(len(rows), 1)
self.assert_strict_snapshot(rows[0])
self.assert_prompt_text(self.api.db, rows[0]["prompt_id"])
pairs = self.api.db.list_prompt_pairs()["pairs"]
self.assertEqual(
[(p["positive_text"], p["negative_text"]) for p in pairs],
[(POSITIVE.strip(), NEGATIVE)],
)
# A repeated scan compares the same strict snapshot: unchanged.
self.assertEqual(self.ingest("nan.png")["unchanged"], 1)
async def test_scan_route_reports_no_failures(self):
from py.config import GalleryConfig
write_png(self.output_dir / "nan.png")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
resp = await self.client.request("POST", "/prompt_manager/scan")
body = await resp.text()
self.assertEqual(resp.status, 200)
self.assertIn('"type": "complete"', body)
self.assertIn('"added": 1', body)
self.assertIn('"failed": 0', body)
class TestMonitorIngestsNonFiniteMetadata(StrictRows, unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.db = PromptDatabase(str(Path(self.temp.name) / "db.sqlite"))
self.pid = self.db.save_prompt(
POSITIVE.strip(), prompt_hash=generate_prompt_hash(POSITIVE)
)
self.tracker = SimpleNamespace(
pop_next_prompt=lambda: None,
get_current_prompt=lambda: None,
extend_prompt_timeout=lambda *args: None,
)
self.handler = ImageGenerationHandler(self.db, self.tracker)
self.handler.logger = logging.getLogger("test.nonfinite.monitor")
self.handler.logger.disabled = True
def tearDown(self):
self.db.close_all()
assert_closed(self, self.temp.name)
self.temp.cleanup()
def test_new_image_with_nan_links_its_pair(self):
# The monitor links only images a PromptManager node produced.
g = graph()
g["1"]["class_type"] = "PromptManager"
path = write_png(Path(self.temp.name) / "out" / "nan.png", g)
self.handler.process_new_image(str(path))
rows = self.image_rows(self.db)
self.assertEqual([row["prompt_id"] for row in rows], [self.pid])
self.assert_strict_snapshot(rows[0])
self.assert_prompt_text(self.db, self.pid)
pairs = self.db.list_prompt_pairs()["pairs"]
self.assertEqual(len(pairs), 1)
self.assertEqual(pairs[0]["image_count"], 1)
def test_legacy_link_stores_strict_json(self):
# The plain prompt link (no exact pair) must not write NaN either.
path = str(Path(self.temp.name) / "plain.png")
metadata = {"prompt": graph(), "workflow": WORKFLOW, "parameters": {"x": NAN}}
self.assertTrue(self.db.link_image_to_prompt(self.pid, path, metadata))
row = self.image_rows(self.db)[0]
self.assert_strict_snapshot(row)
self.assertEqual(strict_loads(row["parameters"]), {"x": None})
def test_stored_images_read_back_without_non_finite_numbers(self):
path = str(Path(self.temp.name) / "legacy.png")
self.db.link_image_to_prompt(self.pid, path, {"prompt": graph()})
with self.db.model.get_connection() as conn:
# A row an older version wrote with Python's default json.dumps.
conn.execute(
"UPDATE generated_images SET prompt_metadata=?", (comfy_text(graph()),)
)
conn.commit()
images = self.db.get_prompt_images(str(self.pid))
strict_loads(json.dumps(images, default=str))
class TestInspectorAnswersStrictJSON(ImageAPITestCase):
async def test_inspect_route_resolves_and_answers_strict_json(self):
write_png(self.output_dir / "nan.png")
response = await self.client.get(
"/prompt_manager/images/inspect/nan.png", params={"root": "0"}
)
self.assertEqual(response.status, 200)
result = strict_loads(await response.text())
self.assertEqual(result["resolution"]["state"], "resolved")
self.assertEqual(
result["prompts"]["positive"],
{"status": "resolved", "text": POSITIVE, "source": "graph-inferred"},
)
async def test_prompt_images_route_answers_strict_json(self):
pid = self.api.db.save_prompt(
POSITIVE.strip(), prompt_hash=generate_prompt_hash(POSITIVE)
)
path = str(write_png(self.output_dir / "linked.png"))
self.api.db.link_image_to_prompt(pid, path, {"prompt": graph()})
with self.api.db.model.get_connection() as conn:
conn.execute(
"UPDATE generated_images SET prompt_metadata=?", (comfy_text(graph()),)
)
conn.commit()
response = await self.client.get(f"/prompt_manager/prompts/{pid}/images")
self.assertEqual(response.status, 200)
result = strict_loads(await response.text())
self.assertEqual(len(result["images"]), 1)
stored = result["images"][0]["prompt_metadata"]
self.assertEqual(stored["1"]["inputs"]["text"], POSITIVE)
self.assertEqual(stored["5"]["is_changed"], [None])
class TestHistoryCapturesNonFiniteGraphs(unittest.TestCase):
"""The queued graph ComfyUI hands the hook may carry float('nan')."""
JOB = "a0c26da3-eb26-405c-b143-c662a84c3112"
def setUp(self):
self.directory = tempfile.TemporaryDirectory()
self.addCleanup(self.directory.cleanup)
self.db = PromptDatabase(os.path.join(self.directory.name, "history.db"))
self.addCleanup(self.db.model.reset_schema_cache)
self.addCleanup(self.db.model.close_all)
def test_queued_graph_with_nan_records_its_run_and_contexts(self):
from tests.test_execution_history_hooks import HostQueue, status
from utils.execution_context_capture import record_runtime_context
from utils.execution_history import register_history_hook
queue = HostQueue()
registration = register_history_hook(
SimpleNamespace(prompt_queue=queue), lambda: self.db
)
self.addCleanup(lambda: registration.dispose(3))
self.addCleanup(registration.writer.invalidate)
deadline = time.monotonic() + 3
while not registration.writer.status().accepting:
self.assertLess(time.monotonic(), deadline, "writer never accepted")
time.sleep(0.005)
g = {
"p": {
"class_type": "PromptManagerPair",
"inputs": {"text": POSITIVE, "negative_text": NEGATIVE},
},
"s": {
"class_type": "KSampler",
"inputs": {"positive": ["p", 0], "negative": ["p", 1], "cfg": INF},
"is_changed": [NAN],
},
"5": graph()["5"],
}
extra = {"extra_pnginfo": {"workflow": WORKFLOW}}
queue.results.append(((0, self.JOB, g, extra, ["s"]), 1))
queue.get()
host = types.ModuleType("comfy_execution.utils")
host.get_executing_context = lambda: SimpleNamespace(
prompt_id=self.JOB, node_id="p", list_index=0
)
with patch.dict(sys.modules, {"comfy_execution.utils": host}):
for role, text in (("positive", POSITIVE), ("negative", NEGATIVE)):
record_runtime_context(
"PromptManagerPair", text, "", "", text, role=role
)
queue.task_done(1, {}, status("execution_success", job=self.JOB))
drain_writer(registration.writer, clock=time.monotonic)
with self.db.model.get_connection() as conn:
conn.row_factory = sqlite3.Row
runs = [dict(r) for r in conn.execute("SELECT * FROM execution_runs")]
self.assertEqual([run["outcome"] for run in runs], ["succeeded"])
contexts = self.db.get_execution_contexts(runs[0]["run_id"])
self.assertEqual(contexts["state"], "complete")
self.assertEqual(contexts["contexts"][0]["positive"]["raw"], POSITIVE)
strict_loads(json.dumps(contexts))
if __name__ == "__main__":
unittest.main()
-66
View File
@@ -160,69 +160,3 @@ class TestPairIngestion(unittest.TestCase):
info = self.db.get_image_prompt_info(self.path)
self.assertEqual(info["pair_context"]["positive_text"], "cat, lora_trigger")
self.assertEqual(info["pair_context"]["text_source"], "execution")
def test_public_ingest_resolved_pair_contract_unchanged(self):
"""The public ingester keeps its result shape and its own transaction;
the extracted ``_ingest_resolved_pair_in`` runs in the caller's."""
from database import pair_ingestion
self.assertTrue(
hasattr(pair_ingestion, "_ingest_resolved_pair_in"),
"_ingest_resolved_pair_in is not extracted",
)
for state in ("ambiguous", "unresolved", "conflict"):
self.assertEqual(
ingest_resolved_pair(self.db, self.path, {}, {"state": state}),
{"status": state, "pairs_added": 0, "links_added": 0},
)
first = ingest_resolved_pair(self.db, self.path, {}, resolved())
self.assertEqual(
set(first),
{"status", "pairs_added", "links_added", "pair_id", "image_id"},
)
self.assertEqual(first["status"], "added")
(link,) = self.db.get_image_prompt_pairs(first["image_id"])
self.assertEqual(
link["provenance"],
{
"resolver_version": 1,
"positive": resolved()["positive"],
"negative": resolved()["negative"],
},
)
conn = self.db.model.get_connection()
self.assertFalse(conn.in_transaction)
# Inside the caller's transaction, with the caller's provenance; an
# exception afterwards rolls the link back with the caller's writes.
provenance = {"kind": "user-selection", "resolver_version": 1}
with self.assertRaises(RuntimeError):
with self.db._pair_transaction() as held:
inner = pair_ingestion._ingest_resolved_pair_in(
held,
self.db,
self.path + "2",
{},
resolved("dog"),
provenance=provenance,
)
self.assertEqual(inner["status"], "added")
raise RuntimeError("caller failed")
self.assertIsNone(self.db.get_image_prompt_info(self.path + "2"))
with self.db._pair_transaction() as held:
inner = pair_ingestion._ingest_resolved_pair_in(
held,
self.db,
self.path + "2",
{},
resolved("dog"),
provenance=provenance,
)
(link,) = self.db.get_image_prompt_pairs(inner["image_id"])
self.assertEqual(link["provenance"], provenance)
with self.db._pair_transaction() as held:
conflict = pair_ingestion._ingest_resolved_pair_in(
held, self.db, self.path, {}, resolved("bird")
)
self.assertEqual(
conflict, {"status": "conflict", "pairs_added": 0, "links_added": 0}
)
+2 -120
View File
@@ -82,8 +82,7 @@ class TestPairScan(AdminAPITestCase):
"class_type": "BasicGuider",
"inputs": {"width": 2048, "height": 2048},
}
# Ambiguous with nothing to choose: unresolved, never a review item.
self.assertEqual(self._scanned("untraceable.png", g), "unresolved")
self.assertEqual(self._scanned("untraceable.png", g), "ambiguous")
self.assertEqual(self.api.db.list_prompt_pairs()["total"], 0)
self.assertEqual(self._texts(), [])
@@ -160,129 +159,12 @@ class TestPairScan(AdminAPITestCase):
"\nSteps: 20, Sampler: Euler"
)
}
# No candidate to choose: reported unresolved, still not guessed.
self.assertEqual(
self.api._ingest_scanned_file(self.output_dir / "ambiguous.png", meta),
"unresolved",
"ambiguous",
)
self.assertEqual(self.api.db.get_recent_prompts()["prompts"], [])
def _nan_scanned(self, name, prompt_graph):
"""Scan ``prompt_graph`` as ComfyUI writes it: ``is_changed`` is NaN."""
stamped = {
node_id: {**node, "is_changed": [float("nan")]}
for node_id, node in prompt_graph.items()
}
text = json.dumps(stamped)
self.assertIn("NaN", text)
return self._scanned_text(name, text)
def _scanned_text(self, name, prompt_text):
meta = {"prompt": prompt_text}
return self.api._ingest_scanned_file(self.output_dir / name, meta)
@staticmethod
def _hires_fix_graph(positive):
"""A PromptManager prompt on a base sampler and a hires-fix sampler.
The base latent comes from a custom batch node and the hires pass from
a custom image filter: no sampler branch traces, so the scan links the
prompt the legacy pages show.
"""
return {
"1": {
"class_type": "CheckpointLoaderSimple",
"inputs": {"ckpt_name": "m.safetensors"},
},
"2": {
"class_type": "MyTriggerWords",
"inputs": {"trigger_words": ["1", 2]},
},
"3": {
"class_type": "PromptManager",
"inputs": {
"text": positive,
"prepend_text": ["2", 0],
"append_text": "",
"clip": ["1", 1],
},
},
"4": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "blur", "clip": ["1", 1]},
},
"5": {"class_type": "MyLatentBatch", "inputs": {"batch_size": 6}},
"6": {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"model": ["1", 0],
"positive": ["3", 0],
"negative": ["4", 0],
"latent_image": ["5", 0],
},
},
"7": {
"class_type": "VAEDecode",
"inputs": {"samples": ["6", 0], "vae": ["1", 2]},
},
"8": {"class_type": "MyImageFilter", "inputs": {"images": ["7", 0]}},
"9": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["8", 0], "vae": ["1", 2]},
},
"14": {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"denoise": 0.25,
"model": ["1", 0],
"positive": ["3", 0],
"negative": ["4", 0],
"latent_image": ["9", 0],
},
},
"15": {
"class_type": "VAEDecode",
"inputs": {"samples": ["14", 0], "vae": ["1", 2]},
},
"16": {"class_type": "SaveImage", "inputs": {"images": ["15", 0]}},
}
def test_hires_fix_prompt_links_with_or_without_nan_is_changed(self):
# Legacy parity: the legacy pages read NaN as null before parsing, so they
# showed this prompt; the scan left NaN-stamped files in review.
for name, scan in (("plain.png", None), ("stamped.png", "nan")):
with self.subTest(name=name):
g = self._hires_fix_graph("cat on a sofa")
if scan:
status = self._nan_scanned(name, g)
else:
status = self._scanned(name, g)
self.assertIn(status, ("added", "linked"))
self.assertEqual(self._texts(), ["cat on a sofa"])
def test_untraced_single_sampler_links_the_legacy_prompt_with_nan(self):
g = self._hires_fix_graph("cat on a sofa")
for node_id in ("7", "8", "9", "14", "15"):
del g[node_id]
g["16"]["inputs"]["images"] = ["6", 0]
self.assertEqual(self._nan_scanned("single.png", g), "added")
self.assertEqual(self._texts(), ["cat on a sofa"])
def test_nan_inside_prompt_text_is_kept(self):
# Only the NaN number reads as null; legacy's regex also rewrote text.
g = self._hires_fix_graph("NaN cat, value: NaN")
self.assertEqual(self._nan_scanned("text.png", g), "added")
self.assertEqual(self._texts(), ["NaN cat, value: NaN"])
def test_infinity_is_still_unreadable_like_the_legacy_pages(self):
text = json.dumps(self._hires_fix_graph("cat")).replace(
'"seed": 7', '"seed": Infinity'
)
self.assertIn(self._scanned_text("inf.png", text), ("ambiguous", "unresolved"))
self.assertEqual(self._texts(), [])
def test_raw_png_workflow_is_stored_as_object_and_repeat_is_unchanged(self):
path = self.output_dir / "workflow.png"
workflow = {"nodes": [{"id": 1, "type": "PromptManagerPair"}]}
-4
View File
@@ -258,10 +258,6 @@ class PairSchemaTests(unittest.TestCase):
],
[(1, 1, 0)],
)
self.assertEqual(
[tuple(r) for r in conn.execute("SELECT * FROM media_review_metadata")],
[(1, 1, 0)],
)
self.assertTrue(PromptModel.verify_database_file(live.db_path)[0])
self.assertTrue(
PromptModel.verify_database_file(live.db_path, require_library_state=True)[
+20 -31
View File
@@ -1245,7 +1245,7 @@ class TestGuiderSamplers(unittest.TestCase):
source["3"]["inputs"]["guider"] = [previous, 0]
self.assertEqual(_inspect(source)["reason_code"], "depth-limit")
def test_custom_advanced_upstream_of_a_sampler_is_a_compared_stage(self):
def test_custom_advanced_upstream_of_a_sampler_is_multistage(self):
source = _guided()
source["41"] = {
"class_type": "KSampler",
@@ -1256,18 +1256,15 @@ class TestGuiderSamplers(unittest.TestCase):
},
}
source["8"]["inputs"]["samples"] = ["41", 0]
# 41 continues 3's latent and reads the same texts as 3's guider.
for anchor in ("9", None):
result = _inspect(source, output_node_id=anchor)
self.assertEqual(
(result["state"], result["sampler_id"]), ("resolved", "41")
)
source["41"]["inputs"]["negative"] = ["6", 0]
result = _inspect(source, output_node_id="9")
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(result["sampler_id"], "41")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason"], "Multistage sampler ancestry is ambiguous")
unanchored = _inspect(source)
self.assertEqual(unanchored["state"], "ambiguous")
self.assertEqual(
sorted(item["sampler_id"] for item in result["candidates"]), ["3", "41"]
sorted(item["sampler_id"] for item in unanchored["candidates"]),
["3", "41"],
)
@@ -2635,19 +2632,17 @@ class TestLatentAncestry(unittest.TestCase):
self.assertEqual(_sides(result), (parity.POS, parity.NEG))
self.assertEqual(_inspect(source, output_node_id="9"), result)
def test_upstream_sampler_through_vaedecode_vaeencode_is_compared(self):
# The latent continues the first stage's through lineage nodes: the
# stages are compared (spec 9.3), equal prompts resolve, others differ.
result = _inspect(_image_to_image(parity.POS), output_node_id="63")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "61"))
self.assertEqual(_sides(result), (parity.POS, parity.NEG))
result = _inspect(_image_to_image(OTHER), output_node_id="63")
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["sampler_id"], "61")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
[item["sampler_id"] for item in result["candidates"]], ["61", "3"]
)
def test_upstream_sampler_through_vaedecode_vaeencode_is_multistage(self):
for positive in (OTHER, parity.POS):
with self.subTest(positive=positive):
result = _inspect(_image_to_image(positive), output_node_id="63")
# Until PR3 compares stage texts, never one stage's prompt.
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["sampler_id"], "61")
self.assertEqual(result["reason"], MULTISTAGE)
self.assertIsNone(result["reason_code"])
self.assertIsNone(result["positive"])
self.assertIsNone(result["negative"])
# An upstream sampler reached through a non-latent input is a stage too.
source = _latent_from(
{
@@ -2964,15 +2959,9 @@ class TestLatentAncestry(unittest.TestCase):
result = _inspect(source, output_node_id="9")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"], {"id": "60", "class_type": "MysteryUpscaler"})
# The listed stage is now an earlier sampler stage, continued through
# VAEEncode: its prompt equals this stage's, so the chain resolves.
# The listed stage is now an earlier sampler stage in the ancestry.
source["60"]["class_type"] = "UltimateSDUpscale"
result = _inspect(source, output_node_id="9")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "3"))
# Reached only through an unlisted image node, it stays unattributed.
source["61"] = {"class_type": "MysteryFilter", "inputs": {"image": ["60", 0]}}
source["5"]["inputs"]["pixels"] = ["61", 0]
result = _inspect(source, output_node_id="9")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_latent_walk_budget_exhaustion_is_work_limit(self):
-419
View File
@@ -1,419 +0,0 @@
"""Multi-stage chains joined through image-space pass-through nodes.
A hires-fix or refiner run decodes sampler A's latent, filters, upscales and
scales the image, then encodes it again for sampler B. The semantics table's
``lineage`` section lists the nodes through which the latent or image carries
on unchanged in kind. When B's latent descends from A only through those nodes
and the stages share one pair, the chain resolves to it. A last stage that adds
conditioning to the base prompt (ConditioningCombine of the base and a style
trigger) keeps the base pair: the floor legacy linked. Everything else stays
ambiguous or unresolved. Ingestion (the default profile) is unchanged (D1).
"""
import json
import unittest
from utils import prompt_pair_resolver as resolver
from utils.prompt_inspector import inspect_prompt_evidence
from utils.prompt_pair_resolver import resolve_prompt_pair
try:
from tests import passthrough_graphs as shapes
except ImportError: # discovered with ``-s tests``
import passthrough_graphs as shapes
def _inspect(graph, **options):
return resolve_prompt_pair(
{"prompt": graph}, profile=resolver.INSPECTION_PROFILE, **options
)
def _texts(result):
return tuple(
result[role]["text"] if isinstance(result[role], dict) else None
for role in ("positive", "negative")
)
class TestHiresFixChain(unittest.TestCase):
def test_shared_pair_resolves_through_image_space_pass_through(self):
graph = shapes.hires_fix()
for anchor in (None, "22"):
with self.subTest(anchor=anchor):
result = _inspect(graph, output_node_id=anchor)
self.assertEqual(result["state"], "resolved")
self.assertEqual(result["sampler_id"], "20")
self.assertEqual(_texts(result), (shapes.POSITIVE, shapes.NEGATIVE))
self.assertIsNone(result["reason_code"])
def test_the_earlier_stage_is_judged_in_its_chain(self):
# Sampler 10 only feeds sampler 20, so the file is 20's: one candidate.
result = _inspect(shapes.hires_fix())
self.assertEqual(result["candidates"], [])
def test_inspector_shows_the_shared_prompt(self):
evidence = inspect_prompt_evidence({"prompt": json.dumps(shapes.hires_fix())})
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(
[evidence["prompts"][role]["text"] for role in ("positive", "negative")],
[shapes.POSITIVE, shapes.NEGATIVE],
)
def test_different_stage_prompts_are_multistage_ambiguous(self):
result = _inspect(shapes.hires_fix(second_positive=shapes.OTHER))
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
sorted(_texts(item)[0] for item in result["candidates"]),
sorted([shapes.POSITIVE, shapes.OTHER]),
)
def test_an_unlisted_image_node_breaks_the_lineage(self):
# The walk still finds the earlier sampler, but not through listed
# pass-through nodes: the stages are never compared.
result = _inspect(shapes.hires_fix(filter_class="MysteryImageEffect"))
self.assertEqual(result["state"], "ambiguous")
later = next(
item for item in result["candidates"] if item["sampler_id"] == "20"
)
self.assertEqual(later["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_a_second_earlier_sampler_in_the_ancestry_stays_unattributed(self):
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["4", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["30", 0]}}
graph["32"] = {
"class_type": "ImageBlend",
"inputs": {"image1": ["14", 0], "image2": ["31", 0], "blend_factor": 0.5},
}
graph["16"]["inputs"]["image"] = ["32", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_a_direct_latent_link_is_a_chain_too(self):
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["10", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "20"))
def test_an_unreadable_earlier_stage_blocks_the_chain(self):
graph = shapes.hires_fix(second_positive=shapes.POSITIVE)
graph["10"]["inputs"]["positive"] = ["6", 0]
graph["6"] = {"class_type": "MysteryPrompt", "inputs": {"seed": 1}}
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "MysteryPrompt")
def test_generated_text_stays_untraceable(self):
result = _inspect(shapes.generated_text_chain())
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "TextGenerate")
def test_ingestion_is_unchanged(self):
# D1: the default profile keeps its historical latent rule.
result = resolve_prompt_pair({"prompt": shapes.hires_fix()})
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(
{item["reason"] for item in result["candidates"]},
{"Unsupported latent ancestry"},
)
class TestFluxThreeStage(unittest.TestCase):
def test_added_style_conditioning_keeps_the_base_pair(self):
graph = shapes.flux_three_stage()
for anchor in (None, "61"):
with self.subTest(anchor=anchor):
result = _inspect(graph, output_node_id=anchor)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, ""))
self.assertEqual(result["reason"], resolver.ADDED_CONDITIONING_REASON)
self.assertEqual(result["sampler_id"], "50")
# D126: the positive says a later stage added conditioning.
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.ADDED_CONDITIONING_SOURCE,
)
self.assertEqual(
result["negative"]["provenance"]["text_source"],
"zeroed-conditioning",
)
def test_any_added_conditioning_is_labelled_never_measured(self):
# Ruling I2: a whole other prompt combined in is labelled the same way;
# no length or "short trigger" heuristic.
graph = shapes.flux_three_stage()
graph["31"] = shapes.encode(shapes.OTHER)
result = _inspect(graph)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result)[0], shapes.POSITIVE)
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.ADDED_CONDITIONING_SOURCE,
)
def test_trigger_words_and_added_conditioning_carry_both_labels(self):
graph = shapes.flux_three_stage()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["2", 0], "text": "x"},
}
graph["8"] = shapes.trigger_toggle()
graph["30"]["inputs"]["prepend_text"] = ["8", 0]
result = _inspect(graph)
self.assertEqual(result["state"], "resolved")
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.RUNTIME_AND_ADDED_SOURCE,
)
def test_a_custom_advanced_last_stage_keeps_its_failure(self):
# Review M2: a guider's combined positive is not floored.
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {"guider": ["63", 0], "latent_image": ["66", 0]},
}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"positive": ["35", 0], "negative": ["33", 0]},
}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["51", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
result = _inspect(graph, output_node_id="61")
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_concat_also_adds_to_the_base_prompt(self):
result = _inspect(shapes.flux_three_stage(composite="ConditioningConcat"))
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, ""))
def test_an_average_blends_the_base_away(self):
result = _inspect(shapes.flux_three_stage(composite="ConditioningAverage"))
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_a_combine_without_the_base_prompt_is_not_the_base_pair(self):
graph = shapes.flux_three_stage(combined=(("37", 0), ("31", 0)))
graph["37"] = shapes.encode(shapes.OTHER)
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_a_different_last_negative_is_not_the_base_pair(self):
graph = shapes.flux_three_stage()
graph["38"] = shapes.encode("watermark")
graph["60"]["inputs"]["negative"] = ["38", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_different_base_stages_are_never_floored(self):
graph = shapes.flux_three_stage()
graph["39"] = shapes.encode(shapes.OTHER)
graph["40"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertNotEqual(result["state"], "resolved")
def test_a_rerouted_combine_is_followed(self):
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "Reroute", "inputs": {"input": ["35", 0]}}
graph["60"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertEqual(result["reason"], resolver.ADDED_CONDITIONING_REASON)
def test_an_unreadable_last_negative_or_encoder_parts_keep_the_failure(self):
graph = shapes.flux_three_stage()
graph["38"] = {"class_type": "MysteryNegative", "inputs": {"seed": 1}}
graph["60"]["inputs"]["negative"] = ["38", 0]
self.assertEqual(_inspect(graph)["reason_code"], "composite")
# Differing encoder parts are a composite too, but not an added stage.
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "CLIPTextEncodeSDXL",
"inputs": {"text_g": shapes.POSITIVE, "text_l": shapes.STYLE},
}
graph["60"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
self.assertEqual(result["node"]["class_type"], "CLIPTextEncodeSDXL")
def test_inspector_shows_the_base_prompt(self):
evidence = inspect_prompt_evidence(
{"prompt": json.dumps(shapes.flux_three_stage())}
)
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(evidence["prompts"]["positive"]["text"], shapes.POSITIVE)
class TestRunTimePromptManagerParts(unittest.TestCase):
"""Maintainer option A: a run-time prepend or append is left out, labelled."""
def test_trigger_words_prepend_resolves_to_the_prompt_manager_text(self):
for anchor in (None, "22"):
with self.subTest(anchor=anchor):
result = _inspect(
shapes.hires_fix_with_trigger_words(), output_node_id=anchor
)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, shapes.NEGATIVE))
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
provenance = result["positive"]["provenance"]
self.assertEqual(
provenance["text_source"], resolver.RUNTIME_PARTS_SOURCE
)
self.assertEqual(result["positive"]["storage_text"], shapes.POSITIVE)
# The negative is untouched by the positive's run-time part.
self.assertEqual(
result["negative"]["provenance"]["text_source"], "graph-inferred"
)
def test_append_and_a_single_sampler_are_labelled_too(self):
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"]["prepend_text"] = "masterpiece"
graph["2"]["inputs"]["append_text"] = ["8", 0]
graph["20"]["inputs"]["latent_image"] = ["4", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "resolved")
# The literal prepend is still combined exactly as before.
self.assertEqual(_texts(result)[0], "masterpiece " + shapes.POSITIVE)
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
def test_the_inspector_shows_the_prompt_and_its_source(self):
evidence = inspect_prompt_evidence(
{"prompt": json.dumps(shapes.hires_fix_with_trigger_words())}
)
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(
evidence["prompts"]["positive"],
{
"status": "resolved",
"text": shapes.POSITIVE,
"source": resolver.RUNTIME_PARTS_SOURCE,
},
)
def test_literal_and_traceable_parts_are_combined_as_before(self):
graph = shapes.hires_fix_with_trigger_words()
graph["8"] = {"class_type": "PrimitiveString", "inputs": {"value": "trig9word"}}
result = _inspect(graph)
self.assertEqual(_texts(result)[0], "trig9word " + shapes.POSITIVE)
self.assertEqual(result["reason"], "")
self.assertEqual(
result["positive"]["provenance"]["text_source"], "graph-inferred"
)
def test_only_a_run_time_feeder_is_left_out(self):
# An unknown feeder stays unsupported, naming it.
graph = shapes.hires_fix_with_trigger_words()
graph["8"]["class_type"] = "MysteryWords"
result = _inspect(graph)
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "MysteryWords")
# A run-time node deeper inside the part makes the whole part run time,
# but it is not a direct feeder: never leave out a traced literal.
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": "kept", "string_b": ["8", 0], "delimiter": ""},
}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
result = _inspect(graph)
self.assertEqual(result["reason_code"], "runtime-value")
def test_the_main_text_from_a_run_time_node_is_never_left_out(self):
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"]["prepend_text"] = ""
graph["2"]["inputs"]["text"] = ["8", 0]
result = _inspect(graph)
self.assertEqual(result["reason_code"], "runtime-value")
def test_the_label_survives_nodes_between_the_prompt_manager_and_sampler(self):
# Review C1: encoders, chained PromptManagers and concatenations.
expected = {
"text-encode": shapes.POSITIVE,
"chained": shapes.POSITIVE,
"concat": shapes.POSITIVE + ", extra",
}
for shape, text in expected.items():
with self.subTest(shape=shape):
result = _inspect(shapes.trigger_words_through(shape))
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result)[0], text)
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.RUNTIME_PARTS_SOURCE,
)
def test_a_partial_prompt_never_equals_a_complete_one(self):
# Review C2: never collapse a branch that left out trigger words with
# one that did not, nor compare such stages as equal.
result = _inspect(shapes.branches_with_and_without_trigger_words())
self.assertEqual(
(result["state"], result["reason"]),
("ambiguous", "Multiple sampler branches"),
)
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = shapes.prompt_manager(shapes.POSITIVE)
graph["20"]["inputs"]["positive"] = ["9", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(
(result["state"], result["reason_code"]),
("ambiguous", "multistage-differs"),
)
# Two PromptManagers leaving out two different toggles differ too.
graph["8b"] = shapes.trigger_toggle("otherword")
graph["9"]["inputs"]["prepend_text"] = ["8b", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["reason_code"], "multistage-differs")
def test_only_listed_run_time_classes_are_left_out(self):
# Review M1: a JoinStringMulti with a linked list flag is run time
# too, but it is not a listed run-time node: never left out.
graph = shapes.hires_fix()
graph["9"] = {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 2,
"delimiter": " ",
"return_list": ["98", 0],
"string_1": "static words",
"string_2": "more",
},
}
graph["98"] = {"class_type": "PrimitiveBoolean", "inputs": {"value": False}}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "runtime-value")
)
self.assertEqual(result["node"]["class_type"], "JoinStringMulti")
def test_ingestion_is_unchanged(self):
result = resolve_prompt_pair({"prompt": shapes.hires_fix_with_trigger_words()})
self.assertEqual(result["state"], "ambiguous")
class TestStillAmbiguous(unittest.TestCase):
def test_edit_batch_with_distinct_prompts_stays_ambiguous(self):
result = _inspect(shapes.edit_batch(4))
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason"], "Multiple sampler branches")
self.assertEqual(len({_texts(item)[0] for item in result["candidates"]}), 4)
if __name__ == "__main__":
unittest.main()
-1053
View File
File diff suppressed because it is too large Load Diff
-335
View File
@@ -1,335 +0,0 @@
"""Needs review queues a file only when there is something to choose.
The maintainer's live queue (#284 / #288) held 39 items whose 11 candidates
all read "unresolved", "No positive / No negative": the ingestion resolver
found several sampler branches but could trace no prompt on any of them.
An ambiguous result is reviewable only when at least two candidates carry a
resolved prompt text. With none, the file could not be traced; with exactly
one, the other branches may still have made the file, so the one text is
not certain either (the resolver never flattens unknown semantics). Both
stay unresolved and are never queued, and a rescan removes the pending,
undecided items such files already have.
"""
import os
import unittest
from unittest.mock import patch
try:
from tests.test_review import DELIMITERS, write_png
from tests.test_review_outcome import OutcomeTestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_review import DELIMITERS, write_png
from test_review_outcome import OutcomeTestCase
from database.review_schema import evidence_offers_choice
from utils import review_capture
def opaque_graph(*opaque, traced=()):
"""Several KSamplers: ``opaque`` positives come from an encoder the
ingestion resolver does not know, ``traced`` ones from CLIPTextEncode.
The first sampler is opaque, so the scan's browser-parser fallback links
nothing either.
"""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
}
sides = [("OpaqueTextEncode", "prompt", t) for t in opaque]
sides += [("CLIPTextEncode", "text", t) for t in traced]
for index, (encoder_class, key, text) in enumerate(sides):
encoder, sampler = str(10 + 2 * index), str(11 + 2 * index)
graph[encoder] = {"class_type": encoder_class, "inputs": {key: text}}
graph[sampler] = {
"class_type": "KSampler",
"inputs": {
"model": ["1", 0],
"positive": [encoder, 0],
"negative": ["2", 0],
},
}
return graph
def _always_reviewable(_resolution):
return True
class ChoosableTestCase(OutcomeTestCase):
def _record_pre_fix(self, name):
"""An item as the pre-fix recorder stored it (no choosable test)."""
with patch.object(review_capture, "reviewable", _always_reviewable):
self._record(name)
def _write(self, name, graph):
write_png(self.output_dir / name, prompt=graph)
return self.output_dir / name
class TestChoosableCandidates(unittest.TestCase):
def _resolution(self, *states):
candidates = []
for index, state in enumerate(states):
side = {"text": f"t{index}"} if state == "resolved" else None
candidates.append(
{"sampler_id": str(index), "state": state, "positive": side}
)
return {"state": "ambiguous", "candidates": candidates}
def test_two_resolved_candidates_are_reviewable(self):
resolution = self._resolution("resolved", "resolved", "unresolved")
self.assertTrue(review_capture.reviewable(resolution))
def test_none_or_one_resolved_candidate_is_not_reviewable(self):
for states in ((), ("unresolved",) * 11, ("resolved", "unresolved")):
with self.subTest(states=states):
resolution = self._resolution(*states)
self.assertFalse(review_capture.reviewable(resolution))
def test_blank_or_missing_text_does_not_count(self):
resolution = self._resolution("resolved", "resolved")
resolution["candidates"][0]["positive"] = {"text": " "}
self.assertFalse(review_capture.reviewable(resolution))
resolution["candidates"][0]["positive"] = None
self.assertFalse(review_capture.reviewable(resolution))
def test_nested_ambiguous_stage_counts_its_own_candidates(self):
inner = self._resolution("resolved", "resolved")["candidates"]
resolution = {
"state": "ambiguous",
"candidates": [{"state": "ambiguous", "candidates": inner}],
}
self.assertTrue(review_capture.reviewable(resolution))
def test_a_result_that_is_not_ambiguous_is_not_reviewable(self):
resolution = self._resolution("resolved", "resolved")
resolution["state"] = "resolved"
self.assertFalse(review_capture.reviewable(resolution))
class TestStoredEvidenceOffersChoice(unittest.TestCase):
def test_unlimited_evidence_needs_two_choosable_candidates(self):
side = {"text": "cat"}
two = [{"state": "resolved", "positive": side}] * 2
self.assertTrue(evidence_offers_choice({"candidates": two}))
one = two[:1] + [{"state": "unresolved", "positive": None}]
self.assertFalse(evidence_offers_choice({"candidates": one}))
def test_limited_evidence_offers_a_choice_only_with_two_candidates(self):
self.assertTrue(
evidence_offers_choice({"limited": True, "candidate_total": 18})
)
for total in (0, 1, None, "18", True):
with self.subTest(total=total):
evidence = {"limited": True, "candidate_total": total}
self.assertFalse(evidence_offers_choice(evidence))
def test_malformed_evidence_offers_nothing(self):
for evidence in (None, [], {"candidates": "x"}, {}):
with self.subTest(evidence=evidence):
self.assertFalse(evidence_offers_choice(evidence))
class TestNothingToChooseIsNotQueued(ChoosableTestCase):
async def test_no_candidate_with_text_could_not_be_traced(self):
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_items"], 0)
(row,) = job["outcomes"]["review"]
self.assertEqual(row["path"], "none.png")
self.assertIn("could not be traced", row["detail"])
self.assertEqual(job["counters"]["review"], 1, job["counters"])
self.assertEqual(self._capture_files(), [])
async def test_exactly_one_candidate_with_text_stays_unresolved(self):
"""One traced branch beside untraced ones is not certain: no link."""
self._roots()
self._write("one.png", opaque_graph("cat", traced=("dog",)))
job = await self._scan()
self.assertEqual(self._items(), [])
(row,) = job["outcomes"]["review"]
self.assertIn("could not be traced", row["detail"])
self.assertEqual(job["counters"].get("added", 0), 0, job["counters"])
self.assertEqual(job["counters"].get("known", 0), 0, job["counters"])
async def test_two_candidates_with_text_are_queued(self):
self._roots()
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
job = await self._scan()
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(item["candidate_count"], 3)
self.assertEqual(job["scope"]["review_items"], 1)
(row,) = job["outcomes"]["review"]
self.assertIn("more than one prompt could match", row["detail"])
async def test_parameters_delimiter_ambiguity_is_not_queued(self):
"""Two A1111 negative delimiters offer no candidate to choose."""
self._roots()
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
job = await self._scan()
self.assertEqual(self._items(), [])
(row,) = job["outcomes"]["review"]
self.assertIn("could not be traced", row["detail"])
def test_recorder_reports_untraced_and_stores_nothing(self):
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._write("one.png", opaque_graph("cat", traced=("dog",)))
for name in ("none.png", "one.png"):
with self.subTest(name=name):
outcome = review_capture.record_ambiguous_file(
self.api.db, self.api._captures, self.output_dir, 0, name
)
self.assertEqual(outcome, "untraced")
self.assertEqual(self._items(), [])
self.assertEqual(self._capture_files(), [])
def test_monitor_does_not_queue_an_untraced_output(self):
self._roots()
path = self._write("none.png", opaque_graph("cat", "dog"))
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
self.assertEqual(self._items(), [])
class TestUntracedItemsLeaveTheQueue(ChoosableTestCase):
async def test_rescan_removes_pending_items_without_a_choice(self):
"""The maintainer's 39 items: queued before the rule, gone on rescan."""
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._write("one.png", opaque_graph("cat", traced=("dog",)))
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
for name in ("none.png", "one.png", "two.png"):
self._record_pre_fix(name)
revision = self._review_revision()
self.assertEqual(self._pending(), 3)
job = await self._scan("all")
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(self._pending(), 1)
self.assertGreater(self._review_revision(), revision)
self.assertEqual(job["scope"]["review_untraced"], 2)
self.assertEqual(job["scope"]["review_items"], 1)
self.assertIn(
"= 2 images left Needs review · nothing to choose", "\n".join(job["log"])
)
async def test_decided_items_of_untraced_files_are_never_touched(self):
self._roots()
for name in ("sel.png", "unres.png"):
self._write(name, opaque_graph("cat", traced=(name, name + "2")))
self._record_pre_fix(name)
items = {os.path.basename(i["locator"]): i for i in self._items()}
self._decide(items["sel.png"], "selected")
self._decide(items["unres.png"], "unresolved")
before = (self._items(), self._decisions())
# The files now carry graphs without a choice, queued pre-fix.
for name in ("sel.png", "unres.png", "pending.png"):
self._write(name, opaque_graph("a cow", "dog"))
self._record_pre_fix(name)
self.assertEqual(self._pending(), 3)
job = await self._scan("all")
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_untraced"], 3)
async def test_an_item_with_a_choice_survives_beside_an_untraced_one(self):
"""Removal judges each item by its own evidence: an item that offers
a choice is not removed although the file's new bytes offer none."""
self._roots()
self._write("f.png", opaque_graph("x", traced=("cat", "dog")))
self._record_pre_fix("f.png")
(choosable,) = self._items()
self._write("f.png", opaque_graph("a cow", "dog"))
self._record_pre_fix("f.png")
self.assertEqual(self._pending(), 2)
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], [choosable["id"]])
self.assertEqual(job["scope"]["review_untraced"], 1)
async def test_limited_items_go_only_without_enough_candidates(self):
"""A pre-fix delimiter item (no candidate) goes; a limited item of 18
candidates may hold a choice and stays, though the bytes now offer
none."""
self._roots()
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
self._record_pre_fix("delims.png")
many = [f"prompt {n}" for n in range(17)]
self._write("many.png", opaque_graph("x", traced=many))
self._record_pre_fix("many.png")
kept = [i["id"] for i in self._items() if "many" in i["locator"]]
self._write("many.png", opaque_graph("a cow", "dog"))
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], kept)
self.assertEqual(job["scope"]["review_untraced"], 1)
async def test_choosable_pending_item_stays(self):
self._roots()
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
await self._scan()
(item,) = self._items()
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
self.assertEqual(job["scope"]["review_untraced"], 0)
async def test_a_failed_removal_is_logged_and_counted_not_fatal(self):
import logging
import sqlite3
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._record_pre_fix("none.png")
with (
patch.object(
self.api.db,
"settle_untraced_review_items",
side_effect=sqlite3.OperationalError("database is locked"),
),
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
):
job = await self._scan("all")
self.assertEqual(job["state"], "finished")
self.assertEqual(job["scope"]["review_errors"], 1)
self.assertTrue(any("could not be removed" in line for line in logs.output))
self.assertEqual(self._pending(), 1)
await self._scan("all")
self.assertEqual(self._pending(), 0)
def test_monitor_removes_the_pending_item_of_an_untraced_output(self):
self._roots()
path = self._write("none.png", opaque_graph("cat", "dog"))
self._record_pre_fix("none.png")
# A pre-fix item of other bytes keeps the stat shortcut out of the way.
self._write("none.png", opaque_graph("a cow", "dog"))
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
self.assertEqual(self._items(), [])
if __name__ == "__main__":
unittest.main()
-374
View File
@@ -1,374 +0,0 @@
"""Needs review takes only files whose final outcome is ambiguous and unlinked.
The maintainer's live scan (#284 / #288) queued every file the strict
resolver found ambiguous, also those the legacy browser-parser fallback then
linked and those already linked, so the queue (10,232) was far larger than
the scan's Needs review group. The rule now: a file is queued only when the
resolver finds it ambiguous, the fallback does not link it and no saved row
links it. A pending, undecided item whose file is linked (by the fallback, a
later resolution or the monitor) is removed on the next scan or monitor
pass; an item that carries a decision is never touched.
"""
import json
import logging
import os
import sqlite3
import unittest
from unittest.mock import patch
try:
from tests.test_review import (
DELIMITERS,
NOW,
ReviewTestCase,
ambiguous_graph,
resolved_graph,
unlinkable_graph,
write_png,
)
except ImportError: # discovered with tests/ as the top-level directory
from test_review import (
DELIMITERS,
NOW,
ReviewTestCase,
ambiguous_graph,
resolved_graph,
unlinkable_graph,
write_png,
)
from utils.hashing import generate_prompt_hash
from utils.image_monitor import ImageGenerationHandler
from utils.prompt_tracker import PromptTracker
from utils import review_capture
def nan_stamped(graph):
"""ComfyUI's ``is_changed: [NaN]`` stamp (written as a bare NaN literal)."""
for node in graph.values():
node["is_changed"] = [float("nan")]
return graph
class OutcomeTestCase(ReviewTestCase):
def _pending(self):
return self.api.db.library_counts()["review_pending"]
def _record(self, name):
"""An item as the pre-fix producers recorded it (any ambiguous file)."""
outcome = review_capture.record_ambiguous_file(
self.api.db, self.api._captures, self.output_dir, 0, name
)
self.assertIn(outcome, ("recorded", "limited"))
def _decide(self, item, outcome):
conn = self.api.db.model.get_connection()
candidate = None
if outcome == "selected":
candidate = json.loads(item["evidence"])["candidates"][0]["hash"]
with conn:
conn.execute(
"UPDATE media_review_items SET state=?, revision=1 WHERE id=?",
(outcome, item["id"]),
)
conn.execute(
"INSERT INTO media_review_decisions(id,item_id,decision_revision,"
"created_at,outcome,candidate_hash,evidence) VALUES(?,?,?,?,?,?,?)",
(
f"00000000-0000-4000-8000-{len(self._decisions()) + 1:012d}",
item["id"],
1,
NOW,
outcome,
candidate,
'{"kind":"user-selection"}',
),
)
def _decisions(self):
conn = self.api.db.model.get_connection()
return [
tuple(row)
for row in conn.execute(
"SELECT id,item_id,outcome FROM media_review_decisions ORDER BY id"
)
]
def _link(self, path, text="already saved"):
prompt_id = self.api.db.save_prompt(
text=text, prompt_hash=generate_prompt_hash(text)
)
self.api.db.link_image_to_prompt(prompt_id, str(path))
return prompt_id
@staticmethod
def _rows(job, group):
return sorted(row["path"] for row in job["outcomes"].get(group, []))
class TestOnlyUnlinkedAmbiguousFilesAreQueued(OutcomeTestCase):
async def test_nan_stamped_graph_the_fallback_links_is_not_queued(self):
self._roots()
path = self.output_dir / "nan.png"
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
self.assertIn(b"NaN", path.read_bytes())
job = await self._scan()
self.assertEqual(job["counters"]["added"], 1, job["counters"])
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
self.assertEqual(self._pending(), 0)
async def test_fallback_link_to_a_known_prompt_is_not_queued(self):
self._roots()
write_png(self.output_dir / "a.png", prompt=ambiguous_graph("cat", "dog"))
await self._scan()
write_png(self.output_dir / "b.png", prompt=ambiguous_graph("cat", "dog"))
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
async def test_already_linked_file_is_not_queued(self):
self._roots()
path = self.output_dir / "linked.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._link(path)
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
self.assertEqual(self._rows(job, "review"), [])
self.assertEqual(self._rows(job, "known"), ["linked.png"])
async def test_genuinely_ambiguous_file_is_queued(self):
self._roots()
path = self.output_dir / "two.png"
write_png(path, prompt=nan_stamped(unlinkable_graph("cat", "dog")))
# Beside it, a file the fallback links: only the first is queued.
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
job = await self._scan()
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(item["state"], "pending")
self.assertEqual(item["candidate_count"], 3)
self.assertEqual(job["scope"]["review_items"], 1)
self.assertEqual(self._rows(job, "review"), ["two.png"])
self.assertEqual(self._pending(), 1)
self.assertIn("? 1 image added to Needs review", "\n".join(job["log"]))
async def test_counter_and_queue_agree_with_the_review_group(self):
"""``review_items`` and the queue total equal the review group less
its unresolved files, on the first scan and on a rescan."""
self._roots()
for name in ("u1.png", "u2.png"):
write_png(self.output_dir / name, prompt=unlinkable_graph("cat", name))
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
unresolved = {"1": {"class_type": "KSampler", "inputs": {}}}
write_png(self.output_dir / "none.png", prompt=unresolved)
write_png(self.output_dir / "one.png", prompt=resolved_graph())
for _ in range(2):
job = await self._scan()
review = self._rows(job, "review")
unresolved_rows = sorted(
row["path"]
for row in job["outcomes"]["review"]
if "could not be traced" in row["detail"]
)
queued = [name for name in review if name not in unresolved_rows]
# delims.png has nothing to choose: "could not be traced".
self.assertEqual(queued, ["u1.png", "u2.png"], job)
self.assertIn("delims.png", unresolved_rows)
self.assertEqual(job["scope"]["review_items"], 2)
self.assertEqual(self._pending(), 2)
self.assertIn("? 2 images already in Needs review", "\n".join(job["log"]))
class TestQueueSelfHeals(OutcomeTestCase):
async def test_pending_item_of_a_fallback_linked_file_is_removed(self):
"""The maintainer's case: queued by the old rule, linked on rescan."""
self._roots()
path = self.output_dir / "nan.png"
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
self._record("nan.png")
revision = self._review_revision()
self.assertEqual(self._pending(), 1)
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
self.assertGreater(self._review_revision(), revision)
self.assertEqual(job["scope"]["review_settled"], 1)
self.assertIn("= 1 image left Needs review · now linked", "\n".join(job["log"]))
async def test_pending_item_of_an_already_linked_file_is_removed(self):
self._roots()
path = self.output_dir / "linked.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._record("linked.png")
self._link(path)
await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
async def test_pending_item_of_a_skipped_known_file_is_removed(self):
"""A file the "new" scope skips as known still settles its old item."""
self._roots()
path = self.output_dir / "one.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._record("one.png")
write_png(path, prompt=resolved_graph())
first = await self._scan()
self.assertEqual(self._rows(first, "added"), ["one.png"])
# An item a pre-fix scan left for the file's old bytes.
self.assertEqual(self._items(), [])
write_png(path, prompt=unlinkable_graph("cat", "dog"), color=(0, 9, 0))
self._record("one.png")
write_png(path, prompt=resolved_graph())
second = await self._scan()
self.assertEqual(self._rows(second, "known"), ["one.png"])
self.assertEqual(self._items(), [])
async def test_a_failed_settle_is_logged_and_counted_not_fatal(self):
self._roots()
write_png(self.output_dir / "nan.png", prompt=ambiguous_graph("cat", "dog"))
self._record("nan.png")
with (
patch.object(
self.api.db,
"settle_linked_review_items",
side_effect=sqlite3.OperationalError("database is locked"),
),
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
):
job = await self._scan()
self.assertEqual(job["state"], "finished")
self.assertEqual(job["scope"]["review_errors"], 1)
self.assertEqual(job["scope"]["review_settled"], 0)
self.assertTrue(any("could not be removed" in line for line in logs.output))
self.assertEqual(self._pending(), 1)
# The next scan settles it.
await self._scan()
self.assertEqual(self._pending(), 0)
async def test_still_unlinked_pending_item_stays(self):
self._roots()
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
await self._scan()
(item,) = self._items()
job = await self._scan()
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
self.assertEqual(job["scope"].get("review_settled", 0), 0)
async def test_decided_items_are_never_touched(self):
self._roots()
for name in ("sel.png", "unres.png"):
write_png(self.output_dir / name, prompt=ambiguous_graph("cat", name))
self._record(name)
items = {os.path.basename(i["locator"]): i for i in self._items()}
self._decide(items["sel.png"], "selected")
self._decide(items["unres.png"], "unresolved")
before = (self._items(), self._decisions())
# A pending neighbour the fallback links: it alone leaves the queue.
write_png(self.output_dir / "pending.png", prompt=ambiguous_graph("x", "y"))
self._record("pending.png")
self.assertEqual(self._pending(), 1)
job = await self._scan("all")
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_settled"], 1)
class TestMonitorRule(OutcomeTestCase):
def _handler(self):
handler = ImageGenerationHandler(self.api.db, PromptTracker(self.api.db))
handler.review_recorder = self.api.record_review_file
return handler
def _context(self):
prompt_id = self.api.db.save_prompt(
text="latest", prompt_hash=generate_prompt_hash("latest")
)
return {"id": prompt_id, "text": "latest", "fallback": True}
async def test_monitor_linked_ambiguous_output_is_not_queued(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._handler().link_image_to_prompt(
str(path), self._context(), {"prompt": graph}
)
self.assertEqual(self._items(), [])
async def test_monitor_link_removes_the_files_pending_item(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._record("two.png")
self.assertEqual(self._pending(), 1)
self._handler().link_image_to_prompt(
str(path), self._context(), {"prompt": graph}
)
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
async def test_monitor_never_touches_a_decided_item(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._record("two.png")
(item,) = self._items()
self._decide(item, "unresolved")
before = (self._items(), self._decisions())
other = self.output_dir / "other.png"
write_png(other, prompt=graph, color=(0, 0, 9))
self._record("other.png")
handler, context = self._handler(), self._context()
handler.link_image_to_prompt(str(path), context, {"prompt": graph})
handler.link_image_to_prompt(str(other), context, {"prompt": graph})
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
async def test_record_review_file_applies_the_scan_rule(self):
self._roots()
write_png(self.output_dir / "fallback.png", prompt=ambiguous_graph("a", "b"))
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
linked = self.output_dir / "linked.png"
write_png(linked, prompt=unlinkable_graph("cat", "dog"))
self._link(linked)
record = self.api.record_review_file
self.assertEqual(record(str(self.output_dir / "fallback.png")), "skipped")
self.assertEqual(record(str(linked)), "linked")
self.assertEqual(record(str(self.output_dir / "two.png")), "recorded")
self.assertEqual(len(self._items()), 1)
if __name__ == "__main__":
unittest.main()
-918
View File
@@ -1,918 +0,0 @@
"""Needs review schema group (#249): install, validation and restore safety."""
import hashlib
import json
import re
import sqlite3
import tempfile
import unittest
from contextlib import closing
from pathlib import Path
from unittest.mock import patch
try:
from tests.open_handles import assert_closed
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed
from database.models import PromptModel
from database.operations import PromptDatabase
try: # guarded so a missing module fails assertions, not test loading
from database.review_schema import (
canonical_evidence,
evidence_digest,
migrate_review_state,
review_schema_error,
)
except ImportError:
canonical_evidence = None
evidence_digest = None
migrate_review_state = None
review_schema_error = None
METADATA = "media_review_metadata"
ITEMS = "media_review_items"
DECISIONS = "media_review_decisions"
TABLES = (METADATA, ITEMS, DECISIONS)
ANNOTATION_TABLES = (
"media_annotation_metadata",
"media_annotation_files",
"media_annotation_bytes",
"media_image_tags",
)
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
# The exact DDL the plan pins; the module must produce precisely these objects.
DDL = {
METADATA: (
f"CREATE TABLE {METADATA}(singleton_id INTEGER PRIMARY KEY "
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
"CHECK(schema_version=1), review_index_revision INTEGER NOT NULL "
"DEFAULT 0 CHECK(review_index_revision>=0))"
),
ITEMS: (
f"CREATE TABLE {ITEMS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
"byte_id TEXT NOT NULL REFERENCES media_annotation_bytes(id) "
"ON DELETE RESTRICT, spelled_path TEXT NOT NULL CHECK(length(CAST("
"spelled_path AS BLOB)) BETWEEN 1 AND 16384), "
"resolver_version INTEGER NOT NULL, "
"evidence_hash TEXT NOT NULL CHECK(length(evidence_hash)=64 AND "
"evidence_hash NOT GLOB '*[^0-9a-f]*'), evidence TEXT NOT NULL "
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
"candidate_count INTEGER NOT NULL CHECK(candidate_count BETWEEN 0 AND 16), "
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), state TEXT NOT NULL "
"CHECK(state IN ('pending','selected','unresolved')), revision INTEGER "
"NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, updated_at TEXT "
"NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
),
DECISIONS: (
f"CREATE TABLE {DECISIONS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
f"item_id TEXT NOT NULL REFERENCES {ITEMS}(id) ON DELETE RESTRICT, "
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
"(length(candidate_hash)=64 AND candidate_hash NOT GLOB '*[^0-9a-f]*')), "
"evidence TEXT NOT NULL "
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
"UNIQUE(item_id, decision_revision), "
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
),
}
INDEXES = {
"idx_media_review_items_queue": (
f"CREATE INDEX idx_media_review_items_queue ON {ITEMS}"
"(state, created_at, id)"
),
"idx_media_review_decisions_pair": (
f"CREATE INDEX idx_media_review_decisions_pair ON {DECISIONS}(pair_id)"
),
"idx_media_review_decisions_image": (
f"CREATE INDEX idx_media_review_decisions_image ON {DECISIONS}(image_id)"
),
}
AUTOINDEXES = {ITEMS: 2, DECISIONS: 2}
LEGACY_TABLES = (
"prompts",
"generated_images",
"tags",
"prompt_tags",
"prompt_pairs",
"prompt_pair_tags",
"image_favorites",
) + ANNOTATION_TABLES
FILE_ID = "0f8e4b2a-1c3d-4e5f-8a9b-0c1d2e3f4a5b"
BYTE_ID = "6a1b2c3d-4e5f-4a6b-8c9d-0e1f2a3b4c5d"
ITEM_ID = "1d2e3f4a-5b6c-4d7e-8f9a-0b1c2d3e4f5a"
DECISION_ID = "2e3f4a5b-6c7d-4e8f-9a0b-1c2d3e4f5a6b"
OTHER_ID = "7b2c3d4e-5f6a-4b7c-9d0e-1f2a3b4c5d6e"
DIGEST = hashlib.sha256(b"img").hexdigest()
CANDIDATE = hashlib.sha256(b"candidate").hexdigest()
NOW = "2026-10-22T00:00:00.000000+00:00"
EVIDENCE = '{"candidates":[],"sha256":"' + DIGEST + '"}'
EVIDENCE_HASH = hashlib.sha256(EVIDENCE.encode("ascii")).hexdigest()
def _compact(sql):
return re.sub(r"\s+", "", sql or "").lower()
def expected_catalog():
objects = {("table", name, _compact(sql)) for name, sql in DDL.items()}
objects |= {("index", name, _compact(sql)) for name, sql in INDEXES.items()}
objects |= {
("index", f"sqlite_autoindex_{table}_{n}", "")
for table, count in AUTOINDEXES.items()
for n in range(1, count + 1)
}
return objects
def catalog(conn):
names = tuple(TABLES) + tuple(INDEXES)
marks = ",".join("?" * len(names))
table_marks = ",".join("?" * len(TABLES))
return {
(kind, name, _compact(sql))
for kind, name, sql in conn.execute(
"SELECT type,name,sql FROM sqlite_master "
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
names + tuple(TABLES),
)
}
def rows(conn, sql, *params):
"""Plain tuples whatever row factory *conn* uses."""
return [tuple(row) for row in conn.execute(sql, params)]
def metadata(conn):
return rows(conn, f"SELECT schema_version,review_index_revision FROM {METADATA}")
def table_rows(conn, tables):
found = {}
for table in tables:
present = conn.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,)
).fetchone()
if present:
found[table] = rows(conn, f"SELECT * FROM {table}") # nosec B608
return found
def insert_byte(conn, file_id=FILE_ID, byte_id=BYTE_ID, locator="/out/img.png"):
conn.execute(
"INSERT INTO media_annotation_files VALUES(?,?,?)", (file_id, locator, NOW)
)
conn.execute(
"INSERT INTO media_annotation_bytes VALUES(?,?,?,?,?,?,?,0)",
(byte_id, file_id, DIGEST, 3, 1_700_000_000_000_000_000, NOW, NOW),
)
def insert_item(conn, *, item_id=ITEM_ID, evidence=EVIDENCE, candidates=2, **extra):
values = {
"id": item_id,
"byte_id": BYTE_ID,
"spelled_path": "/out/img.png",
"resolver_version": 1,
"evidence_hash": hashlib.sha256(evidence.encode("utf-8")).hexdigest(),
"evidence": evidence,
"candidate_count": candidates,
"limited": 0,
"state": "pending",
"revision": 0,
"created_at": NOW,
"updated_at": NOW,
}
values.update(extra)
columns = ",".join(values)
marks = ",".join("?" * len(values))
conn.execute(
f"INSERT INTO {ITEMS}({columns}) VALUES({marks})", # nosec B608
tuple(values.values()),
)
def insert_decision(conn, *, decision_id=DECISION_ID, **extra):
values = {
"id": decision_id,
"item_id": ITEM_ID,
"decision_revision": 1,
"created_at": NOW,
"outcome": "selected",
"candidate_hash": CANDIDATE,
"evidence": EVIDENCE,
"pair_id": None,
"image_id": None,
}
values.update(extra)
columns = ",".join(values)
marks = ",".join("?" * len(values))
conn.execute(
f"INSERT INTO {DECISIONS}({columns}) VALUES({marks})", # nosec B608
tuple(values.values()),
)
def insert_group_rows(conn, **item):
insert_byte(conn)
insert_item(conn, **item)
insert_decision(conn)
def populate_legacy(conn):
conn.execute(
"INSERT INTO prompts(id,text,hash,category,notes,run_count) VALUES"
"(7,'positive','p','keep','notes',12)"
)
conn.execute("INSERT INTO prompts(id,text,hash) VALUES(9,'negative','n')")
conn.execute(
"INSERT INTO generated_images(id,prompt_id,image_path,filename,file_path) "
"VALUES(4,7,'/tmp/image.png','image.png','/tmp/image.png')"
)
conn.execute("INSERT INTO tags(id,name) VALUES(3,'keep-tag')")
conn.execute("INSERT INTO prompt_tags VALUES(7,3)")
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(6,7,9)"
)
conn.execute("INSERT INTO prompt_pair_tags VALUES(6,3)")
conn.execute("INSERT INTO image_favorites VALUES('root','a.png',?)", (NOW,))
def full_capacity_evidence():
"""Sixteen candidates of control, lone-surrogate, astral and combining
text, exactly 8 MiB."""
text = '\x00\x07\x1f
\ud800 é é "q" \\ \U0001f9ea'
candidates = [
{
"hash": hashlib.sha256(str(n).encode()).hexdigest(),
"negative": {"text": text * 3},
"positive": {"text": text * 5},
"sampler_id": str(n),
"state": "resolved",
}
for n in range(16)
]
value = {"candidates": candidates, "pad": ""}
base = len(canonical_evidence(value).encode("utf-8"))
value["pad"] = "x" * (MAX_EVIDENCE_BYTES - base)
return value
class ReviewSchemaTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.addCleanup(assert_closed, self, self.temp.name)
self.addCleanup(PromptModel.reset_schema_cache)
self.models = []
self.addCleanup(self.close_models)
self.path = str(Path(self.temp.name) / "live.db")
def close_models(self):
for model in self.models:
model.close_all()
def model(self, path=None):
model = PromptModel(path or self.path)
self.models.append(model)
return model
def require_module(self):
self.assertIsNotNone(
review_schema_error, "database.review_schema is not implemented"
)
def drop_tables(self, path, tables=TABLES):
"""Remove tables from a closed database file (an old backup)."""
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
for table in reversed(tables):
raw.execute(f"DROP TABLE {table}")
def test_fresh_install_and_reopen_is_idempotent(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(metadata(conn), [(1, 0)])
insert_group_rows(conn)
snapshot = table_rows(conn, TABLES)
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(True, "ok"),
)
model.close_all()
for _ in range(2):
PromptModel.reset_schema_cache()
model = self.model()
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(table_rows(conn, TABLES), snapshot)
# Running the migration again on a current database changes nothing.
migrate_review_state(conn)
self.assertEqual(table_rows(conn, TABLES), snapshot)
model.close_all()
# Items and decisions keep the byte record and the item they cite.
PromptModel.reset_schema_cache()
model = self.model()
with model.get_connection() as conn:
with self.assertRaises(sqlite3.IntegrityError):
conn.execute("DELETE FROM media_annotation_files")
with self.assertRaises(sqlite3.IntegrityError):
conn.execute(f"DELETE FROM {ITEMS}")
self.assertEqual(table_rows(conn, TABLES), snapshot)
def test_startup_validates_stored_evidence_once(self):
"""A reopen streams the evidence once and the annotation group twice
(its own install), never again for the review group (I3)."""
self.require_module()
import database.annotation_schema as annotation_module
import database.review_schema as review_module
model = self.model()
with model.get_connection() as conn:
insert_group_rows(conn)
model.close_all()
for _ in range(2):
PromptModel.reset_schema_cache()
with (
patch.object(
review_module,
"_evidence_error",
wraps=review_module._evidence_error,
) as evidence,
patch.object(
review_module,
"annotation_schema_error",
wraps=review_module.annotation_schema_error,
) as nested,
patch.object(
annotation_module,
"_value_error",
wraps=annotation_module._value_error,
) as annotation_rows,
):
model = self.model()
model.get_connection()
self.assertEqual(evidence.call_count, 1)
self.assertEqual(nested.call_count, 0)
self.assertEqual(annotation_rows.call_count, 2)
model.close_all()
def test_old_database_without_group_migrates_without_touching_legacy_rows(self):
self.require_module()
for label, dropped in (
("annotations only", TABLES),
("predates annotations", ANNOTATION_TABLES + TABLES),
):
with self.subTest(label):
path = str(Path(self.temp.name) / f"{label.replace(' ', '-')}.db")
model = self.model(path)
with model.get_connection() as conn:
populate_legacy(conn)
if dropped == TABLES:
insert_byte(conn)
model.close_all()
self.drop_tables(path, dropped)
with closing(sqlite3.connect(path)) as raw:
self.assertIsNone(review_schema_error(raw))
self.assertEqual(
review_schema_error(raw, allow_missing=False),
"Missing review queue schema",
)
legacy_rows_before = table_rows(raw, LEGACY_TABLES)
self.assertTrue(PromptModel.verify_database_file(path)[0])
self.assertFalse(
PromptModel.verify_database_file(path, require_library_state=True)[
0
]
)
PromptModel.reset_schema_cache()
model = self.model(path)
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(metadata(conn), [(1, 0)])
counts = [
conn.execute(f"SELECT COUNT(*) FROM {t}").fetchone()[0]
for t in (ITEMS, DECISIONS)
]
legacy_rows_after = {
table: value
for table, value in table_rows(conn, LEGACY_TABLES).items()
if table in legacy_rows_before
}
self.assertEqual(counts, [0, 0])
self.assertEqual(legacy_rows_after, legacy_rows_before)
self.assertEqual(
PromptModel.verify_database_file(path, require_library_state=True),
(True, "ok"),
)
model.close_all()
def test_annotations_only_installation_is_valid(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
model.close_all()
self.drop_tables(self.path)
with closing(sqlite3.connect(self.path)) as raw:
self.assertIsNone(review_schema_error(raw))
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(False, "Missing review queue schema"),
)
def test_review_group_without_annotation_group_is_invalid(self):
self.require_module()
model = self.model()
model.close_all()
self.drop_tables(self.path, ANNOTATION_TABLES)
with closing(sqlite3.connect(self.path)) as raw:
for allow_missing in (True, False):
error = review_schema_error(raw, allow_missing=allow_missing)
self.assertIsNotNone(error)
self.assertIn("image annotation", error.lower())
self.assertFalse(PromptModel.verify_database_file(self.path)[0])
PromptModel.reset_schema_cache()
with self.assertRaises(sqlite3.DatabaseError):
self.model()
# An invalid annotation group invalidates the review group beside it.
path = self.tampered(
"bad annotations",
("UPDATE media_annotation_bytes SET sha256=upper(sha256)",),
checks_off=True,
)
with closing(sqlite3.connect(path)) as raw:
self.assertIn("image annotation", review_schema_error(raw).lower())
def tampered(self, name, statements, *, checks_off=False, rows_in=True):
"""A closed current database whose group was altered after the fact."""
path = str(Path(self.temp.name) / f"{name.replace(' ', '-')}.db")
model = self.model(path)
with model.get_connection() as conn:
populate_legacy(conn)
if rows_in:
insert_group_rows(conn)
model.close_all()
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
if checks_off:
raw.execute("PRAGMA ignore_check_constraints=ON")
for statement in statements:
if isinstance(statement, tuple):
raw.execute(*statement)
else:
raw.execute(statement)
return path
def assert_rejected_unrepaired(self, path):
with closing(sqlite3.connect(path)) as raw:
schema_before = raw.execute(
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
).fetchall()
rows_before = table_rows(raw, TABLES)
for allow_missing in (False, True):
error = review_schema_error(raw, allow_missing=allow_missing)
self.assertIsNotNone(error)
self.assertIn("review", error.lower())
self.assertFalse(PromptModel.verify_database_file(path)[0])
self.assertFalse(
PromptModel.verify_database_file(path, require_library_state=True)[0]
)
PromptModel.reset_schema_cache()
with self.assertRaises(sqlite3.DatabaseError):
self.model(path)
PromptModel.reset_schema_cache()
with closing(sqlite3.connect(path)) as raw:
self.assertEqual(
raw.execute(
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
).fetchall(),
schema_before,
)
self.assertEqual(table_rows(raw, TABLES), rows_before)
def test_partial_or_malformed_group_is_rejected_without_repair(self):
self.require_module()
cases = {
"missing metadata": (f"DROP TABLE {METADATA}",),
"missing items": (f"DROP TABLE {DECISIONS}", f"DROP TABLE {ITEMS}"),
"missing decisions": (f"DROP TABLE {DECISIONS}",),
"reserved view": (
f"DROP TABLE {DECISIONS}",
f"CREATE VIEW {DECISIONS} AS SELECT 'x' AS id",
),
"extra trigger": (
f"CREATE TRIGGER media_review_trigger AFTER INSERT ON {ITEMS} "
"BEGIN SELECT 1; END",
),
"extra index": (
f"CREATE INDEX idx_media_review_extra ON {ITEMS}(updated_at)",
),
"extra column": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace("evidence TEXT", "note TEXT, evidence TEXT"),
),
"wrong primary key": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace("id TEXT PRIMARY KEY", "id TEXT", 1),
),
"wrong foreign key": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace(
"generated_images(id) ON DELETE SET NULL",
"generated_images(id) ON DELETE CASCADE",
),
INDEXES["idx_media_review_decisions_pair"],
INDEXES["idx_media_review_decisions_image"],
),
"wrong check": (
f"DROP TABLE {METADATA}",
DDL[METADATA].replace("schema_version=1", "schema_version=2"),
f"INSERT INTO {METADATA} VALUES(1,2,0)",
),
"missing unique": (
"DROP INDEX idx_media_review_decisions_pair",
"DROP INDEX idx_media_review_decisions_image",
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace(" UNIQUE(item_id, decision_revision),", ""),
INDEXES["idx_media_review_decisions_pair"],
INDEXES["idx_media_review_decisions_image"],
),
"missing queue index": ("DROP INDEX idx_media_review_items_queue",),
"wrong index columns": (
"DROP INDEX idx_media_review_items_queue",
INDEXES["idx_media_review_items_queue"].replace(", id", ""),
),
"orphan item": ((f"UPDATE {ITEMS} SET byte_id=?", (OTHER_ID,)),),
"orphan decision": ((f"UPDATE {DECISIONS} SET item_id=?", (OTHER_ID,)),),
"dangling pair": (f"UPDATE {DECISIONS} SET pair_id=404",),
"bad item uuid": (
(f"UPDATE {ITEMS} SET id=?", ("x" * 36,)),
(f"UPDATE {DECISIONS} SET item_id=?", ("x" * 36,)),
),
"uppercase decision uuid": (f"UPDATE {DECISIONS} SET id=upper(id)",),
"null decision id": (f"UPDATE {DECISIONS} SET id=NULL",),
"blob ids": (f"UPDATE {DECISIONS} SET id=CAST(id AS BLOB)",),
"no metadata row": (f"DELETE FROM {METADATA}",),
"long created_at": ((f"UPDATE {ITEMS} SET created_at=?", ("x" * 65,)),),
"blob updated_at": (f"UPDATE {ITEMS} SET updated_at=X'00'",),
"text resolver version": (f"UPDATE {ITEMS} SET resolver_version='one'",),
"blob spelled path": (
f"UPDATE {ITEMS} SET spelled_path=CAST(spelled_path AS BLOB)",
),
"evidence hash mismatch": (
(f"UPDATE {ITEMS} SET evidence_hash=?", (CANDIDATE,)),
),
}
for label, statements in cases.items():
with self.subTest(label):
self.assert_rejected_unrepaired(self.tampered(label, statements))
unchecked = {
"two metadata rows": (f"INSERT INTO {METADATA} VALUES(2,1,0)",),
"wrong schema version": (f"UPDATE {METADATA} SET schema_version=2",),
"negative index revision": (
f"UPDATE {METADATA} SET review_index_revision=-1",
),
"uppercase evidence hash": (
f"UPDATE {ITEMS} SET evidence_hash=upper(evidence_hash)",
),
"bad state": (f"UPDATE {ITEMS} SET state='done'",),
"limited two": (f"UPDATE {ITEMS} SET limited=2",),
"empty spelled path": (f"UPDATE {ITEMS} SET spelled_path=''",),
"negative revision": (f"UPDATE {ITEMS} SET revision=-1",),
"negative candidates": (f"UPDATE {ITEMS} SET candidate_count=-1",),
"bad outcome": (f"UPDATE {DECISIONS} SET outcome='maybe'",),
"short candidate hash": (
f"UPDATE {DECISIONS} SET candidate_hash=substr(candidate_hash,1,63)",
),
"negative decision revision": (
f"UPDATE {DECISIONS} SET decision_revision=-1",
),
}
for label, statements in unchecked.items():
with self.subTest(label):
path = self.tampered(label, statements, checks_off=True)
self.assert_rejected_unrepaired(path)
def test_selected_decision_requires_candidate_hash(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
insert_item(conn)
for outcome, candidate in (("selected", None), ("unresolved", CANDIDATE)):
with self.subTest(outcome):
with self.assertRaises(sqlite3.IntegrityError):
insert_decision(conn, outcome=outcome, candidate_hash=candidate)
insert_decision(conn, outcome="unresolved", candidate_hash=None)
insert_decision(conn, decision_id=OTHER_ID, decision_revision=2)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(
rows(conn, f"SELECT outcome FROM {DECISIONS} ORDER BY 1"),
[("selected",), ("unresolved",)],
)
with self.assertRaises(sqlite3.IntegrityError): # one per revision
insert_decision(conn, decision_id=ITEM_ID, decision_revision=2)
model.close_all()
path = self.tampered(
"selected without hash",
(f"UPDATE {DECISIONS} SET candidate_hash=NULL",),
checks_off=True,
)
self.assert_rejected_unrepaired(path)
def test_oversize_or_noncanonical_evidence_rejected(self):
self.require_module()
self.assertEqual(
canonical_evidence({"b": [1, "é"], "a": None}),
'{"a":null,"b":[1,"\\u00e9"]}',
)
self.assertEqual(
evidence_digest(EVIDENCE), hashlib.sha256(EVIDENCE.encode()).hexdigest()
)
# The store's non-finite policy (utils.json_strict): written as null.
self.assertEqual(
canonical_evidence({"x": float("nan"), "y": [float("-inf")]}),
'{"x":null,"y":[null]}',
)
for bad in (
[],
"text",
{"x": object()},
{"x": object(), "y": float("nan")},
{1: 1, "a": 2},
):
with self.subTest(value=repr(bad)), self.assertRaises(ValueError):
canonical_evidence(bad)
with self.assertRaises(ValueError):
canonical_evidence({"pad": "x" * MAX_EVIDENCE_BYTES})
oversize = '{"pad":"' + "x" * (MAX_EVIDENCE_BYTES - 9) + '"}'
self.assertEqual(len(oversize), MAX_EVIDENCE_BYTES + 1)
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
with self.assertRaises(sqlite3.IntegrityError):
insert_item(conn, evidence=oversize)
model.close_all()
noncanonical = {
"spaces": '{"candidates": [], "sha256": "x"}',
"unsorted": '{"sha256":"x","candidates":[]}',
"raw unicode": '{"text":"é"}',
"duplicate key": '{"a":1,"a":2}',
"not an object": "[]",
"not json": "{",
"nan": '{"a":NaN}',
"empty": "",
}
for label, evidence in noncanonical.items():
for table in (ITEMS, DECISIONS):
with self.subTest(label, table=table):
digest = hashlib.sha256(evidence.encode("utf-8")).hexdigest()
statements = [
(f"UPDATE {table} SET evidence=?", (evidence,)),
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
]
if table == DECISIONS:
statements.pop()
self.assert_rejected_unrepaired(
self.tampered(f"{label} {table}", statements)
)
for table in (ITEMS, DECISIONS):
with self.subTest("oversize", table=table):
digest = hashlib.sha256(oversize.encode("ascii")).hexdigest()
statements = [
(f"UPDATE {table} SET evidence=?", (oversize,)),
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
]
path = self.tampered(f"oversize {table}", statements, checks_off=True)
self.assert_rejected_unrepaired(path)
def test_seventeen_candidates_rejected(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
with self.assertRaises(sqlite3.IntegrityError):
insert_item(conn, candidates=17)
insert_item(conn, candidates=16)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
model.close_all()
path = self.tampered(
"seventeen", (f"UPDATE {ITEMS} SET candidate_count=17",), checks_off=True
)
self.assert_rejected_unrepaired(path)
def test_prompt_or_image_deletion_nulls_navigation_and_keeps_decision(self):
self.require_module()
db = PromptDatabase(self.path)
self.models.append(db.model)
with db.model.get_connection() as conn:
populate_legacy(conn)
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(8,9,NULL)"
)
insert_byte(conn)
insert_item(conn)
insert_decision(conn, pair_id=8, image_id=4)
conn.commit()
self.assertTrue(db.delete_prompt_pair(8))
with db.model.get_connection() as conn:
self.assertEqual(
rows(conn, f"SELECT pair_id, image_id FROM {DECISIONS}"), [(None, 4)]
)
with db.model.get_connection() as conn:
conn.execute("DELETE FROM prompt_pairs WHERE id=6")
conn.commit()
self.assertTrue(db.delete_prompt(7))
with db.model.get_connection() as conn:
self.assertEqual(
rows(
conn,
f"SELECT id, item_id, outcome, candidate_hash, pair_id, image_id "
f"FROM {DECISIONS}",
),
[(DECISION_ID, ITEM_ID, "selected", CANDIDATE, None, None)],
)
self.assertEqual(rows(conn, f"SELECT id FROM {ITEMS}"), [(ITEM_ID,)])
self.assertIsNone(review_schema_error(conn, allow_missing=False))
# The SET NULL actions look decisions up by index, not by a scan.
for column in ("pair_id", "image_id"):
plan = " ".join(
str(row[3])
for row in conn.execute(
f"EXPLAIN QUERY PLAN UPDATE {DECISIONS} " # nosec B608
f"SET {column}=NULL WHERE {column}=?",
(1,),
)
)
self.assertIn(f"idx_media_review_decisions_{column[:-3]}", plan)
def test_full_capacity_unicode_control_pair_round_trips_within_8_mib(self):
self.require_module()
value = full_capacity_evidence()
evidence = canonical_evidence(value)
self.assertEqual(len(evidence.encode("utf-8")), MAX_EVIDENCE_BYTES)
self.assertTrue(evidence.isascii())
source_path = str(Path(self.temp.name) / "source.db")
source = self.model(source_path)
with source.get_connection() as conn:
conn.execute("PRAGMA wal_autocheckpoint=0")
insert_byte(conn)
insert_item(conn, evidence=evidence, candidates=16)
insert_decision(conn, evidence=evidence)
conn.commit()
source_rows = table_rows(conn, TABLES)
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
live = self.model()
backup_file = str(Path(self.temp.name) / "backup.db")
self.assertTrue(source.backup_database(backup_file))
for restore_from in (source_path, backup_file):
with self.subTest(Path(restore_from).name):
live.restore_from_file(restore_from)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
stored = conn.execute(
f"SELECT i.evidence, d.evidence FROM {ITEMS} i "
f"JOIN {DECISIONS} d ON d.item_id = i.id"
).fetchone()
self.assertEqual(json.loads(stored[0]), value)
self.assertEqual(json.loads(stored[1]), value)
PromptModel.close_all_instances()
assert_closed(self, self.temp.name)
def test_install_failure_rolls_back_and_retries(self):
self.require_module()
with patch(
"database.review_schema.review_schema_error",
side_effect=[None, "injected final validation failure"],
):
with self.assertRaisesRegex(sqlite3.DatabaseError, "injected"):
self.model()
with closing(sqlite3.connect(self.path)) as raw:
self.assertEqual(catalog(raw), set())
self.assertIsNotNone(
raw.execute(
"SELECT 1 FROM sqlite_master WHERE name='media_annotation_bytes'"
).fetchone()
)
PromptModel.reset_schema_cache()
model = self.model()
self.assertEqual(
PromptModel.verify_database_file(model.db_path, require_library_state=True),
(True, "ok"),
)
with model.get_connection() as conn:
self.assertEqual(metadata(conn), [(1, 0)])
def test_wal_backup_restore_round_trips_group(self):
self.require_module()
source_path = str(Path(self.temp.name) / "source.db")
source = self.model(source_path)
with source.get_connection() as conn:
conn.execute("PRAGMA wal_autocheckpoint=0")
insert_group_rows(conn)
conn.execute(f"UPDATE {METADATA} SET review_index_revision=3")
conn.commit()
source_rows = table_rows(conn, TABLES)
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
live = self.model()
with live.get_connection() as conn:
insert_byte(conn)
insert_item(conn, item_id=OTHER_ID)
source_before = Path(source_path).read_bytes()
backup_file = str(Path(self.temp.name) / "backup.db")
self.assertTrue(source.backup_database(backup_file))
with closing(sqlite3.connect(backup_file)) as raw:
self.assertEqual(table_rows(raw, TABLES), source_rows)
self.assertIsNone(review_schema_error(raw, allow_missing=False))
safety = live.restore_from_file(source_path)
self.assertEqual(Path(source_path).read_bytes(), source_before)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
self.assertEqual(metadata(conn), [(1, 3)])
with closing(sqlite3.connect(safety)) as raw:
self.assertEqual(rows(raw, f"SELECT id FROM {ITEMS}"), [(OTHER_ID,)])
PromptModel.reset_schema_cache()
live.restore_from_file(backup_file)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
PromptModel.close_all_instances()
assert_closed(self, self.temp.name)
def test_staged_failure_preserves_source_live_safety_and_handles(self):
self.require_module()
live = self.model()
with live.get_connection() as conn:
insert_byte(conn)
insert_item(conn, item_id=OTHER_ID)
live_rows = table_rows(conn, TABLES)
bad = self.tampered(
"restore bad", (f"UPDATE {ITEMS} SET state='done'",), checks_off=True
)
old = str(Path(self.temp.name) / "old.db")
self.model(old).close_all()
self.drop_tables(old)
for source, patched in ((bad, False), (old, True)):
with self.subTest(source=Path(source).name):
before = Path(source).read_bytes()
safety = sorted(Path(self.temp.name).glob("live.db.backup_*"))
if patched:
context = patch(
"database.models.migrate_review_state",
side_effect=sqlite3.OperationalError("staging failed"),
)
expected = sqlite3.OperationalError
else:
context = patch.object(PromptModel, "backup_database")
expected = ValueError
with context as hook, self.assertRaises(expected):
live.restore_from_file(source)
if not patched:
hook.assert_not_called()
self.assertEqual(Path(source).read_bytes(), before)
self.assertEqual(
sorted(Path(self.temp.name).glob("live.db.backup_*")), safety
)
assert_closed(self, source)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), live_rows)
self.assertFalse(
PromptModel.verify_database_file(old, require_library_state=True)[0]
)
self.assertTrue(PromptModel.verify_database_file(old)[0])
def test_verify_requires_group_only_with_require_library_state(self):
self.require_module()
self.model().close_all()
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(True, "ok"),
)
self.drop_tables(self.path)
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(False, "Missing review queue schema"),
)
if __name__ == "__main__":
unittest.main()
+5 -237
View File
@@ -38,11 +38,6 @@ def _graph(text):
}
def _scope_part(scope):
"""The per-source scope keys this module pins (other features add counters)."""
return {key: scope[key] for key in ("files", "roots", "output_folders")}
class ScanScopeTestCase(AdminAPITestCase):
"""Helpers: link files the way the scan and the monitor do, spy on reads."""
@@ -265,7 +260,7 @@ class TestScanScope(ScanScopeTestCase):
async def test_empty_read_body_asks_for_the_default_scope(self):
"""A chunked POST with no bytes can read a body: still the default."""
from py.api.job_routes import _scan_request
from py.api.job_routes import _scan_scope
class Request:
can_read_body = True
@@ -276,21 +271,10 @@ class TestScanScope(ScanScopeTestCase):
async def text(self):
return self._text
default = ({"files": "new", "roots": None}, None)
self.assertEqual(await _scan_request(Request("")), default)
self.assertEqual(await _scan_request(Request(" \n")), default)
self.assertEqual(
await _scan_request(
Request(
'{"scope": "all", "roots": [2, 0, 2],'
' "paths": ["output/c", "output/a", "output/c"]}'
)
),
({"files": "all", "roots": {0: "output/a", 2: "output/c"}}, None),
)
request, error = await _scan_request(Request("{"))
self.assertIsNone(request)
self.assertIsInstance(error, str)
self.assertEqual(await _scan_scope(Request("")), "new")
self.assertEqual(await _scan_scope(Request(" \n")), "new")
self.assertEqual(await _scan_scope(Request('{"scope": "all"}')), "all")
self.assertIsNone(await _scan_scope(Request("{")))
async def test_invalid_scope_is_refused(self):
for body in ({"scope": "everything"}, {"scope": 1}, ["new"]):
@@ -308,222 +292,6 @@ class TestScanScope(ScanScopeTestCase):
self.assertIsNone(self.api._scan_job)
class TestPerSourceScope(ScanScopeTestCase):
"""``roots``: scan only the selected effective sources (#249 S4 Task 10)."""
async def asyncSetUp(self):
await super().asyncSetUp()
self.a = self.output_dir / "a"
self.b = self.output_dir / "b"
self.a.mkdir()
self.b.mkdir()
_write_png(self.a / "in_a.png", prompt_text="a prompt in a")
_write_png(self.b / "in_b.png", prompt_text="a prompt in b")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)]
async def _refused(self, body):
resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body)
data = await resp.json()
self.assertEqual(resp.status, 400, data)
self.assertFalse(data["success"])
self.assertNotIn(self.tmpdir, json.dumps(data))
self.assertIsNone(self.api._scan_job)
self.assertEqual(self.api._jobs.list(), [])
return data
async def test_scoped_scan_only_walks_selected_root(self):
walked = []
original = self.api._collect_output_media_sync
def spy(output_dirs):
walked.append([os.path.basename(str(d)) for d in output_dirs])
return original(output_dirs)
self.api._collect_output_media_sync = spy
reads = self._spy_reads()
job = await self._scan({"scope": "all", "roots": [1], "paths": ["output/b"]})
self.assertEqual(walked, [["b"]])
self.assertEqual(reads, ["in_b.png"])
self.assertEqual(
_scope_part(job["scope"]),
{"files": "all", "roots": [1], "output_folders": 1},
)
self.assertEqual(
job["outcomes"]["added"],
[
{
"path": "in_b.png",
"root": "output/b",
"root_index": 1,
"detail": job["outcomes"]["added"][0]["detail"],
}
],
)
self.assertEqual(job["counters"]["added"], 1)
self.assertEqual(self.api._scan_job.last_event["directories"], ["output/b"])
async def test_duplicate_roots_are_one_selection(self):
job = await self._scan(
{"roots": [1, 0, 1], "paths": ["output/b", "output/a", "output/b"]}
)
self.assertEqual(job["scope"]["roots"], [0, 1])
self.assertEqual(job["scope"]["output_folders"], 2)
self.assertEqual(job["counters"]["added"], 2)
async def test_scope_line_counts_selected_folders(self):
job = await self._scan({"roots": [0], "paths": ["output/a"]})
self.assertTrue(
job["log"][0].endswith(
"Scan started · 1 output folder · new and unresolved files"
),
job["log"][0],
)
self.assertEqual(job["scope"]["output_folders"], 1)
unscoped = await self._scan({"scope": "all"})
self.assertEqual(
_scope_part(unscoped["scope"]),
{"files": "all", "roots": None, "output_folders": 2},
)
self.assertIn("Scan started · 2 output folders · all files", unscoped["log"][0])
async def _stale(self, body):
resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body)
data = await resp.json()
self.assertEqual(resp.status, 409, data)
self.assertEqual(data["code"], "sources-changed")
self.assertFalse(data["success"])
self.assertNotIn(self.tmpdir, json.dumps(data))
self.assertIsNone(self.api._scan_job)
self.assertEqual(self.api._jobs.list(), [])
return data
async def test_index_past_the_sources_is_409_without_job(self):
await self._stale({"roots": [2], "paths": ["output/c"]})
await self._stale({"roots": [0, 64], "paths": ["output/a", "output/x"]})
async def test_stale_index_never_scans_another_source(self):
"""The list changed after the page read it: source 0 was removed, so
old index 1 (b) is past the end and old index 0 now names b."""
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.b)]
await self._stale({"roots": [1], "paths": ["output/b"]})
await self._stale({"roots": [0], "paths": ["output/a"]})
job = await self._scan({"roots": [0], "paths": ["output/b"]})
self.assertEqual(job["outcomes"]["added"][0]["root"], "output/b")
async def test_roots_need_their_paths(self):
for body in (
{"roots": [0]},
{"roots": [0], "paths": None},
{"roots": [0], "paths": "output/a"},
{"roots": [0], "paths": []},
{"roots": [0, 1], "paths": ["output/a"]},
{"roots": [0], "paths": [0]},
{"roots": [0, 0], "paths": ["output/a", "output/b"]},
):
with self.subTest(body=body):
await self._refused(body)
async def test_bool_and_float_roots_rejected(self):
for roots in (
[True],
[False],
[0.0],
[1.5],
["0"],
[None],
[-1],
[0, None],
[],
None,
0,
"0",
{"0": 0},
list(range(65)),
):
with self.subTest(roots=roots):
paths = ["output/a"] * len(roots) if isinstance(roots, list) else []
await self._refused({"roots": roots, "paths": paths})
async def test_legacy_scan_unscoped_unchanged(self):
from py.api.admin import SCAN_COUNT_KEYS
resp = await self.client.request("POST", "/prompt_manager/scan", data="{}")
payloads = _sse_payloads(await resp.text())
final = payloads[-1]
self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"})
self.assertEqual(final["directories"], ["output/a", "output/b"])
self.assertEqual(final["added"], 2)
self.assertEqual(
_scope_part(self.api._scan_job.scope),
{"files": "all", "roots": None, "output_folders": 2},
)
async def test_legacy_scan_attaches_to_a_scoped_job_as_is(self):
import threading
from py.api.admin import SCAN_COUNT_KEYS
gate = threading.Event()
original = self.api._collect_output_media_sync
def gated(output_dirs):
gate.wait(5)
return original(output_dirs)
self.api._collect_output_media_sync = gated
# The legacy route has decided to attach once it streams the job.
attached = asyncio.Event()
stream = self.api._stream_scan_job
async def attaching(request, job):
attached.set()
return await stream(request, job)
self.api._stream_scan_job = attaching
resp = await self.client.request(
"POST",
"/prompt_manager/jobs/scan",
json={"roots": [0], "paths": ["output/a"]},
)
self.assertEqual(resp.status, 202)
started = (await resp.json())["job"]
legacy = asyncio.ensure_future(
self.client.request("POST", "/prompt_manager/scan", data="{}")
)
try:
await asyncio.wait_for(attached.wait(), 10)
finally:
gate.set()
payloads = _sse_payloads(await (await legacy).text())
await asyncio.wait_for(self.api._scan_job.task, 10)
self.assertEqual(self.api._scan_job.id, started["id"])
self.assertEqual(len(self.api._jobs.list()), 1)
final = payloads[-1]
self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"})
self.assertEqual(final["directories"], ["output/a"])
self.assertEqual(final["added"], 1)
async def test_scoped_scan_uses_the_sources_it_was_started_with(self):
"""Sources saved after the start never change what the job walks."""
resolved = []
original = self.api._get_all_output_dirs
def counting():
resolved.append(1)
return original()
self.api._get_all_output_dirs = counting
job = await self._scan({"roots": [1], "paths": ["output/b"]})
self.assertEqual(len(resolved), 1) # the request's own resolution only
self.assertEqual(job["outcomes"]["added"][0]["root_index"], 1)
class TestKnownResolvedPaths(ScanScopeTestCase):
"""``PromptDatabase.get_known_resolved_paths``: one streamed query."""
-959
View File
@@ -1,959 +0,0 @@
"""System › Sources & scans backend (#249 S4 Task 9).
``GET /prompt_manager/settings`` reports the configured, effective and
watched sources and the monitoring setting; ``POST`` saves
``monitoring_enabled`` (a restart applies it) and refuses a source change
while a record-changing job runs; ``POST /prompt_manager/settings/validate-source``
checks one path and writes nothing. No response carries an absolute path.
"""
import asyncio
import functools
import json
import ntpath
import os
import sys
import types
import unittest
from contextlib import contextmanager
from unittest.mock import patch
try:
from tests.test_api_admin import AdminAPITestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_api_admin import AdminAPITestCase
from py import config as config_module # noqa: E402
from py.api import settings_routes, sources # noqa: E402
from py.config import GalleryConfig # noqa: E402
VALIDATE = "/prompt_manager/settings/validate-source"
SETTINGS = "/prompt_manager/settings"
@contextmanager
def no_filesystem_access_for(marker, test):
"""Fail ``test`` if anything resolves, stats or opens a path holding ``marker``.
Trusted directories (ComfyUI's, the anchors) may still be resolved.
"""
real = {
"realpath": os.path.realpath,
"stat": os.stat,
"lstat": os.lstat,
}
def guard(name):
def call(path, *args, **kwargs):
if isinstance(path, (str, bytes)) and marker in os.fsdecode(path):
test.fail(f"{name}({path!r}) touched an outside path")
return real[name](path, *args, **kwargs)
return call
with (
patch("os.path.realpath", guard("realpath")),
patch("os.stat", guard("stat")),
patch("os.lstat", guard("lstat")),
):
yield
class FakeMonitor:
"""The image monitor as get_settings sees it: status and scheduled roots."""
def __init__(self, scheduled, alive=True):
self.monitored_directories = [str(path) for path in scheduled]
self.alive = alive
def get_status(self):
return {
"running": True,
"monitored_directories": self.monitored_directories,
"handler_active": True,
"observer_alive": self.alive,
}
class SourcesTestCase(AdminAPITestCase):
async def asyncSetUp(self):
await super().asyncSetUp()
self.a = self.output_dir / "a"
self.b = self.output_dir / "b"
self.a.mkdir()
self.b.mkdir()
enabled = GalleryConfig.MONITORING_ENABLED
self.addCleanup(setattr, GalleryConfig, "MONITORING_ENABLED", enabled)
GalleryConfig.MONITORING_ENABLED = True
self._install_monitor(None)
def _install_monitor(self, monitor):
import utils.image_monitor as im_mod
orig = im_mod._monitor_instance
im_mod._monitor_instance = monitor
self.addCleanup(setattr, im_mod, "_monitor_instance", orig)
async def _get(self):
resp = await self.client.request("GET", "/prompt_manager/settings")
body = await resp.text()
self.assertEqual(resp.status, 200, body)
self.assertNotIn(self.tmpdir, body)
return json.loads(body)
async def _post(self, url, payload):
resp = await self.client.request("POST", url, json=payload)
body = await resp.text()
self.assertNotIn(self.tmpdir, body)
return resp.status, json.loads(body)
def _config_bytes(self):
if not os.path.exists(self.config_path):
return None
with open(self.config_path, "rb") as f:
return f.read()
def _states(self, data):
return {row["path"]: row["watch"] for row in data["sources"]["effective"]}
class TestSourceStatus(SourcesTestCase):
async def test_sources_list_configured_effective_and_auto_detected(self):
missing = self.output_dir / "gone"
missing.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [
self._root(self.a),
self._root(missing),
]
missing.rmdir() # an unmounted drive: configured, no longer a directory
data = await self._get()
self.assertEqual(
data["sources"]["configured"],
[
{
"label": "a",
"path": "output/a",
"valid": True,
"code": None,
"reason": None,
},
{
"label": "gone",
"path": "output/gone",
"valid": False,
"code": "not-found",
"reason": "Gallery root does not exist",
},
],
)
self.assertEqual(
data["sources"]["effective"],
[{"label": "a", "path": "output/a", "root_index": 0, "watch": "manual"}],
)
self.assertFalse(data["sources"]["auto_detected"])
# The legacy keys are unchanged.
self.assertEqual(
data["settings"]["gallery_root_paths"], ["output/a", "output/gone"]
)
async def test_empty_list_means_auto_detect(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
self._write_config({"gallery": {"monitoring": {"directories": ["x"]}}})
# Detection (the upward search for a ComfyUI tree) finds the output.
self.api._find_comfyui_output_dir = lambda: str(self.output_dir)
status, saved = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": []}
)
self.assertEqual(status, 200, saved)
self.assertEqual(
self._read_config()["gallery"]["monitoring"]["directories"], []
)
data = await self._get()
self.assertEqual(data["sources"]["configured"], [])
self.assertTrue(data["sources"]["auto_detected"])
self.assertEqual(
data["sources"]["effective"],
[
{
"label": "ComfyUI output",
"path": "output",
"root_index": 0,
"watch": "manual",
}
],
)
async def test_watching_requires_live_observer_with_root(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)]
# No monitor in this process: nothing is watched, a restart cannot help.
data = await self._get()
self.assertEqual(
data["monitor"],
{"present": False, "observer_alive": False, "scheduled": []},
)
self.assertEqual(
self._states(data), {"output/a": "manual", "output/b": "manual"}
)
# A dead observer watches nothing, even the roots it scheduled.
self._install_monitor(FakeMonitor([self.a], alive=False))
data = await self._get()
self.assertEqual(
data["monitor"],
{"present": True, "observer_alive": False, "scheduled": ["output/a"]},
)
self.assertEqual(
self._states(data),
{"output/a": "restart_pending", "output/b": "restart_pending"},
)
# A live observer watches its scheduled root; a root configured since
# it started waits for a restart.
self._install_monitor(FakeMonitor([self.a]))
data = await self._get()
self.assertTrue(data["monitor"]["observer_alive"])
self.assertEqual(
self._states(data), {"output/a": "watching", "output/b": "restart_pending"}
)
# Monitoring switched off: the live observer still watches until the
# restart ("stopping"), and nothing else is pending.
GalleryConfig.MONITORING_ENABLED = False
data = await self._get()
self.assertFalse(data["settings"]["monitoring_enabled"])
self.assertEqual(
self._states(data), {"output/a": "stopping", "output/b": "manual"}
)
async def test_scheduled_root_matches_through_its_spelling(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
spelled = os.path.join(str(self.output_dir), "b", "..", "a")
self._install_monitor(FakeMonitor([spelled]))
data = await self._get()
self.assertEqual(self._states(data), {"output/a": "watching"})
self.assertEqual(data["monitor"]["scheduled"], ["output/a"])
async def test_status_failure_is_500_without_paths(self):
def boom(*args, **kwargs):
raise OSError(2, "No such file", f"{self.tmpdir}/secret")
with patch.object(sources, "describe_sources", boom):
resp = await self.client.request("GET", "/prompt_manager/settings")
self.assertEqual(resp.status, 500)
self.assertNotIn(self.tmpdir, await resp.text())
class TestMonitoringSetting(SourcesTestCase):
async def test_monitoring_enabled_strict_bool_and_restart_flag(self):
self._write_config({"custom": {"keep": True}})
before = self._config_bytes()
for value in ("false", 0, 1, None, [], {"on": True}):
with self.subTest(value=value):
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": value}
)
self.assertEqual(status, 400, data)
self.assertFalse(data["success"])
self.assertEqual(self._config_bytes(), before)
self.assertTrue(GalleryConfig.MONITORING_ENABLED)
self._install_monitor(FakeMonitor([self.output_dir]))
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": False}
)
self.assertEqual(status, 200, data)
self.assertTrue(data["persisted"])
self.assertTrue(data["restart_required"])
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
saved = self._read_config()
self.assertIs(saved["gallery"]["monitoring"]["enabled"], False)
self.assertEqual(saved["custom"], {"keep": True})
self.assertFalse(GalleryConfig.MONITORING_ENABLED)
self.assertFalse((await self._get())["settings"]["monitoring_enabled"])
# Back on while the observer still runs: nothing waits for a restart.
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], [])
self.assertIs(self._read_config()["gallery"]["monitoring"]["enabled"], True)
async def test_enabling_without_a_live_observer_needs_a_restart(self):
GalleryConfig.MONITORING_ENABLED = False
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
self.assertTrue(GalleryConfig.MONITORING_ENABLED)
def test_config_file_enabled_must_be_a_bool(self):
GalleryConfig.MONITORING_ENABLED = True
with self.assertLogs("prompt_manager.config", level="WARNING"):
GalleryConfig.update_config({"monitoring": {"enabled": "false"}})
self.assertIs(GalleryConfig.MONITORING_ENABLED, True)
GalleryConfig.update_config({"monitoring": {"enabled": False}})
self.assertIs(GalleryConfig.MONITORING_ENABLED, False)
class TestSourceChangeHold(SourcesTestCase):
async def test_root_save_refused_while_scan_job_runs(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
self._write_config({"gallery": {"monitoring": {"directories": ["kept"]}}})
before = self._config_bytes()
gate = asyncio.Event()
async def runner(job):
await gate.wait()
scan = self.api._jobs.start(
"scan", "Scan for metadata", record_changing=True, runner=runner
)
try:
status, data = await self._post(
"/prompt_manager/settings",
{"gallery_root_paths": [str(self.b)]},
)
self.assertEqual(status, 409, data)
self.assertEqual(data["code"], "busy")
self.assertEqual(
data["running"], {"id": scan.id, "title": "Scan for metadata"}
)
self.assertEqual(self._config_bytes(), before)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [self._root(self.a)])
# A save without sources, or one resending the current sources
# (the legacy page always does), is not a source change.
status, data = await self._post(
"/prompt_manager/settings", {"result_timeout": 9}
)
self.assertEqual(status, 200, data)
status, data = await self._post(
"/prompt_manager/settings",
{"result_timeout": 8, "gallery_root_paths": [str(self.a)]},
)
self.assertEqual(status, 200, data)
finally:
gate.set()
await asyncio.wait_for(scan.task, 5)
async def test_root_save_runs_as_a_source_change_hold(self):
seen = []
original = settings_routes.get_config_store
def store():
seen.extend(
(job.kind, job.title, job.admission, job.cancellable)
for job in self.api._jobs.list()
if job.running
)
return original()
with patch.object(settings_routes, "get_config_store", store):
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]}
)
self.assertEqual(status, 200, data)
self.assertIn(("source_change", "Source change", "records", False), seen)
self.assertFalse(any(job.running for job in self.api._jobs.list()))
async def test_failed_root_save_ends_the_hold_failed(self):
os.makedirs(self.config_path) # config.json is a directory: unreadable
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]}
)
self.assertGreaterEqual(status, 400, data)
held = [j for j in self.api._jobs.list() if j.kind == "source_change"]
self.assertEqual([job.state for job in held], ["failed"])
async def test_mixed_case_windows_paths_dedupe(self):
"""Two spellings of one Windows folder are saved once (normcase)."""
with (
patch.object(sources, "source_problem", return_value=(None, "")),
patch.object(
GalleryConfig, "resolve_gallery_root", side_effect=lambda path: path
),
patch.object(settings_routes, "_source_identity", ntpath.normcase),
patch.object(self.api, "_apply_gallery_roots", return_value=True),
):
roots, error = self.api._parse_gallery_roots(
{
"gallery_root_paths": [
"C:\\ComfyUI\\Output\\Renders",
"c:/comfyui/output/renders",
"C:\\ComfyUI\\output\\other",
]
}
)
self.assertIsNone(error)
self.assertEqual(
roots, ["C:\\ComfyUI\\Output\\Renders", "C:\\ComfyUI\\output\\other"]
)
async def test_case_variant_of_one_folder_is_saved_once_natively(self):
variant = str(self.a)[:-1] + "A"
if not os.path.isdir(variant):
self.skipTest("case-sensitive filesystem")
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.a), variant]}
)
self.assertEqual(status, 200, data)
self.assertEqual(len(GalleryConfig.MONITORING_DIRECTORIES), 1)
class TestValidateSource(SourcesTestCase):
async def test_validate_source_writes_nothing(self):
self._write_config({"gallery": {"monitoring": {"directories": []}}})
before = self._config_bytes()
roots = list(GalleryConfig.MONITORING_DIRECTORIES)
status, data = await self._post(VALIDATE, {"path": str(self.a)})
self.assertEqual(status, 200, data)
self.assertEqual(
data,
{
"success": True,
"valid": True,
"label": "a",
"path": "output/a",
"code": None,
"reason": None,
},
)
self.assertEqual(self._config_bytes(), before)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, roots)
self.assertEqual(self.api._jobs.list(), [])
async def test_relative_path_resolves_against_comfyui(self):
status, data = await self._post(VALIDATE, {"path": "output/b"})
self.assertEqual((status, data["valid"], data["path"]), (200, True, "output/b"))
status, data = await self._post(VALIDATE, {"path": "output"})
self.assertEqual(data["label"], "ComfyUI output")
async def test_invalid_sources_answer_a_code(self):
a_file = self.output_dir / "file.png"
a_file.write_bytes(b"x")
cases = {
"": "empty",
" ": "empty",
str(self.output_dir / "missing"): "not-found",
str(a_file): "not-directory",
str(self.outside_dir): "outside-allowed",
os.path.abspath(os.sep): "outside-allowed",
os.path.expanduser("~"): "outside-allowed",
os.path.join(str(self.output_dir), "b", "..", "a"): "traversal",
"output/../output/a": "traversal",
str(self.a) + "\x00": "invalid-form",
"\\\\?\\C:\\ComfyUI\\output": "invalid-form",
"x" * 5000: "invalid-form",
}
if os.name != "nt":
cases["C:\\ComfyUI\\output"] = "invalid-form"
for path, code in cases.items():
with self.subTest(code=code, path=path[:40]):
status, data = await self._post(VALIDATE, {"path": path})
self.assertEqual(status, 200, data)
self.assertFalse(data["valid"])
self.assertEqual(data["code"], code)
self.assertIsInstance(data["reason"], str)
self.assertIsNone(data["label"])
self.assertIsNone(data["path"])
self.assertNotIn(os.path.expanduser("~"), json.dumps(data))
async def test_outside_paths_answer_alike_whether_they_exist(self):
"""No existence oracle: outside the allowed folders every path looks
the same, existing or not, file or folder."""
a_file = self.outside_dir / "file.txt"
a_file.write_bytes(b"x")
answers = []
for path in (
self.outside_dir,
self.outside_dir / "missing",
a_file,
os.path.join(self.tmpdir, "no", "such", "tree"),
os.path.abspath(os.sep),
os.path.expanduser("~"),
):
status, data = await self._post(VALIDATE, {"path": str(path)})
answers.append((status, data))
self.assertEqual(answers[0][1]["code"], "outside-allowed")
self.assertEqual([answer for answer in answers], [answers[0]] * len(answers))
async def test_settings_save_refuses_outside_paths_alike(self):
"""The legacy save answers every outside path alike too: the
filesystem root, the home directory and a link to it included."""
(self.outside_dir / "x").mkdir()
link = os.path.join(self.tmpdir, "link-home")
try:
os.symlink(os.path.expanduser("~"), link, target_is_directory=True)
except (OSError, NotImplementedError):
link = None
paths = [
str(self.outside_dir / "x"),
self.tmpdir + "/nothing/x",
os.path.abspath(os.sep),
os.path.expanduser("~"),
]
if link:
paths.append(link)
answers = []
for path in paths:
resp = await self.client.request(
"POST", SETTINGS, json={"gallery_root_paths": [path]}
)
body = await resp.text()
self.assertNotIn(os.path.expanduser("~"), body)
data = json.loads(body)
# Only the name the caller sent may differ.
answers.append(
(
resp.status,
{k: v for k, v in data.items() if k not in ("source", "error")},
)
)
self.assertEqual(
data["error"],
f"{data['source']} can't be used: "
f"{config_module.OUTSIDE_ALLOWED_REASON}. Remove it, then save.",
)
self.assertEqual(
answers[0],
(
400,
{
"success": False,
"code": "invalid-source",
"index": 0,
"reason": config_module.OUTSIDE_ALLOWED_REASON,
},
),
)
self.assertEqual(answers, [answers[0]] * len(answers))
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [])
def test_every_caller_folds_root_and_home_alike(self):
for path in (os.path.abspath(os.sep), os.path.expanduser("~")):
with self.subTest(path=path):
self.assertEqual(
GalleryConfig.gallery_root_problem(path),
("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(
GalleryConfig.validate_gallery_root(path),
(False, config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(
sources.validate_source(path)["code"], "outside-allowed"
)
async def test_outside_path_is_never_resolved_or_stat(self):
"""Containment is decided as text first: no realpath, stat or lstat of
an outside path, existing or not, by any caller (I2)."""
probe = self.outside_dir / "probe-x"
probe.mkdir()
(probe / "deep").mkdir()
cases = [str(probe), str(probe / "deep"), str(probe / "missing")]
with no_filesystem_access_for("probe-x", self):
for path in cases:
with self.subTest(path=path):
self.assertEqual(
GalleryConfig.gallery_root_problem(path)[0], "outside-allowed"
)
self.assertEqual(
sources.validate_source(path)["code"], "outside-allowed"
)
_roots, error = self.api._parse_gallery_roots(
{"gallery_root_paths": [path]}
)
self.assertEqual(error["code"], "invalid-source")
status, data = await self._post(VALIDATE, {"path": path})
self.assertEqual(data["code"], "outside-allowed")
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [path]}
)
self.assertEqual((status, data["code"]), (400, "invalid-source"))
# A traversal that normalises outside is refused as text too.
sneaky = os.path.join(
str(self.output_dir), "..", "..", "outside", "probe-x"
)
self.assertEqual(
GalleryConfig.gallery_root_problem(sneaky)[0], "outside-allowed"
)
async def test_settings_save_applies_the_form_checks(self):
"""POST /settings refuses what validate-source refuses, by its form."""
for path, code in {
str(self.a) + "\x00": "invalid-form",
os.path.join(str(self.output_dir), "b", "..", "a"): "traversal",
}.items():
with self.subTest(code=code):
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [path]}
)
self.assertEqual((status, data["code"]), (400, "invalid-source"))
self.assertEqual(data["reason"], sources.FORM_REASONS[code])
windows = functools.partial(sources.source_form_error, windows=True)
with (
patch.object(sources, "source_form_error", windows),
# The probes hold a token no trusted directory does (on Windows the
# temporary ComfyUI tree itself sits on C:).
no_filesystem_access_for("PMProbe", self),
):
for path in (
"\\\\?\\C:\\PMProbe\\output",
"\\\\.\\pipe\\PMProbe",
"C:\\PMProbe\\output\\CON",
"C:\\PMProbe\\output\\a:stream",
"C:PMProbe",
):
with self.subTest(path=path):
_roots, error = self.api._parse_gallery_roots(
{"gallery_root_paths": ["output/a", path]}
)
self.assertEqual(error["code"], "invalid-source")
self.assertEqual(error["index"], 1)
self.assertEqual(
error["reason"], sources.FORM_REASONS["invalid-form"]
)
async def test_inside_paths_say_why_they_are_not_usable(self):
a_file = self.output_dir / "file.png"
a_file.write_bytes(b"x")
status, missing = await self._post(
VALIDATE, {"path": str(self.output_dir / "missing")}
)
status, not_dir = await self._post(VALIDATE, {"path": str(a_file)})
self.assertEqual(
(missing["code"], not_dir["code"]), ("not-found", "not-directory")
)
async def test_malformed_request_is_400(self):
for body in ({}, {"path": 42}, {"path": None}, ["x"]):
with self.subTest(body=body):
status, data = await self._post(VALIDATE, body)
self.assertEqual(status, 400, data)
self.assertEqual(data["code"], "invalid-request")
self.assertFalse(data["success"])
class TestSourceEdges(SourcesTestCase):
async def test_validation_failure_is_500_without_paths(self):
def boom(path):
raise OSError(13, "Permission denied", f"{self.tmpdir}/secret")
with patch.object(sources, "validate_source", boom):
status, data = await self._post(VALIDATE, {"path": str(self.a)})
self.assertEqual(status, 500)
self.assertEqual(data["error"], "Failed to check the source")
async def test_oversized_body_is_413_too_large(self):
resp = await self.client.request(
"POST", VALIDATE, data=json.dumps({"path": "x" * (2 << 20)})
)
self.assertEqual(resp.status, 413)
self.assertEqual((await resp.json())["code"], "too-large")
async def test_unreadable_monitor_counts_as_no_live_observer(self):
class Broken:
def get_status(self):
raise RuntimeError("observer gone")
self._install_monitor(Broken())
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
def test_labels_skip_getters_that_fail_or_are_missing(self):
import sys
import types
def fails():
raise RuntimeError("no input dir")
fake = types.SimpleNamespace(
get_output_directory=lambda: str(self.output_dir),
get_input_directory=fails,
get_temp_directory=lambda: None,
)
with patch.dict(sys.modules, {"folder_paths": fake}):
self.assertEqual(sources.source_label(self.output_dir), "ComfyUI output")
self.assertEqual(sources.source_label(self.a), "a")
with patch.dict(sys.modules, {"folder_paths": None}):
self.assertEqual(sources.source_label(self.output_dir), "output")
class TestOutsideBaseDirectories(SourcesTestCase):
"""ComfyUI folders outside the base directory (``--output-directory``):
their sources round-trip and two folders of one name stay apart (I3)."""
async def asyncSetUp(self):
await super().asyncSetUp()
self.renders = self.outside_dir / "renders"
self.input_dir = self.comfy_dir / "input"
(self.renders / "portraits").mkdir(parents=True)
(self.input_dir / "portraits").mkdir(parents=True)
sys.modules["folder_paths"] = types.SimpleNamespace(
base_path=str(self.comfy_dir),
get_output_directory=lambda: str(self.renders),
get_input_directory=lambda: str(self.input_dir),
)
async def test_source_outside_base_round_trips(self):
status, rendered = await self._post(
VALIDATE, {"path": str(self.renders / "portraits")}
)
self.assertEqual(
(rendered["valid"], rendered["path"]), (True, "renders/portraits")
)
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [rendered["path"]]}
)
self.assertEqual(status, 200, data)
self.assertEqual(
GalleryConfig.MONITORING_DIRECTORIES,
[self._root(self.renders / "portraits")],
)
configured = (await self._get())["sources"]["configured"]
self.assertEqual(
[(row["path"], row["valid"]) for row in configured],
[("renders/portraits", True)],
)
# The page resends what it read: still the same folder, no change.
status, data = await self._post(
SETTINGS, {"gallery_root_paths": ["renders/portraits"]}
)
self.assertEqual((status, data["restart_required_fields"]), (200, []))
async def test_same_name_folders_stay_apart(self):
answers = []
for path in (self.renders / "portraits", self.input_dir / "portraits"):
status, data = await self._post(VALIDATE, {"path": str(path)})
answers.append(data["path"])
self.assertEqual(answers, ["renders/portraits", "input/portraits"])
status, data = await self._post(SETTINGS, {"gallery_root_paths": answers})
self.assertEqual(status, 200, data)
sources_ = (await self._get())["sources"]
self.assertEqual([row["path"] for row in sources_["configured"]], answers)
self.assertEqual([row["path"] for row in sources_["effective"]], answers)
class TestUnusableSourceSave(SourcesTestCase):
async def test_unusable_source_is_named_and_saves_nothing(self):
"""A configured source that went missing blocks the save and is named."""
gone = self.output_dir / "gone"
gone.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(gone)]
self._write_config({"custom": 1})
before = self._config_bytes()
gone.rmdir() # an unplugged drive
status, data = await self._post(
SETTINGS,
{
"gallery_root_paths": ["output/a", "output/gone", "output/b"],
"worker_threads": 2,
},
)
self.assertEqual(status, 400, data)
self.assertEqual(
data,
{
"success": False,
"code": "invalid-source",
"source": "output/gone",
"index": 1,
"reason": "Gallery root does not exist",
"error": "output/gone can't be used: Gallery root does not exist. "
"Remove it or reconnect it, then save.",
},
)
self.assertEqual(self._config_bytes(), before)
async def test_malformed_roots_are_coded_invalid_request(self):
for roots in ("output/a", [1]):
with self.subTest(roots=roots):
status, data = await self._post(SETTINGS, {"gallery_root_paths": roots})
self.assertEqual((status, data["code"]), (400, "invalid-request"))
class TestUnchangedRootsTakeNoHold(SourcesTestCase):
async def test_resent_spelling_of_configured_roots_is_no_change(self):
"""config.json holds a hand-written spelling; the legacy page resends
it resolved: no source change, so no hold and no 409 during a scan (M1)."""
GalleryConfig.MONITORING_DIRECTORIES = ["output/a"]
gate = asyncio.Event()
async def runner(job):
await gate.wait()
job.finish("finished")
self.api._jobs.start(
"scan", "Metadata scan", record_changing=True, runner=runner
)
try:
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [str(self.a)], "result_timeout": 30}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], [])
self.assertEqual([j.kind for j in self.api._jobs.list()], ["scan"])
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [str(self.b)]}
)
self.assertEqual((status, data["code"]), (409, "busy"))
finally:
gate.set()
class TestWindowsContainment(unittest.TestCase):
"""Windows path forms, with ntpath fakes: anything outside the allowed
folders (a UNC share, the verbatim and device namespaces, another drive)
is refused as text, before any path is resolved or opened (I2)."""
ALLOWED = "c:\\comfyui\\output"
NAS = "\\\\nas\\share\\renders"
def setUp(self):
self.touched = []
def realpath(path, *args, **kwargs):
self.touched.append(path)
return ntpath.normcase(ntpath.normpath(path))
def probe(path):
self.touched.append(path)
return True
names = ("normcase", "normpath", "isabs", "join", "dirname", "basename", "sep")
fake_path = types.SimpleNamespace(
**{name: getattr(ntpath, name) for name in names},
realpath=realpath,
exists=probe,
isdir=probe,
expanduser=lambda path: "c:\\users\\someone",
)
fake_os = types.SimpleNamespace(
path=fake_path, environ={}, pathsep=";", name="nt"
)
for target, value in (
(patch.object(config_module, "os", fake_os), None),
(
patch.object(
GalleryConfig,
"_allowed_parent_pairs",
classmethod(
lambda cls: [(self.ALLOWED, self.ALLOWED), (self.NAS, self.NAS)]
),
),
None,
),
(
patch.object(
GalleryConfig,
"path_anchors",
classmethod(lambda cls: ["c:\\comfyui", "\\\\nas\\share"]),
),
None,
),
):
target.start()
self.addCleanup(target.stop)
def test_outside_forms_are_refused_untouched(self):
for path in (
"\\\\attacker\\share\\x",
"//attacker/share/x",
"\\\\?\\C:\\ComfyUI\\output\\a",
"\\\\?\\UNC\\attacker\\share",
"\\\\.\\pipe\\x",
"\\\\.\\C:\\ComfyUI\\output",
"D:\\Renders",
"C:\\Windows",
"C:\\ComfyUI\\output\\..\\..\\Windows",
"C:\\ComfyUI\\output2",
"\\\\nas\\share\\other",
"..\\..\\Windows",
):
with self.subTest(path=path):
self.touched.clear()
self.assertEqual(
GalleryConfig.gallery_root_problem(path),
("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(self.touched, [])
def test_allowed_spellings_are_accepted(self):
for path in (
"C:\\ComfyUI\\output\\a",
"c:/comfyui/OUTPUT/a",
"output\\a",
"\\\\NAS\\share\\renders\\a", # a UNC spelling of an allowed extra root
):
with self.subTest(path=path):
self.assertEqual(GalleryConfig.gallery_root_problem(path), (None, ""))
class TestSourceFormError(unittest.TestCase):
"""The lexical check that runs before any filesystem access."""
def test_windows_forms(self):
err = sources.source_form_error
for path in (
"C:\\ComfyUI\\output",
"C:/ComfyUI/output",
"\\\\server\\share\\output",
"D:\\images",
):
with self.subTest(path=path):
self.assertIsNone(err(path, windows=True))
for path, code in {
"C:output": "invalid-form", # drive-relative
"C:\\out\\a:stream": "invalid-form", # alternate data stream
"C:\\out\\CON": "invalid-form",
"C:\\out\\nul.txt": "invalid-form",
"C:\\out\\folder.": "invalid-form",
"C:\\out\\folder \\sub": "invalid-form",
"\\\\?\\C:\\out": "invalid-form",
"//?/C:/out": "invalid-form",
"\\\\.\\pipe\\x": "invalid-form",
"\\\\server": "invalid-form",
"C:\\out\\..\\in": "traversal",
"output\\..\\input": "traversal",
}.items():
with self.subTest(path=path):
self.assertEqual(err(path, windows=True), code)
def test_posix_forms(self):
err = sources.source_form_error
for path in ("/srv/comfy/output", "output/renders", "out/CON", "a b/c."):
with self.subTest(path=path):
self.assertIsNone(err(path, windows=False))
for path, code in {
"C:\\ComfyUI\\output": "invalid-form",
"C:/ComfyUI/output": "invalid-form",
"\\\\server\\share": "invalid-form",
"out\\renders": "invalid-form",
"/srv/../etc": "traversal",
"../output": "traversal",
"/srv/out\x00": "invalid-form",
"/srv/out\n": "invalid-form",
"": "empty",
}.items():
with self.subTest(path=path):
self.assertEqual(err(path, windows=False), code)
if __name__ == "__main__":
unittest.main()

Some files were not shown because too many files have changed in this diff Show More